![]() We’ve covered all of these topics in some depth before, so go ahead and click on the links to explore further. To provide optimal performance, implement data lake best practices including storing data in columnar formats such as Apache Parquet, rather than as raw CSV and JSON merging small files in a process known as compaction and data partitioning. ![]() ![]() While this likely isn’t material if you’re just querying a few thousand files or a few hundred megabytes, at larger scales it is essential to preprocess the data to ensure Athena performs well and also to control costs. While Athena is serverless and easy to use due to its reliance on regular SQL, its performance can still be impacted by the way data is stored in the underlying Amazon S3 bucket. Optimizing the S3 Storage Layer for Improved Performance in Athena You can read more about Athena tables in the AWS documentation, or peruse our previous blog post covering the differences between Amazon Athena and traditional databases. Athena uses the Apache Hive metastore to define tables and create databases, but doesn’t store any data, in accordance with its serverless architecture. Hence, when we talk about “tables” in Athena we are actually talking about a logical namespace that describes the schema and the location where data is stored on S3. However, unlike a traditional database, Athena uses schema-on-read rather than schema-on-write – meaning a schema is projected on the data when a query is run, rather than when the data is ingested into the database (since Athena doesn’t ingest the data but rather reads it, unmodified, from S3). Since Athena is based on regular SQL, it supports DDL statements and has the concept of “tables.” These behave similarly to tables in a relational database. The results of the Athena-S3 SQL queries can then be read by QuickSight or other visualization tools, which will provide BI and dashboarding to end users. In these cases, Athena can provide a hassle-free way to query the data. Athena S3 – Reference Architecture Source: Athena architecture examplesĬompanies that collect large volumes clickstream data from websites or applications would often choose to store this data on S3 object storage to avoid the high costs of data warehouse storage. This makes life simpler for data engineers as there is no infrastructure to manage, unlike traditional data warehouses such as Amazon Redshift (read more about the differences between Athena and Redshift ). This helps bridge the gap between S3 object storage – which is schemaless and semi-structured – and the needs of analytics users who want to run regular SQL queries on the data (although, as we cover below, data preparation is still required).Īnother advantage of Athena is that it is serverless, with compute resources provisioned by AWS on-demand and as required to answer a specific query. The key advantage of using Athena is that it can read data directly from S3, using regular SQL. Benefits of using AWS Athena when your data is stored on S3 In a typical AWS data lake architecture, S3 and Athena are two services that go together like a horse and carriage – with S3 acting as a near-infinite storage layer that allows organizations to collect and retain all of the data they generate, and Athena providing the means to query the data and curate structured datasets for analytical processing. ![]() ![]() In this guide, we’ll cover the basics of querying Amazon S3 using Athena, including some handy links to previous resources we’ve published on this topic, and end with a quick example and tutorial on querying S3 – specifically Apache Parquet files – as Athena tables. We’re massive fans of Amazon Athena – not just because Upsolver is the only official partner on the Athena webpage on AWS, but also because we see the fascinating, versatile ways in which customers use Athena for business intelligence, ad hoc reporting, and data science. If you want everything in one place, sign up for FREE access to our Athena resource center. This article is part of our Amazon Athena resource bundle, which comes with a host of additional articles, ebooks, and videos to cover everything you always wanted to know about Athena.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |