Build Athena Tables through Glue Tool Suite

Image from AWS Glue Concepts

TL;DR

We first used Glue Crawlers to build data catalogs for S3 source data, and then deployed customized ETL scripts through Glue Jobs. The processed result was saved into a new S3 destination with updated partitions. Next, we ran another crawler on output data so that an Athena table of result data was created/updated for SQL query. Finally, we applied Triggers and Workflows to orchestra and automate the ETL steps as daily job.

  • Glue Crawler: builds data catalog of source data and make S3 data available to Athena in table format no matter what format the source data is, including but…

Jinghan Ma

Full Stack Data Analytics Engineer | AWS Certified | Learning through sharing! Be true to yourself and be kind to others!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store