We first used Glue Crawlers to build data catalogs for S3 source data, and then deployed customized ETL scripts through Glue Jobs. The processed result was saved into a new S3 destination with updated partitions. Next, we ran another crawler on output data so that an Athena table of result data was created/updated for SQL query. Finally, we applied Triggers and Workflows to orchestra and automate the ETL steps as daily job.
Full Stack Data Analytics Engineer | AWS Certified | Learning through sharing! Be true to yourself and be kind to others!