AWS Glue
Introduction
Maintainer: DBA Demo Admins DBA_Demo_Admins@bmc.com
AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.
Use Case Overview
This workflow allows organizations to automate and orchestrate data integration and ETL (Extract, Transform, Load) tasks across both cloud and on-premise environments. Control-M provides centralized control for scheduling, monitoring, and managing dependencies for AWS Glue jobs, ensuring smooth data workflows. This integration ensures real-time visibility, error handling, and auditing for complex data processes, improving operational efficiency and compliance.
Use Case Technical Explanation
This workflow will start by orchestrating the AWS Glue job utilizing IAM (AWS Identity and Access Management) to convert a CSV file to a Parquet file. It will then sleep for 60 seconds and run the same job again utilizing an access key. There is also a dummy SLA job that does not really do anything.
Demo Environment Jobs
| Job Name | Execution | Glue Job | Connection Profile |
|---|---|---|---|
| zzz-glue-job-iam | Executes AWS Glue job using IAM access | customer_csv_to_parquet | AWS_GLUE_IAM |
| zzz-sleep-timer | Sleeps 60 Seconds | N/A | |
| zzz-glue-job-access-key | Executes AWS Glue job using access key | customer_csv_to_parquet | AWS_GLUE_STANDARD |
| zzz-glue-sla-job | Dummy Job | N/A |
Demo Environment Information
| Environment | Status | Folder Name |
|---|---|---|
| Helix Production | Available | zzz-aws-glue |
| VSE CTM PROD | Available | zzz-aws-glue |
| VSE CTM QA | Available | zzz-aws-glue |