Senior ETL Developer

Data Glue(AWS Glue)

Experience :6+ yrs

Job Responsibilities

Please ensure to send candidate profile who has practical hands-on experience with below task / skills:

AWS ETL Pipeline setup:

Candidate should have hands-on experience in creating an ETL pipeline on AWS using S3, Lambda functions, AWS Glue, Redshift. Should be good in Python and PySpark.

PySpark activities like -

join/merge tables,

apply transformations like dedup, Normalization in curation layer.

Should have handled optimizations while handling large volumes of data.

Data Base expertise on -

Knowledge on basic and advanced SQL

Should be able to understand a Data model as shared by an architect.

Able to create complex queries using joins or functions.

Data Warehouse concepts. Understanding of Facts, Dim and their relation.

Other areas where the candidate can have some idea (good to have) -

Data Validations

How data validations are done in ETL eg: Source to target validations, Data type checks, Business rule checks, Data checks during transformations, Duplicate data etc.

API Integration.

Pulling data from external sources using an API.

Eg: pulling Sharepoint or Salesforce data into AWS.

GitHub use.

Airflow exposure. Read more