Lead I Software Engineering

Role Description
• Data Pipeline Development: Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance.
• Develop and maintain ETL workflows for seamless data processing. Azure Cloud Expertise.
• Utilize Azure services such as Azure Data Factory, Azure SQL Database, and Azure Databricks for effective data engineering. Implement and manage data storage solutions on Azure.
• Data Transformation with PySpark Leverage PySpark for advanced data transformations, ensuring high-quality and well-structured output.
• Implement data cleansing, enrichment, and validation processes using PySpark.
• Performance Optimization Optimize data pipelines, queries, and PySpark jobs to enhance overall performance and scalability.
• Identify and address performance bottlenecks within data processing workflows.
• Proven experience as a Data Engineer, emphasizing expertise in Azure.
• Proficiency in Azure Data Factory and other relevant Azure services.
• Expertise in PySpark for data processing and analytics is a must.
• Experience with data modeling, ETL processes, and data warehousing.

Skills

Azure,Azure Data Factory,Pyspark