Lead Data Engineer

I
InfoBeans


Python Automation with Power BI Engineer [Apply Now]
InfoBeans • Pune, Maharashtra • via Jobrapido.com
15 hours ago
Full–time
No Degree Mentioned
Apply on Jobrapido.com
Job description
Job Role - Python Automation with PowerBi Tester
Experience Required: 6 to 10 Years
Location: Pune, Indore
Immediate joiners are preferred.
About the Role:
We are looking for a highly skilled and experienced Senior Python Automation Engineer to significantly enhance our automation capabilities and translate automation outcomes into actionable business intelligence using Power BI. You will be respo...
Show full description
Report this listing

CG-VAK Software & Exports Ltd.


Lead Data Engineer (Scala, Spark, Pyspark, Databricks)
CG-VAK Software & Exports Ltd. • Pune, Maharashtra • via LinkedIn
9 hours ago
Full–time
No Degree Mentioned
Apply on LinkedIn
Job description
Job Title: Data Engineer (Scala, Spark, Pyspark, Databricks)

Remark - final round will be the F2F

Experience: 10+ Years (Lead Role)

Responsibilities
• Design, develop, and maintain robust and scalable data pipelines using Apache Spark and Scala on the Databricks platform.
• Implement ETL (Extract, Transform, Load) processes for various data sources, ensuring data quality, integrity, and efficiency.
• Optimize Spark applications for performance and cost-efficiency within the Databricks environment.
• Work with Delta Lake for building reliable data lakes and data warehouses, ensuring ACID transactions and data versioning.
• Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.
• Implement data governance and security best practices within Databricks.
• Troubleshoot and resolve data-related issues, ensuring data availability and reliability.
• Stay updated with the latest advancements in Spark, Scala, Databricks, and related big data technologies.

Required Skills And Experience
• Proven experience as a Data Engineer with a strong focus on big data technologies.
• Expertise in Scala programming language for data processing and Spark application development.
• In-depth knowledge and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and Spark Core.
• Proficiency in using Databricks platform features, including notebooks, jobs, workflows, and Unity Catalog.
• Experience with Delta Lake and its capabilities for building data lakes.
• Strong understanding of data warehousing concepts, data modeling, and relational databases.
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
• Experience with version control systems like Git.
• Excellent problem-solving and analytical skills.
• Ability to work independently and as part of a team.

Preferred Qualifications (Optional)
• Experience with other big data technologies like Kafka, Flink, or Hadoop ecosystem components.
• Knowledge of data visualization tools.
• Understanding of DevOps principles and CI/CD pipelines for data engineering.
• Relevant certifications in Spark or Databrick

Skills: scala,big data,data,big data technologies,apache,spark,apache spark,lake