Verified Job
IT / Software / Data Analyst
Specialist Software Engineer - Data Engineer Databricks pySpark
IT / Software / Data Analyst
#744924
Remote / WFH
Societe Generale Global Solution Centre · Bengaluru, Karnataka ·
Job Description
Job description
Reference 26000DMZ
Responsibilities
Specialist Software Engineer – Data Engineer (Databricks-pySaprk)
Responsibilities
• Understand business and technical requirements and translate them into scalable, maintainable data engineering solutions.
• Own the design and development of robust, high‑performance data pipelines using Databricks(pySpark).
• Build and optimize data flows within the Hadoop ecosystem (Hive, HDFS, Starburst, Oozie).
• Ensure performance tuning, resource optimization, and efficient cluster usage on Databricks platforms.
• Contribute to data architecture decisions related to storage, processing, security, and exposure.
• Apply industrialization best practices including CI/CD, unit testing, integration testing, and pipeline validation.
• Model data based on business needs and maintain shared data components, dictionaries, and governance rules.
• Perform functional and technical analysis, document specifications, and validate implemented solutions.
• Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams.
• Support production processes, troubleshoot issues, update runbooks, and maintain system stability.
• Ensure compliance with data quality, traceability, and security standards across the data lifecycle.
• Engage in continuous improvement activities, proposing upgrades, refactoring, and automation opportunities.
• Work closely with global teams and support cross‑location coordination when required.
• Demonstrate ownership during key delivery phases, including planned on‑call or deployment activities.
Profile Required
Profile required
Experienced Data Engineer with strong expertise in Databricks, Hadoop ecosystem components, and large‑scale data processing environments.
Minimum Qualifications
• Bachelor of Engineering in Computer Science, Information Technology and equivalent streams.
• Strong hands‑on experience in pySpark for distributed processing and performance optimization.
• Solid understanding of Hadoop ecosystem tools: Hive, HDFS, YARN, Oozie, Starburst.
• Proven experience designing and maintaining data pipelines in complex Big Data environments.
• Good command of Python for supplemental processing and pipeline utilities.
• Strong understanding of data modeling, partitioning strategies, and governance principles.
• Proficiency in Git and modern SDLC practices (branching, PR reviews, tagging).
• Experience with CI/CD pipelines (GitHub Actions, Ansible, AWX).
• Strong documentation skills and ability to produce clear technical specifications.
Preferred Qualifications
• 4+ years of experience in data engineering or large‑scale distributed systems.
• Experience with NiFi/Kafka for batch and real‑time ingestion.
• Working knowledge of JFrog (artifact deployment, vulnerability remediation).
• Strong experience in Apache Hive performance tuning and Spark SQL integration.
• Exposure to Cloudera Manager for monitoring clusters and troubleshooting.
• Familiarity with Data Governance, security policies, and traceability frameworks.
• Experience in Agile @ Scale (SAFe) and backlog contribution (user stories, acceptance criteria).
• Ability to work independently and collaboratively across multicultural teams.
• Strong analytical, communication, and problem‑solving skills.
• cloud technologies (Azure/AWS) and modern platforms like Databricks.
Why join us
“We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, ****** orientation, national origin, genetics, disability, age, or veteran status”
Business insight
At Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious.
Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating and taking action are part of our DNA.
If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!
Still hesitating?
You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices and sharing their skills with charities. There are many ways to get involved.
We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.
Reference 26000DMZ
Responsibilities
Specialist Software Engineer – Data Engineer (Databricks-pySaprk)
Responsibilities
• Understand business and technical requirements and translate them into scalable, maintainable data engineering solutions.
• Own the design and development of robust, high‑performance data pipelines using Databricks(pySpark).
• Build and optimize data flows within the Hadoop ecosystem (Hive, HDFS, Starburst, Oozie).
• Ensure performance tuning, resource optimization, and efficient cluster usage on Databricks platforms.
• Contribute to data architecture decisions related to storage, processing, security, and exposure.
• Apply industrialization best practices including CI/CD, unit testing, integration testing, and pipeline validation.
• Model data based on business needs and maintain shared data components, dictionaries, and governance rules.
• Perform functional and technical analysis, document specifications, and validate implemented solutions.
• Participate in Agile ceremonies and collaborate with Product Owners, architecture, business, and Ops teams.
• Support production processes, troubleshoot issues, update runbooks, and maintain system stability.
• Ensure compliance with data quality, traceability, and security standards across the data lifecycle.
• Engage in continuous improvement activities, proposing upgrades, refactoring, and automation opportunities.
• Work closely with global teams and support cross‑location coordination when required.
• Demonstrate ownership during key delivery phases, including planned on‑call or deployment activities.
Profile Required
Profile required
Experienced Data Engineer with strong expertise in Databricks, Hadoop ecosystem components, and large‑scale data processing environments.
Minimum Qualifications
• Bachelor of Engineering in Computer Science, Information Technology and equivalent streams.
• Strong hands‑on experience in pySpark for distributed processing and performance optimization.
• Solid understanding of Hadoop ecosystem tools: Hive, HDFS, YARN, Oozie, Starburst.
• Proven experience designing and maintaining data pipelines in complex Big Data environments.
• Good command of Python for supplemental processing and pipeline utilities.
• Strong understanding of data modeling, partitioning strategies, and governance principles.
• Proficiency in Git and modern SDLC practices (branching, PR reviews, tagging).
• Experience with CI/CD pipelines (GitHub Actions, Ansible, AWX).
• Strong documentation skills and ability to produce clear technical specifications.
Preferred Qualifications
• 4+ years of experience in data engineering or large‑scale distributed systems.
• Experience with NiFi/Kafka for batch and real‑time ingestion.
• Working knowledge of JFrog (artifact deployment, vulnerability remediation).
• Strong experience in Apache Hive performance tuning and Spark SQL integration.
• Exposure to Cloudera Manager for monitoring clusters and troubleshooting.
• Familiarity with Data Governance, security policies, and traceability frameworks.
• Experience in Agile @ Scale (SAFe) and backlog contribution (user stories, acceptance criteria).
• Ability to work independently and collaboratively across multicultural teams.
• Strong analytical, communication, and problem‑solving skills.
• cloud technologies (Azure/AWS) and modern platforms like Databricks.
Why join us
“We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, ****** orientation, national origin, genetics, disability, age, or veteran status”
Business insight
At Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious.
Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating and taking action are part of our DNA.
If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!
Still hesitating?
You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices and sharing their skills with charities. There are many ways to get involved.
We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.
