Verified Job
IT / Software / Data Analyst
Lead Big Data DevOps Engineer
IT / Software / Data Analyst
#569751
Remote / WFH
Job Description
Location: [Pune / Gurgaon]
Job Type: Full-time
Experience: 6-10 years
Job Description:
We are seeking an accomplished Lead Big Data DevOps Engineer to architect, scale, and
optimize our Big Data infrastructure and DevOps practices. You will drive automation,
reliability, and performance across complex, distributed data environments, collaborating with
cross-functional teams to enable fast, secure, and scalable data delivery. As a technical leader,
you will mentor engineers, set best practices, and ensure the seamless integration of new
technologies and processes.
Key Responsibilities:
Technical Leadership & Strategy:
• Lead the design, implementation, and continuous improvement of Big Data infrastructure (Hadoop, Spark, Kafka, Trino, Airflow, etc.) across on-premises and cloud environments.
• Define and enforce DevOps best practices for automation, CI/CD, monitoring, security, and scalability in Big Data workflows.
• Architect and optimize infrastructure for high availability, disaster recovery, and performance at scale.
• Drive the adoption of infrastructure-as-code (IaC), containerization (Docker), and orchestration (Kubernetes) for reproducible and scalable deployments.
• Evaluate, select, and integrate new tools and technologies to advance the platform’s capabilities.
• Should be able to debug and fine tune workload , processing billions of records
Automation & Operational Excellence:
• Build and maintain robust CI/CD pipelines for Big Data applications, data pipelines, and platform services.
• Oversee automated provisioning, configuration management, and orchestration using tools such as Ansible, Terraform, or similar.
• Implement and maintain comprehensive monitoring, logging, and alerting solutions (e.g., Prometheus, Grafana, ELK, Open Telemetry) for proactive issue detection and resolution.
• Ensure end-to-end data pipeline reliability, including automated data quality checks, lineage, and observability.
Collaboration & Mentorship:
• Collaborate closely with data engineers, data scientists, and application teams to streamline workflows and resolve cross-team dependencies.
• Mentor and upskill DevOps and engineering team members, fostering a culture of knowledge sharing, innovation, and continuous improvement.
• Lead incident response, root cause analysis, and post-mortem processes for critical production issues.
Security & Compliance:
• Implement and enforce security best practices for Big Data environments, including RBAC/ABAC, secrets management, encryption, and compliance with organizational and regulatory standards.
• Oversee access controls, audit trails, and vulnerability management for all Big Data infrastructure components.
Process & Governance:
• Define and document operational processes, disaster recovery/business continuity plans, and SLAs for Big Data platform services.
• Support capacity planning, cost optimization, and resource management across clusters and environments.
• Drive the adoption of DevOps and DataOps culture and practices across the organization.
Required Qualifications:
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 4-8 years of experience in DevOps, SRE, or infrastructure engineering roles, with at least 3 years leading Big Data or DevOps teams.
• Advanced expertise in deploying, managing, and optimizing Big Data frameworks (Hadoop, Spark, Kafka, Hive, Trino, Airflow, etc.).
• Proven experience with containerization (Docker), orchestration (Kubernetes), and infrastructure-as-code (Ansible, Terraform, etc.).
• Strong scripting skills (Python, Bash, or similar) for automation and tooling.
• Deep understanding of CI/CD pipelines, version control (Git), and modern DevOps toolchains (Jenkins, GitLab CI, etc.).
• Experience with cloud platforms (AWS, Azure, GCP) and hybrid cloud architectures.
• Solid understanding of security best practices in Big Data and DevOps environments.
• Excellent problem-solving, communication, and leadership skills.
Preferred Qualifications:
• Experience with advanced monitoring and observability tools (Prometheus, Grafana, ELK, Open Telemetry).
• Familiarity with data cataloging, metadata management, and data lineage tools.
• Exposure to MLOps, GenAI platforms, or data mesh architectures.
• Experience with NoSQL databases (Cassandra, MongoDB, etc.) and distributed storage systems.
• Prior experience in high-availability, mission-critical data environments supporting large-scale, multi-tenant deployments.
Benefits:
• Competitive salary and benefits package.
• Leadership role in building and scaling next-generation Big Data platforms.
• Opportunity to work with cutting-edge technologies in a collaborative, high-impact environment.
• Professional growth and development through challenging projects and continuous learning.
If you are passionate about leading technical teams, driving automation and reliability at
scale, and enabling data-driven innovation, join us as our Lead Big Data DevOps Engineer.
Job Type: Full-time
Experience: 6-10 years
Job Description:
We are seeking an accomplished Lead Big Data DevOps Engineer to architect, scale, and
optimize our Big Data infrastructure and DevOps practices. You will drive automation,
reliability, and performance across complex, distributed data environments, collaborating with
cross-functional teams to enable fast, secure, and scalable data delivery. As a technical leader,
you will mentor engineers, set best practices, and ensure the seamless integration of new
technologies and processes.
Key Responsibilities:
Technical Leadership & Strategy:
• Lead the design, implementation, and continuous improvement of Big Data infrastructure (Hadoop, Spark, Kafka, Trino, Airflow, etc.) across on-premises and cloud environments.
• Define and enforce DevOps best practices for automation, CI/CD, monitoring, security, and scalability in Big Data workflows.
• Architect and optimize infrastructure for high availability, disaster recovery, and performance at scale.
• Drive the adoption of infrastructure-as-code (IaC), containerization (Docker), and orchestration (Kubernetes) for reproducible and scalable deployments.
• Evaluate, select, and integrate new tools and technologies to advance the platform’s capabilities.
• Should be able to debug and fine tune workload , processing billions of records
Automation & Operational Excellence:
• Build and maintain robust CI/CD pipelines for Big Data applications, data pipelines, and platform services.
• Oversee automated provisioning, configuration management, and orchestration using tools such as Ansible, Terraform, or similar.
• Implement and maintain comprehensive monitoring, logging, and alerting solutions (e.g., Prometheus, Grafana, ELK, Open Telemetry) for proactive issue detection and resolution.
• Ensure end-to-end data pipeline reliability, including automated data quality checks, lineage, and observability.
Collaboration & Mentorship:
• Collaborate closely with data engineers, data scientists, and application teams to streamline workflows and resolve cross-team dependencies.
• Mentor and upskill DevOps and engineering team members, fostering a culture of knowledge sharing, innovation, and continuous improvement.
• Lead incident response, root cause analysis, and post-mortem processes for critical production issues.
Security & Compliance:
• Implement and enforce security best practices for Big Data environments, including RBAC/ABAC, secrets management, encryption, and compliance with organizational and regulatory standards.
• Oversee access controls, audit trails, and vulnerability management for all Big Data infrastructure components.
Process & Governance:
• Define and document operational processes, disaster recovery/business continuity plans, and SLAs for Big Data platform services.
• Support capacity planning, cost optimization, and resource management across clusters and environments.
• Drive the adoption of DevOps and DataOps culture and practices across the organization.
Required Qualifications:
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• 4-8 years of experience in DevOps, SRE, or infrastructure engineering roles, with at least 3 years leading Big Data or DevOps teams.
• Advanced expertise in deploying, managing, and optimizing Big Data frameworks (Hadoop, Spark, Kafka, Hive, Trino, Airflow, etc.).
• Proven experience with containerization (Docker), orchestration (Kubernetes), and infrastructure-as-code (Ansible, Terraform, etc.).
• Strong scripting skills (Python, Bash, or similar) for automation and tooling.
• Deep understanding of CI/CD pipelines, version control (Git), and modern DevOps toolchains (Jenkins, GitLab CI, etc.).
• Experience with cloud platforms (AWS, Azure, GCP) and hybrid cloud architectures.
• Solid understanding of security best practices in Big Data and DevOps environments.
• Excellent problem-solving, communication, and leadership skills.
Preferred Qualifications:
• Experience with advanced monitoring and observability tools (Prometheus, Grafana, ELK, Open Telemetry).
• Familiarity with data cataloging, metadata management, and data lineage tools.
• Exposure to MLOps, GenAI platforms, or data mesh architectures.
• Experience with NoSQL databases (Cassandra, MongoDB, etc.) and distributed storage systems.
• Prior experience in high-availability, mission-critical data environments supporting large-scale, multi-tenant deployments.
Benefits:
• Competitive salary and benefits package.
• Leadership role in building and scaling next-generation Big Data platforms.
• Opportunity to work with cutting-edge technologies in a collaborative, high-impact environment.
• Professional growth and development through challenging projects and continuous learning.
If you are passionate about leading technical teams, driving automation and reliability at
scale, and enabling data-driven innovation, join us as our Lead Big Data DevOps Engineer.
