Verified Job
Other Jobs
Streamsets Lead
Other Jobs
#560524
Remote / WFH
Job Description
Position: StreamSet Lead
Permanent Opportunity
Location: Chennai, Tamil Nadu(3 day/week)
Duration: Full Time
Level of experience: 5+ years
Notice Period: Immediate Joiner/ Currently Serving/ Less than 45 days
(Open for Freelancer/Contractor as well if someone can support on need basis with remote option but strong expertise in Streamsets- Min 5+ years in StreamSet and SQL)
Job Responsibilities:
StreamSets Developer/Engineer Lead:
➢ Designing, developing, and deploying data pipelines using StreamSets
➢ Configuring data sources, transformations, and destinations within pipelines.
➢ Monitoring and troubleshooting pipeline performance and errors.
➢ Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency.
➢ Implementing best practices for data governance, security, and compliance.
➢ Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and
consumer groups.
Responsibilities:
Pipeline Development:
➢ Designing and developing data pipelines using StreamSets based on business requirements.
➢ Implementing data transformations, validations, and enrichments within pipelines.
➢ Testing pipelines for performance, reliability, and data quality.
Pipeline Monitoring and Maintenance:
➢ Monitoring pipeline performance metrics such as throughput, latency, and error rates.
➢ Troubleshooting and resolving issues related to pipeline failures or data inconsistencies.
➢ Optimizing pipelines for efficiency and scalability as data volumes grow.
Security and Compliance:
➢ Ensuring that data handling practices within StreamSets comply with organizational security
policies and regulatory requirements.
➢ Implementing encryption, access controls, and auditing mechanisms to protect sensitive data.
Documentation and Knowledge Sharing:
➢ Documenting pipeline configurations, dependencies, and operational procedures.
➢ Sharing knowledge and best practices with team members through training sessions,
documentation, and collaboration tools.
Performance Optimization:
➢ Analyzing pipeline performance bottlenecks and optimizing configurations for improved
efficiency.
➢ Scaling pipelines to handle increasing data volumes and processing requirements.
Collaboration and Communication:
➢ Collaborating with cross-functional teams including developers, data scientists, and business
analysts to understand data requirements and ensure alignment with business objectives.
➢ Communicating updates, issues, and resolutions effectively with stakeholders.
Permanent Opportunity
Location: Chennai, Tamil Nadu(3 day/week)
Duration: Full Time
Level of experience: 5+ years
Notice Period: Immediate Joiner/ Currently Serving/ Less than 45 days
(Open for Freelancer/Contractor as well if someone can support on need basis with remote option but strong expertise in Streamsets- Min 5+ years in StreamSet and SQL)
Job Responsibilities:
StreamSets Developer/Engineer Lead:
➢ Designing, developing, and deploying data pipelines using StreamSets
➢ Configuring data sources, transformations, and destinations within pipelines.
➢ Monitoring and troubleshooting pipeline performance and errors.
➢ Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency.
➢ Implementing best practices for data governance, security, and compliance.
➢ Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and
consumer groups.
Responsibilities:
Pipeline Development:
➢ Designing and developing data pipelines using StreamSets based on business requirements.
➢ Implementing data transformations, validations, and enrichments within pipelines.
➢ Testing pipelines for performance, reliability, and data quality.
Pipeline Monitoring and Maintenance:
➢ Monitoring pipeline performance metrics such as throughput, latency, and error rates.
➢ Troubleshooting and resolving issues related to pipeline failures or data inconsistencies.
➢ Optimizing pipelines for efficiency and scalability as data volumes grow.
Security and Compliance:
➢ Ensuring that data handling practices within StreamSets comply with organizational security
policies and regulatory requirements.
➢ Implementing encryption, access controls, and auditing mechanisms to protect sensitive data.
Documentation and Knowledge Sharing:
➢ Documenting pipeline configurations, dependencies, and operational procedures.
➢ Sharing knowledge and best practices with team members through training sessions,
documentation, and collaboration tools.
Performance Optimization:
➢ Analyzing pipeline performance bottlenecks and optimizing configurations for improved
efficiency.
➢ Scaling pipelines to handle increasing data volumes and processing requirements.
Collaboration and Communication:
➢ Collaborating with cross-functional teams including developers, data scientists, and business
analysts to understand data requirements and ensure alignment with business objectives.
➢ Communicating updates, issues, and resolutions effectively with stakeholders.
