Verified Job
Advertising - Public Relations
Company logo GenAI Data Engineer with GenAI projects expertise, Python, Azure/AWS
Advertising - Public Relations
#765060
Remote / WFH
PwC
Job Description
Must have key skills
Gen AI Frameworks,
Data Engineering,
generative ai,
Azure Databricks,
Python
Other key skills
Pyspark,
CI CD Pipeline,
Data Pipelines,
Spark,
SQL
Job description
What you’ll do
Role & responsibilities
Responsibilities:
- Design, develop, and maintain data pipelines and ETL processes for GenAI projects.
- Collaborate with data scientists and software engineers to implement machine learning models and algorithms.
- Optimize data infrastructure and storage solutions to ensure efficient and scalable data processing.
- Implement event-driven architectures to enable real-time data processing and analysis.
- Utilize containerization technologies like Kubernetes and Docker for efficient deployment and scalability.
- Develop and maintain data lakes for storing and managing large volumes of structured and unstructured data.
- Implement and integrate LLM frameworks (Langchain, Semantic Kernel) for advanced language processing and analysis.
- Collaborate with cross-functional teams to design and implement solution architectures for GenAI projects.
- Utilize cloud computing platforms such as Azure or AWS for data processing, storage, and deployment.
- Monitor and troubleshoot data pipelines and systems to ensure smooth and uninterrupted data flow.
- Stay up-to-date with the latest advancements in GenAI technologies and recommend innovative solutions to enhance data engineering processes.
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
- Document data engineering processes, methodologies, and best practices.
- Maintain solution architecture certificates and stay current with industry best practices.
Requirements:
Python Proficiency: Minimum 3 years of hands-on experience building applications with Python.
Scalable System Design: Solid understanding of designing and architecting scalable Python applications, particularly for Gen AI use cases, with a strong understanding of various components and systems architecture patterns to make cohesive and decoupled, scalable applications.
Web Frameworks: Familiarity with Python web frameworks (Flask, FastAPI) for building web applications around AI models.
Modular Design & Security: Demonstrated ability to design applications with modularity, reusability, and security best practices in mind (session management, vulnerability prevention, etc.,).
Cloud-Native Development: Familiarity with cloud-native development patterns and tools (e.g., REST APIs, microservices, serverless functions).
Cloud Deployments: Experience deploying and managing containerized applications on Azure/AWS (Azure Kubernetes Service, Azure Container Instances, or similar).
Version Control (Git): Strong proficiency in Git for effective code collaboration and management.
CI/CD: Knowledge of continuous integration and deployment (CI/CD) practices on cloud platforms.
3-5 years of relevant technical/technology experience, with a focus on GenAI projects.
Strong programming skills in Python.
Experience with data processing frameworks like Apache Spark or similar.
Proficiency in SQL and database management systems.
Preferred Skills:
Gen AI Frameworks: Experience with LLM frameworks or tools for interacting with LLMs such as LangChain, Semantic Kernel, LlamaIndex
Data Pipelines: Experience in setting up data pipelines for model training and real-time inference.
Industry type
Management Consulting
Department
Consulting
Role
Sr. Consultant
Role category
Management Consulting
Employment type
Full Time, Permanent
Education
Any Graduate, Any Postgraduate
Gen AI Frameworks,
Data Engineering,
generative ai,
Azure Databricks,
Python
Other key skills
Pyspark,
CI CD Pipeline,
Data Pipelines,
Spark,
SQL
Job description
What you’ll do
Role & responsibilities
Responsibilities:
- Design, develop, and maintain data pipelines and ETL processes for GenAI projects.
- Collaborate with data scientists and software engineers to implement machine learning models and algorithms.
- Optimize data infrastructure and storage solutions to ensure efficient and scalable data processing.
- Implement event-driven architectures to enable real-time data processing and analysis.
- Utilize containerization technologies like Kubernetes and Docker for efficient deployment and scalability.
- Develop and maintain data lakes for storing and managing large volumes of structured and unstructured data.
- Implement and integrate LLM frameworks (Langchain, Semantic Kernel) for advanced language processing and analysis.
- Collaborate with cross-functional teams to design and implement solution architectures for GenAI projects.
- Utilize cloud computing platforms such as Azure or AWS for data processing, storage, and deployment.
- Monitor and troubleshoot data pipelines and systems to ensure smooth and uninterrupted data flow.
- Stay up-to-date with the latest advancements in GenAI technologies and recommend innovative solutions to enhance data engineering processes.
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
- Document data engineering processes, methodologies, and best practices.
- Maintain solution architecture certificates and stay current with industry best practices.
Requirements:
Python Proficiency: Minimum 3 years of hands-on experience building applications with Python.
Scalable System Design: Solid understanding of designing and architecting scalable Python applications, particularly for Gen AI use cases, with a strong understanding of various components and systems architecture patterns to make cohesive and decoupled, scalable applications.
Web Frameworks: Familiarity with Python web frameworks (Flask, FastAPI) for building web applications around AI models.
Modular Design & Security: Demonstrated ability to design applications with modularity, reusability, and security best practices in mind (session management, vulnerability prevention, etc.,).
Cloud-Native Development: Familiarity with cloud-native development patterns and tools (e.g., REST APIs, microservices, serverless functions).
Cloud Deployments: Experience deploying and managing containerized applications on Azure/AWS (Azure Kubernetes Service, Azure Container Instances, or similar).
Version Control (Git): Strong proficiency in Git for effective code collaboration and management.
CI/CD: Knowledge of continuous integration and deployment (CI/CD) practices on cloud platforms.
3-5 years of relevant technical/technology experience, with a focus on GenAI projects.
Strong programming skills in Python.
Experience with data processing frameworks like Apache Spark or similar.
Proficiency in SQL and database management systems.
Preferred Skills:
Gen AI Frameworks: Experience with LLM frameworks or tools for interacting with LLMs such as LangChain, Semantic Kernel, LlamaIndex
Data Pipelines: Experience in setting up data pipelines for model training and real-time inference.
Industry type
Management Consulting
Department
Consulting
Role
Sr. Consultant
Role category
Management Consulting
Employment type
Full Time, Permanent
Education
Any Graduate, Any Postgraduate
