Verified Job
Accounting - Finance
Head Data Engineering Bank
Accounting - Finance
#334974
Remote / WFH
Job Description
Lead the data engineering team for designing, developing, implementing and maintaining data systems for the data science team. Design and implement processes to improve data availability, quality and reliability for the consumption of data science team. Responsible for developing and maintaining databases, data sets, improving data quality and efficiency - both on premise and on cloud.
- Manage the data requirements of the data science team
- Guide the data engineering team for designing appropriate data systems, databases and data flow pipelines - both on premise and on cloud
- Build strong data governance practices so as to ensure timely availability of data, data quality, consistency and reliability
- Build and manage existing ETL workflows.
- Building, scheduling, batching and maintaining existing ETL workflows
- Perform logging & unit testing on regular basis on existing ETL workflows
- Implementing open source ETL tool such as SSIS, Airflows
- Design and manage database architecture
- Optimize and maintain existing PostgreSQL database, feeding into SparkBeyond analytics platform.
- Design and implement effective database solutions and models to store and retrieve data from legacy systems
- Design efficient DB architecture (including indexing & vacuuming protocols)
- Writing PostgreSQL queries for data management
- Setting up data governance for database usage and management
- Advance Python expertise - Writing Python 3 coding scripts at advanced levels
- Familiar with Multithreading and Multiprocessing in Python
- Familiar with Pandas and/or Dask
- Maintaining transformation scripts using version control systems (Git)
- Data migration from legacy databases to centralized PostgreSQL data warehouse
- Work jointly with analytics team to identify data sources and liaise with stakeholder to source data into PostgreSQL warehouse
- Monitoring server performance, troubleshooting
- Monthly monitoring of database server performance by performing regular load balancing tests and troubleshooting of issues / errors and publish reports to stakeholders
- Manage the data requirements of the data science team
- Guide the data engineering team for designing appropriate data systems, databases and data flow pipelines - both on premise and on cloud
- Build strong data governance practices so as to ensure timely availability of data, data quality, consistency and reliability
- Build and manage existing ETL workflows.
- Building, scheduling, batching and maintaining existing ETL workflows
- Perform logging & unit testing on regular basis on existing ETL workflows
- Implementing open source ETL tool such as SSIS, Airflows
- Design and manage database architecture
- Optimize and maintain existing PostgreSQL database, feeding into SparkBeyond analytics platform.
- Design and implement effective database solutions and models to store and retrieve data from legacy systems
- Design efficient DB architecture (including indexing & vacuuming protocols)
- Writing PostgreSQL queries for data management
- Setting up data governance for database usage and management
- Advance Python expertise - Writing Python 3 coding scripts at advanced levels
- Familiar with Multithreading and Multiprocessing in Python
- Familiar with Pandas and/or Dask
- Maintaining transformation scripts using version control systems (Git)
- Data migration from legacy databases to centralized PostgreSQL data warehouse
- Work jointly with analytics team to identify data sources and liaise with stakeholder to source data into PostgreSQL warehouse
- Monitoring server performance, troubleshooting
- Monthly monitoring of database server performance by performing regular load balancing tests and troubleshooting of issues / errors and publish reports to stakeholders
