Senior Data Engineer

Key Responsibilities and Role(s):

• Responsible for constructing and maintaining data pipelines that extract, transform, and load data from diverse data sources into centralized repositories, such as data warehouses, lakehouses, and databases.
• Oversee database management activities, including database design, administration, optimization, and performance enhancement.
• Implement physical data models to effectively structure and organize data for efficient storage and analysis, while ensuring data integrity optimization.
• Integrate data from disparate sources and ensure data quality and consistency throughout the data pipeline.
• Develop processes to transform raw data into a usable format for analysis and reporting purposes, including data cleanup and deduplication.
• Monitor the performance of data pipelines and databases, identify bottlenecks, and optimize processes for improved efficiency.
• Manage and sustain the integrity of data infrastructure components, including servers, storage systems, and networking resources.
• Assist in establishing and enforcing data governance policies and procedures to ensure data quality, safeguard sensitive data assets, and comply with regulatory requirements.
• Assist in maintaining documentation, such as business requirements documents, data dictionaries or data catalogs, data mapping documents, and others.
• Perform any additional tasks and duties as reasonably required or instructed by the manager or supervisor based on team and business needs. These may include, but are not limited to, administrative duties, project support, backup developer roles (out-of-role), and other ad-hoc tasks.

Required Competencies:

• Bachelor’s degree in Computer Science, Information Technology, Mathematics, or an equivalent field.
• Minimum of 6 years of hands-on progressive experience in Data Engineering.
• At least 6 years of experience in ETL, ELT, data warehousing, and data modeling.
• A minimum of 3 years of delivering solutions on the Microsoft Azure Platform, with exposure to data solutions and services such as Microsoft Fabric, and Data Factory.
• Proficiency in PySpark or Python, T-SQL scripting and strong technical knowledge of databases.
• Expertise in designing, building, and maintaining data pipelines to extract, transform, and load (ETL) data from various sources into data storage systems such as data warehouses or lakehouses.
• Skilled in different ETL design methodologies.
• Skilled in data modeling techniques and best practices for designing and optimizing data schemas and structures for efficient storage, retrieval, and analysis.
• Understanding of data quality principles and practices to ensure data accuracy, completeness, consistency, and reliability.