Senior Data Engineer

Overview

Reports To: The Senior Data Engineer will report to Line Manager.Job Overview:We are seeking an experienced Senior Data Engineer to join our dynamic team. The ideal candidatewill be responsible for Design and implement the data engineering framework.Responsibilities Strong Skill in Big Query, GCP Cloud Data Fusion (for ETL/ELT) and PowerBI. Need to have strong skill in Data Pipelines Able to work with Power BI and Power BI Reporting Design and implement the data engineering framework and data pipelines using Databricksand Azure Data Factory. Document the high-level design components of the Databricks data pipeline framework. Evaluate and document the current dependencies on the existing DEI toolset and agree amigration plan. Lead on the Design and implementation an MVP Databricks framework. Document and agree an aligned set of standards to support the implementation of acandidate pipeline under the new framework. Support integrating a test automation approach to the Databricks framework in conjunctionwith the test engineering function to support CI/CD and automated testing. Support the development teams capability building by establishing an L&D and knowledgetransition approach. Support the implementation of data pipelines against the new framework in line with theagreed migration plan. Ensure data quality management including profiling, cleansing and deduplication to supportbuild of data products for clients.QualificationsEducational Background: Bachelor’s degree in Software Engineering, Computer Science, or a related field.Skills Knowledge and ExpertiseData EngineerJob Description2 Experience working in Azure Cloud using Azure SQL, Azure Databricks, Azure Data Lake, DeltaLake, and Azure DevOps. Proficient in Python, Pyspark and SQL coding skills. Profiling data and data modelling experience on large data transformation projects creatingdata products and data pipelines. Creating data management frameworks and data pipelines which are metadata and businessrules driven using Databricks. Experience of reviewing datasets for data products in terms of data quality management andpopulating data schemas set by Data Modellers Experience with data profiling, data quality management and data cleansing tools.

Job Type: Full-time

Pay: ₹472,241.21 - ₹1,821,771.18 per year

Benefits:

• Cell phone reimbursement
• Health insurance
• Provident Fund

Schedule:

• Morning shift

Application Question(s):

• Experience working in Azure Cloud using Azure SQL, Azure Databricks, Azure Data Lake, and Delta

Lake, and Azure DevOps.

• Proficient in Python, Pyspark and SQL coding skills.
• Profiling data and data modelling experience on large data transformation projects creating

data products and data pipelines

• Creating data management frameworks and data pipelines which are metadata and business

rules driven using Databricks.

• Experience of reviewing datasets for data products in terms of data quality management and

populating data schemas set by Data Modellers

• Experience with data profiling, data quality management and data cleansing tools.

Work Location: In person