Azure Data Engineer (SQL, ADF, Data verse)

Overview

Job Title: Azure Data Engineer (SQL, ADF, Dataverse)

Location: Remote

Department: Data Engineering / Data Management

Job Overview

We are seeking an experienced Azure Data Engineer with expertise in SQL, Azure Data Factory (ADF), and Microsoft Dataverse to join our data engineering team. The ideal candidate will be responsible for designing and developing data integration solutions, leveraging Dataverse for data management, orchestrating data workflows using ADF, and ensuring efficient data processing and storage. This role combines data engineering, cloud architecture, and business intelligence technologies to help the organization manage and analyze its data.

Key Responsibilities

· Design & Implement Data Pipelines: Develop and maintain robust data pipelines using Azure Data Factory (ADF) to move, transform, and load data between different cloud environments (Dataverse, Azure SQL Database, etc.).

· Dataverse Integration: Build and manage data integration workflows within Dataverse, ensuring seamless data flow between Dataverse and other systems, including Azure SQL Database and Azure Data Lake.

· SQL Development & Optimization: Write and optimize SQL queries (T-SQL, PL/SQL) for data extraction, transformation, and reporting. This includes complex joins, stored procedures, and performance optimization in both Dataverse and Azure SQL.

· ETL Process Development: Develop and optimize ETL (Extract, Transform, Load) processes using ADF to automate and streamline data ingestion and transformation workflows.

· Data Quality & Governance: Ensure the integrity and consistency of data during transformations and across systems. Implement data validation and quality checks in both ADF pipelines and Dataverse environments.

· Cloud Data Architecture: Design and implement scalable cloud-based data solutions using Azure technologies, ensuring performance, security, and data accessibility across systems.

· Reporting & Analytics Integration: Collaborate with data analysts and business intelligence teams to integrate data into reporting and analytics platforms like Power BI for decision-making.

· Automation & Scripting: Automate data workflows and processes using PowerShell, Azure CLI, or custom scripts within Azure Data Factory and Dataverse.

· Monitor & Troubleshoot: Continuously monitor data pipelines for performance issues and resolve any integration or performance-related problems. Ensure the operational efficiency of data workflows.

· Documentation: Document the data pipeline architecture, integration processes, and solutions, maintaining clear and concise documentation for team collaboration and future reference.

Required Skills & Qualifications

· Azure Data Factory (ADF): Strong hands-on experience in building, managing, and optimizing Azure Data Factory pipelines for data orchestration.

· SQL Expertise: Proficiency in writing complex SQL queries, stored procedures, and performance tuning for Azure SQL Database, Dataverse, and other relational databases.

· Microsoft Dataverse: Experience with Dataverse in the Power Platform ecosystem, including integrating Dataverse with other Azure services and external data sources.

· ETL Development: Experience in developing and maintaining ETL processes using Azure Data Factory, particularly for cloud-based systems.

· Data Management in Azure: Knowledge of Azure data services, including Azure SQL Database, Azure Data Lake, Azure Synapse Analytics, and Azure Blob Storage.

· Cloud Data Solutions: Experience with cloud-based data architecture and solutions, particularly within the Azure ecosystem.

· Scripting & Automation: Familiarity with scripting languages such as PowerShell or Azure CLI for automation within ADF and Dataverse environments.

· Version Control: Experience with version control systems like Git and Azure DevOps for managing code and deployment processes.

Preferred Skills & Qualifications

· Advanced Analytics: Familiarity with advanced analytics tools like Azure Synapse Analytics, Azure Machine Learning, or DataBricks.

· Big Data Technologies: Understanding of big data processing frameworks (such as Spark or Hadoop) and their integration with Azure services.

· Data Governance & Security: Knowledge of data governance, compliance, and security best practices within cloud environments, particularly for Azure and Dataverse.

· Certifications: Microsoft certifications such as Microsoft Certified: Azure Data Engineer Associate, Microsoft Certified: Power Platform App Maker Associate, or related certifications are a plus.

Education & Experience

· Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent experience).

· 2+ years of hands-on experience in data engineering, with specific experience in Azure Data Factory, SQL, and Dataverse.

· Previous experience working with cloud data platforms and business intelligence solutions.

Job Type: Full-time

Pay: Up to ₹400,000.00 per year

Benefits:

• Work from home

Schedule:

• Monday to Friday

Application Question(s):

• Please provide your expected CTC?

Experience:

• SQL: 2 years (Required)
• Azure Data Factory (ADF): 2 years (Required)

Work Location: Remote