Senior Data Engineer

Overview

Senior Data Engineer
Location: Hyderabad / Remote
Experience: 6+ years
Employment Type: Full-time

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data processing solutions. The ideal candidate will have deep expertise in ETL/ELT development, cloud platforms (Azure or AWS), and strong skills in Python, SQL, and Spark. This role involves enabling advanced analytics and data-driven decision-making across the organization by ensuring clean, reliable, and high-performance data systems.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines for multi-source data ingestion and transformation.
• Build and optimize cloud-native data processing frameworks using Azure Synapse, Databricks, or equivalent AWS services.
• Develop robust data models ensuring quality, consistency, and governance across the ecosystem.
• Write and optimize complex SQL queries and Spark jobs for efficient data processing.
• Collaborate with Data Scientists, Analysts, and Architects to deliver trusted datasets for analytics and reporting.
• Automate workflows to improve scalability, performance, and reliability.
• Monitor and troubleshoot pipelines to ensure smooth and efficient operations.

Requirements

• 6–8 years of proven experience in Data Engineering or Big Data development.
• Strong proficiency in Python and advanced SQL.
• Hands-on experience with Apache Spark and cloud data platforms (Azure Synapse, Azure Data Factory, Databricks, or AWS Glue).
• Solid understanding of ETL/ELT processes, data warehousing, and data lake architectures.
• Familiarity with version control (Git) and data orchestration tools (Airflow, ADF, Prefect).
• Knowledge of data governance, security, and performance optimization best practices.

Preferred Qualifications

• Experience with Azure Data Lake, AWS Redshift, or similar technologies.
• Exposure to real-time data streaming (Kafka, Event Hubs).
• Familiarity with DevOps and Infrastructure-as-Code (Terraform, ARM templates).
• Cloud certifications such as Microsoft Azure Data Engineer Associate or AWS Data Analytics Specialty.