Free cookie consent management tool by TermsFeed Senior Data Engineer - Python/PySpark | Antal Tech Jobs
Back to Jobs
2 Days ago

Senior Data Engineer - Python/PySpark

decor
Delhi, Delhi, India
Information Technology
Full-Time
qpact

Overview

We are looking for a highly skilled Senior Data Engineer with 710 years of professional experience in data engineering and cloud-based data platforms. The ideal candidate will be proficient in Azure technologies and have strong expertise in data architecture, data integration, data modeling, and advanced ETL/ELT pipelines. You will play a key role in building scalable, reliable, and high-performing data solutions that enable advanced analytics and business insights.

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines and data integration workflows using Azure Data Factory, Azure Databricks, and Azure Synapse.
  • Implement data warehouse and lakehouse solutions leveraging medallion architecture and modern best practices.
  • Perform data profiling, anomaly detection, and quality checks to ensure reliability and accuracy of data.
  • Design and optimize ETL/ELT processes for batch and real-time data ingestion.
  • Build robust data models (Dimensional/Kimball) for analytics, BI, and reporting solutions.
  • Optimize and manage relational database systems for performance, scalability, and availability.
  • Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions.
  • Drive the adoption of data engineering best practices, coding standards, and agile methodologies.
  • Implement CI/CD pipelines and data DevOps practices for automated deployments and monitoring.
  • Ensure compliance with data governance, security, and privacy policies.
  • Mentor junior engineers, conduct code reviews, and contribute to knowledge-sharing across the team.

Required Skills & Qualifications

  • Bachelors or Masters degree in Computer Science, Information Technology, Engineering, or related field.
  • 7 - 10 years of professional experience in data engineering.

Proven Expertise With Azure Data Services

  • Azure Data Factory (ETL/ELT workflows)
  • Azure Databricks (big data processing, PySpark/Scala)
  • Azure Synapse Analytics (data warehousing and analytics)
  • Strong knowledge of data warehouse and lakehouse methodologies including medallion architecture.
  • Hands-on experience with ETL/ELT pipelines, data integration, data profiling, and anomaly detection.
  • Expertise in SQL and relational databases (SQL Server, Oracle, PostgreSQL, or equivalent).
  • Solid understanding of data modeling techniques (Dimensional/Kimball, Star/Snowflake

schemas).

  • Experience working in Agile/Scrum environments.
  • Strong understanding of DevOps practices, CI/CD pipelines, and automated deployments.
  • Excellent problem-solving, analytical, and communication skills.
  • Experience with cloud-native data platforms (AWS Redshift, GCP BigQuery) in addition to Azure.
  • Familiarity with streaming data technologies (Kafka, Event Hubs, Spark Streaming).
  • Knowledge of data governance frameworks, data security, and compliance (GDPR, HIPAA, etc.).
  • Exposure to machine learning pipelines and integration with advanced analytics platforms.
  • Experience with Python, PySpark, or Scala for data engineering and automation.

(ref:hirist.tech)
Share job
Similar Jobs
View All
1 Day ago
Chief Technology Officer (CTO)
Information Technology
  • 9 - 15 Yrs
  • Mumbai (All Areas)
Role & responsibilities * Develop and execute the company's technology roadmap aligned with business goals. * Lead and mentor the engineering and IT teams to ensure high performance. * Identity and implement emerging technologies to enhance ...
decor
2 Days ago
Website Developer
Information Technology
  • 700000 - 1200000 INR - Annual
  • 5 - 10 Yrs
  • Udaipur
Job Title: Website Developer Location: Udaipur, Rajasthan (HO) Job Type: Full-time Experience Level: Senior (5 TO 10 Years) About the Role: We are looking for a skilled and detail-oriented Website Developer to join our team. The ideal candidat...
decor
2 Days ago
Business Analyst
Information Technology
  • Delhi, Delhi, India
Job DescriptionBusiness Analyst, Sales Operations:As Gartner Conferences continues its rapid expansion, we are seeking a highly motivated and analytical Sales Operations Analyst to join our dynamic team. This pivotal role supports the operational ne...
decor
2 Days ago
ChicMic Studios - Python/Django Developer
Information Technology
  • Delhi, Delhi, India
Profile : Python developer.Experience : 2-6 Years.Location : Mohali, Punjab (WFO).ChicMic Studios is on the lookout for a skilled Python Developer who loves turning complex problems into elegant solutions.If you have a passion for Django, Flask, AWS...
decor
2 Days ago
Sr Data Engineer
Information Technology
  • Delhi, Delhi, India
Job DescriptionAbout HighspotHighspot is a software product development company and a recognized global leader in the sales enablement category, leveraging cutting-edge AI and GenAI technologies at the core of its robust Software-as-a-Service (SaaS)...
decor
2 Days ago
ChicMic Studios - Angular Developer - TypeScript/Javascript
Information Technology
  • Delhi, Delhi, India
Key Responsibilities Develop and maintain responsive, high-performance web applications using Angular. Collaborate with UX/UI designers and back-end developers to implement user-friendly features. Optimize application for maximum speed and scalab...
decor
2 Days ago
Oracle PL-SQL Developer
Information Technology
  • Delhi, Delhi, India
Role SummaryWe are seeking a skilled Oracle PL/SQL Developer to design, develop, optimize, and maintain PL/SQL code for a regulatory database application which integrates with middleware and external APIs of MHA and DOT. The ideal candidate will hav...
decor
2 Days ago
Software Engineer Senior
Information Technology
  • Delhi, Delhi, India
The CompanyGentrack provides leading utilities across the world with innovative cleantech solutions. The global pace of change is accelerating, and utilities need to rebuild for a more sustainable future. Working with some of the world’s biggest ene...
decor

Talk to us

Feel free to call, email, or hit us up on our social media accounts.
Social media