Pune, Maharashtra, India
Information Technology
Full-Time
LeewayHertz
Overview
Job Description
Job Description :
We are seeking a highly experienced and innovative Senior Data Engineer with a strong background in hybrid cloud data integration, pipeline orchestration, and AI-driven data modeling. This role is responsible for designing, building, and optimizing robust, scalable, and production-ready data pipelines across both AWS and Azure platforms, supporting
modern data architectures such as CEDM and Data Vault 2.0.
Responsibilities
Essential Skills:
Job
Job
check(event) ; career-website-detail-template-2 => apply(record.id,meta)">
Job Description :
We are seeking a highly experienced and innovative Senior Data Engineer with a strong background in hybrid cloud data integration, pipeline orchestration, and AI-driven data modeling. This role is responsible for designing, building, and optimizing robust, scalable, and production-ready data pipelines across both AWS and Azure platforms, supporting
modern data architectures such as CEDM and Data Vault 2.0.
Responsibilities
- Design and develop hybrid ETL/ELT pipelines using AWS Glue and Azure Data Factory (ADF).
- Process files from AWS S3 and Azure Data Lake Gen2, including schema validation and data profiling.
- Implement event-based orchestration using AWS Step Functions and Apache Airflow (Astronomer).
- Develop and maintain bronze → silver → gold data layers using DBT or Coalesce.
- Create scalable ingestion workflows using Airbyte, AWS Transfer Family, and Rivery.
- Integrate with metadata and lineage tools like Unity Catalog and OpenMetadata.
- Build reusable components for schema enforcement, EDA, and alerting (e.g., MS Teams).
- Work closely with QA teams to integrate test automation and ensure data quality.
- Collaborate with cross-functional teams including data scientists and business stakeholders to align solutions with AI/ML use cases.
- Document architectures, pipelines, and workflows for internal stakeholders.
Essential Skills:
Job
- Experience with cloud platforms: AWS (Glue, Step Functions, Lambda, S3, CloudWatch, SNS, Transfer Family) and Azure (ADF, ADLS Gen2, Azure Functions,Event Grid).
- Skilled in transformation and ELT tools: Databricks (PySpark), DBT, Coalesce, and Python.
- Proficient in data ingestion using Airbyte, Rivery, SFTP/Excel files, and SQL Server extracts.
- Strong understanding of data modeling techniques including CEDM, Data Vault 2.0, and Dimensional Modeling.
- Hands-on experience with orchestration tools such as AWS Step Functions, Airflow (Astronomer), and ADF Triggers.
- Expertise in monitoring and logging with CloudWatch, AWS Glue Metrics, MS Teams Alerts, and Azure Data Explorer (ADX).
- Familiar with data governance and lineage tools: Unity Catalog, OpenMetadata, and schema drift detection.
- Proficient in version control and CI/CD using GitHub, Azure DevOps, CloudFormation, Terraform, and ARM templates.
- Experienced in data validation and exploratory data analysis with pandas profiling,
- AWS Glue Data Quality, and Great Expectations.
- Excellent communication and interpersonal skills, with the ability to engage with teams.
- Strong problem-solving, decision-making, and conflict-resolution abilities.
- Proven ability to work independently and lead cross-functional teams.
- Ability to work in a fast-paced, dynamic environment and handle sensitive issues with discretion and professionalism.
- Ability to maintain confidentiality and handle sensitive information with attention to detail with discretion.
- The candidate must have strong work ethics and trustworthiness
- Must be highly collaborative and team oriented with commitment to excellence.
Job
- Proficiency in SQL and at least one programming language (e.g., Python, Scala).
- Experience with cloud data platforms (e.g., AWS, Azure, GCP) and their data and AI services.
- Knowledge of ETL tools and frameworks (e.g., Apache NiFi, Talend, Informatica).
- Deep understanding of AI/Generative AI concepts and frameworks (e.g., TensorFlow, PyTorch, Hugging Face, OpenAI APIs).
- Experience with data modeling, data structures, and database design.
- Proficiency with data warehousing solutions (e.g., Redshift, BigQuery, Snowflake).
- Hands-on experience with big data technologies (e.g., Hadoop, Spark, Kafka).
- Demonstrate proactive thinking
- Should have strong interpersonal relations, expert business acumen and mentoring skills
- Have the ability to work under stringent deadlines and demanding client conditions
- Ability to work under pressure to achieve the multiple daily deadlines for client deliverables with a mature approach
- Bachelor’s in Engineering with specialization in Computer Science or Artificial Intelligence or Information Technology or a related field.
- 9+ years of experience in data engineering and data architecture.
check(event) ; career-website-detail-template-2 => apply(record.id,meta)">
Similar Jobs
View All
3 Hours ago
IOT - Product Manager
Information Technology
- 4 - 6 Yrs
- Mumbai
Our client is a pioneering Cleantech platform that bridges finance and technology to accelerate the adoption of rooftop solar energy in India. Backed by top-tier investors, our client combines a digital lending platform and IoT-enabled asset manageme...
1 Day ago
MERN Stack Developer - Remote
Information Technology
- 2 - 4 Yrs
- Anywhere in India/Multiple Locations
Job responsibility:
Hire and mentor a team of high-calibre front-end and Backend engineers
Design and develop new user-facing full-stack applications.
Building reusable components and front-end, backend libraries for future use
Optimizing...
1 Day ago
Robotics Software Engineer 3 (Senior-Level)
Information Technology
- Surat, Gujarat, India
Title: Robotics Software Engineer 1 (Junior)DescriptionAuzmor is redefining workforce training by seamlessly integrating human and robotic skill development to empower the hybrid workforce of tomorrow. As a Robotics & AI Research Engineer in our Rob...
1 Day ago
Security Analyst - CISA/CISM
Information Technology
- Surat, Gujarat, India
Kansoft is looking for a Security Analyst who will play a vital role in strengthening our information security framework, supporting audit readiness, and ensuring alignment with global security standards. The ideal candidate will have a strong under...
1 Day ago
Java Developer - Spring Boot/Microservices Architecture
Information Technology
- Surat, Gujarat, India
Java DeveloperWe are seeking a skilled and experienced Java Developer with 6+ years of hands-on development experience to design, develop, and maintain scalable Java-based applications. The ideal candidate will have deep expertise in Java 8+, Spring...
1 Day ago
Codemonk - Python Backend Developer - Django/FastAPI
Information Technology
- Surat, Gujarat, India
About RoleWe are seeking a skilled Backend Engineer with 2+ years of experience to join our dynamic team, focusing on building scalable web applications using Python frameworks (Django/FastAPI) and cloud technologies. You'll be instrumental in devel...
1 Day ago
Roadcast - Senior Python Developer - Flask/Django Framework
Information Technology
- Surat, Gujarat, India
We are seeking a highly skilled and experienced Senior Python Developer to join our dynamic team. The ideal candidate will have a strong background in designing, developing, and deploying robust and scalable Python applications.You will be responsib...
1 Day ago
Haqdarshak - Data Engineer - ETL
Information Technology
- Surat, Gujarat, India
Skillsets And AttitudesMust have : Educational Background : Bachelors degree in computer science, Information Technology, or a related field. Technical Proficiency Hands-on experience with database technologies (e.g., Oracle, SQL Server, MySQL, P...
Talk to us
Feel free to call, email, or hit us up on our social media accounts.
Email
info@antaltechjobs.in