Free cookie consent management tool by TermsFeed Data Engineer – GCP, Airflow & PySpark,BigQuery | Antal Tech Jobs
Back to Jobs
3 Days ago

Data Engineer – GCP, Airflow & PySpark,BigQuery

decor
Gurugram, Haryana, India
Information Technology
Full-Time
UST

Overview

Role Description

Key Responsibilities:

  • Design, develop, and automate scalable data processing workflows using Apache Airflow, PySpark, and Dataproc on Google Cloud Platform (GCP).
  • Build and maintain robust ETL pipelines to handle structured and unstructured data from multiple sources and formats.
  • Manage and provision GCP resources including Dataproc clusters, serverless batches, Vertex AI instances, GCS buckets, and custom images.
  • Provide platform and pipeline support for analytics and product teams, resolving issues related to Spark, BigQuery, Airflow DAGs, and serverless workflows.
  • Collaborate with data scientists, data analysts, and other stakeholders to understand data requirements and deliver reliable solutions.
  • Deliver prompt and effective technical support to internal users for data-related queries and challenges.
  • Optimize and fine-tune data systems for performance, cost-efficiency, and reliability.
  • Conduct root cause analysis for recurring pipeline/platform issues and work with cross-functional teams to implement long-term solutions.

Must-Have Skills

  • Strong programming expertise in Python and SQL
  • Deep hands-on experience with Apache Airflow (including Astronomer)
  • Strong experience with PySpark, SparkSQL, and Dataproc
  • Proven knowledge and implementation experience on GCP data services:
    • BigQuery, Vertex AI, Pub/Sub, Cloud Functions, GCS
  • Strong troubleshooting skills related to data pipelines, Spark job failures, and cloud data environments
  • Familiarity with data modeling, ETL best practices, and distributed systems
  • Ability to support and optimize large-scale batch and streaming data processes
Good-to-Have Skills

  • Experience with SQL dialects like HiveQL, PL/SQL, and SparkSQL
  • Exposure to serverless data processing and ML model deployment workflows (using Vertex AI)
  • Familiarity with Terraform or Infrastructure-as-Code (IaC) for provisioning GCP resources
  • Knowledge of data governance, monitoring, and cost control best practices on GCP
  • Previous experience in healthcare, retail, or BFSI domains involving large-scale data platforms

Educational Qualification

  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field
  • Certifications in GCP Data Engineer, GCP Professional Cloud Architect, or Apache Spark are a plus

Skills

Gcp,Pyspark,Airflow
Share job
Similar Jobs
View All
1 Day ago
Software Engineer - Backend Development
Information Technology
  • Indore, Madhya Pradesh, India
Job Title : Software Engineer BackendLocation : Bengaluru, IndiaPosition Summary : We are looking for an experienced Software Engineer Backend with expertise in Python, object-oriented programming, Django, and Node.js. The ideal candidate should als...
decor
1 Day ago
Python Full Stack Developer in Hyderabad
Information Technology
  • Indore, Madhya Pradesh, India
Key Responsibilities Write reusable, testable, and efficient code Design and implement low-latency, high-availability, and performant applications Integrate user-facing elements developed by front-end developers with server-side logic Implement ...
decor
1 Day ago
Studio Diseno Innovative - Senior Business Analyst
Information Technology
  • Indore, Madhya Pradesh, India
Job DescriptionThe impact that you will be makingThe role would require you to give end-to-end support to the product development and analytics services teams and maintain a strong relationship with our clients.What Lands You In This Role 3-6 years...
decor
1 Day ago
Technical Project Manager - Full Stack Development
Information Technology
  • Indore, Madhya Pradesh, India
Job Title : Full Stack ( With (Technical Project Manager)Location : Bengaluru, IndiaExperience : 7-8 yearsRole OverviewWe are looking for a proactive and technically strong Technical Project Manager (TPM) to drive the successful delivery of cloud-na...
decor
1 Day ago
Nihilent Technologies - Data Architect - ETL Platform
Information Technology
  • Indore, Madhya Pradesh, India
Designation : Data Architect.Location : Pune.Experience : 10-15 years.Job DescriptionRole & Responsibilities : The architect should have experience in architecting large scale analytics solutions using native services such as Azure Synapse, Data La...
decor
1 Day ago
Interesting Job Opportunity: Full Stack Developer - .Net/AngularJS
Information Technology
  • Indore, Madhya Pradesh, India
Job DescriptionWe are seeking an experienced and .NET full stack Developer with 3 to 6 years of hands-on development experience to join our growing team. The ideal candidate will be a skilled software engineer who can drive the development of .NET C...
decor
1 Day ago
Studio Diseno Innovative - Java Software Engineer - Spring/Microservices Architecture
Information Technology
  • Indore, Madhya Pradesh, India
Job DescriptionResponsibilities : Understand project requirements and translate them into technical solutions. Develop and maintain Java-based applications with a focus on efficiency, reliability, and scalability. Utilize Core Java, JSP, and Serv...
decor
1 Day ago
Data Engineer - Spark/Apache Flink
Information Technology
  • Indore, Madhya Pradesh, India
Required Skills Proficiency in multiple programming languages ideally Python Proficiency in at least one cluster computing frameworks (preferably Spark, alternatively Flink or Storm) Proficiency in at least one cloud data lakehouse platforms (pre...
decor

Talk to us

Feel free to call, email, or hit us up on our social media accounts.
Social media