Free cookie consent management tool by TermsFeed Data Engineer – GCP, Airflow & PySpark,BigQuery | Antal Tech Jobs
Back to Jobs
2 Days ago

Data Engineer – GCP, Airflow & PySpark,BigQuery

decor
Chennai, Tamil Nadu, India
Space Exploration & Research, Information Technology
Other
UST Global

Overview

    9 - 12 Years
    1 Opening
    Trivandrum


Role description

Key Responsibilities:

  • Design, develop, and automate scalable data processing workflows using Apache Airflow, PySpark, and Dataproc on Google Cloud Platform (GCP).

  • Build and maintain robust ETL pipelines to handle structured and unstructured data from multiple sources and formats.

  • Manage and provision GCP resources including Dataproc clusters, serverless batches, Vertex AI instances, GCS buckets, and custom images.

  • Provide platform and pipeline support for analytics and product teams, resolving issues related to Spark, BigQuery, Airflow DAGs, and serverless workflows.

  • Collaborate with data scientists, data analysts, and other stakeholders to understand data requirements and deliver reliable solutions.

  • Deliver prompt and effective technical support to internal users for data-related queries and challenges.

  • Optimize and fine-tune data systems for performance, cost-efficiency, and reliability.

  • Conduct root cause analysis for recurring pipeline/platform issues and work with cross-functional teams to implement long-term solutions.

Must-Have Skills:

  • Strong programming expertise in Python and SQL

  • Deep hands-on experience with Apache Airflow (including Astronomer)

  • Strong experience with PySpark, SparkSQL, and Dataproc

  • Proven knowledge and implementation experience on GCP data services:

    • BigQuery, Vertex AI, Pub/Sub, Cloud Functions, GCS

  • Strong troubleshooting skills related to data pipelines, Spark job failures, and cloud data environments

  • Familiarity with data modeling, ETL best practices, and distributed systems

  • Ability to support and optimize large-scale batch and streaming data processes

Good-to-Have Skills:

  • Experience with SQL dialects like HiveQL, PL/SQL, and SparkSQL

  • Exposure to serverless data processing and ML model deployment workflows (using Vertex AI)

  • Familiarity with Terraform or Infrastructure-as-Code (IaC) for provisioning GCP resources

  • Knowledge of data governance, monitoring, and cost control best practices on GCP

  • Previous experience in healthcare, retail, or BFSI domains involving large-scale data platforms

Educational Qualification:

  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field

  • Certifications in GCP Data Engineer, GCP Professional Cloud Architect, or Apache Spark are a plus

Skills

Gcp,Pyspark,Airflow


About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
Share job
Similar Jobs
View All
1 Day ago
Lead Python Developer
Information Technology
  • 8 - 12 Yrs
  • Anywhere in India/Multiple Locations
My Client is revolutionizing the hospitality industry by transforming how businesses manage their finances through cutting-edge Accounts Payable automation. Our mission? To eliminate manual work from business expenses and empower operators with real-...
decor
1 Day ago
Business Analyst I, DeviceOS
Information Technology
  • Chennai, Tamil Nadu, India
This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. DescriptionAmazon is looking for a data-savvy professional to crea...
decor
1 Day ago
IT Support Mgr II
Information Technology
  • Chennai, Tamil Nadu, India
This job is with Amazon, an inclusive employer and a member of myGwork – the largest global platform for the LGBTQ+ business community. Please do not contact the recruiter directly. DescriptionAre you passionate about helping people solve IT proble...
decor
1 Day ago
EverestDX - Senior Python Developer
Information Technology
  • Chennai, Tamil Nadu, India
Job Title : Sr. Python DeveloperJob Location : Hyderabad Type : Full : 8+ Period : Immediate to 15 days joiners are highly : Design, develop, and maintain scalable Python applications and AI-powered solutions. Collaborate with cross-functional tea...
decor
1 Day ago
.NET Application Developer
Information Technology
  • Chennai, Tamil Nadu, India
🌟 We're Hiring: .NET Application Developer! 🌟We are looking for a skilled .NET Application Developer with at least 5 years of experience to join our dynamic team. The ideal candidate will have a strong background in developing robust applications us...
decor
1 Day ago
Full Stack Developer - Node.js/React.js
Information Technology
  • Chennai, Tamil Nadu, India
Designation : Full Stack Developer( React.js+ Node.js)Job Location : PrabhadeviJob Day & Timing : Mon - Fri , 9 AM -6 PMJob Description Perform hands-on computer program improvement, counting composing code and unit tests, doing verification of co...
decor
1 Day ago
Sr. Test Engineer - QA Job
Information Technology
  • Chennai, Tamil Nadu, India
We use cookies to offer you the best possible website experience. Your cookie preferences will be stored in your browser’s local storage. This includes cookies necessary for the website's operation. Additionally, you can freely decide and change any...
decor
1 Day ago
QA Engineer
Information Technology
  • Chennai, Tamil Nadu, India
Flowserve is a world-leading manufacturer and aftermarket service provider of comprehensive flow control systems. Join a company whose people are committed to building a more sustainable future to make the world better for everyone. With 16,000+ emp...
decor

Talk to us

Feel free to call, email, or hit us up on our social media accounts.
Social media