Data Engineer – GCP, Airflow & PySpark,BigQuery

Candidates ready to join immediately can share their details via email for quick processing.

📌 CCTC | ECTC | Notice Period | Location Preference - nitin.patil@ust.com

Act fast for immediate attention! ⏳📩 .

Key Responsibilities:

1. Design, develop, and automate scalable data processing workflows using Apache Airflow, PySpark, and Dataproc on Google Cloud Platform (GCP).
2. Build and maintain robust ETL pipelines to handle structured and unstructured data from multiple sources and formats.
3. Manage and provision GCP resources including Dataproc clusters, serverless batches, Vertex AI instances, GCS buckets, and custom images.
4. Provide platform and pipeline support for analytics and product teams, resolving issues related to Spark, BigQuery, Airflow DAGs, and serverless workflows.
5. Collaborate with data scientists, data analysts, and other stakeholders to understand data requirements and deliver reliable solutions.
6. Deliver prompt and effective technical support to internal users for data-related queries and challenges.
7. Optimize and fine-tune data systems for performance, cost-efficiency, and reliability.
8. Conduct root cause analysis for recurring pipeline/platform issues and work with cross-functional teams to implement long-term solutions.

Must-Have Skills:

• Strong programming expertise in Python and SQL
• Deep hands-on experience with Apache Airflow (including Astronomer)
• Strong experience with PySpark, SparkSQL, and Dataproc
• Proven knowledge and implementation experience on GCP data services:
• BigQuery, Vertex AI, Pub/Sub, Cloud Functions, GCS
• Strong troubleshooting skills related to data pipelines, Spark job failures, and cloud data environments
• Familiarity with data modeling, ETL best practices, and distributed systems
• Ability to support and optimize large-scale batch and streaming data processes

Good-to-Have Skills:

• Experience with SQL dialects like HiveQL, PL/SQL, and SparkSQL
• Exposure to serverless data processing and ML model deployment workflows (using Vertex AI)
• Familiarity with Terraform or Infrastructure-as-Code (IaC) for provisioning GCP resources
• Knowledge of data governance, monitoring, and cost control best practices on GCP
• Previous experience in healthcare, retail, or BFSI domains involving large-scale data platforms