Data Engineer - Python/ETL

Overview

We are seeking a highly skilled Data Engineer with expertise in Python, Apache (Spark/Hadoop), and Airflow to design, develop, and optimize data pipelines that power analytics and business insights.

The ideal candidate should have a strong background in data architecture, ETL development, and workflow orchestration, along with hands-on experience managing large-scale data systems.

Key Responsibilities

• Design, build, and maintain scalable ETL pipelines and data architectures.
• Develop efficient data processing workflows using Apache Spark/Hadoop and Airflow.
• Collaborate with data scientists, analysts, and software engineers to define data requirements and integration strategies.
• Ensure data reliability, accuracy, and security across multiple sources and systems.
• Optimize workflows for performance, scalability, and cost-efficiency.
• Implement monitoring and alerting for data quality and pipeline performance.
• Work with cloud platforms (AWS, Azure, GCP) for data storage, processing, and orchestration.
  
  

• Bachelors or Masters degree in Computer Science, Information Technology, or a related field.
• 6+ years of experience in data engineering or ETL development.
• Strong programming skills in Python.
• Hands-on experience with Apache Spark, Hadoop, or similar big data frameworks.
• Expertise in Airflow or equivalent orchestration tools.
• Strong SQL and data modeling proficiency.
• Familiarity with cloud data services (AWS Glue, Redshift, Azure Data Factory, GCP BigQuery, etc.).
• Experience with Git, CI/CD pipelines, and data versioning tools.
  
  

• Experience with containerization (Docker, Kubernetes).
• Knowledge of data lake and data warehouse architectures.
• Exposure to real-time data processing tools (Kafka, Flink, etc.).
• Strong analytical and problem-solving skills with attention to detail.
• Excellent communication and collaboration abilities.
  
  

(ref:hirist.tech)