Data Engineer

Job Title : Data Engineer

Location: Gurugram, Haryana

Experience: 3-4 Years

Role Overview:

We are seeking a skilled Data Engineer with experience in designing, building, and optimizing big data solutions to meet business and technology needs. The ideal candidate will have a strong background in Hadoop, Hive, Spark, and Python/Java, with proven expertise in handling large-scale datasets, implementing data governance best practices, and ensuring data quality across all layers.

Key Responsibilities:

• Design, develop, and maintain scalable big data pipelines and architectures to support business and analytical needs
• Build and optimize data processing systems using technologies such as Hadoop, Hive, Spark, and Python/Java
• Implement data governance principles, ensuring data quality, integrity, and security across all data layers.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Develop and manage data workflows and orchestration using tools like Airflow, Azkaban, or Luigi
• Integrate and manage data from diverse sources including SQL, NoSQL databases, APIs, and streaming data.
• Design and maintain real-time data streaming and message queuing systems (e.g., Kafka, Spark Streaming, Storm).
• Support the deployment and monitoring of data systems on cloud platforms such as AWS or Azure.
• Conduct data profiling, cleansing, and validation to ensure accuracy and reliability.
• Continuously improve system performance, scalability, and efficiency through automation and optimization.

Qualification:

• Bachelor’s or Graduate degree in Computer Science, Statistics, Informatics, Information Systems, or a related quantitative field.
• 4+ years of hands-on experience as a Data Engineer or in a related data engineering
• role.
• Proven expertise with big data technologies such as Hadoop, Hive, Spark, and Kafka.
• Strong programming skills in Python, Java, or Scala
• Proficiency with SQL and NoSQL databases (e.g., MySQL, PostgreSQL, Cassandra).
• Experience with data pipeline and workflow management tools such as Airflow, Azkaban, or Luigi.
• Familiarity with cloud data services on AWS or Azure (e.g., EC2, EMR, RDS, Redshift, HDInsights, DataLake, DataBricks).
• Working knowledge of stream-processing frameworks such as Spark Streaming or Storm.
• Strong analytical and problem-solving skills with the ability to design solutions for complex business use cases.
• Excellent project management and organizational skills with experience working in dynamic, cross-functional environments.