Data Engineer

Overview

• Design, develop, and maintain efficient and reliable data pipelines.
• Identify and implement process improvements, automating manual tasks and optimizing data delivery.
• Build and maintain the infrastructure for data extraction, transformation, and loading (ETL) from diverse sources using SQL and AWS cloud technologies.
• Develop data tools and solutions to empower our analytics and data science teams, contributing to product innovation.
  
  

• 2-3 years of experience in a Data Engineering role.
• Familiarity with data pipeline and workflow management tools (e.g., Airflow, Luigi, Azkaban).
• Experience with AWS cloud services.
• Working knowledge of object-oriented/functional scripting in Python
• Experience building and optimizing data pipelines and datasets.
• Strong analytical skills and experience working with structured and unstructured data.
• Understanding of data transformation, data structures, dimensional modeling, metadata management, schema evolution, and workload management.
• A passion for building high-quality, scalable data solutions.
  
  

• Experience with stream-processing systems (e.g., Spark Streaming, Flink).
• Working knowledge of message queuing, stream processing, and scalable data stores.
• Proficiency in SQL and experience with NoSQL databases like Elasticsearch and Cassandra/MongoDB.
  
  

Experience with big data tools such as HDFS/S3, Spark/Flink, Hive, HBase, Kafka/Kinesis.

Skills:- Apache Airflow, azkaban, Amazon Web Services (AWS), Python, Pipeline management, Data Structures, Data Transformation Services, Data modeling, Meta-data management, Spark, Microsoft Message Queuing and SQL