Data Engineer

Overview

This role is for one of the Weekday's clients

We are seeking a highly skilled Data Engineer with a strong background in building and managing scalable data pipelines. You will play a critical role in designing, developing, and optimizing data infrastructure to support analytics and real-time processing needs. This role requires expertise in relational and NoSQL databases, data warehouses, real-time streaming, ETL processes, and cloud-based data solutions.

Requirements:
Key Responsibilities:

• Design, develop, and optimize scalable data pipelines for structured and unstructured data.
• Work with relational databases (MySQL, PostgreSQL) and NoSQL databases (MongoDB, Cassandra) to manage and process large datasets efficiently.
• Build and maintain data pipelines with data warehouses like Snowflake, Redshift for efficient storage and retrieval.
• Process unstructured data stored in S3 using Athena, Glue, and other AWS services to derive meaningful insights.
• Implement real-time data streaming and messaging solutions using Kafka to enable low-latency data processing.
• Develop ETL pipelines to ensure smooth data integration, transformation, and optimization for analytics and reporting.
• Write high-performance code using Python, Java, or Scala to process and analyze large datasets efficiently.
• Utilize Apache Spark for big data processing and analytics to enhance data engineering workflows.
• Work with cloud platforms like AWS, GCP, or Azure to build scalable and resilient data infrastructure.

• 5+ years of experience in data engineering, with a focus on data pipeline development, data processing, and optimization.
• Strong expertise in Relational Databases (MySQL, PostgreSQL) and NoSQL databases (MongoDB, Cassandra).
• Experience working with data warehouses such as Snowflake and Redshift for analytical workloads.
• Hands-on experience in processing unstructured data using tools like Athena, Glue, and S3.
• Proficiency in real-time data streaming using Kafka for handling large-scale event-driven data processing.
• Strong knowledge of ETL processes, data integration techniques, and pipeline performance optimization.
• Proficiency in programming languages like Python, Java, or Scala for data processing and automation.
• Experience with Apache Spark for distributed big data processing is a plus.
• Familiarity with cloud platforms (AWS, GCP, or Azure) for data infrastructure management is an advantage.