Senior Data Engineer

ROLES & RESPONSIBILITIES

• Assume full responsibility for the entire lifecycle of the "data processing platform."
• Maintain and enhance a transformative platform that empowers non-technical colleagues to independently build and manage end-to-end Airflow data processing pipelines.
• Utilize the scalability of Spark clusters on AWS to ensure platform performance and reliability.
• Maintain and improve an intuitive, virtually no-code interface for data assembly, transformation, and serialization.
• Support sophisticated media planning solutions through the data processing platform.
• Democratize data access and manipulation across diverse departments.
• Expedite the delivery of valuable insights by enabling self-service data processing.
• Work in an agile environment, collaborating closely with software developers, data scientists, and client-facing solution managers.
• Adhere to software engineering best practices in all aspects of platform development and maintenance.
• Utilize proficiency in distributed computational engines such as Spark, Presto, and Trino for data processing tasks.
• Work with cloud platforms, particularly AWS, and potentially Azure and Google Cloud Platform (GCP).
• Employ containerization technologies (Docker, Kubernetes, etc.) for deployment and management of platform components.
• Implement and maintain continuous integration pipelines for platform updates and deployments.
• Interact effectively with Data Science teams to understand their data processing needs and requirements.
• Design and implement scalable data processing pipelines capable of efficiently handling data volumes from megabytes to terabytes while consistently applying business rules.
• Continuously learn and develop skills in areas such as big data, cloud computing, data lake architecture, and data orchestration.
• Collaborate effectively as a team player, demonstrating flexibility, a proactive approach, and pragmatism.
• Apply excellent problem-solving and analytical skills to address technical challenges.
• Communicate effectively and collaborate with team members and stakeholders from various disciplines.

QUALIFICATIONS

• 5+ years of experience in data engineering.
• Bachelor or master’s degree in Computer Science Data Engineering, or proven experience in the field;
• Knowledge of and experience with software engineering best practices is crucial.
• Proficiency in distributed computational engines such as Spark, Presto, and Trino.
• Familiarity with cloud platforms like AWS, Azure, and Google Cloud Platform (GCP).Experience with containerization technologies (Docker, Kubernetes, etc.).
• Knowledge of continuous integration pipelines.
• Ability to interact with Data science teams as key stakeholders.
• Capability to design scalable data processing pipelines that can efficiently handle data volumes ranging from a few megabytes to terabytes, while maintaining consistency in applying the same business rules.
• Fast learner
• Team player with flexible, proactive and pragmatic attitude;
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.