Senior Data Engineer - Python/Spark

Overview

• Design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation
• Build and support data platform on the cloud
• Define and implement automation of jobs and testing
• Optimize the data pipeline to support workloads and use cases.
• Support mission critical applications and near real time data needs from the data platform.
• Build and support reusable framework to ingest, integration and provision data.
• Capture and publish metadata and new data to subscribed users
• Work collaboratively with product managers, data scientists as well as business partners and actively participate in solution thinking sessions
• Participate in design and code reviews
• Leads data disaster recovery planning and execution with the disaster recovery team and internal team members.
• Mentors more junior members of the team
• Determines risk of various configurations and options for data administration and is able to communicate issues to managers and team.
• Leads and develops internal standards for data administration and data system documentation.
• Develops physical data design.
• Defines, documents and executes, referential integrity requirements, constraints and techniques.
• Identifies index requirements and determines storage requirements
  
  

• Bachelors degree in computer science, Information Systems, Engineering, or a related field
• 4+ years of hands-on experience in data engineering and building data pipelines
• 4+ years of experience in writing complex SQL and Python
• Solid hands-on experience with data warehousing concepts and implementations
• 4+ years of experience with Snowflake or another modern cloud data warehouse
• 4+ years of hands-on Python development
• Understanding of Data modeling and Data warehousing concepts
• 2+ years experience with cloud computing (Azure)
• Familiarity with at least one cloud platform (AWS, Azure, or GCP)
• Strong analytical, problem-solving, and communication skills
• Retail/e-comm experience
  
  

• Experience with DBT (Data Build Tool) for data transformations
• Exposure to real-time/streaming platforms (Kafka, Spark Streaming, Flink)
• Familiarity with CI/CD and version control (Git) in data engineering projects
• Exposure to the e-commerce or customer data domain
• Experience working on multiple concurrent deliverables
  
  

(ref:hirist.tech)