Senior Data Engineer

Job Description:

We are looking for an experienced Data Engineer to join our Platform Team within the developer organization. This role is pivotal in creating libraries, tools, and data pipeline templates that will be used by other development teams. You will collaborate closely with the Data Platform Team, which maintains the organization's analytics platform, to ensure seamless integration and optimal use of our data streams. Additionally, you will play a key role in kickstarting our data governance structure and guide business application teams to leverage our data and analytics infrastructure effectively.

Key Responsibilities:

Data Tool and Pipeline Development:

• Lead the design and development of reusable libraries, tools, and data pipeline templates.
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable solutions.
• Optimize data workflows for performance, scalability, and reliability.

Data Governance and Collaboration:

• Participate actively in data governance forums to establish best practices and standards.
• Ensure compliance with data security and privacy regulations.
• Work closely with the Data Platform Team to align on infrastructure and tool usage.

Guidance and Support:

• Provide expertise and guidance to business application teams on utilizing the data and analytics infrastructure.
• Develop documentation and training materials to facilitate the adoption of data tools and practices.
• Mentor junior engineers and promote a culture of continuous learning.

Infrastructure Optimization:

• Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.
• Implement monitoring and alerting systems for proactive issue detection.
• Evaluate new technologies and tools to enhance our data engineering capabilities.

Qualifications:

• Education:
• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or a related field.

Experience:

• 8+ years of experience in data engineering, with a strong focus on building data pipelines and tools.
• Proven experience working with cloud-based data platforms.
• Experience in participating in data governance initiatives.
• Technical Skills:
• Proficiency in programming languages such as Python, Java, or Scala.
• Strong experience with data pipeline frameworks (e.g., Apache Airflow, AWS Glue).
• Expertise in SQL and experience with data warehousing solutions like Snowflake or Databricks.
• Familiarity with big data technologies (e.g., Spark, Hadoop) and cloud data services.
• Knowledge of data modeling, ETL processes, and data integration techniques.

Soft Skills:

• Excellent communication and interpersonal skills.
• Strong problem-solving abilities and attention to detail.
• Ability to work collaboratively in a fast-paced, agile environment.
• Leadership skills with the ability to mentor and guide team members.

Preferred Qualifications:

• Experience with data visualization tools (e.g., Tableau, Power BI).
• Data Engineer certifications or courses.