Senior Data Engineer (Python + Snowflake)

Overview

Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making.

As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.

If your passion for data engineering solutions that make a real-world impact, consider this your pass to the captivating world of Data Science and Engineering! 🌍🔥

Responsibilities:

• Lead the design and implementation of scalable ETL/data pipelines using Python for data processing.
• Ensure efficient data processing for high-volume clickstream, demographics, and business data.
• Lead the strategic planning, execution, and optimization of large-scale data migration initiatives to Snowflake, ensuring data integrity, security, and minimal business disruption.
• Use Snowflake components/tools - Snowpipe, SnowSQL, SnowPark etc
• Guide the team in adopting best practices for data pipeline development, code quality, and performance optimization.
• Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security.
• Support data storage and retrieval workflows using S3 and SQL-based storage solutions.
• Provide architectural guidance for cloud-native data solutions and infrastructure design.
• Oversee legacy framework maintenance, identify improvement areas, and propose comprehensive cloud migration or modernization plans.
• Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints.
• Develop and implement robust monitoring solutions to track system health, performance, and data pipeline accuracy.
• Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues.
• Lead efforts in incident response, root cause analysis, and post-mortem processes for complex data engineering challenges.
• Document workflows, troubleshooting procedures, and code for system transparency and continuity.
• Provide mentoring and training to team members on best practices and technical skills.
• Foster a culture of continuous learning, knowledge sharing, and technical excellence within the data engineering team.
  
  

• Experience: 6 years of experience in data engineering or a related technical field, with at least 4 years in a Snowflake projects.
• Cloud Data Warehousing: Proven expertise with Snowflake data warehousing, including schema design, efficient data loading (e.g., Snowpipe, COPY into), performance tuning, and robust access control mechanisms.
• Programming & Scripting: Strong programming skills in Python for automation and data workflows.
• Data Processing & Storage: Expertise in managing SQL databases for data storage and query optimization.
• Monitoring & Alerting Tools: Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines.
• Experience with the AWS Cloud and data engineering services