Lead Data Engineer (Databricks)

About Rearc

Founded in 2016, we pride ourselves on fostering an environment where creativity flourishes, bureaucracy is non-existent, and individuals are encouraged to challenge the status quo. We're not just a company; we're a community of problem-solvers dedicated to improving the lives of fellow software engineers.

Our commitment is simple - finding the right fit for our team and cultivating a desire to make things better. If you're a cloud professional intrigued by our problem space and eager to make a difference, you've come to the right place. Join us, and let's solve problems together!

As a Lead Data Engineer at Rearc, you'll play a pivotal role in establishing and maintaining technical excellence within our data engineering team. Your deep expertise in data architecture, ETL processes, and data modelling will be instrumental in optimizing data workflows for efficiency, scalability, and reliability. You'll collaborate closely with cross-functional teams to design and implement robust data solutions that meet business objectives and adhere to best practices in data management. Building strong partnerships with both technical teams and stakeholders will be essential as you drive data-driven initiatives and ensure their successful implementation.

What You Bring

• With 10+ years of experience in data engineering, data architecture, or related fields, you offer a wealth of expertise in managing and optimizing data pipelines and architectures.
• Extensive experience in writing and testing Java and/or Python
• Proven experience with data pipeline orchestration using platforms such as Airflow, Databricks, DBT or AWS Glue.
• Hands-on experience with data analysis tools and libraries like Pyspark, NumPy, Pandas, or Dask.
• Proficiency with Spark and Databricks is highly desirable.
• You have a proven track record of leading complex data engineering projects, including designing and implementing scalable data solutions.
• Your hands-on experience with ETL processes, data warehousing, and data modeling tools allows you to deliver efficient and robust data pipelines.
• You possess in-depth knowledge of data integration tools and best practices.
• Your strong understanding of cloud-based data services and technologies (e.g., AWS Redshift, Azure Synapse Analytics, Google BigQuery).
• You bring strong strategic and analytical skills to the role, enabling you to solve intricate data challenges and drive data-driven decision-making.
• Proven proficiency in implementing and optimizing data pipelines using modern tools and frameworks, including Databricks for data processing and Delta Lake for managing large-scale data lakes.
• Your exceptional communication and interpersonal skills facilitate collaboration with cross-functional teams and effective stakeholder engagement at all levels.

What You’ll Do

As a Lead Data Engineer at Rearc, your role is pivotal in driving the success of our data engineering initiatives. You will lead by example, fostering trust and accountability within your team while leveraging your technical expertise to optimize data processes and deliver exceptional data solutions. Here's what you'll be doing:

• Understand Requirements and Challenges: Collaborate with stakeholders to deeply understand their data requirements and challenges, enabling the development of robust data solutions tailored to the needs of our clients.
• Implement with a DataOps Mindset: Embrace a DataOps mindset and utilize modern data engineering tools and frameworks, such as Apache Airflow, Apache Spark, or similar, to build scalable and efficient data pipelines and architectures.
• Lead Data Engineering Projects: Take the lead in managing and executing data engineering projects, providing technical guidance and oversight to ensure successful project delivery.
• Mentor Data Engineers: Share your extensive knowledge and experience in data engineering with junior team members, guiding and mentoring them to foster their growth and development in the field.
• Promote Knowledge Sharing: Contribute to our knowledge base by writing technical blogs and articles, promoting best practices in data engineering, and contributing to a culture of continuous learning and innovation.

At Rearc, we're committed to empowering engineers to build awesome products and experiences. Success as a business hinges on our people's ability to think freely, challenge the status quo, and speak up about alternative problem-solving approaches. If you're an engineer driven by the desire to solve problems and make a difference, you're in the right place!

Our approach is simple — empower engineers with the best tools possible to make

an impact within their industry.

We're on the lookout for engineers who thrive on ownership and freedom, possessing not just technical prowess, but also exceptional leadership skills. Our ideal candidates are hands-on-keyboard leaders who don't just talk the talk but also walk the walk, designing and building solutions that push the boundaries of cloud computing.