Lead Data Engineer

About The Team

As a Lead Data Engineer, you will be joining the newly formed AI, Data & Analytics team, whose mission

is to drive increased value from the data InvestCloud captures to enable a smarter financial future for our

clients, focused on “enhanced intelligence”. Ensuring we have fit-for-purpose modern capabilities is a key

goal for the team.

This is a unique opportunity to shape the architecture and the tech execution patterns of a green field

ecosystem to create a next-generation advisor and client experience. We're building for scale. As such,

much of what we design and implement today will be the technology/infrastructure which will serve

thousands of clients and petabyte-level volumes of data.

The core stack we use and are building is:

• AWS as our cloud provider
• Oracle as our legacy data warehouse
• Snowflake as our next-gen data warehouse
• Mage AI for data ingestion and processing
• Kafka as our message bus
• Terraform for building infrastructure
  
  

Key Responsibilities

• Ask the right questions to deeply understand stakeholder needs, ensuring solutions are aligned
  
  

with real problems.

• Drive the technical strategy of the team, and its execution through prioritization, and delivery
  
  

management.

• Lead a team to build reliable and scalable pipelines and capabilities across the platform, as well
  
  

as monitor and support the capabilities we offer.

• Inspire, teach and guide your team members; own design sessions, upskill the team where needed,
  
  

maintain the team’s cohesion.

• Own complex architectures tying multiple services, SaaS tooling and third-party data together,
  
  

leveraging a strong understanding of a cloud-based stack.

• Uphold high-quality standards across documentation, testing, resiliency, monitoring, and code
  
  

quality.

• Drive towards efficiency, lower our cloud spend, tackle our tech debt and look for ways to simplify
  
  

code, infrastructure and data models across the platform.

• Promote data quality, governance and security as a first-class citizen of the platform, complying
  
  

with relevant regulations

Required Skills

• You have at least 10 years of relevant professional experience in Data Engineering
• You have shaped the architecture of a mature cloud data platform (AWS, GCP, Azure), designed
  
  

for different consumer types and providing quantifiable business value.

• You have served as a leader in technology in the past; you’ve built a roadmap, planned data/code
  
  

migrations and led execution, made mistakes and learned from them.

• You have interacted with a large community of stakeholders before; you understand the business
  
  

use cases and can tailor your communication to ICs and Senior Management.

• You have designed and implemented performant, reusable, and scalable data models in a cloud
  
  

data warehouse (dbt, BigQuery, Snowflake) and in a legacy ecosystem (Oracle, Postgres)

• You have hands-on experience in building resilient batch (Airflow, Fivetran, Mage AI, Airbyte) and
  
  

streaming (Kafka, Kinesis, Flink, Spark) data pipelines at scale (> 1 TB/day)

• You can build and maintain your own infrastructure through IaC (Terraform, OpenTofu, Ansible),
  
  

containerization (Docker) and CI/CD (Jenkins, Github Actions).

• You preach operational procedures from data and infrastructure observability (Monte Carlo,
  
  

Datadog, Prometheus), alerting and incident management (PagerDuty, incident.io)

• You bring deep expertise in SQL and Python, and use them to lead by example in technical
  
  

execution

• You’re AI-proficient or AI-curious, eager to explore how emerging technologies can enhance team
  
  

productivity.

• You have experimented with Machine Learning frameworks (TensorFlow, PyTorch, Scikit-learn)
  
  

and LLM frameworks (e.g. Langchain)

Apply without meeting all requirements statement

If you don’t meet every requirement but believe you’d thrive in this role, we’d still love to hear from you.

We're always keen to speak to people who connect with our mission and values.

Location and Travel

The ideal candidate will be expected to work from the office on a regular basis (3 days minimum per week).

Occasional travel may be required.

Compensation

The salary range will be determined based on experience, skills, and geographic location.

Equal Opportunity Employer

InvestCloud is committed to fostering an inclusive workplace and welcomes applicants from all

backgrounds