Data & AI/ML Architect

Company Overview

Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility. Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential.

Our seasoned professionals deliver services based on Milestone’s best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Job Overview

Job Summary:

We are seeking a highly experienced and visionary Databricks Data Architect with over 14 years in data engineering and architecture, including deep hands-on experience in designing and scaling Lakehouse architectures using Databricks. The ideal candidate will possess deep expertise across data modeling, data governance, real-time and batch processing, and cloud-native analytics using the Databricks platform. You will lead the strategy, design, and implementation of modern data architecture to drive enterprise-wide data initiatives and maximize the value from the Databricks platform.

Key Responsibilities

• Lead the architecture, design, and implementation of scalable and secure Lakehouse solutions using Databricks and Delta Lake.
• Define and implement data modeling best practices, including medallion architecture (bronze/silver/gold layers).
• Champion data quality and governance frameworks leveraging Databricks Unity Catalog for metadata, lineage, access control, and auditing.
• Architect real-time and batch data ingestion pipelines using Apache Spark Structured Streaming, Auto Loader, and Delta Live Tables (DLT).
• Develop reusable templates, workflows, and libraries for data ingestion, transformation, and consumption across various domains.
• Collaborate with enterprise data governance and security teams to ensure compliance with regulatory and organizational data standards.
• Promote self-service analytics and data democratization by enabling business users through Databricks SQL and Power BI/Tableau integrations.
• Partner with Data Scientists and ML Engineers to enable ML workflows using MLflow, Feature Store, and Databricks Model Serving.
• Provide architectural leadership for enterprise data platforms, including performance optimization, cost governance, and CI/CD automation in Databricks.
• Define and drive the adoption of DevOps/MLOps best practices on Databricks using Databricks Repos, Git, Jobs, and Terraform.
• Mentor and lead engineering teams on modern data platform practices, Spark performance tuning, and efficient Delta Lake optimizations (Z-ordering, OPTIMIZE, VACUUM, etc.).
  
  

Technical Skills

• 10+ years in Data Warehousing, Data Architecture, and Enterprise ETL design.
• 5+ years hands-on experience with Databricks on Azure/AWS/GCP, including advanced Apache Spark and Delta Lake.
• Strong command of SQL, PySpark, and Spark SQL for large-scale data transformation.
• Proficiency with Databricks Unity Catalog, Delta Live Tables, Autoloader, DBFS, Jobs, and Workflows.
• Hands-on experience with Databricks SQL and integration with BI tools (Power BI, Tableau, etc.).
• Experience implementing CI/CD on Databricks, using tools like Git, Azure DevOps, Terraform, and Databricks Repos.
• Proficient with streaming architecture using Spark Structured Streaming, Kafka, or Event Hubs/Kinesis.
• Understanding of ML lifecycle management with MLflow, and experience in deploying MLOps solutions on Databricks.
• Familiarity with cloud object stores (e.g., AWS S3, Azure Data Lake Gen2) and data lake architectures.
• Exposure to data cataloging and metadata management using Unity Catalog or third-party tools.
• Knowledge of orchestration tools like Airflow, Databricks Workflows, or Azure Data Factory.
• Experience with Docker/Kubernetes for containerization (optional, for cross-platform knowledge).
  
  

Preferred Certifications (a Plus)

• Databricks Certified Data Engineer Associate/Professional
• Databricks Certified Lakehouse Architect
• Microsoft Certified: Azure Data Engineer / Azure Solutions Architect
• AWS Certified Data Analytics – Specialty
• Google Professional Data Engineer
  
  

Compensation

Estimated Pay Range:

Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.

Our Commitment to Diversity & Inclusion

At Milestone we strive to create a workplace that reflects the communities we serve and work with, where we all feel empowered to bring our full, authentic selves to work. We know creating a diverse and inclusive culture that champions equity and belonging is not only the right thing to do for our employees but is also critical to our continued success.

Milestone Technologies provides equal employment opportunity for all applicants and employees. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, gender, gender identity, marital status, age, disability, veteran status, sexual orientation, national origin, or any other category protected by applicable federal and state law, or local ordinance. Milestone also makes reasonable accommodations for disabled applicants and employees. 

We welcome the unique background, culture, experiences, knowledge, innovation, self-expression and perspectives you can bring to our global community. Our recruitment team is looking forward to meeting you.