Senior AI Engineer- (NPW)

Company Overview

Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility. Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential.

Our seasoned professionals deliver services based on Milestone's best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Job Overview

Job Description

Job Overview

We're hiring an Senior AI Engineer to build production-grade components for an AI-first, data-centric platform. You will implement agentic capabilities (intent, planner, router/composer), integrate knowledge-graph reasoning alongside a strong RAG baseline, and instrument robust evaluation and observability. The ideal candidate writes clean, reliable code, understands LLM systems and data retrieval trade-offs, and can optimize for latency, quality, and cost.

Key Responsibilities

• Agent Implementation: Build and harden Intent, Planner, and Router/Composer agents with typed JSON I/O, retries/timeouts, and idempotency; emit call-graph traces and correlation IDs.
• Knowledge-Graph Reasoning: Generate correct graph queries (SPARQL/Gremlin/PGQL) from planner outputs; perform subgraph extraction; encode rationale and references in responses.
• RAG Baseline & Retrieval: Implement document prep, chunking/embeddings, hybrid retrieval and (where available) reranking; maintain a high-quality baseline path for side-by-side comparisons.
• Prompt/Config Tuning: Version and tune prompts, routing policies (smalllarge model escalation), temperature/top-p settings, and caching; document routing outcomes and cost/latency budgets.
• Evaluation Hooks: Integrate test sets and scoring (faithfulness/correctness, precision/recall, multi-hop coverage, latency); enable automated re-evaluation on any change (model/agent/prompt/data).
• Observability & Cost Controls: Instrument traces/metrics/logs (token usage, latency P50/P95, error codes); surface cost-per-answer dashboards; implement backpressure and graceful degradation.
• Security & Guardrails: Enforce policy-as-code and entitlement checks (role/row/column), PII/PHI handling, content moderation, and HITL approval prompts for state-changing actions.
• Quality & CI/CD: Write unit/integration/contract tests; participate in PR reviews; ship via CI/CD with feature flags and environment promotion; maintain API/connector schemas and docs.

Required Skills

• Applied LLM Engineering: 1-2+ years building production services; hands-on with LLM tool/function-calling, agent frameworks, and prompt/version management.
• Knowledge & Retrieval: Practical experience with Knowledge Graphs (RDF/SPARQL or property graph/Gremlin) and RAG pipelines (chunking, embeddings, retrieval/reranking).
• Data/Model Ecosystem: One or more vector DBs (pgvector, Pinecone, Weaviate, Milvus) and search (OpenSearch/Elasticsearch); familiarity with major model platforms (Azure OpenAI, Vertex, Anthropic, open-weights).
• Backend Skills: Proficiency in Python and/or TypeScript/Node.js; strong REST/gRPC API design, JSON Schema/OpenAPI, retries/backoff/idempotency, and error taxonomies.
• Observability & Reliability: OpenTelemetry (traces/metrics/logs), performance profiling, resiliency patterns (circuit breakers, bulkheads, DLQ/queues).
• Security by Design: OIDC/SSO, secrets management, least-privilege access, audit logging, and secure coding for AI/data services.
• CI/CD & Testing: Git-based workflows, automated pipelines, unit/integration/contract tests, and environment promotion practices.

Good to Have Skills

• Ontology & Data Quality: SHACL/OWL basics, ontology stewardship, lineage/provenance capture, and data quality checks for KG/RAG pipelines.
• Evaluation Engineering: Judge-model setups, A/B testing, rubric design, and regression dashboards.
• Performance & FinOps: Async I/O, caching strategies, connection pooling, and token/runtime budget enforcement.
• Runtime & Platform: Containers/Kubernetes, service mesh/API gateways, feature flags, blue/green or canary releases.
• UX for Explainability: Collaborating on rationale/explanations (source lists, subgraph summaries) and clear HITL approval prompts.

Compensation

Estimated Pay Range:

Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.

Our Commitment to Diversity & Inclusion

At Milestone we strive to create a workplace that reflects the communities we serve and work with, where we all feel empowered to bring our full, authentic selves to work. We know creating a diverse and inclusive culture that champions equity and belonging is not only the right thing to do for our employees but is also critical to our continued success.

Milestone Technologies provides equal employment opportunity for all applicants and employees. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, gender, gender identity, marital status, age, disability, veteran status, sexual orientation, national origin, or any other category protected by applicable federal and state law, or local ordinance. Milestone also makes reasonable accommodations for disabled applicants and employees.

We welcome the unique background, culture, experiences, knowledge, innovation, self-expression and perspectives you can bring to our global community. Our recruitment team is looking forward to meeting you.