Senior Software Engineer - LLM Platform

Are you ready to power the World's connections?

About The Role

Kong’s API Management suite including the Kong Enterprise AI Gateway and Konnect support multiple AI capabilities for customers.

The AI Gateway offers a unified API for managing multiple Large Language Models (LLMs), allowing users to switch between different AI providers seamlessly. The AI Gateway also allows a variety of AI policies to be enforced such as semantic caching, advanced routing and load balancing, promt guards and Personally Identifiable Information (PII) sanitization. The Konnect SaaS platform exposes these same AI capabilities via Cloud Gateways and provides customers a variety of AI-driven insights into the operations of a customer’s API surface.

We are building out a new platform team - LLMOps - in our Bangalore office to help all the other engineering teams in Kong work with Large Language Models (LLMs) and build new features into our product portfolio.

What You'll Do

• Be the founding member of the LLMOps team
• Provide an LLM SaaS platform for all the engineering teams at Kong, giving them access to a variety of LLMs based on their needs
• Manage model lifecycle with high reliability and uptime SLOs (at least 99.99 %)
• Fine tune the platform to provide AI security and high throughput for LLM queries
• Own the end-to-end deployment architecture of the LLM SaaS platform
• Play a pivotal role in shaping the technical direction for AI innovation in Kong
• Collaborate closely with product leadership to define and refine strategy, roadmap, and objectives for the LLM SaaS platforms, ensuring alignment between engineering efforts and business goals.
• Assist in the hiring process and help grow and mentor the team
  
  

What You'll Bring

• Bachelor's or Master's degree in Computer Science or a related field
• 5+ years of experience in building and operating highly reliable SaaS/PaaS systems
• Prior ownership of uptime SLOs for a platform operating at 99.99% or higher
• Hands-on experience with at least one of the major cloud platforms (AWS, Azure, or GCP)
• Strong experience with AI platforms in public clouds (like Bedrock, Vertex AI, etc.).
• Familiarity with observability tools such as Datadog, Prometheus, Grafana, Victoria Metrics, Loki, or similar technologies.
• Expertise in designing and developing highly scalable distributed systems
• Experience managing incidents and communicating effectively under high-pressure situations
• Strong verbal and written communication skills to effectively collaborate across teams
  
  

About Kong

Kong Inc., a leading developer of cloud API technologies, is on a mission to enable companies around the world to become “API-first” and securely accelerate AI adoption. Kong helps organizations globally — from startups to Fortune 500 enterprises — unleash developer productivity, build securely, and accelerate time to market. For more information about Kong, please visit www.konghq.com or follow us on X @thekonginc.