QA Engineer III

Overview

We are looking for an experienced QE‑III (Senior Quality Engineer) to lead quality validation for our agentic AI models product to assess compliance, reasoning outcomes, and evaluation status.

This role requires strong expertise in test automation using Selenium WebDriver, CI/CD enablement, and solid experience validating AI-driven systems, including model outputs, confidence scores, and decision traceability. The QE‑III will act as a quality owner, partnering closely with Product, Engineering, and Data Science teams.

Responsibilities

AI & Compliance Quality Ownership

• Own end‑to‑end quality strategy for the AI agent models/ Agentic AI platform, covering UI, backend services, and AI model outputs.
• Validate AI agent behavior, including:
• Decision correctness
• Evaluation status
• Confidence thresholds
• Explainability / traceability of responses
• Define test approaches for non‑deterministic AI behavior (golden datasets, tolerance-based assertions, confidence validation).
• Partner with Data Science teams to validate model inference results, regression drift, and version-to-version consistency.
  

• Design, develop, and maintain robust Selenium WebDriver automation frameworks (Java preferred).
• Implement automation for:
• Web UI workflows
• Compliance dashboards
• Agent evaluation flows
• Admin & configuration screens
• Ensure automation follows industry best practices: Page Object Model (POM), reusable utilities, proper wait strategies, and stable selectors.
• Page Object Model (POM)
• Data‑driven and keyword‑driven testing
• Reusable libraries and utilities
• Robust synchronization and wait strategies to reduce flakiness
• Integrate automation suites into CI/CD pipelines using:
• Maven
• Jenkins
• GitHub
  

• Own Jenkins pipelines for automated test execution (smoke, regression, AI evaluation suites).
• Enable automation execution as part of:
• Pull request validation
• Nightly regression
• Release readiness checks
• Analyze test failures, improve flakiness, and ensure high signal‑to‑noise ratio in automated results.
• Produce actionable test metrics (coverage, defect leakage, automation ROI).
  

• Drive functional, integration, and regression testing across multiple services and AI agents.
• Validate:
• API integrations (REST)
• Data flows across agent pipelines
• Evaluation status lifecycle (queued → processed → evaluated → completed).
• Perform exploratory testing on new AI capabilities and edge cases.
  

• Act as QE SPOC for the pod / product area.
• Mentor junior engineers on automation, testing strategies, and AI validation approaches.
• Participate in requirement reviews and proactively identify quality risks early in the SDLC.
  
  

• 6–9+ years of experience in Software Quality Engineering.
• Strong hands‑on experience in test automation using Selenium WebDriver.
• Proficiency in Java (or similar OOP language) for automation.
• Solid experience working with:
• Maven (build & dependency management)
• Jenkins (CI pipelines)
• GitHub (version control, PR workflows)
  

• Experience testing AI/ML‑driven applications, analytics platforms, or rule/decision engines.
• Understanding of:
• Model inference pipelines
• Confidence scores
• Evaluation metrics
• Handling non‑deterministic outputs
• Ability to design validation strategies for AI agents, beyond simple pass/fail testing.
  

• Strong experience with:
• API testing (REST)
• TestNG / JUnit
• Agile/Scrum methodologies
• Familiarity with test management and defect tracking tools (Azure DevOps TFS, etc.).
• Experience validating complex workflows across UI + API layers.
  

• Experience in compliance, audit, or regulatory systems.
• Exposure to performance testing (e.g., JMeter) for dashboards or AI pipelines.
• Knowledge of logging, monitoring, and observability tools for troubleshooting AI behavior.
• Experience validating data privacy and security aspects of AI systems.