Test Engineer II (AI Test Engineer )

Overview

Build a brighter future while learning and growing with Siemens company at the intersection of technology, community, and sustainability. Our global team of innovators is always looking to create meaningful solutions to some of the toughest challenges facing our world. Find out how far your passion can take you.

About The Job

Brightly Software continues to grow and needs amazing engineers ready to embrace the future of AI-powered Quality Assurance! This is an excellent fit for talented engineers who thrive in a fast-paced environment and are passionate about testing Generative AI and traditional ML applications using Generative AI and advanced automation to redefine software quality. You will work alongside top-notch engineers and product teams to design, implement, deliver, and support highly ambitious products and integrations, while championing modern QA practices.

The AI Test Engineer (Automation) supports application platforms in the Engineering department. This role is critical to overall program success, with aggressive timelines and direct impact on our customers. You will play a key role in implementing AI-assisted automation frameworks and driving intelligent testing strategies.

What You’ll Be Doing

Lead and execute thorough testing of LLM, AI, and ML applications, ensuring comprehensive coverage and reliability across diverse platforms and use cases.

Demonstrate strong inclination towards product quality and AI-first testing mindset.

Develop and maintain AI-assisted test scripts from functional and technical designs, leveraging Gen AI tools for faster script generation and self-healing capabilities.

Apply data-driven analysis and AI-powered insights for pragmatic decision-making aligned with business and technology needs.

Understand user requirements and implement intelligent test case generation for ML and AI systems using Gen AI platforms.

Perform automated testing, debug automation code, and provide comprehensive reports enriched with AI-driven defect analysis.

Work on improving test automation coverage using next-gen automation platforms that integrate AI co-pilots.

Collaborate with cross-functional teams to ensure product quality and advocate for AI-supported shift-left testing.

Investigate defects for root cause using AI-powered anomaly detection.

Execute test cases, log defects, and maintain test documentation with intelligent defect tracking systems.

Mentor associate engineers on AI-driven QA practices and automation best practices.

What You Need

Bachelor’s in Computer Science or related discipline; or equivalent work experience.

3–4 years of experience in test automation and QA.

Strong knowledge of Python testing frameworks such as Pytest, REST Assured, Selenium, API testing, CI/CD pipelines such as Jenkins and Github Actions.

Solid understanding of AWS services such as Sagemaker, EKS and lambda.

Solid understanding of data structures, algorithms, OOP, databases, SQL, and web programming.

Hands-on experience with AI-assisted automation tools or willingness to learn emerging Gen AI QA platforms.

Strong communication and collaboration skills.

Fundamentals of ML and AI systems such as the ML Lifecycle, model monitoring, testing and testing of LLM applications.

Experience validating AI model monitoring pipelines for drift, degradation, and anomaly signals.

Experience with prompt engineering and evaluation frameworks (e.g., DeepEVAL, OpenAI Evals).

Ability to design evaluation metrics for LLM‑generated outputs, including hallucination detection, grounding accuracy, factual consistency, and robustness testing.

Familiarity with ethical AI testing: bias detection, toxicity testing, safe‑prompt evaluations, and guardrail validation.

Hands‑on expertise building automated evaluation harnesses for GenAI outputs integrated within CI/CD pipelines.

Exposure to self‑healing automation frameworks enhanced by LLMs.

Knowledge of LLM failure patterns, adversarial prompt testing, and structured red‑teaming.

What Makes You Standout

Exposure to public cloud platforms (AWS, GCP).

Certifications like ISTQB, Python, or AI/ML fundamentals.

Experience with AI-powered defect prediction or self-healing automation frameworks.

The Brightly Culture

We’re guided by a vision of community that serves the ambitions and wellbeing of all people, and our professional communities are no exception. We model that ideal every day by being supportive, collaborative partners to one another, conscientiously making space for our colleagues to grow and thrive. Our passionate team is driven to create a future where smarter infrastructure protects the environments that shape and connect us all. That brighter future starts with us.