AI/LLM QA Engineer – Agentic AI & RAG at Diverse Lynx in Indianapolis, IN

Responsibilities

• Design and build agentic AI systems capable of multi-step reasoning| tool usage| and autonomous decision-making
• Implement and manage MCP (Model Context Protocol) or similar frameworks for structured context sharing between models and tools
• Develop multi-agent architectures| orchestration layers| and communication protocols
• Integrate LLMs (OpenAI| Azure OpenAI| Anthropic| etc.) with enterprise systems| APIs| and databases
• Build pipelines for context grounding| memory management| and stateful interactions
• Develop reusable AI toolkits| plugins| and function-calling frameworks
• Eligible to workimize agent performance through prompt engineering| evaluation frameworks| and fine-tuning strategies
• Ensure scalability| observability| and governance of AI systems
• Collaborate with product managers| data scientists| and engineering teams to translate business needs into AI solutions

Requirements

• Validate AI/LLM behavior| prompts| and conversational workflows
• Design tests for non-deterministic outputs and model variability
• Ensure data quality| lineage| and model output accuracy
• Build and manage test automation frameworks for AI pipelines
• Integrate testing into CI/CD and MLOps workflows
• Perform performance| scalability| and reliability testing of AI services
• Enforce Responsible AI checks (bias| fairness| compliance| auditability)Lead defect management| test planning| and Agile POD QA execution
• Provide quality metrics| validation reports| and release readiness sign-off

Skills

Required

• Design and build agentic AI systems
• Multi-agent architectures
• CI/CD and MLOps workflows