Why This Role Matters Caseware is a leading Fintech company with a global audit and accounting software platform. We are accelerating an AI‑first future by embedding generative AI and autonomous agents to deliver smarter, faster user experiences. What You’ll Be Doing 1. AI‑Driven Quality Strategy & Architecture Architect a comprehensive “Quality Intelligence” platform to predict defect hotspots, optimize regression suites, auto‑generate tests, and enable self‑healing automation. Define an enterprise‑wide AI‑first testing strategy, including continuous monitoring for drift, hallucination, and bias. Establish ethical governance for AI testing, aligning with emerging standards. 2. LLM & Agent Evaluation Frameworks Design advanced benchmarks, red‑team protocols, and adversarial testing for AI agents. Build evaluation pipelines using tools such as LangFuse, LangSmith, DeepEval, RAGAS, or Arize Phoenix for faithfulness, context precision, and safety compliance. Architect harnesses for agentic workflows, multi‑agent simulations, and post‑deployment observability. 3. Infrastructure & Automation Architecture Embed AI‑based testing into GitHub‑based CI/CD pipelines. Lead design of self‑healing test frameworks that adapt to UI and model changes. Architect synthetic data generation, maintain gold‑data sets, and provide AI‑powered data masking for privacy compliance. 4. Cross‑Functional Leadership & Evangelism Collaborate with product, data science, ML engineering, and security teams to influence AI feature design. Mentor QA engineers into AI‑augmented testers through workshops and playbooks. Drive adoption of AI quality best practices organization‑wide with DORA + AI indicators. 5. Observability, Metrics & Continuous Evolution Implement AI‑specific quality telemetry integrated with tools like Langfuse. Establish feedback loops for model iteration and proactive risk mitigation. Success in the First 6‑12 Months Launch the “Quality Intelligence” platform covering 70% of critical paths with AI‑augmented pipelines. Reduce high‑severity AI risks by 40% via red‑team processes. Upskill 50% of QA/engineering teams on AI testing fundamentals. Establish a 90%+ faithfulness baseline for RAG‑powered features. What You Will Bring 8+ years in Quality Engineering/Test Architecture within cloud‑native SaaS environments, 2+ years focused on AI/ML/LLM testing. Deep expertise in AWS, Terraform/CloudFormation, and GitHub CI/CD. Proficiency with LLM‑based applications and testing frameworks (LangChain, LangGraph, LangSmith). Strong programming skills in JavaScript/TypeScript and/or Python. Experience with LLM evaluation tools like Bedrock Evaluations, Prompt Management, Guardrails, DeepEval, RAGAS, Arize Phoenix, and Langfuse. Proven leadership and cross‑functional change drive. Bachelor’s/Master’s in Computer Science, AI/ML, or equivalent. Strong English communication and collaboration skills. Perks & Benefits Contrato a termino Indefinido with all legal benefits. Prepaid medicine and life insurance. Home office stipend and internet allowance. Competitive compensation above market average. 100% remote work environment with excellent work‑life balance. Mentorship, training budget, and career growth opportunities. About Caseware Caseware’s industry‑leading software products are designed for accounting firms, corporations, and governments. We are building technology that shapes the future of audits, financial reporting, and data analytics. EEO Statement Caseware welcomes and encourages candidates of all backgrounds to apply. We are committed to diversity and inclusion. #J-18808-Ljbffr