Caseware is one of Canada's original Fintech companies, having led the global audit and accounting software industry for over 30 years, with more than 500,000 users across 130 countries and available in 16 different languages. Why This Role Matters As a leader in cloud-native SaaS, we are accelerating our shift to an AI‑first future—embedding generative AI and autonomous agents across our platform to deliver smarter, faster user experiences. We’re looking for a visionary AI Test Architect to build the next‑generation “Quality Intelligence” platform: one that leverages generative AI for automated test creation, self‑healing execution, predictive defect analytics, and rigorous validation of our AI features built in‑house for our global audience. As our foundational AI Test Architect, you’ll design scalable, ethical frameworks that ensure reliability, safety, and compliance while accelerating release velocity (targeting 30‑50% faster cycles through AI‑augmented testing). Your work will reduce risk in production AI agents, minimize hallucinations, bias, and security exposures, and empower the entire engineering organization to adopt AI‑augmented quality practices that supplement traditional mature frameworks. What You’ll Be Doing 1. AI‑Driven Quality Strategy & Architecture Architect a comprehensive "Quality Intelligence" platform using generative AI to predict defect hotspots, intelligently optimize regression suites, auto‑generate tests, and enable self‑healing automation. Define enterprise‑wide AI‑first testing strategy, including non‑deterministic evaluation paradigms, continuous monitoring for drift and hallucination, and integration across the full SDLC. Establish governance for ethical AI testing, aligning with emerging standards. 2. LLM & Agent Evaluation Frameworks Design and implement advanced benchmarks, red‑team protocols, and adversarial testing for internal AI agents and generative features—focusing on hallucination rates, bias/fairness, prompt injection, jailbreaks, and goal misalignment. Build evaluation pipelines with statistical rigor (e.g., multi‑trial runs, LLM‑as‑judge, human‑in‑the‑loop) using tools such as LangFuse, LangSmith, DeepEval, RAGAS, or Arize Phoenix for metrics such as faithfulness, context precision, and safety compliance. Architect harnesses for agentic workflows, tool‑calling, planning, multi‑agent simulations, and post‑deployment observability. 3. Infrastructure & Automation Architecture Partner with DevOps to embed AI‑based testing into GitHub‑based CI/CD pipelines (e.g., AI‑generated tests, predictive flakiness detection, automated gating with quality signals). Lead design of self‑healing test frameworks (integrating AI plugins with Playwright or Cypress) that adapt to UI/model changes with minimal maintenance. Architect synthetic data generation, maintain golden data sets, and AI‑powered data masking solutions to enable privacy‑compliant, high‑fidelity testing at scale. 4. Cross‑Functional Leadership & Evangelism Collaborate with product, data science, ML engineering, and security teams to influence AI feature design with quality guardrails from day one. Evangelize and mentor: upskill traditional QA engineers into AI‑augmented testers through workshops, playbooks, and communities of practice. Drive adoption of AI quality best practices organization‑wide, including metrics dashboards for DORA + AI‑specific indicators (e.g., hallucination rate, red‑team success rate, self‑healing coverage). 5. Observability, Metrics & Continuous Evolution Define and implement AI‑specific quality telemetry (e.g., drift detection, faithfulness scoring, compliance incidents) integrated with tools like Langfuse. Establish feedback loops for model iteration, A/B testing guardrails, and proactive risk mitigation in production. Challenges You’ll Architect Solutions For Building reliable evaluation for non‑deterministic, agentic AI in a fast‑moving SaaS landscape. Scaling self‑healing and generative test automation without introducing new flakiness or security debt. Balancing innovation speed with rigorous red‑team testing and ethical safeguards for customer‑facing AI. Success in the First 6‑12 Months Launch the "Quality Intelligence" platform foundation with AI‑augmented pipelines covering 70%+ of critical paths. Establish red‑team / red‑team‑as‑code processes that reduce high‑severity AI risks by 40%+. Upskill 50%+ of QA/engineering teams on AI testing fundamentals and deliver measurable velocity and safety gains. Establish a baseline of 90%+ faithfulness for all RAG‑powered features. What You Will Bring 8+ years in quality engineering/test architecture within cloud‑native SaaS environments, with 2+ years focused on AI/ML/LLM testing and validation. Deep expertise in AWS (serverless, microservices, IaC with Terraform or CloudFormation) and GitHub CI/CD ecosystems. Proficiency architecting LLM‑based applications and testing frameworks (LangChain, LangGraph, LangSmith strongly preferred). Mastery of modern automation (Playwright, Cypress) with hands‑on experience integrating self‑healing AI plugins or generative test tools. Strong programming skills in JavaScript/TypeScript and/or Python; solid understanding of foundational AI concepts (transformers, embeddings, RAG, evaluation trade‑offs). Experience with LLM evaluation tools such as Bedrock Evaluations, Prompt Management, Guardrails, DeepEval, RAGAS, Arize Phoenix, Langfuse. Experience with red‑team frameworks/tools (Cobalt Strike, Sliver, Nmap) and knowledge of adversarial testing methodologies is a plus. Proven leadership: mentoring teams, defining standards, and driving cross‑functional change in ambiguous, high‑growth settings. Bachelor's or Master’s in Computer Science, AI/ML, or equivalent; relevant certifications a strong plus. Strong English language communication and collaboration skills. Perks & Benefits Contract a término indefinido with all the legal benefits. Prepaid medicine. Life insurance and funeral assistance. Internet allowance and home office stipend. Competitive compensation above the market average. 100‑percent remote work environment and an excellent work‑life balance. Opportunity to work for a growing global SaaS leader company. A culture that promotes independence, innovation, trust, and accountability. Mentorship by a highly experienced professional. Budget for training and professional growth. Five personal time‑off days per year. Sick leave top‑up to 100% of salary paid by the employer from Day 3 to Day 90. Recognition award, additional paid time off in recognition of the corresponding year of service. Upgrade vacation starting at five years of service. Location This is a fully remote position located in Colombia. EEO Statement We welcome and encourage candidates of all backgrounds to apply. If you require accommodations or have any questions at any point during the application or interview process, please email our People Operations team at #J-18808-Ljbffr