Senior Solutions Architect

About AGS
AGS Health is a leading strategic growth partner to healthcare providers across the U.S. Working alongside each client as one team, we improve revenue cycle performance by orchestrating data, AI, automation, and human expertise across every workflow. Our proven model complements customers' existing teams, technology, and processes, deploying the right capabilities to improve efficiency, strengthen financial performance, and enhance the patient financial experience. Supported by a global team of more than 16,000 clinical and revenue cycle experts across onshore, nearshore, and offshore locations, AGS Health helps customers achieve measurable results across diverse care settings and specialties
About the Role
Own the accuracy bar for AI-enabled healthcare automation products - designing the audit methodology and evidence package that gates every production launch, independent of delivery timeline pressure.
Key Responsibilities
Define and own the accuracy bar for an AI/automation product prior to and after production launch.
Design statistically sound audit-sampling methodology used to certify a workflow, specialty, or customer account as ready to go live.
Produce go/no-go evidence packages ahead of any new launch, using defined accuracy, precision/recall, and audit-sample results.
Partner with AI/ML engineering's evaluation harness and data-monitoring pipelines to ensure production accuracy continues to match what was certified pre-launch.
Independently gate launch decisions on evidence, separate from delivery/build timeline pressure.
Required Qualifications
Experience designing audit or statistical sampling methodology, ideally in a healthcare, claims, or clinical-documentation-accuracy context.
Comfortable being an independent, evidence-based go/no-go gate, separate from the team delivering the product.
Familiarity with core ML/AI evaluation concepts (precision, recall, F1, confidence/calibration).
Preferred Qualifications
Background in clinical documentation integrity, medical coding audit, or ML/LLM model validation.
Experience with LLM evaluation tooling and methodologies (e.g., LLM-as-judge, automated regression testing of prompts/models).
Technology & Tools
Python or R for statistical sampling design and analysis (e.g., stratified/confidence-interval-based sampling).
SQL for pulling audit samples and production accuracy data directly.
Working familiarity with ML/LLM evaluation tooling (Ragas, DeepEval, LangSmith, or promptfoo) sufficient to partner with engineering on harness design, not necessarily to build it.
Audit-tracking/case-management tooling for documenting go/no-go evidence packages.
Education
Bachelor's degree in a quantitative field (statistics, data science, health informatics, or similar). Master's degree a plus.
Certifications
CPMA (Certified Professional Medical Auditor) or a quality-methodology certification (e.g., Lean Six Sigma) is a plus. No ML-specific certification required.
Relevant Background
A coding/claims audit firm, a healthcare compliance department, or an AI/ML quality-assurance function at a health-tech company - with direct experience gating a production launch on audit or statistical evidence, not only performing after-the-fact quality reviews.

Work arrangement
No