Interaction Designer

Our client, a leader in financial and tax software, is seeking an Interaction Designer/AI Evaluation to join their team!
**This is a 5-month W2 contract with benefits and paid holidays**
Team focused on digital help experiences and generative AI, helping improve how AI-powered assistants support customers across products. In this role, you will evaluate AI-generated conversations, establish quality standards, coordinate human evaluation efforts, and translate findings into actionable improvements.
Responsibilities
AI Evaluation & Quality: Lead evaluations of generative AI outputs, ensuring responses meet established quality guidelines and deliver effective customer experiences.
Conversational Design: Assess conversational flow, language quality, relevance, clarity, and the overall effectiveness of AI-generated customer interactions.
Human-in-the-Loop Operations: Coordinate evaluators and subject-matter experts, providing guidance, feedback, and training to improve annotation quality and consistency.
Cross-Functional Collaboration: Partner with product managers, engineers, AI teams, and designers to generate and evaluate AI responses across different use cases.
Data Analysis & Insights: Summarize evaluation results, identify trends and recurring issues, and recommend actionable improvements to AI experiences.
Prioritization & Planning: Coordinate evaluation and training data requests with product managers, balancing priorities and business needs.
Quality Management: Develop and refine evaluation processes, communicate quality status and risks, and support continuous improvement of generative AI experiences.
Required Qualifications
3+ years of relevant experience in conversational design, AI evaluation, interaction design, or a related field.
Experience evaluating chatbot interactions, AI-generated responses, or conversational quality.
Foundational understanding of LLMs, generative AI, and NLP.
Experience with human annotation, data labeling, or AI quality evaluation.
Familiarity with Claude, AWS SageMaker Ground Truth, or similar evaluation tools.
Understanding of RAG and AI quality improvement processes.
Experience in customer support, fintech, or digital help experiences.
Motion Recruitment Partners is an Equal Opportunity Employer, including disability/vets. All applicants must be currently authorized to work on a full-time basis in the country for which they are applying, and no sponsorship is currently available. Employment is subject to the successful completion of a pre-employment screening. Accommodation will be provided in all parts of the hiring process as required under Motion Recruitment Employment Accommodation policy. Applicants need to make their needs known in advance.

Work arrangement
Yes