Finance Assistant: Evaluating AI-Powered Guidance at Scale

The business sought to evaluate the effectiveness of a generative AI-based Finance Assistant embedded within COSMIC workbenches. The goal: understand how agents use, perceive, and derive value from the tool—and where opportunities lie for improvement.

Research Goals

Evaluative Goals

Measure agent satisfaction with the Finance Assistant.
Identify barriers to satisfaction and optimal usage.
Collect feedback on desired improvements.
Evaluate how often agents refer to work instructions.
Pinpoint scenarios where guidance is most useful.

Generative Goals

Identify the most appealing features of the Assistant.
Understand perceptions of usability, usefulness, and trust.
Gather UX insights to inform future releases and adoption strategy.

Methods, Approach & Key Findings

Methodology & Approach

I use a foundational mixed-methods approached to map satisfaction, uncover unmet needs, and evaluate product performance.

Phase 1: Stakeholder Interviews – informed research design
Phase 2: 60-minute global agent interviews (N=10–12)
Phase 3: Online survey distributed to all remaining vendor agents

Participant Sampling

Phase 1: Stakeholder Interviews – informed research design
Phase 2: 60-minute global agent interviews (N=10–12)
Phase 3: Online survey distributed to all remaining vendor agents

Key Findings

Accuracy & Relevance = Critical: Users accept and adopt the AI assistant when outputs are reliable, contextual, and easy to act on.
High Interest, Low Depth: While agents appreciated AI speed, many lacked clarity on how to best use the assistant for complex tasks.
Instruction Utility Gaps: Work instruction referrals were low unless tied to immediate workflow friction.

Strategic Impact

Informed product’s feature prioritization roadmap
Recommended workflow-aware enhancements to reduce drop-off
Influenced internal documentation strategy tied to AI assistant behavior
Highlighted a need for role-specific onboarding prompts

Reflection

"This project underscored the tension between AI output velocity and human trust. Even in a high-performance tool, perceived value hinges on how clearly the assistant integrates into a real-world task. I’d push for integrated measurement of user trust in future releases."