Finance Assistant: Evaluating AI-Powered Guidance at Scale
The business sought to evaluate the effectiveness of a generative AI-based Finance Assistant embedded within COSMIC workbenches. The goal: understand how agents use, perceive, and derive value from the tool—and where opportunities lie for improvement.
Research Goals
Evaluative Goals
-
Measure agent satisfaction with the Finance Assistant.
-
Identify barriers to satisfaction and optimal usage.
-
Collect feedback on desired improvements.
-
Evaluate how often agents refer to work instructions.
-
Pinpoint scenarios where guidance is most useful.
Generative Goals
-
Identify the most appealing features of the Assistant.
-
Understand perceptions of usability, usefulness, and trust.
-
Gather UX insights to inform future releases and adoption strategy.
Methods, Approach & Key Findings
Methodology & Approach
​I use a foundational mixed-methods approached to map satisfaction, uncover unmet needs, and evaluate product performance.
-
Phase 1: Stakeholder Interviews – informed research design
-
Phase 2: 60-minute global agent interviews (N=10–12)
-
Phase 3: Online survey distributed to all remaining vendor agents
Participant Sampling
-
Phase 1: Stakeholder Interviews – informed research design
-
Phase 2: 60-minute global agent interviews (N=10–12)
-
Phase 3: Online survey distributed to all remaining vendor agents
Key Findings
-
Accuracy & Relevance = Critical: Users accept and adopt the AI assistant when outputs are reliable, contextual, and easy to act on.
-
High Interest, Low Depth: While agents appreciated AI speed, many lacked clarity on how to best use the assistant for complex tasks.
-
Instruction Utility Gaps: Work instruction referrals were low unless tied to immediate workflow friction.
Strategic Impact
-
Informed product’s feature prioritization roadmap
-
Recommended workflow-aware enhancements to reduce drop-off
-
Influenced internal documentation strategy tied to AI assistant behavior
-
Highlighted a need for role-specific onboarding prompts