With most of the team based in Pune, India, the Mobile/Voice zone were fantastic at execution, but most validation was done via survey. As they started to expand the roadmap into voice capabilities including Ambient, they needed insight into how providers did — or would — use voice in their day to day work, which was impossible to get with their usual methods.
My initial mission was to support the team with foundational research that helped them design and evaluate experiences that went well beyond screens. As time went on, I continued providing deeper insights and strategy that led to a successful launch of Ambient Notes, which allowed users to choose from a selection of commercially available models to turn their conversations into clinical notes directly in the patient’s encounter.
I wanted to thank you for your holistic perspective, as well as the unparalleled passion and commitment to improving the user's experience that you bring to the table. It greatly accelerated our Ambient User Research & Design. - Jenny James Zacharias, Product Manager, Clinicals Mobile/Voice
Putting dictation behavior into context
Timeline: Fall 2023
Team: 1 product manager plus Zone Product lead
Question: How do providers in Orthopedic practices use Dictation and Ambient tools to create their clinical documentation?
When I first started working with the team, they were focused on the Ortho specialty, which were heavy users of both the mobile app and Voice services such as Dictation. I started with three site visits to clients in Washington State to observe mobile and dictation use in context.
The site visits helped contextualize problems the team had only seen in Voice of the Customer feedback by mapping them to the user’s daily experience. They also helped the team understand exactly how dictated content in the note combines with content from accelerators and content put in by medical assistants.
Evaluating the Ambient Notes experience in context
Timeline: Summer 2024
Team: 2 product managers, 1 designer, 1 product architect
Question: How do Ambient Listening apps work for the providers using them today? What will differentiate our offering from the others?
The following summer, the team was hard at work integrating Ambient listening platforms into the Mobile app as part of the upcoming Ambient Notes launch. They had done usability tests on Figma prototypes to validate the interaction model. However, they hadn’t been able to see how the actual models held up against real-world use. I was called in to help a few months before our launch at Thrive, our annual customer conference.
We started with a set of site visits to clients in Texas and North Carolina, where a visiting product manager and I used a demo version of the app on our phones while shadowing providers. After getting consent from provider and patient, we recorded the visit using the Ambient feature, and asked the provider to give feedback on the generated summary.
After each visit, I wrote up a complete report in Confluence to send to the product team in Pune. By the next morning, they were eagerly consuming the findings and logging JIRA tickets for bugs we had found in testing.
A critical finding
The most important thing we saw in site visits wasn’t about the quality of the Ambient summary. It was about how providers understood this new technology, and the potential disaster waiting if the technology failed them. Providers were excited about the potential for an Ambient solution that delivered the summary directly into their clinical note. However, multiple issues with the UI and the limitations of cell networks in doctor’s offices led to repeated failures. This led to visible panic in the providers testing the system.
This finding led the team to focus design and dev effort on UX copy, error messaging, and data loss prevention. We also made sure that training and marketing materials for the feature recommended using Wi-Fi for the app instead of the user’s cell network.
Testing the model’s output across specialties
After the site visits, we needed to pressure test the model’s output beyond Orthopedics. I worked with ChatGPT to brainstorm scripts for common clinic visit conversations, then tapped our Patient Safety clinicians to record the visits, using my improv skills to riff on the script. Finally, I ran the recordings through the Ambient functionality to get the resulting clinical note.
Once we had the content, a designer and I mocked up each visit in a comprehensive Figma prototype, using variables for each data element to keep things easy for participants to switch between visits as needed for testing.
The full test protocol led participants through the process of recording the visit all the way to reviewing the output. They were then asked what they’d need to do to finish the note, and gave feedback on some enhancements we were exploring.
The lead designer and I tested the finished prototype with over 20 participants of varying specialties over 4 weeks. After each week’s calls, we would meet to review the findings, and decide what to change based on what we saw. The lead designer would make the changes, and testing would begin again. The designer worked closely with Product during this time to prioritize fixes we were confident in along the way.
By the end of the four weeks, we had a clear design direction for launch.
Validating at scale during our customer conference
In fall of 2024, Ambient notes was launched at Thrive, athenahealth’s annual user conference. I was there with several members of the product team, who flew in from Pune. They worked the Ambient booth while I stayed in the Innovation Lab, testing our prototype at scale with a constant stream of attendees.
The conference also gave us an opportunity to validate fast-follow hypotheses on the solution, including user management, ambient-inferred diagnoses, and preference settings.
