Evaluating the Ambient Notes experience in context

Timeline: Summer 2024

Team: 2 product managers, 1 designer, 1 product architect

Question: How do Ambient Listening apps work for the providers using them today? What will differentiate our offering from the others?

The following summer, the team was hard at work integrating Ambient listening platforms into the Mobile app as part of the upcoming Ambient Notes launch. They had done usability tests on Figma prototypes to validate the interaction model. However, they hadn’t been able to see how the actual models held up against real-world use. I was called in to help a few months before our launch at Thrive, our annual customer conference.

We started with a set of site visits to clients in Texas and North Carolina, where a visiting product manager and I used a demo version of the app on our phones while shadowing providers. After getting consent from provider and patient, we recorded the visit using the Ambient feature, and asked the provider to give feedback on the generated summary.

A demo version of the app allowed us to test the Ambient feature against real visits directly on a test patient. It also gave us the ability to see what the model created against a transcript of the visit.

After each visit, I wrote up a complete report in Confluence to send to the product team in Pune. By the next morning, they were eagerly consuming the findings and logging JIRA tickets for bugs we had found in testing.

The team was surprised and delighted by how quickly I could turn around site visit findings.

A critical finding

The most important thing we saw in site visits wasn’t about the quality of the Ambient summary. It was about how providers understood this new technology, and the potential disaster waiting if the technology failed them. Providers were excited about the potential for an Ambient solution that delivered the summary directly into their clinical note. However, multiple issues with the UI and the limitations of cell networks in doctor’s offices led to repeated failures. This led to visible panic in the providers testing the system.

Adding a bit of humor helps draw people into the story of a critical finding.

This finding led the team to focus design and dev effort on UX copy, error messaging, and data loss prevention. We also made sure that training and marketing materials for the feature recommended using Wi-Fi for the app instead of the user’s cell network.

Testing the model’s output across specialties

After the site visits, we needed to pressure test the model’s output beyond Orthopedics. I worked with ChatGPT to brainstorm scripts for common clinic visit conversations, then tapped our Patient Safety clinicians to record the visits, using my improv skills to riff on the script. Finally, I ran the recordings through the Ambient functionality to get the resulting clinical note.

These were two of what ended up as ten mock visits we were able to generate notes for. Some I recorded, others were donated from salespeople who had done demos already.

Once we had the content, a designer and I mocked up each visit in a comprehensive Figma prototype, using variables for each data element to keep things easy for participants to switch between visits as needed for testing.

A complex Figma prototype with variables helped us iterate after each week’s testing, so we got a validated solution to engineering teams more efficiently. Recreated to preserve confidentiality.

The full test protocol led participants through the process of recording the visit all the way to reviewing the output. They were then asked what they’d need to do to finish the note, and gave feedback on some enhancements we were exploring.

The lead designer and I tested the finished prototype with over 20 participants of varying specialties over 4 weeks. After each week’s calls, we would meet to review the findings, and decide what to change based on what we saw. The lead designer would make the changes, and testing would begin again. The designer worked closely with Product during this time to prioritize fixes we were confident in along the way.

By the end of the four weeks, we had a clear design direction for launch.

Validating at scale during our customer conference

In fall of 2024, Ambient notes was launched at Thrive, athenahealth’s annual user conference. I was there with several members of the product team, who flew in from Pune. They worked the Ambient booth while I stayed in the Innovation Lab, testing our prototype at scale with a constant stream of attendees.

The Ambient Notes team was the belle of the ball at Thrive; we stayed busy answering questions and further validating the solution with customers.

The conference also gave us an opportunity to validate fast-follow hypotheses on the solution, including user management, ambient-inferred diagnoses, and preference settings.

Outcome

The Ambient Notes launch was objectively successful, and the functionality is popular among our customers. As of April 2026, athenahealth’s own Ambient Notes page reports the following metrics:

2M+ — notes created using Ambient Notes

90% — report a meaningful reduction in burnout

86% — say they can give patients full attention

74% — report spending less time documenting

OrthoLoneStar, an early adopter, saw the following outcomes:

36% — reduction in after-hours documentation time

12% — increase in Same-Day Encounter Close rate

3.6 minutes — saved per encounter