A New Innovation: First Mobile App Use of Amazon Transcribe

Ryan Furness, Lead Engineer, Northwestern Mutual

Jul 29, 2020

With the AI-enabled smart devices and virtual assistants that are being used more and more each day, voice technology has become an integral part of our lives. So, when one of our internal ideation campaigns identified a significant business need for a better transcription solution to create compliant case notes, an interesting idea took root. What if we could use the latest voice-to-text technology to make this task easy, fast and, most importantly, accurate?

The need for financial representatives to generate compliant case notes for each of their clients, paired with the new Regulation Best Interest rule recently put into place, meant creating a tool to capture more accurate and complete case notes would have a substantial business impact.

In search of solutions

As our team began a journey to engineer the ideal voice-to-text solution for case notes, one thing became very clear – mobile was the best use case. Think about it: Your smartphone is always with you, so it’s the ideal device for taking notes right after a meeting while the details are still fresh in your mind.

Thinking about the voice technologies that are available from big tech companies, two main obstacles emerged. First, could these technologies be integrated into a custom mobile app? Second, would the level of accuracy be high enough, considering our industry has a lot of unique terms and acronyms?

A few years ago, Northwestern Mutual created an app called Posting Plus that offered a voice-to-text feature using a phone’s native capabilities. However, we discovered only 12 percent of financial representatives used it due to issues like missing prepositions, problems with capitalization and poor punctuation – they found it took longer to correct the transcribed text than to type it in themselves.

To best solve the low adoption problem and provide far greater accuracy, we needed to think of a new approach. Little did we know we’d be creating a new innovation.

Thinking differently to find a voice-to-text solution

Our four-person team of software engineers and innovators set out on our path with a divide-and-conquer strategy in different workstreams. We did the technical research related to iOS and Android platform microphones – information an audio engineer would know well, but was completely new to us.

Diving into the various transcription technologies, we ran a micro-experiment with 35 financial representatives who created case notes for client meetings by answering a set of prepared questions. Three different voice-to-text services were compared, with 25 field supervisors reviewing the sample case notes to confirm compliance. In the end, Amazon Transcribe stood out to the reviewers as the most accurate platform with the fewest transcription issues – for our use case, we estimated the accuracy to be around 95 percent, whereas the other services were in the 70-percent range. Another benefit of Amazon Transcribe is its custom vocabulary feature that gave us the ability to add unique industry or company terms and acronyms.

Because Northwestern Mutual uses Electrode Native, a Walmart Labs open-source framework leveraging React Native, we were able to make our app work on both iOS and Android devices. Our greatest challenge, however, was implementing a microphone bridge for both platforms that provided the audio in the format Amazon Transcribe (or any transcription service) was expecting. Think of the complexity of achieving this dual-platform capability, especially when you consider all the scenarios where you might use your smartphone mic: with your car speakers, for your plugin or Bluetooth headset, pausing phone calls – the list goes on.

After a few sprints of struggling to get the service working on a mobile device, we reached out to AWS for assistance. To our surprise, they told us this was the first mobile use case of its kind. For several weeks, we tried different iterations and continued to collaborate with AWS until we finally came up with a solution that worked. Rigorous testing is now under way, and extensive piloting of our new mobile app, Smart Notes, will kick off in a few weeks.

The big takeaway: Innovation can come from anywhere

Smart Notes is the first mobile application to use Amazon Transcribe. Although our team was simply looking to solve a business problem and didn’t set out to break new ground, our efforts illustrate that it’s not just startup companies coming up with innovative solutions.

I think of all the potential users of this new tool – social workers, physicians, lawyers, or any person whose job requires taking notes “on-the-fly.” The innovation team here at Northwestern Mutual hasn’t just created a new way to help solve issues with dictated case note compliance; I’m proud to say we were able to think of a potential new ­­– and better – way of working that applies to many industries.

View All Articles

720 East Wisconsin Avenue

Milwaukee, WI 53202

(414) 271-1444


Copyright © 2021 The Northwestern Mutual Life Insurance Company, Milwaukee, WI. All Rights Reserved. Northwestern Mutual is the marketing name for The Northwestern Mutual Life Insurance Company and its subsidiaries.