case study

How we built an AI-powered speech-to-text appfor medical scribes to streamline documentation workflows in healthcare

Domain:

Medical

Country:

USA

Team size:

Collaboration:

2020-now

challenge:

Apart from performing their direct clinical duties, like examining patients and diagnosing, doctors spend hours typing clinical data and entering it into Electronic Health Record (EHR) systems. According to research, physicians spend about 4.5 hours a day on electronic health records (EHR), which leads to burnout and reduces the quality of patient care. Medical scribes are a solution to this challenge. These are trained professionals who manually transcribe information during clinical visits, relieving physicians from the tedious task of data entry.

Recent advancements in AI and natural language processing have opened up new possibilities in this field. By leveraging speech recognition software to assist medical scribes, we can further enhance their productivity and streamline medical documentation workflows. HRZN tech, a USA-based company that helps businesses in advancing their concepts through technology, recognized the potential of this idea and approached Uplab to implement it.

Illustration of speech captured on a phone flowing into an EHR system, so the same documentation is finished faster and more time is left for patients

project evolution:

After discussing project details with the client, we decided to create a proof of concept (PoC) to check if building an automated medical scribe was doable. It helped us test the product and get feedback on how we could make it better. With the user feedback, we added a web interface to the PoC and turned it into a minimum viable product (MVP) that we could take to market.

As a result, Uplab helped the serial entrepreneur release a cross-platform mobile application that captures doctor-patient conversations and transcribes them into text format.

The app's provider dashboard with visit metrics on a tablet, beside the visit overview and its list of recordings on a phone

Let’s dive in to learn more

From proof of concept to a HIPAA-compliant release

Project highlights

Built a PoC to assess the feasibility of an intelligent speech recognition application
Designed a multi-tenant software architecture to ensure proper isolation of data
Implemented security policies to comply with HIPAA regulations
Developed an AI-powered speech transcription app

Building a PoC to test the project's feasibility

Is it possible to convert an audio recording of a doctor-patient conversation into a comprehensive text that can be entered into the EHR system?

To verify the feasibility of this idea, our initial focus was on developing a proof of concept.
We identified technical requirements, developed a project roadmap, and moved on to the implementation.
The idea behind the PoC was to create a mobile app that could capture conversations and instantly convert them into text format with just one click.
Within two months, we delivered the cross-platform mobile application to our client.
Over the next few months, our client used the app to test its core features and the accuracy of the audio-to-text transformation.
With the expected results achieved, we proceeded to further scale and refine the app for its public launch.

The recording screen of the proof of concept beside the refined MVP version of the same screen

Refining the speech-to-text app for public release

After creating the PoC, our team's next goal was to enhance the app and prepare it for release.
To achieve this, we used:

Artificial intelligence (AI)
Natural language processing (NLP)
Automatic speech recognition (ASR)

These technologies helped improve the app's ability to accurately comprehend and interpret human language.

As the software deals with sensitive patient data, our team also prioritized implementing robust security measures to safeguard this information from unauthorized access.

We carefully selected the most suitable tenancy model (a way of organizing the system's data and users into isolated groups, or "tenants," to ensure that each tenant's data and information are kept separate and secure), and followed HIPAA standards to ensure that the app adheres to the highest level of data privacy and security.

The patient overview screen on desktop and mobile, stamped “protected”

HIPAA compliance

Healthcare providers that electronically transmit protected healthcare information (PHI) should meet the requirements of the Health Insurance Portability and Accountability Act (HIPAA compliance). Otherwise, in case confidential data were to be compromised, the company could face legal and financial penalties, which could damage its reputation.

As we were developing an app that involves storing patients' personal data, it was crucial for us to implement security measures to ensure compliance with HIPAA regulations. To achieve this, we conducted a thorough security risk assessment and implemented robust security policies and procedures to safeguard data transmission and encrypt data storage.

By taking these steps, we were able to develop software that is 100% HIPAA-compliant, thereby reducing the risk of data breaches and loss.

Multi-tenant architecture

Our speech transcription app is designed as multi-tenant software, which means that it allows multiple doctors and scribes (tenants) to have access to it. Since each tenant stores their confidential information within the system, it was crucial for us to select a tenancy model that ensures the isolation of data between tenants, preventing unauthorized access to data by other tenants or users.

After analyzing the advantages and disadvantages of various models, we ultimately decided on a multi-tenant system with table-level isolation architecture. This approach enabled us to keep costs low while also ensuring proper segregation of tenants.

Diagram of three tenants connecting to one system, each with their data held in a separate store

AI algorithms

To create a system capable of transforming human speech into text, we leveraged various AI capabilities, such as automatic speech recognition, natural language processing, and machine learning. These technologies allowed us to recognize spoken words and patterns and convert them into textual records.

By applying NLP techniques and algorithms, the app was also able to analyze and process unstructured data, converting it into a format that can be understood by machines.

Diagram of a person speaking into a device, where AI turns the speech into a written note on a laptop

results:

AI-powered speech recognition app for healthcare

Uplab's team created a software application that helps doctors and medical scribes to focus more on patients, leading to improved patient care. The application is designed with different access levels for doctors and scribes.

Here is how it works:

Once the doctor logs in to the application, they can access the patient's database which includes their name, medical history, and other relevant information.

The sign-in screen beside the patient overview, listing the patient’s number, date of birth and visit history

During the appointment, the doctor starts the recording and the app continues to record the conversation in the background.

Three states of the recorder: ready to record, recording in progress, and paused with reset and done controls

Then, the app converts the audio recording into text format and sends it to the medical scribe.

The recordings screen playing back a visit with the transcript below it, shown as a conversation between doctor and patient

A scribe receives the recording and the text from the doctor, reviews it, writes SOAP notes (Subjective, Objective, Assessment, and Plan based on the patient’s examination), and sends them back to the doctor.

The note overview form with subjective, objective, assessment and plan fields filled in, ready to submit for review

The speech-to-text app significantly optimizes the medical workflow and saves time for both doctors and scribes.

We’re still in the game: cooperation on other projects

The speech transcription app is on its way to getting its first clients and capturing the market. Our Uplab team nailed it with our technical know-how and good communication during the project. We continue working together with this client on implementing his new business ideas.

They came up with creative solutions to even the most unique technological challenges.

I expect us to grow into a bigger team as we continue down our roadmap. All team members work very well together under the leadership of the project manager.