Customer Story
BioticsAI Scales Lifesaving AI with Label Studio
In Conversation With
Robhy Bustami
CEO
Customer Story
In Conversation With
CEO
BioticsAI was founded with a mission to enable obstetricians to provide the highest possible quality of care to expectant mothers. Today, the misdiagnosis rate for fetal malformation can reach over 50%, primarily due to errors made during the ultrasound process. The importance of getting an ultrasound right so doctors and families can make accurate, informed care decisions can't be overstated.
To solve this lifesaving problem, BioticsAI develops models to identify errors when conducting fetal ultrasound screenings, localize problematic anatomical views, identify fetal anomalies, generate high-quality documentation, and provide real-time feedback to clinicians.
BioticsAI's models are trained on millions of prenatal images, a complex computer vision use case that requires a high degree of domain expertise when annotating data. Models built on this training data have to be cleared by the FDA and have to work across a variety of demographics and patient-specific situations. As a result, BioticsAI needs a large volume of high quality data to meet the exceptionally high standards required for product development.
At the end of the day, the value we're providing is novel machine learning algorithms that can solve problems for patients. None of that is possible without being able to create proprietary datasets that are highly accurate and labeled in an efficient, compliant, and streamlined way.
Robhy Bustami, CEO
High quality models depend on high quality training data, continuous evaluation and fine-tuning. That's especially true when it comes to problem spaces such as healthcare, which require domain expertise and traceability.
"Our product depends on our ability to efficiently generate a large volume of novel, accurately-labeled ultrasound imagery," explained Robhy Bustami, CEO of BioticsAI. "There are no publicly available prenatal ultrasound datasets that meet our needs in terms of volume and diversity, so we had to start from scratch and build a proprietary dataset."
To enable accurate models and meet regulatory requirements, Robhy knew that having a human in the loop during model development was a must as they started looking at annotation solutions. "We need highly qualified annotators, typically medical students, to label the bulk of the data, as well as board-certified physicians that specialize in medical imaging to review the annotators' work and sign off on our datasets."
From an efficiency standpoint, the company wanted to pre-annotate as much of the data as possible with existing models developed internally, as well as quickly organize, prioritize, and prepare tasks for annotators.
The company initially evaluated desktop applications, but quickly found them lacking in quite a few areas including transparency, the ability to monitor agreement across annotators, and the ability to enable annotation review by physicians. They then turned to evaluating cloud-based annotation solutions like Labelbox but also found that they were missing key features, particularly for the computer vision use case that they had.
Robhy chose Label Studio because "it accomplished everything we needed to do from a feature/functionality standpoint." Notably, Label Studio's support for pixel-wise semantic segmentation enabled BioticsAI to annotate ultrasound images down to the pixel level. The company was also attracted to the open source model for transparency and to future proof support for a broader range of capabilities as the company grew.
Using Label Studio community edition, BioticsAI quickly scaled their annotation to millions of images over the first 2 years. But this growth came with challenges.
BioticsAI' DevOps teams were frequently called to patch and upgrade their system. "We didn’t want our DevOps teams running around trying to figure out why we're running out of memory," noted Robhy. Uptime also became a problem where busy medical students and busier Chief Medical Officers would experience high latency and challenges to use the product reliably - a pain point Robhy would experience when CMOs would text him directly when they couldn't log in during their limited window of availability.
The open source solution also lacked configurable review workflows and quality assurance reporting.
To meet the needs of his growing business, Robhy evaluated several enterprise-grade labeling tools. Label Studio Enterprise was a natural fit as annotators and reviewers were already familiar with Label Studio's workflow.
But the biggest benefit from choosing Label Studio Enterprise was the integrated support for role-based access, multiple reviewers, and annotator agreement features to improve quality and efficiency. "When you're doing these large scale clinical studies, you need expert reviewers to sign off on the underlying annotations. The ability to define the workflow between annotator and reviewer has been huge for us." The company is also using the annotator agreement matrix as an additional layer of QA for their annotators.
As Robhy noted, staying on Label Studio open source would have required custom scripting and a lot of manual effort to manage an ever growing pool of annotators and reviewers. Now they have a solution that removes the burden from their DevOps team and, more importantly, they have a solution that ensures they build the best models possible for their mission-critical use case.
Since the core product we use to power all of this is Label Studio Enterprise, it's the most important technology in our stack.
Robhy Bustami, CEO
Summarizing his journey with Label Studio open source and then with Label Studio Enterprise, Robhy concluded: "At the end of the day, the value we're bringing are these novel machine learning algorithms that can solve problems for patients. And none of that is possible without being able to create proprietary datasets that are highly accurate and labeled in an efficient, compliant, and streamlined way. Since the core product we use to power all of this is Label Studio Enterprise, it's the most important technology in our stack."