From Manual to Automated: Geberit's Leap Towards Efficient Data Annotation with Label Studio and HumanSignal

Introduction

As a major supplier of sanitary technology and bathroom ceramics, the Geberit Group uses custom AI models to help improve customer service and operational efficiency. When generative AI and LLMs came to the forefront of the data labeling conversation, Geberit saw an opportunity to leverage this technology to accelerate their data labeling operations.

So they embarked on integrating GPT-3.5 into their labeling process on their own, beginning with an email labeling use case. However, early implementations were prone to inaccuracy and incurred significant effort. Fortunately, Geberit had already been using Label Studio as a part of their ML operations stack for a number of years, and so they looked to HumanSignal to help them adopt this technology into their AI workflow.

The clarity and efficiency of the POC process with HumanSignal were remarkable. We're excited about the potential of Label Studio in revolutionizing our data annotation and active learning processes.

Dr. Tilo Sperling, Head of AI-Projects Business Applications

Using Label Studio’s integrated platform to automate label creation and trigger workflows for human review, they were able to achieve significant time- and cost-savings while achieving 93% accuracy relative to manually-labeled and reviewed data.

Problem

Handling vast datasets, especially with intricate structures - like emails - are always a challenge. Manual data annotation methods were becoming cumbersome, time-consuming, and inefficient. Geberit needed to find a solution that could streamline this process, introducing accuracy and speed.

Generative AI models from companies like OpenAI seemed to offer a clear solution to these issues. By having an LLM label their large datasets, Geberit hoped to increase their labeling speed while maintaining the necessary quality benchmarks to make the labeled data useful. However, using OpenAI to automate the classification of thousands of emails proved to be too cost-prohibitive. In addition, there were significant concerns about the quality of the data labels being produced by the automation, particularly in the form of model “hallucinations.” Geberit determined that they needed a solution that would provide the same degree of automation, but with much higher accuracy at a much lower cost.

Solution

In the pursuit of optimizing their email labeling process and as longtime Label Studio users, Geberit turned to HumanSignal for help. Recognizing the challenges Geberit faced, HumanSignal delivered a multi-faceted approach:

• Integration of Advanced AI Models: Label Studio facilitated the integration of a GenAI stack, including Foundation Models (FMs) and Large Language Models (LLMs). This allowed for a more interactive labeling process, termed "copilot labeling," enhancing the efficiency and accuracy of the task.
• Interactive Labeling Experience: The introduction of "Labeling Copilot" transformed the labeling process. This feature enables semi-automated labeling, where users could benefit from prompt inputs and chat functionalities. The ability to extend prompts related to target classes and roll back auto-generated changes provided Geberit’s labelers and labeling managers with greater control and flexibility.
• Labeling Automation: HumanSignal's solution also allows for batch inference using a machine learning model. Geberit could easily review and convert AI-labeled tasks into final annotations. The automation process was refined with quality control measures, ensuring the predictions were consistently accurate.
• Enhanced Model Integration: Label Studio provides integration options for public services APIs, including notable names like OpenAI, Anthropic, and Replicate, as well as custom models, which allowed Geberit to integrate its ML pipelines more easily.
• Guardrails Functionality: To ensure the quality of the output, Label Studio helps provide guardrails. This feature structures the output from the LLMs, significantly reducing errors and enhancing the overall quality of the labeled data. This, in particular, was critical to Geberit’s success.

By implementing these solutions, Geberit was not only able to streamline its email labeling process but also ensure that the results were of the highest quality. Geberit and HumanSignal's collaboration showcases innovative machine learning solutions' transformative power to automate data labeling and improve efficiency.

HumanSignal's solutions not only streamlined our email categorization but also empowered us with near-instant feedback, ensuring continuous quality improvement.

Dr. Tilo Sperling, Head of AI-Projects Business Applications

Key Benefits

Geberit saw immediate benefits from their work with HumanSignal:

• Seamless integration of generative AI models, allowing for much faster labeling.
• Subject matter experts able to set up their own labeling prompts with a short introduction, no data science/ML background needed.
• Efficient AI labeling after the base prompt was written - a prompt design based on 50 emails was far more efficient vs. hand-labeling thousands of emails.
• Improved data quality with the introduction of guardrails, completely eliminating any noticeable occurrences of AI "hallucinations."
• Flexibility in labeling with the ability to revert to previous edits and add detailed label descriptions.

Overall, Label Studio enabled Geberit to achieve impressive accuracy in labeling their data with almost human-level accuracy, increased efficiency in the process, improved scalability, and enabled a smooth user experience with an intuitive interface.

Outcomes

The collaboration between Geberit and HumanSignal redefined how Geberit approached its challenges. Using Label Studio’s automation features, Geberit estimates that they were able to achieve 5x faster throughput (speed per 1000 items) relative to straight manual labeling, at 4-5x lower cost savings across LLM charges and annotator time.

One of the most notable achievements was the impressive accuracy attained through LLM-automated classifications. Geberit achieved a labeling performance almost on par with human capabilities, reaching up to 93% human-reviewed accuracy. This was a significant milestone, considering the vast volume of data they dealt with.

Furthermore, the efficiency of Geberit’s tailored BERT-based model became evident as it consistently outperformed in latency, cost, and accuracy in subsequent iterations. This custom model not only streamlined the process but also ensured that the quality of results was consistently high, reducing the need for manual oversight and corrections.

The interactive labeling screen interactions with LLM emerged as a pivotal feature. It ensured a continuous quality improvement mechanism, allowing Subject Matter Experts (SMEs) to provide near-instant feedback. This feedback loop was crucial in refining the AI models and ensuring they evolved per Geberit's requirements.

Beyond these tangible outcomes, Geberit realized several additional benefits. The flexibility and adaptability of the system meant that Geberit could scale its operations without compromising on quality. The enhanced security measures ensured that data integrity and compliance were always maintained, giving Geberit the peace of mind to focus on its core operations. Moreover, the intuitive user interface and features provided by HumanSignal ensured that Geberit's team could easily adapt to the new system.

Conclusion

Geberit's collaboration with HumanSignal is a compelling example of the transformative impact of advanced technological solutions on business operations. By addressing and resolving Geberit's immediate data-labeling challenges, HumanSignal has provided a solution for the present and paved the way for a more streamlined and efficient future.

This partnership has highlighted the immense potential of harnessing the right tools, ensuring that data processing remains accurate and of the highest quality. As Geberit moves forward, they are better equipped with innovative tools and strategies to elevate its data-driven initiatives. The success of this collaboration emphasizes the pivotal role that strategic technological partnerships can play in empowering businesses to push their operational limits and achieve unparalleled success.

With Label Studio Enterprise, we've been able to bootstrap our labeling performance to near-human accuracy, transforming our data processing like never before.

Dr. Tilo Sperling, Head of AI-Projects Business Applications