Blog

Large language models can reason and plan—but they can’t act. Model Context Protocol (MCP) changes that. Learn how to integrate MCP with Label Studio to automate and simplify your annotation workflows.

LLM-as-a-Judge promises speed, scale, and lower costs—but at what price? This post explores where automated evaluation works, where it fails, and how to make it trustworthy. From subtle biases to hybrid evaluation strategies, we break down how to evaluate the evaluator itself.

Learn how you can use Plugins to create a simple content moderation interface that will alert annotators of disallowed text in task submissions.

Krippendorff’s alpha is one of the most flexible ways to measure inter-annotator agreement. Unlike other metrics, it works with incomplete data and supports various label types. This guide walks through a real example—showing you how to calculate observed and expected agreement step by step, and what to do when scores are low.

AI benchmarks are quietly failing us. Studies reveal that data leakage, leaderboard manipulation, and misaligned incentives are inflating model performance. This blog explores four pillars of reform, governance, transparency, broad-spectrum metrics, and oversight, and outlines how enterprises can build trust through a centralized benchmark management platform.

The latest In the Loop episode breaks down the Model Context Protocol (MCP)—a new standard for connecting LLMs to tools, data, and real-world actions. Learn how MCP enables practical, production-ready AI.

Learn how to automatically mitigate suspicious annotations like spam, bots, and fraud with Plugins in Label Studio Enterprise.

Not every ML team looks the same and your labeling workflow shouldn’t either. In this guide, we break down three common annotation team setups and how to tailor your tools and processes to boost quality, speed, and scale.

Learn how to generate synthetic Q&A pairs using Label Studio’s Prompts feature. This step-by-step guide walks through creating a project, writing prompts, and reviewing model-generated data—perfect for bootstrapping RAG systems or training LLMs when labeled data is limited.

Apply powerful customizations to your Label Studio projects with only a few clicks. Browse a library of popular Plugins for advanced use cases or write your own using JavaScript.

Automated metrics can tell you what’s wrong with your RAG system—human review tells you why.
This post walks through a structured approach to evaluating RAG outputs using tools like Ragas and Label Studio. Learn how to prioritize weak responses, streamline human-in-the-loop review, and use feedback to iteratively improve your retrieval, prompts, and models.

Training a RAG system starts with clean, structured QA data—but messy source material makes that tough. In this post, we share how we used Label Studio and its Prompts feature to break down tasks, synthesize QA pairs, and build a reliable RAG assistant. If you're looking to scale RAG with better data and smarter workflows, this guide is for you.

Using LLMs to label data is fast—but is it accurate? This post shows two ways to evaluate model performance in Label Studio, with or without ground truth. Learn how to compare outputs, track accuracy and cost, and choose the right model for your workflow.

In this post, we walk through a creative use case for Label Studio’s AI Assistant and Prompts: generating rich, image-based narratives. Follow along to see how you can build a project that combines structured labeling with a touch of storytelling.

When Stop the Traffik lost years of labeled data, they needed a faster, smarter way to rebuild. With Label Studio, they transformed their approach—bringing structure to messy reports, integrating AI for pre-labeling, and uncovering trafficking patterns hidden in plain sight.

RAG systems promise more accurate AI responses, but they often fall short due to retrieval errors, hallucinations, and incomplete answers. This post explores seven common RAG failures—from missing top-ranked documents to incorrect formatting—and provides practical solutions to improve retrieval accuracy, ranking, and response quality. Learn how to optimize your RAG system and ensure it delivers reliable, context-aware AI responses.

SmolDocling is designed for end-to-end document conversion, extracting text, tables, and layout with high efficiency. But how well does it perform on real documents? In this post, we walk through testing SmolDocling’s OCR capabilities using Label Studio and a step-by-step notebook to help you evaluate its accuracy.

Mistral OCR is setting a new standard for document understanding, but how well does it perform on your data? Using Label Studio, you can evaluate its accuracy, compare outputs, and fine-tune results. In this post, we walk through the process and share a hands-on handbook to get started.

With Label Studio's Prompts, you can now handle multiple image annotation types - from basic object classification to nuanced caption generation - in a unified workflow that maintains both efficiency and quality.

RAG is transforming how businesses use AI, but without human oversight, its accuracy and reliability suffer. This blog explores the biggest challenges in RAG implementation and how human expertise improves data quality, retrieval relevance, and AI-driven decision-making.

Accelerate pre-labeling and model evaluation with LLM-powered workflows, while ensuring quality and managing costs. The Prompts interface is now generally available in Label Studio SaaS with support for more data types, evaluation metrics, and models.

Explore the fundamentals of RAG, its advantages over fine-tuning, and the challenges of implementation.

Subject matter experts can save time labeling multiple tasks at once. Filter to label all matching tasks, or select from a grid view using Data Preview for closer visual inspection.

Learn about Reinforcement Learning with Verifiable Rewards, one of the leading training strategies for injecting learning signals into LLMs, successfully employed by models such as DeepSeek R1 and Tülu 3.

Learn the data curation and human supervision techniques that we believe are crucial to DeepSeek’s success by examining technical reports from DeepSeek-R1, DeepSeek-V3, and its predecessors.

Get started with content moderation labeling to classify potentially harmful data for training purposes, or evaluate the output of a model you already have in place.

Evaluating multi-turn conversations with Label Studio helps identify performance gaps, refine responses, and improve user experience for a more effective virtual assistant.

Large Language Models (LLMs) are revolutionizing how we tackle data annotation, offering speed and efficiency that were previously unattainable. With Label Studio's Prompts, you can now handle multiple natural language processing (NLP) annotation types for a single data task in a unified Label Studio workflow.

Annotator Performance Dashboards make it easer to manage labeling teams at scale and orchestrate all the internal and external resources needed to improve data quality, without sacrificing speed.

While data annotation for LLMs may look and feel somewhat different than the data annotation of the past, it’s still a crucial step of the machine learning process.

Go through the entire fine-tuning process on OpenAI’s platform—from preparing recently-posted Wikipedia data to estimating costs and deploying your fine-tuned model.

In this post, we’ll guide you through the process of using Prompts in Label Studio Enterprise to pre-annotate data for Named Entity Recognition (NER) tasks.

Learn the intricacies of data quality, strategies to build the data you need for training and fine-tuning ML/AI models, and how you can use Label Studio Enterprise to engineer your AI/ML success.

We’re excited to release our latest improvements to Label Studio Enterprise’s quality workflow: the ability to attach comments to a specific piece within an annotation and more granular reviewer rejection options.

Learn when you should use webhooks vs. the API in Label Studio, and see examples of what you can do with webhooks.

Learn how to use Ultralytics YOLOv8 object detection model with Label Studio.

OpenAI’s new Structured Outputs feature allows you to ensure outputs conform to a defined JSON structure. In this blog, we’ll explore how to leverage this feature for various labeling tasks.

We’ve released a new version of the Ultralytics YOLO ML backend connector designed for YOLOv8 and YOLO11, which now supports advanced object detection, segmentation, classification, and video object tracking with Label Studio.

We’re excited to announce a new feature that enhances Label Studio’s video labeling capabilities: video frame classification.

Explore the topic of evaluation for LLMs, its importance, and how we should approach it. Learn how integrating systematic evaluations can help teams iteratively refine their models to meet real-world needs.

We’ve updated the reviewer workflow to make it easier and more intuitive.

In this tutorial, we'll guide you through the process of setting up and using Label Studio in combination with Ragas (Retrieval-Augmented Generation Answer Scoring) and GPT-4 to build an optimized QA application.

Implementing RAG-based systems comes with challenges to be aware of, particularly in assessing the quality of generated responses. This article will walk you through some of those challenges.

Label Studio is already the most customizable labeling platform. We’re making it even more flexible with custom scripts.

Evaluate the output of LLMs and RAG pipelines with Label Studio using five new templates designed for human supervision of AI models.

Connect Segment Anything 2 (SAM2) with Label Studio to accelerate image and video data labeling.

Delve into three effective methods to automate your labeling using Label Studio, including examples and resources.

We just released exciting functionality that could transform the way your data science teams work: fully-automated data labeling!

Introducing Evaluations, Prompts, and the new HumanSignal platform. These new features make it easier to build reliable generative AI for the enterprise. Read on to learn more!

In this article, we want to demonstrate a method of curating large datasets to reduce but not remove the cost for curating a high quality medical Q&A dataset in Label Studio and fine-tuning Llama 3 on this data.

This article is part of a longer series that will teach you how to develop and optimize a question answering (QA) system using Retrieval-Augmented Generation (RAG) architecture. In this tutorial, we are going to show you how to create a generator that builds responses based on those documents.

An ongoing challenge for Large Language Models (LLMs) is their tendency to hallucinate. In this article, we explore four methods to automatically detect these errors.

In this introduction to our tutorial series on optimizing RAG pipelines, we'll introduce an example question answering (QA) system leveraging a Retrieval-Augmented Generation (RAG) architecture and outline three methods for optimizing your RAG pipeline utilizing Label Studio.

The short answer is: it depends. Read on as we explore this topic further, uncovering the advantages and drawbacks of each approach to help you make an informed decision.

This post will take you through the intricacies of data quality, the strategies employed to build top-tier datasets, and how to use Label Studio Enterprise to engineer your AI/ML success.

New reports & graphs inside Label Studio provide the data you need to accurately pay annotators, track performance, and allocate resources.

Understanding the distinction between regular datasets and ground truth datasets is crucial for leveraging data effectively in machine learning and data analysis tasks. This article explores both concepts and digs deeper into the importance of ground truth datasets.

Generalist models, like GLiNER, provide an excellent starting point for the tasks that they aim to solve. Fine-tuning these models offers us a way to improve their performance in the areas that we care about to solve business problems.

Different models are naturally going to excel at different tasks (just like humans). For users — especially those building products — having visibility into those tradeoffs is going to be a critical part of the decision-making process.

Sure, benchmarks are cool, but they don’t give you the feel or the intuition of how a model actually works. To get that, you’ve got to hack around with the model and throw real-world prompts at it — like you’d do in day-to-day tasks.

Harness Generative AI and ML models for pre-labeling, interactive labeling, and model evaluation.

Today we’re launching a new feature to get your most challenging tasks in front of additional annotators—automatically.

RLHF has enabled language models trained on a general corpus of text data to be aligned with complex human values. This article details how you can train a reward model for RLHF on your own data.

These five tips for using Label Studio's API and SDK demonstrate these tools' powerful capabilities and flexibility for managing data labeling projects. From efficient project creation and task imports to advanced configurations and bulk data exports, Label Studio provides a comprehensive and streamlined approach suitable for beginners and advanced users.

Get a brief overview of RAG and how it relates to LLMs, learn when you might consider using RAG, and get a summary of some challenges based on current research you should be aware of should you choose to travel down this path.

From precise disease diagnoses to personalized treatment plans, accurately labeled data profoundly impacts healthcare. This guide explores the fundamentals of medical data labeling, its applications, and its evolution through AI.

Introducing new filters for managing users at the organizational levels. Also new are collapsible cards for the ranker interface, making it easier to work with high volumes of answers and cards containing lots of text.

When training Large Language Models and utilizing machine learning, the significance of precise and efficient data labeling cannot be overstated. Here are ten actionable tips to elevate your data labeling processes.

The newest version of Label Studio Enterprise includes support for large-scale taxonomies from external sources. This allows teams to load, manage, and maintain well-defined taxonomies of hundreds of thousands of choices in less than a second.

The newest version of Label Studio Enterprise includes two updates that provide granular visibility into outliers and reduce security risks from churned employees: label distribution donut charts for label groups and user soft delete.

We're delighted to share our latest open source project with you! Meet Adala: a groundbreaking new framework for implementing agents specialized in advanced data processing, starting with data labeling and generation.

From active learning to autonomous agents, learn the use cases, strategies, and tradeoffs for automated data labeling.

At HumanSignal, our top priority is the security and privacy of our customers' data. Today, we're proud to announce that we have achieved HIPAA compliance.

This month, we've released an update that will streamline project setup. Labeling Configuration Autocomplete eliminates the need to code when creating custom labeling interfaces or modifying existing templates.

We're excited to release Project-Level Roles. These provide more granular access to your data and simplify managing internal and third-party annotator permissions.

We are excited to share some new functionality that will enhance your data labeling experience with Label Studio - read on to learn more!

The realm of data labeling is undergoing significant transformations, reflecting the dynamic nature of the tech industry. Here are some of the most notable trends and their implications.

Integrating a machine learning (ML) backend into the data labeling process for a labeling platform can significantly enhance the efficiency and accuracy of the process.

With the introduction of Project Performance Dashboards, we're making it easier than ever to track and optimize your data labeling projects.

We're excited to showcase some new features we've added to Label Studio Enterprise specifically designed to help create datasets for fine-tuning Large Language Models (LLMs) like ChatGPT or LLaMA.

In our four-year journey as Heartex, we've successfully built Label Studio, a top-notch data labeling platform used by tens of thousands of organizations. Today, we're taking a bold step forward as HumanSignal, harmonizing human insights and feedback with AI progression.

Learn how building a scalable data labeling process ensures that your ML models have enough accurately-labeled training data to be effective and efficient.

Explore the essential steps and guidelines to create a data annotation team that can actively contribute to creating reliable data models.

We recently held a webinar with Dr. Vera Dvorak, Machine Learning Operations Manager at Yext. We’ve pulled out a few key takeaways for you.

As we wrap 2022, the Label Studio community survey reveals trends, investments and technology choices for data science teams in the year ahead.

We've added comments and notifications to Label Studio Enterprise.

The Heartex team celebrated growth and milestones hit in 2022 at our first team offsite—join us in 2023!

Learn the four core pillars of data labeling — data, process, people, and technology — and how to build a successful data labeling practice.

Enterprise customers can feel confident that their high standards for security and compliance are met while experiencing the convenience of SaaS.

Learn why going from manual data labeling to intelligent data labeling could be the key to saving time and cost.

Learn about data labeling from Heartex founder and CEO Michael Malyuk.

Get started with sample labeling projects for image annotation, natural language processing (NLP), audio annotation, and time series data with a free trial.

The newest version of Label Studio Enterprise includes a major update to our annotations UI that makes the tool much more ergonomic, efficient, and ready to support larger, more complex tasks with dozens to hundreds of regions.

Improved user, workspace and role management with new SCIM integration, and UX improvements to speed up your team’s annotation and review workflows.

An overview of the common ways to annotate data based on the type of data and business goals.

Better ML/AI performance starts with accurate and consistent data, labeled by domain experts, accelerated by active learning.

Joe Alfaro joins as VP of Engineering, Lauren Sell as VP of Marketing & Ecosystem, and Brandi Bergstrom as Head of Talent

Read about an important milestone in our ongoing commitment to operational excellence and data security.

We’ve hit a big milestone for the company—securing our next funding round of $25 million in funding led by Redpoint, with participation from all our existing investors, Unusual Ventures, Bow Capital, and Swift Ventures.

Data-centric AI is a rapidly growing, data-first approach to building AI systems using high-quality data from the start and continually enhancing the dataset to improve the model's performance. Data-centric AI is a modern approach to building AI where model accuracy is primarily dependent on data quality.

Learn about the most popular technologies and tools data scientists and ML teams leverage to power data-centric machine learning.

Data labeling may seem simple, but it isn’t always easy to implement at scale. And getting it wrong will delay your entire model training process. Learn how to develop your labeling strategy for scale and accuracy.

Building and managing a data science team has some interesting and unique challenges. How do you structure your team? What are the right roles?

2021 was a monumental year for Heartex and Label Studio. We innovated, built the largest data labeling community, and hired an amazing team. What's in store for 2022?

Zhuoru Lin, Data Scientist at Bombora, the leader in B2B intent data and Heartex customer, sat down with us to discuss how Bombora uses Heartex Label Studio to test and validate new NLP models.

To be fully effective, data scientists need to work with other roles as part of a team. As companies fully embrace data and build their data science departments, it is essential to establish the right processes and workflows first before proceeding to hire people with the right skills needed to implement these processes. Here are some important roles to consider when structuring a data science team.

The latest updates to Label Studio Enterprise, featuring custom agreement metrics and export snapshots to enhance annotation evaluation for your data science and machine learning projects.