Guide October 1, 2024

Webhooks in Label Studio: When And How To Use Them

Webhooks allow you to set up integrations that subscribe to certain events occurring inside Label Studio. When an event is triggered, Label Studio sends an HTTP POST request to the configured webhook URL. This enables you to automate workflows and seamlessly integrate Label Studio with other systems.

Label Studio supports the following webhook events:

TASK_CREATED: Triggered when new tasks are created.
TASK_DELETED: Triggered when tasks are deleted.
ANNOTATION_CREATED: Triggered when new annotations are created.
ANNOTATION_UPDATED: Triggered when an annotation is updated.
ANNOTATION_DELETED: Triggered when an annotation is deleted.
PROJECT_CREATED: Triggered when a project is created.
PROJECT_UPDATED: Triggered when project settings are updated.
PROJECT_DELETED: Triggered when a project is deleted.
START_TRAINING: Triggered when the Start Training action is initiated for a connected model.

For more information about setting up and developing Label Studio webhooks, see our documentation.

Using Webhooks vs. the API

When should you use webhooks rather than SDK or other API workflows?

The SDK is ideal for the following scenarios:

Batch Operations: Performing bulk operations or querying data in batches.
Custom Integrations: **Building custom integrations where you need more control over the data flow and operations.
On-Demand Actions: When you need to perform actions on-demand rather than in response to specific events.

However, you might want to consider webhooks for the following:

Real-time Notifications: When you need to receive real-time notifications about events happening in Label Studio, such as task creation, annotation updates, or project changes. Webhooks automatically send HTTP POST requests to your configured URL whenever these events occur.
Automated Workflows: If you want to automate workflows based on specific events in Label Studio. For example, starting a machine learning model training process after a certain number of annotations are completed, or triggering data versioning in a dataset repository.
Integration with External Systems: When you need to integrate Label Studio with external systems that can handle HTTP POST requests. Webhooks are ideal for notifying other services or applications about events in Label Studio without the need for continuous polling.
Event-Driven Actions: If your use case requires actions to be taken immediately after an event occurs in Label Studio, webhooks provide a direct and efficient way to achieve this.

Example Webhook Use Cases

Model Pipeline Automation

Use webhooks with your ML backend to automate your training pipeline. For this use case, you would use the ANNOTATION_CREATED and/or ANNOTATION_UPDATED events.

Because the annotation webhook payloads includes the annotation result, you can get really creative with how you use them with your ML pipelines. For example, you can use webhooks to automate prompt engineering tasks so that whenever a negative/false review is captured, you can trigger an event to rewrite the prompt using an SME or another LLM.

As another example, you can use webhooks to send annotation data to an external machine learning system such as Amazon Sagemaker. For a full tutorial of this process, see From raw data to a trained model: Automate your ML pipeline with Label Studio & Amazon SageMaker.

Create and configure your webhook endpoint in AWS.
From the project settings in Label Studio, configure the webhook to send the event to your endpoint.
When an annotation is created, the endpoint listener triggers a script to start training a machine learning model in Amazon SageMaker.
Monitor the training process and update the model registry upon completion.

Tip: In Label Studio Enterprise, users can identify ground truth annotations. This allows you to track model performance against ground truth annotations and trigger actions based on performance metrics.

Data Versioning

Create versions of training data as your project progresses. Creating versioned datasets may be a requirement within your organization to support reproducibility and traceability.

For this use case, you would use the TASK_CREATED and/or ANNOTATION_UPDATED events.

Create and configure your webhook endpoint.
From the project settings in Label Studio, configure the webhook to send the event to your endpoint.
When new tasks are created, the endpoint listener triggers a data preprocessing pipeline.
The preprocessed data is then stored in a versioned dataset repository like DVC or Pachyderm.

Notification System

Notify annotators when there is a new project ready for labeling.

For this use case, you would use the PROJECT_CREATED webhook.

Create and configure your webhook endpoint.
From the project settings in Label Studio, configure the webhook to send the event to your endpoint.
When a new project is created, the endpoint listener sets up a new data management pipeline.
Notify annotators that a new project is ready for labeling.

Tip: Label Studio Enterprise includes a number of built-in queue management and automated workflow features. For example, you can automatically assign annotators to a task that has a low agreement score.

Ontology Creation

Update an external database with information submitted by annotators. For example, you might use this to help build an internal knowledge base as annotators submit feedback on LLM responses.

For this use case, you would use the ANNOTATION_CREATED event.

Create and configure your webhook endpoint.
From the project settings in Label Studio, configure the webhook to send the event to your endpoint.
The endpoint listener parses the payload to extract the updated list of labels.
Update the label list in an external database to keep it in sync with Label Studio.

Active Learning & Labeling Queue Management

Trigger active learning and reorganize the labeling queue based on annotation results.

For this use case, you would use the ANNOTATION_CREATED and/or ANNOTATION_UPDATED events.

Create and configure your webhook endpoint.
From the project settings in Label Studio, configure the webhook to send the event to your endpoint.
The endpoint listener triggers a script to perform active learning, selecting the most uncertain tasks for further annotation.
Update the model with the new annotations and repeat the process. Reorganize the queue (e.g. tabs in the Data Manager) according to the prediction score.