Spam and fraud annotations are becoming a growing problem in the data labeling space, especially for companies that are scaling rapidly or working with low trust annotators. Finding spam annotations can be challenging – how do you look for patterns that would fit the definition? With Plugins in Label Studio Enterprise, it’s easier than ever to build spam detection directly into your projects. In this tutorial, we’ll show you how!
The first thing you’ll do is create your project as you usually do, adding any data you have and any labeling config that you need. For this example, we’ll use the Labeling Config and sample data found in the plugin documentation. Then, head to your Project Settings and go to Labeling Interface, and click on the Plugins tab at the top of the screen.
With Plugins, you can code any JavaScript that you’d like to work on a task by task basis as your annotators work. To help you get started, we’ve provided a set of out of the box plugins, including one for Spam and Bot Detection! To find it, click on the Insert Plugin dropdown menu on the top left of the JavaScript coding box, and select Spam and Bot Detection. This will automatically insert the code you need into the code editing window!
Out of the box, this code is looking for a few different things:
If any of these conditions are met, an annotator will automatically be paused in your project until a manager or above un-pauses them. All of this code is configurable in your implementation – you’ll want to make sure that you change the field names to match the names of the control tags in your labeling configuration, but you can also change the number of times that you look for similarity, what “too similar” means, and the definition of “too fast”, among other things.
Now that we have our Plugin set up, you can test it by clicking on the “run plugin” button at the bottom of the screen and doing sample annotations in the window. For this Plugin in particular, you need to have the annotator or reviewer role to be paused!
Now, we can go ahead and annotate. If the conditions for pause are met, the annotator will see a pop up window appear telling them that they have been paused for the project and the reason why – in this case, we had too many similar comments in a row.
And that’s it! You just added Spam and Bot detection to your project in a few minutes.
Ready to try Plugins for yourself? Talk to us to enable Plugins on your Label Studio Enterprise account, or check out our Plugin Gallery for more inspiration!
Happy Labeling!
Using LLMs to label data is fast—but is it accurate? This post shows two ways to evaluate model performance in Label Studio, with or without ground truth. Learn how to compare outputs, track accuracy and cost, and choose the right model for your workflow.
In this post, we walk through a creative use case for Label Studio’s AI Assistant and Prompts: generating rich, image-based narratives. Follow along to see how you can build a project that combines structured labeling with a touch of storytelling.
OpenAI’s new Structured Outputs feature allows you to ensure outputs conform to a defined JSON structure. In this blog, we’ll explore how to leverage this feature for various labeling tasks.
