Why Internal Data Labeling is the Right Choice for Data-Centric AI
Mark Brandsetter
According to a VentureBeat study, 87% of data science projects fail to make it to production. We can find the reason for this failure in Alation's State of Data Culture Report, where 87% of employees cite data quality issues as the number one reason their organizations failed to implement AI and machine learning successfully.
To enhance data quality, organizations need to focus on optimizing all the data-related aspects within AI projects, an approach referred to as "Data-centric-AI ." Part of the Data-centric-AI approach focuses on improving data labeling, where informative labels are added to raw data samples to provide proper context for the training of machine learning models. Today's business leaders know that accurate data labeling leads to more accurate and competitive machine learning and AI outcomes. However, many companies looking to invest in data labeling will have a choice to make early on: whether to do data labeling internally (in-house) or outsource it.
Both approaches have pros and cons. However, organizations looking to enhance their data labeling process should consider investing in an internal data labeling practice, and here's why.
In-house data labeling ensures high data labeling quality, consistency, and accuracy, which are critical to the success of any machine learning project. Quality data labeling also requires subject matter expertise, which is better managed internally. In-house domain experts can provide the ground truth for the specific business use cases where you want to apply AI and determine if the dataset accurately represents the problem.
For instance, a doctor/healthcare professional performing data labeling for an AI product for diagnosing medical conditions based on patient images will provide better context and insight than a regular data labeler without such medical experience. In an article published in Nature Machine Intelligence, University of Cambridge machine learning researcher Derek Driggs and his colleagues investigated the application of deep learning models for COVID-19 virus diagnosis.
Driggs' group found their model was flawed because they trained it on a data set that included scans of patients lying down while scanned and not patients standing up. The patients lying down were much more likely to be seriously ill, so the algorithm learned to identify COVID risk based on the person's position in the scan. They could have avoided something like this if the data labeling had been done in-house with collaboration from medical professionals.
Research carried out by data science company Hivemind showed that crowdsourced data labelers had an error rate of more than 10x the managed (in-house) labelers. Additionally, managed labelers' accuracy was 25% higher than that of the crowdsourced team. Managing your data labeling in-house will ultimately lead to more accurate algorithms. With internal data labeling, the company can perform regular quality assurance audits on data labeling tasks at specific intervals ensuring that the process is accurate and of high quality.
Additionally, the labeler's inherent bias comes into play when labeling data sets. Their experience, geography, language, cultural interpretations, and more can impact their interpretation of data. For example, a UK annotator could label a piece of clothing as a "trouser." Another annotator from America could label it as a "pant." This inconsistency could lead to inaccurate outcomes. Performing data labeling internally—and using a platform like Label Studio to drive common workflows and consensus amongst annotators—makes it easier to ensure clear standards, guidelines, and consistency when labeling images and other data.
Internal data labeling keeps sensitive customer and business data private, and mitigates against data breaches or attacks. In its 16th annual Data Breach Report, the Identity Theft Resource Center says the number of data breaches at corporations was up more than 68% in 2021, beating the previous record, set in 2017, by 23%. With data breaches increasing, companies should be wary of exposing sensitive customer data to third-party vendors during the data labeling process.
Internal data labeling using a platform with SSO integration mitigates the risk of sensitive data falling into the wrong hands. You can ensure that only the team members with the required security clearance can access the data, and ensure data is not tampered with or transferred via third party. Also, some datasets can span multiple regions, each having its data laws (GDPR, DPA, CCPA), governing institutions, and privacy restrictions. As a result, ensuring proper data regulation and compliance when outsourcing data labeling, especially in critical sectors and with sensitive information, can be difficult.
Having direct oversight over the data labeling process, as in the case of internal data labeling, ensures more efficient communication and collaboration between teams. With the ever-changing nature of data and its use cases, scenarios will arise where data labeling techniques, conventions, and processes would need to be refreshed. For example, in companies with diverse product offerings, data labeling for product A might not be suitable for product B. Because there will be an extra layer of people and processes to work through, you'll lose the ability to respond quickly to changes in the business environment when the data labeling is outsourced.
Data labelers, data scientists, and other stakeholders can easily collaborate, share ideas, and suggest improvements with in-house data labeling. It's also easier for teams to adhere to quality control workflows such as model-specific labeling techniques established during the workforce onboarding and training. The company can thoroughly educate every member of the in-house data labeling team on the project rules and requirements, and it becomes easier to monitor quality metrics accordingly.
In continuous data projects, the initial costs of investing in a data labeling tool, training, and hiring the right workforce pay dividends in the long run. You can avoid costly data quality errors, ensure properly-labeled data is on schedule and on your timeline, avoid expensive data breaches, and prevent messy vendor contracts.
In the same research by Hivemind, they discovered that in 7% of the cases, crowdsourced workers transcribed at least one of the figures incorrectly. In comparison, in-house workers only made a mistake in 0.4% of cases, an essential difference given its implication for data quality. This study reveals that hiring the cheapest outsourced data labeling provider is less effective since they lack the quality assurance mechanisms to ensure high accuracy. Having to re-do data labeling tasks leads to added costs in terms of wages, lost time, and a more prolonged product launch.
For smaller companies handling short-term, high-volume data projects, active learning can help reduce data labeling budgets. Active learning is a procedure in which you manually label a subset of the available data, and the remaining labels are automatically inferred using a machine learning model. Active learning is often referred to as a smarter data labeling strategy for greatly reducing the time and cost required to perform labeling in high-volume data. Additionally, tools like Label Studio make the labeling process faster, more efficient, and more accurate.
Every AI project's success begins with access to high-quality training data, which is where investing in internal data labeling helps. By collaborating with domain experts, hiring the right people, giving them proper training and resources, and fostering a productive environment for communication and collaboration, you are setting your organization up for data labeling success.
Find out how Heartex's Label Studio can help accelerate your in-house data labeling workflow.
Different models are naturally going to excel at different tasks (just like humans). For users — especially those building products — having visibility into those tradeoffs is going to be a critical part of the decision-making process.
Learn how building a scalable data labeling process ensures that your ML models have enough accurately-labeled training data to be effective and efficient.
As we wrap 2022, the Label Studio community survey reveals trends, investments and technology choices for data science teams in the year ahead.
