At the iMerit MLDataOps Summit 2022, Beata Kouchnir, Director of Machine Learning Science at Glassdoor, and Anna Bethke, Ethical AI Data Scientist at Salesforce, discuss why human-in-the-loop and high-quality data go hand-in-hand to achieve widespread production AI applications. Learn why humans cannot be taken out of the loop in machine learning.
What is Human-in-the-Loop Machine Learning?
To better understand the implications of this session, it’s crucial to introduce the idea of human-in-the-loop machine learning (HITL). AI models are prepared by humans, but human interaction must continue throughout a mechanism. Because AI is based largely on statistics, it will never be 100% accurate. Rather, the journey towards better accuracy and high-quality data is the result of feedback from humans.
HITL allows humans and machines to accomplish what neither can do alone. This panel discussion with Beata Kouchnir and Anna Bethke brings to light the necessity of HITL machine learning.
“I don’t think that we can take humans out of the loop”
– Beata Koouchnir
Automation vs. Human Interaction
There is a time and place for automation, but there are times when human interaction is necessary for machine learning. As Beata explains, automation is good for specific tasks. In particular, she highlights the 3 D’s – Dull, dirty, and dangerous.
When the data fall into one or more of these categories, it is well-suited for an automated response. However, when tasks require cognitive processing, reasoning, and even creativity, AI tends to fail without human feedback.
As we increasingly rely on few-shot and zero-shot learning, the quantity of labeled data necessary for a successful model is reduced. Therefore, the human-in-the-loop process isn’t as tedious because the time to label data is often cut down from weeks to hours. With that, it makes sense to keep humans involved in structured tasks that act as quality assurance against machine learning models.
Test and R&D Mode → Production
When moving from testing to production environments in machine learning, there are a few things to consider. Beata and Anna offer advice related to this, coming from companies that are further ahead with HITL:
“It’s really important to have human-in-the-loop part of the design.”
– Beata Koouchnir
- Even though human-in-the-loop tends to be an afterthought, it’s going to need to be a part of the model design and every step of the lifecycle.
- By being involved at all stages of the model development and testing edge cases, humans ensure that enterprises aren’t deploying AI that will perform poorly.
Businesses moving in this direction have a competitive advantage in the machine learning space.
Changes in the Future
In the next two to three years, Beata and Anna explain that there are some large forecasts related to HITL and machine learning. The main point that was discussed in the panel was that job requirements are going to get harder. What they mean by that is humans need to lean into learning machine learning and AI because they can’t just treat it as a black box. In essence, humans need to wear more hats.
It’s no longer about just drawing a bounding box around an image – humans need to understand how the model works and what it is based on.
What Human Skills are Needed?
As discussed earlier, humans need to know more about machine learning algorithms, but they also need to understand their place in the model development and deployment process. For example, rather than just looking at whether the results are accurate compared to the training data, humans also have the ability to consider whether the results are interesting and useful.
Part of drawing these conclusions is about being a subject matter expert in the business. Understanding things like the terminology and jargon used on a day-to-day basis, as well as the business rules and objectives, is no longer optional for the HITL team.
Leveraging Human-in-the-Loop Machine Learning
This panel session established the importance of HITL as it relates to successful business applications of machine learning. While certain tasks are optimal for being automated, there is no way to take humans “out of the loop”. From the model design to the testing and deployment, humans play a key role in the success of AI and will continue to do so in the future. It’s important that companies plan for this by following ethical AI practices when it comes to HITL machine learning.
For small or medium-sized companies with less experience than large enterprises like Glassdoor or Salesforce, working with a quality data annotation company can make it easier to navigate the machine learning space.
