Want a Better Data Loop? Try Humans.

January 05, 2022

At the iMerit ML Data Ops summit, Cruise’s Head of Artificial Intelligence Hussein Mehenna had a conversation with Facebook AI’s Ragavan Srinivasan titled Emerging AI Companies are Driving a Paradigm Shift.

The conversation focused on the importance of fusing together the AI data loop with the human data loop, and how humans play a vital role in identifying and solving edge cases. The combination of human-in-the-loop best practices, seamless data loop collaboration, and a safety-first mentality will ultimately yield the highest levels of AI and ML success.

The Importance of Humans-in-the-Loop

The success of AI-native products like autonomous vehicles hinges on the sophistication of the data loop they’re built upon. Strong data loops simultaneously generate, annotate, and continually apply new data into production. But the best data loops, like Cruise’s, integrate human involvement.

Data loops depend on human involvement to ensure AI solutions perform high-level activities safely and effectively. Humans-in-the loop will constantly evaluate the performance of the car, ensuring every event that happens around the vehicle is handled in a way that a human would. 

When certain maneuvers make human passengers uncomfortable, it takes human insight to spot these instances. Once these are identified, the humans-in-the-loop can fine-tune the model to perform differently. Cruise refers to these as “data instances”, which are instances where the developers can learn from the vehicle based on how it is performing. 

Cruise partners with iMerit so they can inject human expertise into the data loop. In keeping with Cruise’s ultimate mission of safety, fairness, and ethical AI, experts-in-the-loop also identify bias in the performance of the model. To do this, performance must be understood at a granular level, which can only truly be done with humans-in-the-loop. 

This combination of a strong data loop with human involvement is the recipe for superhuman autonomous vehicle performance.

How Humans Solve Edge Cases

There’s an infinite number of scenarios an autonomous vehicle may face on the road. Cruise understands this, and has a system in place that calls for human involvement in real time while a vehicle is on the road. 

In one particular case a Cruise vehicle was confused by road obstacles in a construction zone. Rather than having the model do its best to solve the problem, the vehicle contacted a human operator. This operator took control of the vehicle remotely and guided it around the construction zone. This human intervention system ensures that the vehicle will behave safely, and is a perfect example of how humans-in-the-loop can help deal with edge cases in real time. 

Each time this happens, there is a follow-up around the edge case. What was it about this construction zone that challenged the vehicle? How can we prevent this from happening again?  

Cruise works with AI data solution companies like iMerit to optimize their loop and help their experts solve edge cases using relevant information, ongoing distraction minimization, and reducing unnecessary UI interactions. This allows Cruise’s experts to clearly identify and service their vehicles for edge cases. 

The ultimate goal will be to create a model that’s capable of solving its own edge cases. Cruise employs techniques that use the model to solve the edge case as much as possible before human intervention. This way, the model will generate as much data as possible for the data loop, and also give all humans-in-the-loop the chance to judge the response.

Instead of having human experts deal with these situations from scratch, Cruise lets their models do as much as they can before the human expert becomes involved. This way the model learns, and the human can observe how it’s handling challenges and deem whether the response was sufficient or not. 

Provided this is done within ethical safety standards, this will be the way for companies to generate invaluable data that will truly move the needle on autonomous vehicle performance.

Seamless Human/Machine Collaboration

With humans, edge cases are dealt with either in real time or from the rearview mirror. In an effort to expose AI models to edge cases, Cruise has designed a simulation system that creates edge cases. This allows the humans to observe the behavior of the car in the safest environment possible while also improving performance. This human/machine interaction and collaboration highlights the importance of seamless human and machine collaboration.

While studying the introduction of robotics into manufacturing, Hussein learned that the best automation systems are built with seamless collaboration between human operators and the robotics system itself. Instead of a siloed interaction, they’re engaging together. The sophisticated data loops needed to make autonomous vehicles successful must be developed with this same seamlessness in mind. 

The siloed evolution of machine learning tools is an extreme pain point for the end-to-end machine learning lifecycle. Building a holistic and integrated solution that allows the human experts, machine learning engineers, software engineers, and data scientists to smoothly hop between each phase of the life cycle is important. 

Ultimately, this is the north star for any comprehensive end-to-end machine learning platforms.

How 2022 Will Drive the Paradigm Shift

Each year more autonomous vehicles will be on the road than ever before. The combination of human-in-the-loop best practices, seamless data loop collaboration, and a safety-first mentality will continue to drive the paradigm shift that took place this past year. 

To learn more about how iMerit’s AI data solutions work with autonomous vehicle industry leaders, listen to the full discussion.