At the iMerit MLDataOps Summit 2021, iMerit VP of Engineering Sudeep George moderated a panel with Eldar Khaliullin the Principal Engineer leading the data team for perception at Leap, Keith Fieldhouse the Director of Cyber Physical Systems at Kitware, and Georg Poier the Machine Learning Team Lead at Reactive Reality.
The panelists talked about the importance of data in progressing the alternate/virtual reality arena, a market which is set to reach $454B by 2030. Their diverse expertise gives a multi-dimensional view of the technical challenges that AI leads and entrepreneurs face when building the metaverse of tomorrow.
Data is the Foundation for Alternate Reality
The development and adoption of an alternate reality (AR) world will require a paradigm shift from the current 2D standard to a fully-immersive 3D experience. Rather than flattening our virtual experiences to fit a touchscreen, a different framework will need to be adopted to fully understand how to modify the controls and interactions between humans and a 3D space.
This requires teaching computers how to use human movements as controls, such as the motions we can do with our hands, as well as our visual perception.
“As we’re trying to deal with reality, we need to be sure that we’re capturing the inherent messiness and unexpectedness of the reality that we’re operating in”
– Keith Fieldhouse, Director of Cyber Physical Systems at Kitware
As a core computer vision problem, AR requires large amounts of adequate data to design solutions. It is vital that the data is not sanitized and captures the messiness of the world. Otherwise, we can end up with incomplete virtual representations that will face too many edge cases.
The lack of data challenge can be overcome in two ways:
- Embarking on a long data-gathering phase
- Manually generating synthetic data that mimics real-world data
Synthetic data is touted as a potential solution to ramping up the quantity of data, and it’s easy to see why. In certain problem spaces, getting labeled 3D images requires a very high effort that could be saved by generating the images themselves. Full controls over the generated data can also ensure that datasets are complete for each use case that the AR solution needs to solve.
Of course, synthetic data also has drawbacks, such as requiring significant work to prepare a framework to generate realistic, task-specific data. In addition, it poses limitations in the variety of data and a lack of edge cases.
Well annotated data is crucial to improve the semantic performance and accuracy of the model. Regardless of the strategy for acquiring the computer vision data, there is no question that it would be the distinguishing factor in whether AR/VR companies succeed and a key enabler for this new generation of communication.
Agile Data Management
The sheer quantity of data needed to create a high performance computer vision algorithm poses a whole new set of challenges. Large players in the space – i.e. multinational enterprises – have an enormous competitive advantage because of their ability to own the entire data management pipeline from the collection, curation, annotation, and management to storage, manipulation, and regulation.
“We need to think very carefully about making sure that not just muscle is what allows the advancements in computer vision/alternate reality, but agileness and cleverness and, you know, other ways of developing competitive advantages. “
– Keith Fieldhouse, Director of Cyber Physical Systems at Kitware
New entrants into the AR ecosystem need to be creative in their data usage. This might mean clever ways to use existing image data, perhaps for purposes not initially intended. Competing with well-established data powerhouses requires agile thinking rather than brute force.
Another opportunity can be carved out by cross-communicating throughout the entire solution space. This means increased collaboration starting from data generation to the architecture of the solution, from model tuning to the training infrastructure. We need to ensure coherence between how we build the model and how it relates to the data generation process itself.
Enterprise Alternate Reality to Pave the Way
Intuitively, AR applications might seem best for a consumer space geared towards entertainment. The quirky VR headsets gaining popularity in gaming arcades and theatrical avenues probably influenced this perception. These fun use cases of VR hide the potential it has in B2B scenarios. Of course, any AR interface will eventually be experienced by a consumer, but the content of the experience is bound to be populated by businesses.
For example, a portal that allows users to try on clothes virtually requires retailers to be onboarded and display their wares in the virtual dressing room. In the future, any immersive digital experience is bound to offer sponsorship and advertising opportunities to businesses.
There’s also the fact that solving any major consumer problem with AR would require data and resources at a scale not possible for most new companies. Pivoting to an enterprise problem with a narrow and precise scope might be the most lucrative opportunity. It will enable the technologies to mature as well as break new ground. The consumer space is still worth keeping an eye on, as it drives the AR conversation and would be the context for any new regulations.
Managing AI Biases
“When it comes to data, if we have bias in the data, it’s just bad data. It’s unrepresentative of the world that we’re trying to capture”
– Eldar Khaliullin, Principal Engineer at Leap
Increased social justice reforms and awareness have also shaken the AI/ML world. It is now understood that human interfacing AI systems should be designed with a particular focus on bias training. The fact that the problem has been acknowledged is a good sign. When it comes to virtual reality, the bias could have two sources:
- There could be bias in the data, meaning it is unrepresentative of the world in which we live. This is relatively easy to solve – get better data.
- The data is representative and captures real biases in the world. This is a bigger problem as it is harder to identify. Bias from the real world could get amplified by algorithms, as demonstrated by the plethora of racist chatbots.
Biases cannot be prevented single-handedly or in isolation. To address the risks of bias, AI projects need to define or adhere to a set of ethical guidelines and regulations.
In Conclusion
The panel addressed the challenges and solutions of intersecting the world of AR/VR with ML data. The three key takeaways from the session are:
- Set up a robust data pipeline to ensure a good flow of accurate, good quality (either synthetic or human-annotated) computer vision data.
- As a lean startup, adopt a creative mindset to overcome the challenges of data.
- Target enterprise use cases to break ground in the industry. Enterprises may have more resources to invest and adopt AR solutions, while the consumer market will likely follow suit.
