It’s official: Not all domain experts are created equal

June 18, 2020

Where do domain experts come from? The words “expert” and “experience” share the same root, and indeed, to become a domain expert one must acquire an intimate familiarity with the particular segment of the world that a machine learning model requires. For autonomous vehicles, that expertise is widespread. Virtually all drivers (and even many non-drivers) have the necessary level of familiarity to annotate images relevant to this domain. We begin cultivating that expertise as children, looking out the car window as our parents drive us around town. The ubiquity of expertise, the relative ease of recruiting domain experts, and hence the resultant high volume of training data able to be generated is an important reason why autonomous vehicles are making such rapid progress among real world applications of AI technology.

From the preceding example it is easy to see why medical AI faces a critical shortage of domain experts, and will continue to face a shortage in the years ahead. Unlike domains in the external world, medical domains are hidden from view. Even medical students attain their expertise over years of study, and continue to hone their skills long after they gain certification. Whether looking at radiologic images, pathologic slides, or surgical videos, full depth of understanding resides in the hands of a relative few. Relying solely on medical professionals or students to annotate images of tumors in a CT scan, for example, is not a scalable strategy to meet the demand that is already here, much less future demand. These individuals are limited in number and have limited time to devote to image annotation, leading to lengthy project timelines and high costs. 

But what if it were possible to train lay people to annotate medical images? The strategy is not as radical as it might seem. Consider the field of civil engineering. While it takes years of training to design a bridge, bridge inspectors can quickly identify weak rivets, cracked footings, rusted beams, and other dangerous signs of wear, all without the advanced training required of certified engineers. Domains involving medical imaging, such as radiology, pathology, endoscopy, or robotic surgery, can be far more heterogeneous than civil engineering, which helps explain why engineering technology like computer aided design (CAD) is ubiquitous, while computer assistance still plays only a marginal role in clinical practice.