This Startup Is Using AI to Help Keep Store Shelves Stocked

In that old system, trainees were in charge for several hours of each procedure. It wasn’t much different for laparoscopic surgery (sometimes called “minimally invasive surgery”), in which tools and cameras are put into the patient via tiny slits. In those surgeries, trainees did much of the preliminary work and cleanup as well. This system of master-apprentice cooperation was so entrenched that hours spent in the operating room (OR) are still seen as a proxy for skill development.

That’s not working in robotic surgery. Surgical robots have become increasingly prevalent in hospitals ever since the
da Vinci Surgical System was approved by the U.S. Food and Drug Administration in 2000. The da Vinci robot, from the Silicon Valley–based company Intuitive Surgical, dominates the market today. Intuitive has more than 6,700 machines in hospitals around the world, and the company says that in the United States, da Vinci machines are used in 100 percent of top-rated hospitals for cancer, urology, gynecology, and gastroenterology diseases. There are also a variety of specialized robotic systems from other companies that are used in fields such as orthopedics, neurology, and ophthalmology.

In robotic surgeries, the most dangerous times are at the beginning and the end, when the surgical team “docks” the massive robot to the patient. For the current generation of da Vinci systems, that means positioning four robotic arms tipped with surgical tools and creating “ports” for those tools by inserting metal cylinders into the patient’s abdomen via small incisions. The first port allows the entry of the camera; the ports are used for scalpels, graspers, cauterizing instruments, staplers, or other tools.

Once the robotic arms are in place and instruments are inserted, the surgeon “scrubs out” and takes up position perhaps 15 feet away from the patient in the immersive da Vinci control console, which provides a stereoscopic view. The surgeon’s hands are on two multipurpose controllers that can move and rotate the instruments in all directions; by switching between instruments, the surgeon’s two hands can easily control all four robotic arms.

The da Vinci Surgical System has four arms tipped with exchangeable surgical tools. One arm typically inserts the camera while others insert tools such as scalpels, graspers, cauterizing instruments, and staplers.Spencer Lowell

And the trainee… well, the trainee gets to watch from another console, if there is one. While the lead surgeon could theoretically give the trainee one of the robot arms to control, in practice it never happens. And surgeons are reluctant to give the trainee control over all the arms because they know that will make the procedure take longer, and the risk to the patient goes up nonlinearly with elapsed time under anesthesia.

I began researching the impact of surgical robots on surgical technique and education in 2013. My studies have found that hospitals that adopted the technology have most often turned trainees into optional assistants in the OR, meaning that they begin practicing as “real” surgeons without enough skill. Reversing this trend would require sweeping institutional change, which I don’t expect to happen anytime soon. So, I’m working with collaborators on an alternate solution for surgical skill learning. The platform we create could turn out to be broadly useful, perhaps even turning into a blueprint for 21st-century apprenticeship.

Surgical robots are marvels of engineering in many ways. The da Vinci system gives surgeons a magnified view and robotic hands that never shake, enabling very precise surgical maneuvers. It also provides more efficient and intuitive control than surgeons get from laparoscopic tools: Those operate on fulcrums, so moving a hand to the left moves the tool to the right. The da Vinci robot also provides haptic feedback, with earlier models vibrating the controllers if the software detected instrument “clashes,” and more recent models providing similar feedback when surgeons move too quickly or operate out of the visual field. And the ergonomic consoles are certainly easier on surgeons’ bodies; they no longer have to hunch over an operating table for hours at a time. The robots have also been a marketing phenomenon that has led to a robotic-surgery arms race, with mid-tier hospitals advertising their high-tech capabilities.

Many people assume that patient outcomes must be better with robotic surgery. It’s not obvious that’s true. In fact, a recent survey of 50 randomized control trials that compared robotic surgery to conventional and laparoscopic surgeries found that
outcomes were comparable, and robotic surgeries were actually a bit slower. From my perspective, focusing on education, it’s something of a miracle that outcomes aren’t worse, given that residents are going to their first jobs without the necessary experience. It may be that the outcomes of inexperienced junior surgeons are counterbalanced by those of senior surgeons—or it may be that junior surgeons are really learning on their first patients “in the wild,” which is a somewhat uncomfortable idea. This is a hot research area, so we should know more soon.

It may seem counterintuitive that surgical trainees need more training time. To become a surgeon, a person must first spend four years in medical school and then
at least five years in a residency program. Medical residents are famously overworked and sleep-deprived, to the extent that the United States passed regulations in 2003 limiting their workweek to 80 hours. But although surgical residents spend many hours in the OR, my findings show that those hours aren’t giving them the skills they need. And because they’re always racing from one patient-related task to the next, they spent almost no time on simulator programs, though they are available. The last time I checked on this situation, about a year ago, most hospitals mandated that residents spend about four hours per year on simulators. That’s like asking someone to play a video game for four hours per year to prepare for a life-or-death situation.

In many ways, the issues arising in robotic surgery mirror those confronted by other professions as they have come to rely increasingly on automation. The situation is summed up as the “
automation paradox”: The more advanced and reliable the automated system, the more crucial the contributions of the human operator. That’s because the system will inevitably encounter unexpected circumstances that fall outside its design parameters or will fail in some way. In those rare but critical moments, the operator must detect the failure and take over, quickly bringing the very human faculties of creativity and problem solving to bear on a tricky situation. Airline pilots became familiar with this issue as autopilot became ubiquitous, and the promise of self-driving cars is bringing this conversation to the general public. Surgical robots have quite limited autonomy at this point, so the surgical profession should learn from these examples and act now, changing the human-machine relationship to both preserve surgical skill and avert tragic crashes in the OR.

My conclusions come from two years spent studying the impact of robots on surgical training. I spent a great deal of time at five hospitals, observing 94 surgeries that took a total of 478 hours. I next conducted interviews at 13 more top-tier teaching hospitals around the United States, gathering information from senior surgeons and sets of trainees that the surgeons deemed high-performing or average. The paper I published in 2019 summarized my findings, which were dismaying. The small subset of trainees who succeeded in learning the skills of robotic surgery did so for one of three reasons: They specialized in robotics at the expense of everything else, they spent any spare minutes doing simulator programs and watching YouTube videos, or they ended up in situations where they performed surgeries with little supervision, struggling with procedures that were at the edge of their capabilities. I call all these practices “shadow learning,” as they all bucked the norms of medical education to some extent. I’ll explain each tactic in more detail.

Residents who engaged in “premature specialization” would begin, often in medical school and sometimes earlier, to give short shrift to other subjects or their personal lives so they could get robotics experience. Often, they sought out research projects or found mentors who would give them access. Losing out on generalist education about medicine or surgery may have repercussions for trainees. Most obviously, there are situations where surgeons must turn off the robots and open up the patient for a hands-on approach. That situation almost never occurs because of a robotic failure; it’s more likely to occur if something goes wrong during the robotic procedure. If the surgeon accidently nicks a vein or cuts through a tumor in a way that causes a leakage of cancerous cells, the recovery mode is to undock the robot rapidly, cut the patient open, and fix the problem the old-fashioned way. My data strongly suggest that residents who prematurely specialize in robotics will not be adequately prepared to handle such situations.

The robots are a marketing phenomenon that has led to a robotic-surgery arms race, with mid-tier hospitals advertising their high-tech capabilities.

The second practice of successful trainees was abstract rehearsal, spending their spare moments in simulators and carefully reviewing surgical videos. One resident told me that he watched a one-hour video of a certain procedure perhaps 200 times to understand every part of it. But passively watching videos only helped so much. Many recordings had been made public because they were particularly good examples of a procedure, for example. In other words, they were procedures where nothing went wrong.

Practicing on the simulator was helpful for trainees, giving them fluency in the basics of robotic control that might impress a senior surgeon in the OR and cause the trainee to get more time on the console. But in the case of the da Vinci system, the simulator software was often only available via the real console, so residents could only practice with it when an OR was empty—which typically meant staying at the hospital into the evening. A few elite institutions had simulation centers, but these were often some distance from the hospital. Most residents didn’t shirk other responsibilities to make the time for such dedicated practice.

An additional drawback of the simulators, some senior surgeons told me, was that they don’t include enough examples of the myriad and compounding ways in which things can go wrong during surgery. Even the best surgeons make errors, but they recover from them: For example, a surgeon might accidentally nick a small blood vessel with a scalpel but quickly seal the cut and move on. In surgery and many other occupations, one of the most important things that trainees need to learn is how to make errors and recover from them.

The final practice of successful trainees was finding situations in which they were able to operate on a patient with little supervision, often working near the edge of their competency and often in violation of hospital policies. Some were working under “superstar” surgeons who were officially in charge of several simultaneous procedures, for example. In such cases, the expert would swoop in only for the trickiest part of each operation. Others rotated from high-status hospitals to departments or hospitals that had relatively little experience with robotic surgery, making the trainees seem competent and trustworthy. Middle-tier hospitals also put less pressure on surgeons to get procedures done quickly, so handing control to a trainee, which inevitably slows things down, was seen as more acceptable. Residents in all these situations were often tense and nervous, they told me, but their struggle was the source of their learning.

To change this situation in a systematic way would require overhauling surgical residency programs, which doesn’t seem likely to happen anytime soon. So, what else can be done?

In the past five years, there has been an explosion of apps and programs that enable digital rehearsal for surgical training (including both robotic techniques and others). Some, like
Level EX and Orthobullets, offer quick games to learn anatomy or basic surgical moves. Others take an immersive approach, leveraging recent developments in virtual reality like the Oculus headset. One such VR system is Osso VR, which offers a curriculum of clinically accurate procedures that a trainee can practice in any location with a headset and Wi-Fi.

I’m working on something different: a collaborative learning process for surgical skill that I hope could be analogous to
GitHub, the platform for hosting open-source software. On GitHub, a developer can post code, and others can build on it, sometimes disagreeing about the best way forward and creating branching paths. My collaborator Juho Kim and I are in the early stages of building a crowdsourced repository for annotated and annotatable surgical videos, not only eliminating the time required to search for useful videos on YouTube but also giving watchers a way to interact with the video and increase their active learning. Thankfully, we have a superb industry collaborator as well: the Michigan Urological Surgery Improvement Collaborative. They curate an open library of robotic urologic surgical videos that is known worldwide.

One somewhat similar platform exists for video-based learning: the
C-SATS platform, which is now a subsidiary of Johnson & Johnson. That subscription-based platform enables surgeons to securely upload their own videos and uses AI to scrub out all personally identifying information, such as images of a patient’s face. It then gives surgeons personalized feedback on their performance.

If C-SATS is the Encyclopedia Britannica, we’ll be Wikipedia. We’re currently testing an alpha version of our free and open-source platform, which we call Surch. Recently, we’ve been testing an alpha version with groups of surgeons and residents at select top-tier teaching hospitals to determine which features would be the most valuable to them. We’ve asked testers to complete tasks they typically struggle with: finding good quality surgical videos that match their learning objectives, processing videos actively by making notes on things like surgical phases and anatomy, and sharing those notes with others for feedback. It’s still an academic project, but based on the enthusiastic response we’ve gotten from testers, there seems to be demand for a commercial product. We may try to embed it in a surgical residency program for a year to test the platform further.

I believe that we need a 21st-century infrastructure for apprenticeship.

I believe that we need a 21st-century infrastructure for apprenticeship. The problems I found in robotic skill development have arisen because surgeons are relying on an apprenticeship model that was invented many thousands of years ago: Watch an expert for a while, get increasingly involved, then start to help more junior members along. This process goes by many names—in surgery, it’s called “see one, do one, teach one”—but it always requires one-on-one collaboration in real work, and it’s therefore not remotely scalable.

Since the 1990s, our societies have invested heavily in the infrastructure needed to scale formal learning of explicit knowledge; think of the proliferation of online lectures, documents, quizzes, group chats, and bulletin boards. We need the equivalent infrastructure for embodied skill if we’re going to build the capabilities we need for new kinds of work.

My collaborators and I imagine our Surch platform evolving into an AI-enabled global GitHub for skill learning. Any form of procedural knowledge could be captured, studied, and shared on this kind of platform—supported by AI, people could efficiently and collaboratively learn how to shuck oysters, remove tree stumps, change the oil in their cars, and countless other tasks. Of course, we’ll be grateful and excited if our system makes a difference just for surgeons. But the world requires many skills that you can’t write down, and we need to find a modern way to keep these capabilities alive.