Google, DoD create AI-powered microscope to help doctors detect cancer

In his office at the VA Hospital in Seattle, Dr. Nadeem Zafar needed to settle the controversy.

Zafar is a pathologist, who performs clinical lab tests on bodily fluids and tissues to diagnose conditions such as cancer. It’s a specialty that often goes behind the scenes, but it’s an important backbone of medical care.

Late last year, Zafar’s colleague consulted him about a case of prostate cancer. It was clear that the patient had cancer, but the two doctors disagreed on how serious it was. Zafar believed that the cancer was more aggressive than his colleague.

Zafar turned to his microscope—a genuinely beloved tool in pathology that doctors use to help make their diagnoses. But this device is no ordinary microscope. It is an artificial intelligence-powered microscope developed by Google and the US Department of Defense.

The pair ran the case through a special microscope and Zafar was right. Within seconds, the AI ​​flagged the exact part of the tumor that Zafar thought was more aggressive. After the machine backed him up, Zafar told his colleague that he was convinced.

“He has a smile on his face and he agrees to it,” Zafar said in an interview with CNBC. “That’s the beauty of this technology, it’s kind of an intermediary.”

The AI-powered device is called the Augmented Reality Microscope, or ARM, and Google and the Department of Defense have been quietly working on it for years. The technology is still in its early days and not yet actively used to help diagnose patients, but early research is promising, and officials say it could prove to be a useful tool for pathologists without easy access to a second opinion.

Currently 13 ARMs exist and one is located at a Miter facility just outside of Washington, DC Miter is a nonprofit organization that works with government agencies to address major technology issues. Researchers there are working with ARM to identify vulnerabilities that could cause problems for pathologists in a clinical setting.

At first glance, the ARM looks like a microscope that might be found in a high school biology classroom. The device is beige, with a large eyepiece and tray for examining traditional glass slides, but it’s also connected to a boxy computer tower that houses the AI ​​models.

When a glass slide is prepared and fixed under a microscope, the AI ​​is able to outline where the cancer is. The outline appears as a bright green line that the pathologist can see through their eyepieces and on a separate monitor. The AI ​​also indicates how bad the cancer is and creates a black and white heat map on the monitor that shows the extent of the cancer in pixelated form.

CNBC demonstrated ARM with researchers at the Miter facility in August.

Patrick Minot, senior autonomous systems engineer at Mitre, said that because the AI ​​is directly overlaid on the microscope’s field of view, it doesn’t disrupt the pathologist’s established workflow.

Ease of use is a deliberate design choice. In recent years, pathologists, like many other corners of health care, have struggled with staff shortages. But as the general population grows, so do pathologists’ case loads.

This is a dangerous combination of features. If pathologists are stretched too thin and something goes wrong, it can have serious consequences for patients.

Many organizations are trying to digitize pathologists’ workflows as a way to increase efficiency, but digital pathology comes with its own challenges. Digitizing a single slide can require more than a gigabyte of storage, so the infrastructure and costs associated with large-scale data collection can quickly balloon. For many small health systems, digitization is not yet a problem.

ARM isn’t meant to replace digital pathology systems, but Minot said it could help healthcare organizations avoid the need for them. Pathologists have the option of screen-capturing slides using ARM’s software, for example, which are much less expensive to store.

An ARM will typically cost a health system between $90,000 and $100,000.

Minot added that the ARM ensures the physical microscope, not just the computer, is an integral part of the pathologist’s process. Many have warned him not to mess with their microscopes, he joked.

Chief Medical Officer of the Defense Innovation Unit or DIU in the Department of Defense Dr. Few understand the challenges faced by pathologists like Niels Olsson.

The DIU was created in 2015 as a way for the military to integrate cutting-edge technologies developed by the commercial world. The organization negotiates contracts with companies so they can collaborate and avoid long bureaucratic hang ups.

Olson is a pathologist and served in the US Navy before starting his role at DIU. In 2018, he was deployed to Guam, a US island territory in Micronesia, where he served as Laboratory Medical Director and Blood Bank Director at the Naval Hospital.

During his two years on Guam, Olson was one of two pathologists on the island and the only pathologist at the Naval Hospital. This meant he often made big decisions and diagnosed himself.

It’s not just your job to say, ‘This is cancer, this is cancer.’ Part of the job is to say, ‘It’s not totally cancer,’ and when you’re alone, that can be nerve-wracking,” Olson told CNBC in an interview. “I would have loved to have an augmented reality microscope in Guam, so someone would be there helping.”

ARMs are meant to serve as a second line of defense for pathologists, and Olson said they won’t replace doctors themselves. He added that an obvious initial use case for the microscope would be in small, remote labs, and that it could also serve as a training resource for pathology residents.

But Olson dreamed of a tool like the ARM long before he lived in Guam. On August 10, 2016, while working as a resident at the Naval Medical Center in San Diego, Olson decided to message a connection he had on Google. In an email seen by CNBC, Olson described a rough idea of ​​what an ARM-like microscope could be.

For a while, Olson said he heard nothing. But a few months later, he was standing in a Google office building in Mountain View, Calif., trapped in a locked room that only a handful of people in the company had access to. There, he watched as the AI-powered microscope successfully identified cancer on a small set of slides he had brought with him.

Olson said the room was sweltering because everyone inside was so “pumped”.

“I don’t want to say it’s like seeing your kid for the first time, but it’s like, it’s cool, it’s something,” Olson said.

Around the time he was sent to Guam, a product manager at DIU noticed Olson’s research. The pair co-wrote an article in 2019 about how the Department of Defense and Silicon Valley can work together to take advantage of AI. He said millions of patients are registered in the federal government’s health care systems, making it “the most comprehensive health care dataset in the world.” That data has a clear commercial use.

“Big data is what Silicon Valley does best, and the potential for spillover into civilian health care systems is immense,” he wrote.

Soon after, DIU began looking for commercial partners to help build and test the ARM. The organization chose optical technology company Genoptic to handle the hardware and, after evaluating 39 companies, chose Google to develop the software.

Ashima Gupta, global director of health care strategy and solutions at Google Cloud, said the company has launched four algorithms for ARM that can detect breast cancer, ovarian cancer, prostate cancer and mitosis. AI models are trained on data from DIU, and Gupta said neither Google employees nor Google infrastructure have access to it.

“It’s encrypted all the way,” Gupta told CNBC in an interview. “How data is collected, how it’s stored and analyzed, and anything in between.”

In terms of hardware and software, DIU is conducting initial research to test the functionality of ARM.

In the fall of 2022, the institute published a peer-reviewed paper in the Journal of Pathology Informatics. The paper found that the breast cancer AI algorithm performed reasonably well in a large area of ​​samples, but there are some caveats, said David Jin, the paper’s lead author and the Defense Department’s deputy director of AI evaluation for Chief Digital and Artificial Intelligence. Intelligence Office.

The paper specifically examined how well the AI ​​performed at detecting breast cancer metastasis in lymph nodes, and Jin said it did better on certain types of cells than others. He said the study is promising, but still needs “a large amount” of rigorous testing before it can help pathologists care for real patients.

“Some of these things have tremendous potential for benefit, but there are also a lot of risks,” because it will change how cancer is diagnosed, Jin told CNBC in an interview.

Olson, who returned from Guam and began working at DIU in 2020, is also listed as an author on the paper. He said three other models for prostate cancer, mitosis and cervical cancer have not yet been independently evaluated at DIU.

Research is ongoing with ARM, and DIU is also seeking feedback from organizations like Miter and health systems like Veterans Affairs. There is still work to be done, but with DIU validating the initial concept, the organization has begun to think about how to scale up the technology and collaborate with regulators.

DIU negotiated agreements with Google and Jenoptik to allow the technology to be distributed militarily and commercially. DIU hopes to make ARM available to all government users through the General Services Administration website this fall.

Zaffer, of VA Puget Sound, said that ultimately, ARM will certainly help pathologists, but the general public will benefit the most from the technology. He said ARM’s accuracy, speed and cost-effectiveness will all contribute to better care.

“AI is here, and it will continue to evolve,” Zafar said. “The point is not to fear these technologies, but to try to make the best use of them for our medical and health care needs.”