A New Way to Find Polyps

EndoKED is a major step forward because it automates a process that has been a major barrier to developing AI in medicine.

Traditionally, doctors find precancerous growths called polyps during a colonoscopy by visually inspecting thousands of images. This method is the gold standard for detecting polyps but has a major drawback: it's incredibly time-consuming and relies on a person's ability to spot a tiny detail. The sheer number of images and reports generated by these procedures makes it impossible to manually look through all the data in a hospital's archive.

A groundbreaking new study published in the peer-reviewed journal Nature Biomedical Engineering has found a way to solve this problem. Researchers have developed a new system called EndoKED that uses two different types of artificial intelligence (AI) to automatically identify and mark polyps in colonoscopy records. This system works without any human input, which means it can process millions of images and reports from a hospital's database.

The AI Team's Process

EndoKED works by having two AIs, a language-based one and a vision-based one, work together to get a complete picture. The process begins with a Large Language Model (LLM), like ChatGPT or Claude, which acts as a virtual medical archivist. It's tasked with reading a patient's free-text colonoscopy report and answering a simple question, "Was a polyp found?". This initial "yes or no" answer, or report-level label, is extracted with 100% accuracy. This report-level knowledge is then passed to a second AI model called EndoKED-MIL. This model's job is to look at all the images from that patient's colonoscopy and figure out which specific ones are most likely to show a polyp. It achieves high performance, with an image-level average precision (AP) of up to 0.901. The final step involves a powerful Large Vision Model (LVM), such as the Segment Anything Model (SAM). Using information from the previous steps, the LVM draws a precise outline around the polyp in the image, a task that would normally require a trained human expert. The model keeps refining this outline through an iterative process until it is highly accurate.

The Promising Results

The models trained using this "human-free" method showed impressive results. When it came to outlining polyps, the EndoKED-SEG model performed exceptionally well, achieving an average score of 0.827 across six different public datasets. This performance is on par with, or even better than, existing models that were trained with detailed human-made annotations.

The study also created an optical biopsy model, EndoKED-PATH, which learned to distinguish between harmless and potentially harmful polyps. This model performed at the same level as experienced senior doctors and was highly efficient, needing much less data to learn than other AI models. On a prospective test set, EndoKED-PATH achieved an excellent area under the curve (AUC) of 0.911, showcasing its diagnostic power.

Why This Matters

By using existing data to train AIs, it makes it possible to create highly accurate diagnostic tools without the need for expensive and labor-intensive manual annotation. This could help make advanced diagnostic tools more accessible in more places and could even be used to help discover new insights from existing medical records. It frames AI as a powerful partner in scientific discovery—a tool that can reveal truths hidden in plain sight.