A Comparative Study in Surgical AI:
Potential and Limitations of Data, Compute, and Scaling

Kirill Skobelev1, Eric Fithian1, Yegor Baranovski1, Jack Cook2, Sandeep Angara2, Shauna Otto2, Zhuang-Fang Yi2, John Zhu2, Neeraj Mainkar2, Margaux Masson-Forsythe2, Daniel A. Donoho2,3, X.Y. Han1

NeurIPS 2026 The Third Workshop on GenAI for Health: Agentic Systems, Clinical Trust, and Future Potential at The Fortieth Annual Conference on Neural Information Processing Systems

Abstract

Background. Recent Artificial Intelligence (AI) models match or exceed human experts on many biomedical benchmarks, yet surgery—which requires visual recognition beyond text question-answering—is largely absent from prominent medical benchmark suites. Whether scaling general-purpose models suffices to deliver capable surgical AI, or whether specialized data is the binding constraint, remains unknown.

Methods. We studied surgical tool detection on SDSC-EEA, a dataset of 67,634 annotated frames from 66 endoscopic endonasal neurosurgical procedures (20,016 held-out validation frames, split by procedure). We evaluated 26 models—20 open-weight Vision Language Models (VLMs; 2B–235B parameters) and 6 frontier proprietary models—under zero-shot prompting and LoRA fine-tuning, swept LoRA rank from 2 to 1024, and compared a 26-million-parameter specialized object detector (YOLOv12-m). The evaluation protocols were replicated on three public datasets (CholecT50, PitVis-2023, SurgVU). Performance is reported as exact-match accuracy and micro-averaged F1.

Results. Among 20 open-weight VLMs evaluated zero-shot on 20,016 held-out frames, none surpassed a majority-class baseline of 47.3% micro-averaged F1 (range, 1.0% to 44.9%); the highest general-benchmark scorer reached only 39.2%. The 26-million-parameter specialized detector (YOLOv12-m) achieved 54.7% exact-match accuracy (95% CI, 54.0 to 55.4) and 80.5% micro-F1—exceeding every zero-shot VLM by more than 35 percentage points in micro-F1—and matched a fine-tuned 27B-parameter VLM (51.1% exact match [95% CI, 50.4 to 51.8]; 81.4% micro-F1) using approximately 1,000-fold fewer parameters. Increasing LoRA adapter capacity by three orders of magnitude raised training micro-F1 from 66.3% to 99.5% but validation micro-F1 only to 69.9%, a persistent train–validation gap of roughly 30 percentage points. Across the three public datasets, the specialist and the fine-tuned VLM led every dataset (74.9% to 92.8% micro-F1), whereas all six frontier proprietary VLMs trailed both.

Conclusions. Current general-purpose models face substantial obstacles in surgical perception that scaling alone does not overcome; the dominant bottleneck appears to be specialized data rather than model size or compute.

Main results

Figure 1. Validation frames from the SDSC-EEA endoscopic endonasal neurosurgery dataset with zero-shot Gemma 3 27B-it predictions. (a) Five frames in which the predicted tool set matches the ground-truth set. (b) Five frames in which the predicted set differs from the ground truth. SDSC-EEA comprises 67,634 annotated frames from 66 endoscopic endonasal approach neurosurgical procedures. Source: Authors' analysis of the SDSC-EEA dataset (Surgical Data Science Collective).
Figure 2. Zero-shot surgical tool detection on SDSC-EEA does not improve with parameter count or general multimodal benchmark score. Micro-averaged F1 over the 15-tool SDSC-EEA evaluation subset (n = 20,016 frames) for 20 open-weight vision–language models released between September 2023 and April 2026. (a) Micro F1 as a function of model parameter count (log scale). (b) Micro F1 as a function of MMBench general-multimodal score (range 0–100). The dashed line indicates a majority baseline that predicts the single most frequent tool (Suction) on every frame (47.3% micro F1). Filled stars and crosses respectively show fine-tuned Gemma 3 27B with a LoRA classification head (81.4%) and YOLOv12-m (80.5%), both trained on SDSC-EEA. No zero-shot model reaches the baseline; the strongest is Gemma 3 27B-it at 44.9%, and the highest MMBench scorer (Gemma 4 31B, MMBench 90.9) reaches just 39.2% micro F1. Source: Authors' analysis of the SDSC-EEA dataset (Surgical Data Science Collective).
Figure 3. Scaling LoRA adapter capacity by three orders of magnitude does not close the procedure-level train–validation gap on SDSC-EEA or SurgVU. Train and validation micro-averaged F1 (over the 15 evaluable tools for SDSC-EEA and the full ontology otherwise) of Gemma 3 27B with a LoRA classification head as a function of LoRA rank r ∈ {2, 4, 8, 16, 32, 64, 128, 256, 512, 1024} on (a) SDSC-EEA, (b) CholecT50, (c) PitVis-2023, and (d) SurgVU. Dashed lines indicate per-dataset majority-tool micro F1 baselines. Values are point estimates. Source: Authors' analysis of the SDSC-EEA, CholecT50, PitVis-2023, and SurgVU datasets.
Figure 4. Cross-dataset replication: a 26M-parameter object detector and a fine-tuned 27B-parameter VLM are tied at the top of every dataset. Validation micro-averaged F1 (over the 15 evaluable tools for SDSC-EEA and the full ontology otherwise) on SDSC-EEA (n = 20,016), CholecT50 (n = 19,923), PitVis-2023 (n = 30,896), and SurgVU (n = 18,919). YOLOv12-m and Gemma 3 27B with a LoRA classification head are essentially tied at the top of every dataset (74.9–92.8% F1). The six frontier closed-weight VLMs (Gemini 3.1 Pro Preview, Gemini 3 Flash Preview, Claude Opus 4.6, Claude Sonnet 4.6, Claude Fable 5, GPT-5.4) trail both leaders by a wide margin (roughly 10–45 F1 points). Closed-weight models were not evaluated on SDSC-EEA because the data are private (entries marked “n/a”). Claude Fable 5 was evaluated on a random seed-42 subsample of each validation set (~1,000 frames) rather than the full validation set used for the other models. Dashed lines indicate per-dataset majority-tool micro F1 baselines. Values are point estimates. Source: Authors' analysis of the SDSC-EEA, CholecT50, PitVis-2023, and SurgVU datasets.