A 2015 CNN Beats GPT-6 and Claude on 200 Skin Lesions
A fine-tuned Inception v3 CNN correctly diagnosed 62% of 200 biopsy-proven skin lesions, against 42% for two frontier LLMs, though the test is small and each LLM ran once.
A consultant dermatologist tested two frontier LLMs and a fine-tuned Inception v3 CNN on 200 biopsy-proven skin lesions from MILK10k. The CNN identified the correct diagnosis among six options for 125 lesions (62%), while GPT-6 Astra and Claude Fable 5.1 managed 84 and 85 (42%).
The CNN was pretrained on everyday photographs, fine-tuned on 68,011 dermoscopic and clinical images from the ISIC Archive on one H100 GPU, with test lesions and same-patient images excluded. With a second choice allowed, the CNN reached 173/200 against 134 and 138, and it named 13 of 30 melanomas against 8 and 11.
Caveats include small numbers, single LLM runs, and a deliberately non-state-of-the-art CNN; MILK10k has been public since 2025, so the LLMs may have seen it in training.
Key takeaways
- The test set drew 200 biopsy-proven lesions from MILK10k: 100 unsafe (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, 20 actinic keratoses) and 100 safe (50 naevi, 50 seborrhoeic keratoses). Because every lesion was biopsied, the benign cases had already worried a clinician.
- The CNN was Inception v3 pretrained on everyday photographs, then fine-tuned on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. Test lesions and every other image of the same lesion or patient were removed from training and validation.
- The CNN scored both images per lesion, weighting dermoscopy most, and chose among six diagnoses. It reached 125/200 (62%) correct first choices, 173/200 with a second choice, and 13/30 melanomas, against 84/200, 134, and 8/30 for GPT-6 Astra and 85/200, 138, and 11/30 for Claude Fable 5.1.
- Error patterns diverged: GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of 30 melanomas; Claude Fable 5.1 called 53 actinic keratosis when 20 were. The CNN's 17 melanoma misses split into 8 naevi and 6 seborrhoeic keratoses.
- Caveats include small numbers, single runs per LLM, and a deliberately non-state-of-the-art CNN. MILK10k has been public since 2025, so the LLMs may have seen it in training; the CNN had not. When asked as a patient, both LLMs advised seeing a doctor in almost every case, including harmless lesions.
What people think
Analysis details
17 comments analyzed. Analyzed .
The sample selects active discussion branches and includes available parent comments. It may omit parts of the full thread. The analysis included 17 of 18 usable stored comments because of the input limit. Selected themes do not measure community opinion or how common a view is.
Dedicated classifiers versus general LLMs
For high-volume tasks, a purpose-built classifier can outperform a general LLM, and LLMs can lower the cost of producing one. The counter is that training a good classifier remains hard and requires knowing the right terms, while an API-based decision model is one call. Jev's launch is credited with making decision models seem like a new breakthrough, exposing a monoculture where LLMs are the only AI tool many developers consider.
Benchmark scope and generalization doubts
The 200-lesion test is challenged as benchmaxxing: a model fine-tuned on a train/test split can win on that dataset yet remain brittle on out-of-domain images. The comparison is also called not useful because it pits a fine-tuned CNN against general-purpose models on a tiny dataset. Reported caveats acknowledge small numbers and single LLM runs.
Tool augmentation changes the comparison
A calculator analogy frames the result as unsurprising: specialized tools beat LLMs at arithmetic. Giving an LLM a calculator MCP can erase that advantage, so the relevant question is whether the LLM is used bare or with tools. The exchange also treats the result as a reality check against treating LLMs as universal boxes.
LLMs predict tokens, not compute
One argument holds that LLMs are designed to emit plausible tokens rather than perform mathematical analysis, citing teams that feed spreadsheets to ChatGPT for reports instead of using regression. A reply notes that frontier models can write Python to run regression and report results, making the outcome depend on the prompt and harness rather than an inherent inability.