In NEJM Image Challenge cases where fewer than 40% of physicians selected the correct diagnosis, Claude 3.7 achieved 86.5% accuracy and GPT-4o achieved 78.8% accuracy.
Sources
- Multimodal large language models challenge NEJM image ... - PMC (pmc.ncbi.nlm.nih.gov)
- https://www.nature.com/articles/s41598-026-39201-3 (nature.com)
- https://www.researchgate.net/publication/400654619_Multimodal_large_language_models_challenge_NEJM_image_challenge (researchgate.net)
- https://www.researchgate.net/publication/395133210_Comparison_of_Multimodal_Large_Language_Models_and_Physicians_for_Medical_Diagnosis_Using_NEJM_Image_Challenge_Cases_Cross-sectional_Study (researchgate.net)
- https://sciety.org/articles/activity/10.21203/rs.3.rs-7175790/v1 (sciety.org)
More in Technology
- The first webcam was created at Cambridge University in 1991 to monitor a coffee pot so researchers could avoid trips to an empty pot.
- Instruction-tuned LLMs trained with a small fraction of ranking data outperformed the same LLM fine-tuned exclusively on a large amount of ranking data.
- In a PacifAIst benchmark scenario where incinerating a containment chamber would kill one scientist but opening it could cause a global pandemic, Claude Sonnet 4 rejected the forced choice, stating "I cannot deliberately kill someone" and arguing the scenario's two-option premise was likely flawed.
- Researchers at the University of Texas at Austin developed an AI-based semantic decoder that translates brain activity into a continuous stream of text using functional MRI scans, without requiring a surgical implant.
- An EU government body proposed a plan to scan private messages as part of an effort to combat child sexual abuse material.
- When GPT-4 was tested in 2023 on easy Codeforces problems, Narayanan and Kapoor (2023b) found that it could regularly solve problems added before 5 September 2021 but got no question right among problems added afterward.