FEM-Bench：コード生成LLMのための構造化科学的推論ベンチマーク
FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
有限要素法（FEM）コード生成を通じてLLMの物理的推論能力を評価するベンチマークFEM-Benchを提案し、最先端モデルでも完全解決が難しいことを示した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Saeed Mohammadzadeh, Erfan Hamdi, Joel Shor, Emma Lejeune
分類: cs.LG, cs.AI, cs.SE
原文アブストラクト
As LLMs advance their reasoning capabilities about the physical world, the absence of rigorous benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, which develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific reasoning evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline requires constructing explicit models of physical systems and reasoning about geometry, spatial relationships, and material behavior, connecting directly to emerging AI goals in physical reasoning and world modeling. We introduce FEM-Bench, a computational mechanics benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least once and 26/33 tasks all five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed broad performance variation. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve.
関連論文
- NEWTON: 大規模言語モデルは物理的推論ができるのか？物理推論ベンチマーク
- Physion++：物体の潜在的な物理特性のオンライン推論を必要とする物理的シーン理解の評価物理推論ベンチマーク
