Become A Wiser, More Profitable Trader

🗞️   Re-Evaluating Code LLM Benchmarks Under Semantic Mutation

science

:2506.17369v1 Announce Type: cross Abstract: In the era of large language models (LLMs), code benchmarks have become an important research area in software engineering and are widely used by practitioners. These benchmarks evaluate the performance of LLMs on specific code-related tasks, such as code understanding and generation. A critical step in constructing code benchmarks is the design of prompts. However, as existing code benchmarks typically rely on a single prompt template per task, they are prone to the issue of prompt sensitivity, where minor prompt variations could result in substantial performance variations, leading to unreliable evaluations of model capabilities. While previous studies have explored prompt sensitivity, their experimental designs and findings are limited to ...

arXiv     Jun 24, 2025