Researchers investigated the reliability of using large language models (LLMs) to upgrade software by analyzing their ability to transition legacy PHP code to version 8.3. The study focused on an 'easy shortcut' scenario: taking a single file from WordPress 4.0—a project with significant legacy footprint—and requesting an LLM to "modernize this" while preserving its original behavior. To ensure an objective baseline, the researchers evaluated five different LLMs against two benchmarks: the untouched original files and results produced by Rector, a dedicated rule-based modernization tool.
The findings suggest that current LLMs frequently degrade working code rather than improving it. While 83 out of the 100 test files passed all behavioral checks in their original state, the best-performing LLM left only 59 files functional after attempted modernization. In contrast, each model broke between 32 and 51 previously working files, while repairing at most 10. When compared to the rule-based Rector tool, which only broke 7 files and repaired 6, the gap in reliability becomes stark. Most notably, for files that required no actual changes to be compatible with PHP 8.3, LLMs introduced breaking regressions in 36% to 60% of cases, whereas Rector caused such errors in only 8% of those instances.
A primary cause for these failures is the introduction of basic syntax errors that the PHP engine itself rejects, specifically regarding type declarations. Because the study utilized a single-file setting where the AI sees internal structure but lacks project context or external dependencies, the models struggled to make safe assumptions about how variables interact across the broader system. The researchers conclude that evaluations of LLM-based modernization must distinguish between successful 'modernization' (new features/syntax) and 'behavioral preservation.' Without execution feedback or an agentic workflow—where an AI can iterate on its own mistakes—single-prompt modernization remains prone to introducing unnecessary damage to stable legacy systems.