Evaluating LLM-powered coding assistants for refactoring test assets to the Page Object Model
Volkov, Roman (2025)
2025
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-2025121637084
https://urn.fi/URN:NBN:fi:amk-2025121637084
Tiivistelmä
The thesis investigated the capability of LLM-powered assistants to refactor legacy Robot Framework test suites to the Page Object Model under controlled conditions, addressing the challenge of maintaining monolithic suites through automation with minimal supervision.
Claude Code, Codex CLI, and GitHub Copilot were evaluated on the same repository with a standardized prompt and identical acceptance criteria. Each run was observed until the output met the Definition of Done (DoD), requiring selector encapsulation, functional stability, static quality, and separation of concerns. Metrics included execution time, cost, and review scores. The best-performing assistant was rerun with Model Context Protocol tooling (RobotMCP) to assess added value.
Results showed that Claude Code achieved a DoD-compliant refactor with high autonomy and modest cleanup, completing the task about three times faster than Codex CLI. Codex produced functional but less disciplined structures requiring normalization, while Copilot failed to complete the migration without continuous intervention. The RobotMCP configuration increased latency and context window pressure without quality gains. Findings show that LLMs can automate complex restructuring tasks effectively when scope and acceptance criteria are clearly defined. Claude Code offers the best trade-off between cost and productivity. LLM-assisted refactoring is feasible for medium-sized, well-scoped tasks, whereas MCP integration is not beneficial for existing suites. Future research should examine generative test creation, where MCP’s discovery features may offer greater benefit.
Claude Code, Codex CLI, and GitHub Copilot were evaluated on the same repository with a standardized prompt and identical acceptance criteria. Each run was observed until the output met the Definition of Done (DoD), requiring selector encapsulation, functional stability, static quality, and separation of concerns. Metrics included execution time, cost, and review scores. The best-performing assistant was rerun with Model Context Protocol tooling (RobotMCP) to assess added value.
Results showed that Claude Code achieved a DoD-compliant refactor with high autonomy and modest cleanup, completing the task about three times faster than Codex CLI. Codex produced functional but less disciplined structures requiring normalization, while Copilot failed to complete the migration without continuous intervention. The RobotMCP configuration increased latency and context window pressure without quality gains. Findings show that LLMs can automate complex restructuring tasks effectively when scope and acceptance criteria are clearly defined. Claude Code offers the best trade-off between cost and productivity. LLM-assisted refactoring is feasible for medium-sized, well-scoped tasks, whereas MCP integration is not beneficial for existing suites. Future research should examine generative test creation, where MCP’s discovery features may offer greater benefit.