How to teach an AI agent to test Windows apps through a real GUI
Exploring the Deskcheck architecture: connecting Claude Code and Delphi apps using OS-level input and a local protocol adapter.

In modern software development, artificial intelligence frequently writes code and builds projects, but verifying desktop application functionality remains a blind spot. Code compilation does not guarantee that the UI actually works as expected for a human user. To solve this issue, Deskcheck was developed to allow AI agents to test Windows applications through a genuine graphical user interface.
- Developer Igor Belykh created Deskcheck for testing Delphi applications using AI agents.
- The architecture separates "hands" (OS input driver) and "eyes" (DIP protocol for state reading).
- The system supports seamless transition from AI generation to deterministic test runs in virtual machines without token costs.
Deskcheck Architecture: Driver and Adapter
For an agent to fully interact with an application, it needs control mechanisms and feedback. The driver simulates real operating system input using functions like SendInput, clicking buttons and typing just like a human. This approach catches real layout bugs and overlapping windows, although it operates slower than direct event handler calls.
To feed interface state data back, a lightweight adapter is embedded into the application, running a local HTTP server over the DIP (Deskcheck Identity Protocol) specification. The core rule of this protocol is read-only isolation: the adapter can report information but cannot click, focus, or change application state. All control actions are performed strictly by the external driver.
Read alsoWindows 11 is secretly tracking coding agents and developer tools usageElement Cataloging and Interface Handling
A major challenge when integrating into legacy projects, such as a large Delphi application with hundreds of forms, was establishing proper control names. Because static form definition files do not account for runtime scaling and visual themes, the control catalog is dynamically built from the live window instance.
Control bounding rectangles serve as a fallback reference for the driver when precise name-based lookup is temporarily unavailable. This ensures high test stability regardless of screen resolution, DPI scaling, or operating system display settings across different environments.
Scenarios, Assertions, and Running Without AI
Recorded test scenarios are saved as standard text files containing steps, comments, and handling rules for potential pop-up interruptions. The assertion suite covers not only visual elements but deep business logic, including DevExpress grid states, direct SQL queries against test databases, and validation of exported files.
Once a scenario is successfully recorded and verified with AI, subsequent regression runs are executed by a regular test runner without requiring language models. This saves token expenses and guarantees reproducible results, while built-in visual screenshot comparison mechanisms capture any subtle interface regressions.
Source: Все публикации подряд на Хабре



