Autopilot: On-Device AI Desktop Agent for Hands-Free and Accessible Windows Automation
- Gonzalo Torras Serrano
- Supervisor: Dr. Moisés Martínez Muñoz
- Universidad Francisco de Vitoria · Computer Science · Research Track
- Graded 10/10 · 2026
Abstract
Autopilot takes a single voice command and carries out the requested Windows task end to end on the local device, so that people with limited motor function can operate a computer by voice alone: no further intervention, no internet connection and no data leaving the machine. It combines deterministic and non-deterministic AI. A local speech model transcribes the command; a language model extracts the intent and its parameters; a deterministic planner (PDDL, Fast Downward) produces an explainable plan; specialised vision-language agents execute it against the real graphical interface through visual grounding, mouse and keyboard; and a verification agent confirms completion without human supervision.
Results
Evaluated on seven task templates across 21 runs and compared with five alternative desktop-automation approaches, Autopilot reached a 100% task success rate with zero planning failures and zero data exposure, outperforming every alternative in autonomy, execution accuracy and privacy.
- 100%
- task success
- 21
- runs, 7 task templates
- 0
- planning failures
- 0
- bytes leaving the device
It beat Claude, Gemini, OpenClaw and Windows Voice Access.
Same seven task templates, three runs each, no human help. Autopilot was the only system that completed every task autonomously while keeping every byte on the machine.
- Claude CoworkClaude Sonnet 4.6
- Gemini Computer UseGemini 3.5 Flash, model-requested screenshots
- Gemini, screenshot-assistedGemini 3.5 Flash, screenshots supplied after each action
- OpenClawOpen-source local computer-use agent
- Windows Voice AccessBuilt-in Windows 11 voice control
Keywords
- Artificial intelligence
- Speech-to-text
- Vision-language models
- Automated planning
- Desktop automation
- Accessibility
- On-device AI
The full report and the code are not public yet: the work is being turned into a paper. If you would like the details, write to me.
Ask about Autopilot




