Microsoft might add the UFO, a highly customizable AI assistant, to its next Windows

The AI tool can complete tasks without human input.

3 min. read

Published onFebruary 28, 2024

published onFebruary 28, 2024

Share this article

Read our disclosure page to find out how can you help Windows Report sustain the editorial teamRead more

Microsoft recently released UFO, a highly customizable AI assistant, capable of fulfilling users’ requests tailored to different operating systems, including Windows.

The AI assistant is based on and uses the capabilities of GPT-vision to visualize and understand various visual elements, including graphical user interface (GUI) and control information of Windows applications, and it can provide Windows users with additional assistance without the need to have direct audio input.

In other words, UFO is a special kind of computer program that helps users interact with other programs on their Windows computers. It uses a clever system to understand what’s happening on their screen and can perform tasks for them, like clicking buttons or typing text. UFO can do all this automatically, without needing any human input.

The AI tool was developed by a team of researchers working for Microsoft Research, and the paper can be read in its entiretyhere.

The abstract reads:

We introduceUFO, an innovative UI-focused agent to fulfill user requests tailored to applications on WindowsOS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users’ daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment.

Not only Microsoft UFO doesn’t need human input to work, but it can also be customized to each user, meaning the assistant can be highly personalized to fit the needs of each Windows user, and it can be automatized to run certain tasks without having to explicitly let it know.

This goes hand in hand with the idea ofWindows coming alive, something Microsoft might be interested in exploring inthe next version of Windows, which is reportedly AI-based.

Microsoft has also made the open-source code for UFO available on GitHub, and you can find ithere.

What do you think? Would you like to have UFO as your Windows assistant? Let us know in the comments section below.

With KB5043178 to Release Preview Channel, Microsoft advises Windows 11 users to plug in when the battery is low#

Copilot in Outlook will generate personalized themes for you to customize the app#

Microsoft will raise the price of its 365 Suite to include AI capabilities#

Death Stranding Director’s Cut is now Xbox X|S at a huge discount#

Outlook will let users create custom account icons so they can tell their accounts apart easier#

Microsoft might add the UFO, a highly customizable AI assistant, to its next Windows#

With KB5043178 to Release Preview Channel, Microsoft advises Windows 11 users to plug in when the battery is low

Copilot in Outlook will generate personalized themes for you to customize the app

Microsoft will raise the price of its 365 Suite to include AI capabilities

Death Stranding Director’s Cut is now Xbox X|S at a huge discount

Outlook will let users create custom account icons so they can tell their accounts apart easier

Microsoft might add the UFO, a highly customizable AI assistant, to its next Windows