Ui.Vision

New Dec 5, 2024 Update: Anthropic Claude Computer Use Integration

Open-Source Ui.Vision has consistently been at the forefront of visual web automation. With Claude’s integration, we’re taking the next step forward. The aiComputerUse command allows you to automate complex tasks with a single line of code that would traditionally require hundreds of lines of classic Ui.Vision commands (such as XClick, OCRExtractScreenshot, If/then statements, and more). For example, you can teach Ui.Vision to play TicTacToe with just one short "Play this game..." prompt.

-- Ui.Vision is an open-source automation RPA software that combines classic browser automation with modern computer vision and OCR:

(1) Visual Browser Automation

Ui.Vision's visual UI testing commands assist web designers and developers in checking and ensuring the accuracy of website layouts and canvas elements. It can identify and read images and text within canvas elements, images, and videos.

(2) Visual Desktop Automation for Windows, Mac, and Linux

Beyond web browser automation, Ui.Vision uses image and text recognition (OCR) to automate browser extensions and desktop environments as well. It can interpret images and text on the desktop, executing actions like clicking, moving, dragging and dropping the mouse, and simulating keyboard inputs.