ATLAS researchers have developed a software framework that enables developers who lack experience programming robots to repurpose off-the-shelf toy and educational devices to serve as input-output devices for desktop applications.
Dan Szafir, assistant professor of computer science in the ATLAS Institute, Shaun Kane, assistant professor of computer science and ATLAS faculty fellow, and doctoral student Darren Guinness, presented the GUI (graphical user interfaces) Robots framework at the Designing Interactive Systems (DIS 2017) conference in Edinburgh this summer.
The GUI Robots toolkit enables developers to transform low cost robots (often less than $100 USD) into wireless controllers for desktop applications. The toolkit supports connecting such devices with a wide range of software applications, including web browsers, 3D modeling tools and video games, extending the application user interfaces into the physical world in interesting new ways. 聽聽
With most software applications confined to screen-based graphical user interfaces, tactile input and haptic feedback are an important frontier for interaction design.
Szafir says, 鈥渄evelopers can use our framework to quickly prototype tangible interactions and attach them to existing applications. As consumer-oriented, wirelessly-connected robots become ubiquitous, our work can enable new user experiences in which an ecosystem of helpful robots extends traditional graphical user interface applications.鈥澛犅� 聽聽聽 聽聽聽 聽
Similar to a Nintendo Wii console where the controller detects movements in three dimensions, providing realistic on-screen action games, users can repurpose devices to apply the same type of movement in any existing application. For instance, as part of the project a Sphero Ollie was configured as a controller for Rovio Entertainment鈥檚 popular Angry Birds game: rolling the robot back and then moving it forward launched a bird. 聽
Robots can also be programmed to provide physical feedback, such as vibrating when a bird is launched. Or, a user could manipulate a 3D object on the screen by moving and rotating the robot controller in the air.
To test the GUI Robots toolkit, researchers asked twelve developers to build controllers for two applications: Angry Birds and Windows Movie Maker. 聽All of the developers were able to build working prototypes of the Angry Birds controller within a half hour; the Movie Maker controller took them a little longer.
As an extension to their work, the team is exploring the use of GUI Robots to provide haptic displays for visually impaired users. Movement or tactile feedback from a robot would be deployed to communicate the kind of information that is typically displayed in charts, diagrams, movies and even interactive simulations. Other promising uses include educational software, controllers for musical instruments and 3D modeling.
