demonstrational

An action inferring facility for a multimodal interface called Edward is described. Based on the actions the user performs, Edward anticipates future actions and offers to perform them automatically. The system uses inductive inference to anticipate actions. It generalizes over arguments and results, and detects patterns on the basis of a small sequence of user actions, e.g. “copy a lisp file; change extension of original file into .org; put the copy in the backup folder”. Multimodality (particularly the combination of natural language and simulated pointing gestures) and the reuse of patterns are important new features. Some possibilities and problems of action inferring interfaces in general are addressed. Action inferring interfaces are particularly useful for professional users of general-purpose applications. Such users are unable to program repetitive patterns because either the applications do not provide the facilities or the users lack the capabilities.

Conventional interface builders allow the user interface designer to select widgets such as menus, buttons and scroll bars, and lay them out using a mouse. Although these are conceptually simple to use, in practice there are a number of problems. First, a typical widget will have dozens of properties which the designer might change. Insuring that these properties are consistent across multiple widgets in a dialog box and multiple dialog boxes in an application can be very difficult. Second, if the designer wants to change the properties, each widget must be edited individually. Third, getting the widgets laid out appropriately in a dialog box can be tedious. Grids and alignment commands are not sufficient. This paper describes Graphical Tabs and Graphical Styles in the Gild interface builder which solve all of these problems. A “graphical tab” is an absolute position in a window. A “graphical style” incorporates both property and layout information, and can be defined by example, named, applied to other widgets, edited, saved to a file, and read from a file. If a graphical style is edited, then all widgets defined using that style are modified. In addition, because appropriate styles are inferred, they do not have to be explicitly applied.

Many tasks performed using computer interfaces are very repetitive. While programmers can write macros or procedures to automate these repetitive tasks, this requires special skills. Demonstrational systems make macro building accessible to all users, but most provide either no visual representation of the macro or only a textual representation. We have developed a history-based visual representation of commands in a graphical user interface. This representation supports the definition of macros by example in several novel ways. At any time, a user can open a history window, review the commands executed in a session, select operations to encapsulate into a macro, and choose objects and their attributes as arguments. The system has facilities to generalize the macro automatically, save it for future use, and edit it.