Personal Wizards: Learning Procedural Knowledge by Demonstration

Innovation Matters

Providing a software framework for capturing procedural knowledge from demonstrations, by recording the steps users take when performing procedures, combining multiple recordings and producing executable procedures that can be re-executed or disseminated.

There are currently three vehicles for disseminating procedural knowledge:

Programs include traditional applications, scripts and macros. They are typically easy to use and can guide users through complex tasks, but suffer from high development costs, ongoing maintenance expenses and brittleness.
Documentation can be disseminated through hardcopy and in electronic form, and provides a complete view of the procedure. It can be inspected and understood by people. However, documentation has high development costs and can quickly become obsolete.
Word-of-mouth

The Main Functionalities of the Personal Wizards Architecture

The goal of the Personal Wizards project is to develop a new approach for capturing procedural knowledge that has most of the advantages of both programs and documentation and overcomes many of their disadvantages.

The core technology of Personal Wizards is programming by demonstration and machine learning. To automatically construct executable procedures, Personal Wizards observes one or more users perform the procedure, records the actions and the system reactions, and collects information that describes the content of the screen and the configuration of the computer system. It then combines the recordings of multiple executions of the same procedure into an executable procedure model by segmenting each recording into a sequence of actions that can be easily understood, aligning these sequences (i.e., identifying equivalent steps), and generalizing the aligned steps (i.e., explaining the differences in terms of state of the computer system and of the applications with which the user interacts). The executable procedure can be dynamically constructed and updated by adding new recordings.

The result of this learning process is procedure that can be executed, inspected and distributed. The Personal Wizards execution engine loads an executable procedure and takes a snapshots of the contents of the screen. Based on this information, it decides which steps to take. After taking each step, the execution engine observes the result of the action before deciding which action to take next. The user is allowed to take over at any point, in which case the execution engine observes the user actions, records them, and tries to match them to the existing procedure. While the matching is successful, the user can allow the execution engine to continue at any point. When the execution engine cannot decide what action to take, for example when it encounters a situation that cannot be generalized from the recorded demonstrations, it can request the user to take over the execution.

The Personal Wizards approach can reduce the cost of acquiring procedural knowledge by minimizing the need for actual programming or for writing documentation. Since the executable procedure model can be dynamically updated by adding new recordings, the approach also reduces the cost of maintaining the model and the risk of obsolescence. During execution, the system makes decisions based on the actual content of the screen, and can ask the user to take over, thereby reducing the brittleness that characterizes macros and scripts. Finally, the execution framework allows for a seamless mixed-initiative mode, where the user can both take over and relinquish the control of the execution at any given point, which sets this approach apart from traditional programs, scripts and macro recorders.

Related Publications

Lawrence Bergman, Tessa A. Lau, Vittorio Castelli and Daniel A. Oblinger. Programming-by-Demonstration for Behavior-Based User Interface Customization. Workshop on Behavior-based User Interface Customization at IUI 2004 - Intelligent User Interfaces. January 2004.

Lawrence Bergman, Tessa Lau, Vittorio Castelli and Daniel Oblinger. Personal Wizards: collaborative end-user programming. Workshop on End User Development, CHI 2003, Fort Lauderdale, FL. April 2003.

Tessa Lau, Daniel A. Oblinger, Lawrence Bergman, Vittorio Castelli and Corin Anderson. Learning Procedures for Autonomic Computing. Workshop on AI and Autonomic Computing (IJCAI 2003 - Eighteenth International Joint Conference on Artificial Intelligence). IBM, April 2003.

Tessa A. Lau, Lawrence Bergman, Vittorio Castelli and Daniel A. Oblinger. Sheepdog: Learning Procedures for Technical Support. IUI 2004 - Intelligent User Interfaces. ACM, October 2003.

Rate this article

Innovator's corner

Larry Bergman Researcher

What is the most exciting potential future use for the work you're doing?
The potential for sharing know-how about how to use computer systems. Imagine "procedural wikis" where anyone can contribute knowledge about how to do things, not just as text, but as executable procedures. Documentation becomes active and can perform tasks automatically, or that can teach the user by guiding them through the procedure. I think that such collaboratively acquired and distributed executable knowledge has the potential for revolutionizing the end-user experience.

What is the most interesting part of your research?
I'm intrigued by the user-interface challenges. One of the big difficulties in making a system like Personal Wizards truly usable is to build trust in the system. The user must be able to examine what the procedure is doing and why, but without getting swamped in irrelevant details. Providing the right information at the right time is a fascinating task.

What inspired you to go into this field?
The desire to make computers easier to use. I want computers to be as easy to use as a hammer, as easy to drive as an automobile, but am dubious about attempts to turn computers into intelligent assistants by building smart systems. Personal Wizards seeks to ease computer use by using the intelligence and know-how of the community of existing computer users.

What is your favorite invention of all time?
The screw - deceptively simple, yet not at all obvious. Try to imagine modern civilization without it!