Demonstrations are Not Always Self Evident
by Roy Latham

A demonstration is living proof of something, but it is not always clear of exactly what.

First, let us all agree that demonstrations are powerful methods for conveying information. They are powerful because they involve us; they teach us by example. We can see and hear just what is happening, we become involved, and we learn quickly. One might be presented with an elegant description of the theory of how a boomerang is able to return to its thrower. This description might be accompanied by sincere claims that indeed the boomerang works according to theory. Neither the theory nor the sincere claims is likely to have the impact of seeing a single successful toss.

Now, if you or I see a successful toss of a boomerang, neither of us is likely to conclude that we could immediately be so successful in making it return, at least not on the first try. A demonstration has two essential parts: a thing being demonstrated and a demonstrator. A good demonstration, in fact, is usually one in which the thing being demonstrated is shown to its full capacity. The demonstration should be limited by the device, not the demonstrator.

It is easy to figure out the importance of the demonstrator in the context of a demonstration of a boomerang or a violin, where the skill of the demonstrator is apparent and the skill practiced is probably not closely related to any of our skills. Of course, a good demonstrator of less exotic products may perform with such ease that we start to blur this understanding. We start to think that anything that looks so easy perhaps cannot in fact be so difficult.

For high-tech products, the lure of apparent ease can be seductive. At SIGGRAPH there is always a demonstration of some great new three-dimensional program with which in a five minute blaze of mouse clicks someone constructs a lovely shaded model of St. Paul's Cathedral. In fact, learning to throw a boomerang will certainly be easier to master than the software in question, and possibly learning the violin might be easier as well.

I recollect visiting a simulator company that produced genuinely first class real time databases. They were quite proud of the custom database tools they had developed, and they spent some time showing me the tools and how they worked. The tools turned out to be quite peculiar, and it was soon apparent to me that the fine work they did was much in spite of their tools, not because of their tools. Their demonstrated databases showed that it was within the capacity of the tools to produce good results, but the results did not prove that the tools were easy to use. In fact, when they demonstrated the tools they did operate them with great facility and often repeated the sincere belief that they were great tools. It took a fairly knowledgeable comparison to other tools to sort out the merits of the tools and the operators.

If the tools had been just a little bit better, but still quite bad, I would not have detected it from the facile demonstration. The correct way to judge ease of use is not from demonstrations, but by inquiry of in-depth users. For popular computer software and the like, magazine reviews are valuable information sources. Unfortunately, many products in the simulation business are too specialized to make it economical to review them. (Real Time Graphics would very much like to do in-depth product reviews, but our small subscriber base cannot bear the costs involved.) One alternative is to find and talk to individuals who have used the products, and even that avenue has limitations. Not many people have used a variety of similar products to serve as a basis of comparison; a product which is not very easy to use may nonetheless be the best available in its product category. Another choice, by the way, is to hire a consulting firm to do an in-depth comparison. (We do such studies in our alter-ego capacity as consultants, but sponsoring a study is not a viable option unless you are going to buy a lot of something.)

Have you ever seen a demonstration of a household vacuum cleaner picking up a bowling ball? Presumably no one who sees such a demonstration wants to pick up a bowling ball with a vacuum cleaner, but some who see the demonstration will make the connection that any vacuum cleaner powerful enough to pick up a bowling ball must be powerful enough to do a good job of household cleaning. The implied connection is false; unlike when cleaning, there is very little airflow when the ball is in its adapter and, static pressure being what it is, any vacuum cleaner could pick up a luxury car using a suitable adapter.

I call the strong-enough-to-pick-up-a-bowling-ball type of demonstration a "fallacy of implied capability" demo. In an image generator demonstration, one may see many moving objects rendered with correct occlusion, and then, perhaps incorrectly, conclude that the image generator will always provide correct occlusion. The image generator may well always render moving objects correctly, but not necessarily. If the image generator uses a list priority algorithm for occlusion, the algorithm may work perfectly for objects that are easily separated from terrain, but not well if fixed objects are mixed with the moving ones. If the objects are on predefined paths in the demonstration, the algorithm may be adjusted for the demo case, but not work well in general. In a hybrid architecture machine, the moving objects may be inserted using Z-buffering, so the occlusion will be correct. However, if the Z-buffering capacity is minimal then there may be a problem if some of the moving objects are too close. When too close, the objects are too large on the screen and the pixel capacity of the Z-buffering is exceeded.

Perhaps the ultimate in the fallacy of implied capability is the notion that if an image generator makes beautiful pictures for a demonstration, then it will make beautiful pictures in its ultimate application. The demonstration is likely to use a database crafted with more skill and proportionately more labor than the one in the final application. The operative principle is that if you are selling ceiling paint, get Michelangelo to paint a demo ceiling. In addition to being designed for maximum appeal, a demonstration can be scripted to avoid the limitations of the system. Creative database design can be implemented and painstakingly hand-tuned for a small demonstration database, whereas the small techniques would be impractical to apply to a large database.

It would be unrealistic to expect any product producer to stage a demonstration in which a system was not shown to its best advantage. Such a vigorous pursuit of honesty would be unlikely to inspire competitors to do likewise. Besides, customers fully expect to see a system demonstrated at its best, and will to some extent discount whatever they are shown. If one showed less than the best, it would be discounted even further. In terms of avoiding known system limitations, vendors always have good reasons for supposing that whatever the limitation, they have a good solution that will prevent the limitation from coming into play in the final application. There are a few cases of outright trickery, wherein a new compact image generator sits proudly atop a desk while cables run out of sight into a back room having an older fire-breathing monster of a graphics engine that actually does the processing. Such fraud is lamely justified in the mind of the zealous presenters by the notion that the new model will do all the capability shown, but it just isn't quite ready yet.

Good demonstration watchers must focus on sorting out exactly what they see from what it would be obvious to suppose they saw.

Notices Received

SMART WEAPONS COURSE General Notice: "The Guidance and Control Information Analysis Center (GACIAC) will conduct a three-day Smart Weapons Training Course at STRICOM in Orlando, FL on 13 through 15 August 96." This makes one wonder exactly how smart a weapon must be to benefit from attending this course. For example, could a weapon without a high school diploma really get anything out of it? And what will a smart weapon learn? Will, for example, the training include teaching the weapon how to ask directions from a receptionist and then use an elevator?

The U.S. Government sponsors Small Business Innovation Research (SBIR) projects to answer the pressing needs of government. The Department of Defense issues a book of topics to identify areas of interest to the military. There are always a few topics in the book that are especially interesting for one reason or another.

Among the recent Army topics is "Extruded, Shelf-Stable, Intermediate-Moisture Meat Jerky Analogs," with the following problem description: "Extruded meat-in-carbohydrate-matrix, intermediate-moisture products are under development as eat-out-of-hand ration components. These products are designed to have flavor and texture comparable to commercially available meat-jerky items, but also to provide carbohydrates as an energy source. Current development has yielded ration prototypes containing approximately a 1:3 meat:starch ratio with a moisture content of approximately 20%. These products are highly acceptable shortly after extrusion, but toughen significantly after short periods of storage (i.e., a 10-fold increase in elastic modulus after 4 weeks accelerated [125¡ F] storage)." They suggest the bidder "conduct thermal-analysis/nuclear magnetic resonance-evaluation to assess changes in component state . . ."

More high tech, the Advanced Research Projects Agency is seeking technology for the development of a remotely piloted vehicle, not exceeding 15 cm (about six inches) in length "to perform military missions that cannot be accomplished by other means." The exact nature of those missions being left to the reader's imagination. The technology sought includes everything from aerodynamic analysis to propulsion and navigation, and, of course, a systems integrator to put it all together. Pondering this problem, an initial thought was to develop a backpack for a sparrow, then the sparrow could eat when hungry, thereby providing extended range refueling. A second thought was that maybe the central problems were too difficult, and we ought to instead focus on the baggage handling system.