Your supermarket cashier may not know a kiwano from a tamarillo, but Veggie Vision does.
Although it is
a practice generally frowned upon, Jon Connell is very good at comparing apples and oranges. He is also expert at the differences between cabbages and cauliflowers, lemons and lingonberries. For Connell and his colleagues at the Exploratory Vision Group at IBM's Thomas J. Watson Research Center have invented a computer vision system, dubbed Veggie Vision, that can recognize the 150 or so different fruits and vegetables found in the produce section of a well-stocked supermarket. With a Veggie Vision scanner, even an inexperienced cashier can ring up a basket of exotic produce like a pro, speeding checkout times and reducing the losses caused by cashier errors.
In fact, the system is so accurate and easy to use that it can be used without a cashier. Several supermarkets have shown an interest in Veggie Vision, as has Optimal Robotics, a Canadian manufacturer of self-checkout systems.
The debut of bar-code scanners in 1974 revolutionized supermarkets. Not only did they speed up lines, but they greatly improved checkout accuracy and provided a source of valuable marketing and inventory information. But bar codes are too large to stick onto fruits and vegetables, so markets laboriously label each item with a coded sticker or twist tie that is often difficult to remove. Some leafy greens resist even these measures, meaning that cashiers have to be trained to tell Boston lettuce from red, parsley from cilantro. When they're stumped, they have to flip through a little photo album containing mug shots of fruits and vegetables, while shoppers wait impatiently. Little wonder that, even though the produce department is a significant part of their business, according to Connell, "if asked, most grocers and supermarket owners would admit that they would rather not carry produce."
Seven years ago, in conjunction with IBM's Raleigh Store Systems Division, Connell and his colleagues began to piece together a vision system to help solve the grocers' problem. The most obvious difference between fruits and vegetables is their color: apples are red and carrots are orange. But, Connell points out, "There's a lot of stuff that's green. How can you tell the green stuff apart?" It turns out there are different shades and mixtures of green, green with blue, green with yellow. So the challenge was making the color distinctions fine enough and stable enough to distinguish among the different shades.
Measuring the colors of fruits and vegetables in a controlled laboratory setting is relatively simple. But in a supermarket, ambient light can confuse the color balance, so a scanning system had to be invented to reduce this problem. The produce is placed on the glass surface of a scale built into the checkout counter. An inexpensive CCD camera looks upward through the glass at the produce, which is illuminated from below by a ring-shaped fluorescent tube. Two images of the produce are captured, one with the light on and the other with it off. The produce can be segmented out from the background by analyzing the difference between the two images. And because the scanner provides its own light source, the color of the illumination is constant. Unwanted reflections are eliminated with polarizing filters, enabling the system to recognize produce through a transparent plastic bag.
Once the image of the produce is captured, its color is analyzed into hue, saturation and intensity, yielding a color signature. By itself, color is enough to tell apples from oranges and can even distinguish between Macintosh and Empire apples. But to tell lemons from bananas takes a little more effort. Shape, texture and size information derived from the images provides the final clues.
Even with these techniques, Veggie Vision can be stumped. Sometimes, Connell explains, the operator will set the produce down "in a really weird way. For example, a pineapple placed on its butt end appears as a small brown circle." The solution is for the system to ask the operator to reorient the object. When this fails, the computer displays a small set of candidates, complete with pictures, for the operator to choose from. Using this feedback, a knowledgeable cashier can gradually train Veggie Vision so it can learn from its mistakes.
After their initial effort to teach a computer to tell veggies apart a few years ago, Connell says he and his colleagues "got to a point where the accuracy was kind of mediocre and the system was expensive." The problem was largely one of hardware cost; because supermarkets are a notoriously low-margin business, they must use inexpensive systems. In order to be fast enough, the cash register PC -- typically a 286- or 486-based machine -- had to be augmented by expensive digital signal processing boards. But in the last five years, Moore's law has come to the rescue and the typical cash register contains a Pentium® 200 MHz chip, not fast by today's standards but sufficient to identify produce in about a second. Since the produce has to be weighed, as well, this is more than fast enough. Meanwhile, the researchers have improved the algorithms to the point that, according to Ruud Bolle, the group's manager, it "can do better than the average checker, but not better than the experienced checker, es
pecially one who works in a specialty store."
Still, this level of accuracy would be a boon to most markets, where the turnover of cashiers is great. Moreover, Veggie Vision allows pricing strategies that are impossible with human cashiers. For example, by looking at the color, the computer can, says Connell, "judge how ripe your bananas are. Yellow bananas might be the most expensive, with green ones cheaper, and black ones cheaper still." Different-size apples cost the market different amounts, but "cashiers really can't tell the sizes apart. The machine can easily measure the fruit so they can charge differently for the bigger apples."
While most shoppers and grocery store owners are sure to benefit from the introduction of Veggie Vision, not everyone's life will be easier: spouses who avoid shopping on the grounds that they don't know beans about produce will be out of excuses.
Bruce Schechter, author of My Brain Is Open: The Mathematical Journeys of Paul Erdös, squeezes the tomatoes in Brooklyn, New York.
