Blue Gene, a dazzlingly fast supercomputer that's five years away, will tackle one of the great unsolved problems of biology.
The folding of a protein is one of nature's underappreciated wonders. In less than a second, the protein -- a chain of hundreds of amino acid molecules strung end to end -- twists and crumples itself into exactly the right configuration, each of its thousands of atoms ending up just where it is supposed to be. For humans and other organisms, it is a vitally important second, for proteins are the basic building blocks of life, and the particular shape of a protein determines its properties and what sort of job it performs.
Scientists would love to see how the protein does it. "For biologists, this has always been one of the 'grand challenge' problems," says Sharon Nunes, manager of the Computational Biology Center at IBM's Thomas J. Watson Research Center. If researchers understood protein folding, she says, it could give doctors a better understanding of many diseases, lead to the development of a new generation of highly effective drugs, and open an unprecedented window on life at its most basic level. But proteins are far too small for their folding to be observed directly, and although in theory it is possible to model the folding accurately with a computer, in practice the process is so complex that it would take centuries to calculate on even the most powerful of today's supercomputers.
Tomorrow's supercomputers will be different, however -- or at least one of them will be. In December 1999, IBM Research launched a $100 million, five-year effort to build Blue Gene, a supercomputer that is intended to crack the protein-folding problem. Based on a radically different type of computer architecture called SMASH (simple, multiple and self-healing), the Blue Gene machine will be hundreds of times faster than anything operating today. It will be rated at 1 petaflop -- that is, capable of performing 1 quadrillion (1 followed by 15 zeros) floating-point operations per second. And the SMASH architecture will allow Blue Gene to be made for considerably less money, in terms of dollars per flop, than any other supercomputer to date.
It is an ambitious project, acknowledges Ambuj Goyal, vice president of systems and software for IBM Research, but it is meant to be. "We are trying to take the next leap in architectural innovation," he says. If the project is successful, not only will it provide biological researchers an invaluable tool with which to study proteins, but it could fundamentally reshape how the most powerful computers are made in the future.
Blue Gene represents the coming together of two quite different lines of research at IBM. The first is computational biology, a relatively new area of interest for the company. Although IBM has for many years had a small effort in bioinformatics -- the handling, manipulation and analysis of biological data -- it has only recently come to be a major concentration. "In the past, biology was focused on wet-laboratory science," Nunes notes. Today, however, an increasing amount of biology is being done on the computer instead of in the test tube, as biologists have become inundated with data: protein structures, gene sequences from humans and dozens of other organisms, and much more. "Information technology is becoming the language of biology," Nunes says.
In the summer of 1997, at Nunes's recommendation, IBM unified its biological research
worldwide into the Computational Biology Center and began to build that group by attracting
experts from outside. The group now contains some 30 biologists, computer scientists,
mathematicians, chemists and physicists, about a third of whom study protein folding
using several different techniques (see Think Research, Number 2, 1999, "Decoding proteins"). One approach, the use of brute-force calculation to simulate the folding process at the atomic level, has been impractical because of the huge computational requirements. It is that limitation that Blue Gene is intended to overcome.
The second line leading to Blue Gene was a more traditional research program at IBM: the development of state-of-
the-art supercomputers to solve computationally intensive problems. The best-known and most recent example is
the chess-playing supercomputer Deep Blue™, which defeated the chess champion Garry Kasparov, but there have been many others, such as the GF-11, used to calculate the mass of the proton.
FROM MOVIES TO MOLECULES
The idea for the computer that would later evolve into Blue Gene was hatched in 1995 by Watson computer designers Monty Denneau and Peter Hochschild, when they began thinking about a machine for generating photo-quality computer graphics. The production of high-quality graphics, such as those used in the computer-animated film Toy Story, relies on a computationally intensive technique called ray tracing. A single frame of such images can take a day or more to calculate, Denneau says, and companies that produce such computer animation have "whole farms of workstations." Denneau and Hochschild set out to devise a machine that could compute each frame in a fraction of a second rather than over many hours, making it possible to produce real-time video images.
"We looked at characteristics of a machine that could do this sort of problem," Denneau remembers. Since ray tracing involves calculating the trajectories of millions of light rays as they bounce from object to object in a scene, it would be possible to break the problem into millions of calculations, or threads, that could be performed independently and simultaneously. This would demand a massively parallel
computer with thousands of chips and millions of independent processors. Holding a scene in memory, with the positions and visual characteristics of all its objects, would require a great deal of memory, but if the memory could be spread out among the individual chips, the memory requirements for each would be relatively small. Spreading out the memory, however, would demand that the chips be able to communicate with one another very rapidly, or else the calculations would be slowed while the processors were waiting for data.
Most important, the chips should be fast. Denneau decided to start from scratch on the chip design, leaving out all the instructions that were not necessary for the problem at hand. Processors today, even RISC (reduced instruction-set computing) processors, have a large number of specialized instructions that improve overall performance but greatly complicate the design. Denneau, working with Hank Warren, simplified the instruction set to include only those that were absolutely necessary. By 1997, Denneau had a rough layout of the proposed chip, complete with memory and a couple of dozen processors. It would be a gigaflop chip, able to do a billion floating-point operations a second, and assembling several thousand of them would create a computer that ran at several teraflops, or trillions of operations per second -- quite satisfactory for ray tracing. But then other demands slowed progress. Two collaborators were reassigned, and Denneau was for a time the only researcher working on the project.
Meanwhile, the project was transferred to a new manager, Christos Georgiou. As Denneau continued working on the design of the chip, and Warren on the development of a C compiler for generating executable code for it, Georgiou and Denneau pursued efforts to keep the project alive. "We were convinced that this was a really good technology, and we tried very hard to find partners within IBM who might be interested in using the technology," says Georgiou.
Some six or eight months ago, things began to fall into place. Goyal was looking for a project that would define a new state of the art for supercomputers. "I wanted us to do out-of-the-box thinking," he says. Seizing on protein folding as a goal -- suitable because it was unattainable with today's technology and was already tantalizing Nunes's computational biology group -- Goyal challenged IBM Research's computer architects to find a way to build a petaflop computer that met several key criteria: it must be inexpensive, it must be capable of being completed within five years, and it must be applicable to a variety of problems besides protein folding instead of being dependent on specialized chips. "I put a stake in the ground," Goyal recalls.
Shortly afterward, Denneau received a visit from Mark Snir, senior manager of Watson's scalable parallel systems group, who had been charged with assessing the technical challenges of building such a supercomputer. Would it be possible, he asked, for Denneau to scale up his planned ray-tracing machine -- intended to run at several teraflops -- to petaflop power and put it to work on protein folding? "I said, 'Of course,' without really thinking about it," Denneau remembers. Then he had to figure out just how to do it.
ENTERING THE FOLD
Although ray tracing and protein folding are quite different problems, they make similar demands on a computer, and, as Denneau realized, the computer design he had developed was flexible enough to be turned easily from one to the other. In a protein-folding problem, the computer must calculate the movements of the thousands of atoms that make up the hundreds of amino acid molecules in the long chain that is the protein. Tugged on by the forces among its atoms and between them and the surrounding water molecules, the protein will fold in upon itself, contracting into its characteristic form.
Surprisingly, it doesn't matter where the protein starts -- it will always end up in the same shape. Proteins are created in a cell's ribosomes, tiny "factories" that string together amino acids in the order specified by a gene. As the protein emerges from the ribosome -- a process that takes anywhere from seconds to minutes -- it is presumed to twist and fold into various shapes but apparently does not take its final form until the last amino acid emerges and the protein disengages from the ribosome. The shape is not dependent on this process, however, for when researchers heat up proteins to make them unfold and then cool them back down, the molecules typically return to their original shape. "Moreover, we can even make proteins purely chemically now, in a completely different fashion than the way the ribosome does it, and they still fold up correctly," adds IBM Distinguished Engineer Barry Robson, who is a strategic adviser to the Computational Biology Center.
So the folding calculations on Blue Gene will probably start with a protein chain in some random, loosely packed form, according to Ajay Royyuru, a molecular biologist at IBM's Computational Biology Center. Indeed, Royyuru says, the strategy might be to try dozens or hundreds of initial structures, watch each for a billionth of a second or so, and then attempt to pick out one that seems likely to fold faster than the others.
Of course, Blue Gene will have to accomplish all this within a reasonable time -- say, one year. To make sure this would be possible, Dennis Newns, a Watson physicist who is leading the molecular dynamics effort, and Pratap Pattnaik, manager of the parallel system research group at Watson, analyzed a state-of-the-art molecular dynamics program developed by Michael Klein, a professor of computer science at the University of Pennsylvania. By carefully examining this program -- and by using it to study fundamental biological processes in the influenza virus -- they were able to establish the computational requirements for various steps used in such a program, and to show that it would execute efficiently on Blue Gene.
Modeling the folding process itself is in principal straightforward, says Newns. Nonetheless, he admonishes, "the process is fraught with subtleties whereby many have come to grief. Protein folding is such a delicate balance of energy and entropy of both protein and solvent that mastering it must be considered a challenging science project, into which the computational power of Blue Gene gives us an entree but provides no guarantee." To calculate how the atoms in the protein will move as they are pushed and pulled into shape, it is necessary to compute the electrostatic forces between all pairs of atoms, both in the protein and in the drop of water holding the protein, add up the forces acting on each atom, and calculate from that how the atom will move over the next time step, which for Blue Gene is 5 quadrillionths of a second.
The difficulty of the calculations is compounded by the sheer number of them: with several thousand atoms in the protein and surrounding water, there are tens of millions of forces to calculate and add up at each time step, and 200 billion time steps are necessary to follow the protein's movements through the thousandth of a second it requires to fold.
This was exactly the type of problem that Denneau's design was intended for. The calculation could be divided naturally into millions of threads, one for each pair of atoms, with each thread requiring relatively little memory -- mainly a record of the locations of the two atoms. But because protein folding demanded so much more computation than ray tracing, Denneau would have to find a way to make a computer 100 times more powerful than he had planned.
GRAND DESIGN
The supercomputer that Denneau mapped out was daringly ambitious. He would beef up the processors to 1 gigaflop each -- as powerful as the supercomputers of a decade ago --
and put 32 processors along with 16 megabytes of memory onto a chip. Sixty-four of those chips would be placed onto a board about 20 inches square, cramming 2 teraflops of computing power -- nearly as much as today's most powerful supercomputers -- into an area smaller than a desktop. Eight of the boards would then be placed in a tower, and 64 such towers would be connected to create Blue Gene.
For the plan to work, Denneau and his colleagues had to solve a variety of problems. Denneau had already done much of the work necessary for the 32-gigaflop chips, but he needed to improve the precision of the floating-point operations and design the switching circuitry that would route information around 32,000 chips.
Perhaps most important, a way was needed to make the machine tolerant of errors and breakdowns. Because the calculation is expected to take a year to finish, it wouldn't do to have to start over anytime something went wrong, but with a million processors and eight threads of calculation per processor, glitches were inevitable. "Every four days, on average, a processor will fail," Denneau says. "There's nothing you can do about that." The solution is to make the computer self-healing. As Snir explains, the software will regularly check the state of each computational thread. If an error is detected, the thread can retry the last phase of the computation. If the error persists, the failed component is isolated and the software will rejigger the system to work around it. The challenge is to perform this reconfiguration in a distributed manner, avoiding the bottlenecks that would arise if 8 million threads were controlled from a single point.
PAYOFFS FOR SCIENCE
If all works as planned, the payoff will be substantial. For biologists, Blue Gene promises to open a new window on the behavior of biological molecules. "The essential characteristic of Blue Gene in molecular dynamics," says Newns, "is that it opens up a new field of simulating biomolecules on the time scales of their own intrinsic processes, of which protein folding is only one of many hugely important problems. Up to now, it's been possible to simulate only a few very-short-time-scale wiggles."
Even before the protein-folding calculation is performed, the computer will be used to solve other, somewhat easier simulations as it is being assembled and tested. Two years from now, for instance, the goal is to have assembled a single board containing 64 chips. With a speed of 2 teraflops, Newns says, the board will be able to simulate such things as the passage of ions through the walls of a cell. For the process to occur, a protein embedded in the cell wall acts as a channel, allowing potassium, calcium and other ions to pass into and out of the cell. That passage lasts only about a 10-millionth of a second, short enough to be calculated by a single board, and researchers who study ion channels would love to see the phenomenon in action.
Royyuru says that when Blue Gene is completed, the first protein to be studied will be one whose structure is already known from X-ray crystallography, so researchers can verify the answer. The simulation will provide plenty of useful information. "We don't know how a protein folds or why a protein folds as it does," he says, and the how and the why are important both to basic researchers and to medical scientists trying to understand various maladies. "Several important diseases are apparently due to changes in the shape of proteins, affecting the folding pattern," Royyuru notes. In Alzheimer's disease, for instance, an aggregated form of proteins known as beta-amyloid is formed from shape changes in the natural brain form of that protein. In mad cow disease and possibly in the related Creutzfeldt-Jakob disease, which affects human brain tissue and is inevitably fatal, it is believed that a malformed protein induces proteins to change shape and form another kind of amyloid plaque. Understanding the forces
determining the conformation process could help researchers find ways to block the shape changes and fight such diseases.
Ultimately, however, the greatest promise of protein-folding simulations is being able to determine protein structures that are not known. The structure of a protein can sometimes be learned by growing a crystal of the protein, exposing the crystal to X-rays, and analyzing the resulting diffraction pattern. But such a process is quite laborious -- it generally takes one to two person-years of work, Nunes says -- and not every protein can be crystallized. If Blue Gene is successful, it will be possible to calculate a protein's structure simply by knowing its sequence of amino acids, which in turn can be determined from the genetic information specifying the protein.
Once a protein's structure is known, says Robson, researchers can develop drugs to act on that protein. In general, most drugs work by latching onto a protein, triggering it to perform or block some action. "If you want to design a key efficiently," Robson says, "you need to know the structure of the lock." Once a protein's structure is known, it is possible to design drugs that act on it, and one of the future uses of Blue Gene may be to model how well various candidate drugs latch onto a target protein. By watching the interaction of drug and protein, it should be possible to develop more effective drugs.
Eventually, it should also be possible to tailor drugs to individuals. Proteins often vary subtly from one person to the next, and that subtle difference is often enough to make a drug that is effective for some people ineffective for others. With an efficient way to determine protein structure, one could take a DNA sample from a person, calculate the shape of the relevant protein, and then determine which drugs would be effective and which would not. "It could be a watershed point in human history," Robson says.
The field of materials science might benefit from Blue Gene, as well. A deeper understanding of how preprogrammed chains of atoms fold up could enable scientists to design novel polymers and smart new materials. Ultimately, this knowledge could open up a practical approach to designing and producing those tiny machines and molecular-scale computer chips and storage devices envisaged by nanotechnology.
For the computing industry, Blue Gene could have far-reaching implications. Goyal compares the computer's SMASH architecture to two earlier IBM innovations: RISC computing in the 1980s and SP® (for scalable parallel) supercomputers in the 1990s. RISC brought about a huge leap in computing power by reducing the size of the instruction set on a chip, and scalable parallelism has made it possible to build computers with anywhere from a few processors to many thousands. Thanks to the SP architecture, Goyal notes, IBM has become the world's leading supplier of supercomputers, and the number of IBM machines on the list of the world's 500 most powerful computers has nearly doubled, to 141.
The SMASH approach takes RISC-like thinking and applies it not just to the instruction set but to the whole architecture of the computer, Goyal says. "Typically, people think that the way to get performance out of a chip is to add special-purpose functions," he explains. "Instead, we are going to take simplification to the extreme." By simplifying the complete architecture, he predicts, it will be possible to build machines that are many times more powerful than today's best computers at a small fraction of the cost. Although Blue Gene is intended solely as a research tool and not as a prototype for a commercial product, the lessons learned from building both the hardware and the software should give IBM a head start on the next generation of computers, Goyal says. "If we are successful, the twenty-first century will be a very different place."
More on Blue Gene
Robert Pool is a freelance science and technology writer based in Tallahassee, Florida. His most recent book is Beyond Engineering: How Society Shapes Technology.
