Interconnected Servers Can Balance the Load
On June 10, 1999, not a creature was stirring on the Internet's most popular auction site — not even a mouse. For 22 hours, eBay was shut down by computer trouble, and bargain hunters looking for everything from sofas to Smurfs had to surf elsewhere. When the smoke finally cleared and the computers came back online, eBay had lots of angry customers on its hands. Moreover, it had lost the confidence of Wall Street: The glamour stock plummeted from $180 to $136 a share.
The crash at eBay focused attention on the crucial importance — especially for an e-business — of continuous access to critical data. Though inadequate backup of the company's disk drives didn't cause the crash, it made it more difficult to recover, turning the outage from a short-term glitch into a day-long fiasco. Disasters will happen, and one job of a storage system is to pick up the pieces quickly. As Wall Street's verdict showed, 22 hours is not quick enough.
Restoring a single computer is easy, if you've recently backed up all your files and the backup copy is readily available on a fast storage device. But for today's e-business networks, it is not so straightforward. For starters, simply finding the time to make backups is a challenge. The Beatles may have had eight days a week, but e-businesses don't. The traditional down time after midnight, when enterprises could back up their data onto tape, is a thing of the past on the Web, where customers can arrive at any time of day or night.
The need for continuous access to data is only one aspect of a much more complicated data management problem. A typical e-business has dozens of servers running at the same time, all accessing and modifying the same data. At present, e-businesses often maintain a separate copy of many of the data files on each server, a solution that can waste storage space and requires every change in a file to be made in all the copies. While servers can share files over a local area network (LAN), doing so degrades performance because the data must go through a dedicated file server that controls access to the storage device.
Then there is the issue of scalability: How do you design a storage system that can expand easily as the company grows? Although this is a special concern for rapidly growing dot-coms, where storage needs can double in 90 days, it is relevant to nearly any business.
In response to these concerns, the world is moving away from direct-attached storage to storage networks, and companies like IBM, Compaq, EMC, Hewlett Packard, Legato, Network Appliance, Veritas and many others are investing heavily to compete in this rapidly emerging segment of the storage business. Currently growing at 50 to 60 percent a year, it is expected to become a $20 billion market by 2003.
Today, there are two very different methods for attaching storage to networks — storage-area networks, or SANs, and network-attached storage, or NAS, says Jai Menon, department group manager of the computer science:storage systems department at IBM's Almaden Research Center and co-director of the Storage Systems Institute, a joint program between Research and the Storage Systems Group. A SAN consists of a special "fabric" of hardware and software for interconnecting a business's servers with its storage systems. Although these systems may be individual disk drives, more often they are stand-alone units, such as IBM's Shark™ Enterprise Storage Server, a storage controller that manages an array of up to 384 disk drives with 11.2 terabytes of memory, or tape controllers, such as IBM's Virtual Tape Server. A NAS is a storage system that connects to an existing LAN, and it possesses both advantages and disadvantages compared to a SAN (see "Toasters to the Rescue," page 14).
The SAN fabric forms a layer separate from the LAN that already links the servers together. Before SANs, only the servers and clients — the personal computers and workstations that employees work on — were networked together, whereas each server was directly attached to its storage devices. To access data on a storage device attached to another server, it was necessary to go through the other server. With a SAN, any server can access data from a storage device directly, without having the data go through an intervening server.
Today, SANs are built using Fibre Channel, a new industry-standard network — which, despite its name, does not necessarily involve optical fibers — that can run at 1 gigabit per second. It is different from the LAN used to attach clients to servers, which today is typically based on Ethernet technology. Thus, clients are networked to servers using an Ethernet LAN, whereas servers are networked to storage devices using a Fibre Channel SAN.
In theory, at least, SANs promise to solve many of the storage management problems mentioned above, from continuous availability and remote mirroring — creating copies of all the files at distant locations for disaster recovery — to scalability and file sharing. For example, backup becomes easier because the network not only connects the servers to the storage devices but also connects the devices to each other. That means that storage devices can communicate directly with each other to move data from one device to another, such as from a disk to a tape. In addition, it becomes easier to create remote mirrors, because a storage device can send a copy of any data written to it directly to another remote storage device on the SAN. SANs also address the issue of scalability by allowing an almost unlimited number of storage devices to be added to the network and automatically integrated into the SAN.
Although the SAN business is booming, many of these features, such as high-performance, heterogeneous file sharing, are not available today. By developing the technology to implement the new features, Menon hopes to help IBM recapture an important part of the data storage business that has slipped from IBM's grasp in recent years. There are two main SAN initiatives currently under way in Research. The first is devoted to enhancing the capabilities of SANs significantly beyond what is possible today, whereas the second is aimed at creating SANs using Ethernet and TCP/IP (Transport Control Protocol/Internet Protocol) instead of Fibre Channel, thereby reducing the cost of installing a SAN and making it easier to manage. Both have significant implications for the future of networked storage.
DOWN ON THE (STORAGE) FARM
Before it became a corporate park, the land on which Almaden Research Center sits was home to a farm. The window from David Pease's office still looks out on a postcard-perfect green hillside kept neatly trimmed by grazing cows. In a nearby office, a "cattle cam" records their progress and posts photos on the Internet (www.almaden.ibm.com/cattlecam/ index.htm).ÊMostÊofÊtheÊtime, though, Pease, the manager of the storage software department at Almaden, devotes his attention to the needs of a different kind of farm: the "server farms" operated by today's e-businesses. These farms are a little more difficult to manage than a herd of cows because, unlike cows, which can understand one another, one server cannot understand data that has been created by another server. SANs promise to help provide a solution to that fundamental problem.
The SAN hardware itself — the storage devices, switches, hubs, cables and network adapter cards — cannot do this alone. Software is needed to manage the flow of data among the servers and the storage devices. "But today's software," says Pease, "does not allow a SAN to do the kinds of things a SAN ought to do." That is about to change. At Research and Tivoli®, an IBM company that is one of the industry leaders in SAN management software, efforts are under way to enable SANs to reach their full potential.
One objective is high-performance data sharing across different platforms. A credit card company, for example, might want to use a fraud detection program on a Windows NT® server to access transaction records stored on a UNIX® server. Today, this requires either using a shared file server to store the transaction data, or having the NT server ask the UNIX server to send the file over the LAN so that the data can be processed locally on the NT server. Both introduce extraneous steps that impair performance.
Another goal is to enable a company to balance the load across all its disk drives. Today, this is still a manual process that requires a systems administrator to monitor the load on each drive and decide whether to copy certain heavily used files from one disk drive to another. If the company in question could not afford a systems administrator, a SAN that could do this automatically would be the ideal — and perhaps the only — solution.
All of these capabilities are on the agenda for Storage Tank, a new software package for SAN management that is already running in Pease's laboratory. Storage Tank owes its advanced management abilities to its unique architecture, which includes a special "location server" to control traffic without getting in the way. The other servers verify their credentials and find out where files are stored on the SAN from the location server, but aside from that they are autonomous: they access the data they need by themselves, no matter where it resides in the network. The difference between directly accessing a file and accessing it through another server may seem trivial, but it adds milliseconds to every storage operation and keeps servers busy doing things they don't have to. "About 30 percent of the computing resources on application servers are used by storage traffic today," says Menon. "Those are resources that are not available to run the applications they're supposed to be running."
Perhaps the most attractive selling point of Storage Tank, in Pease's opinion, will be its ability to provide central, policy-based management of storage. With policy-based storage, users can specify the amount of storage they need, as well as the characteristics of that storage, such as its reliability and the rate at which data can be accessed. The system will then automatically find the requisite disk space and make sure it meets the user's specifications for performance and reliability. This represents a huge advance over today's manual solution, which requires a systems administrator to carry out this task —a lengthier, more costly and potentially error-prone process. Furthermore, instead of using each disk as needed — which may result in some drives becoming overloaded while others are idle — the network will dynamically balance the load among the disks.
In the mainframe world, centralized management of storage is old news. On IBM's System/390® mainframes, a program called System Managed Storage (SMS) managed the storage. But SMS assumed that everything in the network ran on System/390. In the SAN world, where the network may contain servers running on several different platforms, centralized management hasn't been done before. Pease believes that IBM's mainframe experience gives it an advantage over competitors such as EMC.
BEYOND FIBRE CHANEL
One of the mysteries of storage-area networks, for a business that is installing one for the first time, is why one needs a separate network for storage at all. Remember, there's a LAN on top, linking the client machines and the servers, and a SAN on the bottom, connecting the servers and the storage devices. The typical e-business is very familiar with LANs, which run over Ethernet cables and use the same protocols, such as TCP/IP, that the Internet uses. IBM Fellow Steve Hetzler was the first person in IBM to ask, "Why should a SAN require a special network? Why should a customer be stuck with two different infrastructures to install and manage?"
More to the point, why didn't the SAN community use Ethernet and TCP/IP in the first place instead of going to the trouble to invent Fibre Channel? "The primary reason is performance," says Menon. "At the time that Fibre Channel was introduced, it ran at 1 gigabit per second, whereas Ethernet was running at 10 and 100 megabits per second. Another reason was the lack of a standard — one that would provide a compatibility between SAN software and Ethernet." Hetzler did not see these as insuperable obstacles, and he proposed that a new scheme be developed for running SAN traffic over existing Ethernet LANs.
The key to this scheme depends on the "stack" nature of networks, which can be thought of as having four basic layers. The bottommost is simply the physical wires and cables. Next come the networking and transport layers, which consist of the various rules, or protocols, for transmitting the data reliably, and, finally, there is an application layer, which deals with higher-level communication issues. For example, Ethernet is a physical layer, IP a network layer and TCP a transport layer.
For SANs, the application layer is a command language called SCSI (Small Computer Systems Interface), which is used to communicate between servers and storage devices. In today's SANs, the SCSI commands are adapted to run on top of the Fibre Channel transport, network and physical layers. Hetzler's idea was to adapt the SCSI commands so that they would run on top of Ethernet and TCP/IP. By keeping the command language unchanged, any application software designed for a Fibre Channel SAN would also run on an Ethernet-based SAN. Since today client computers are attached to Ethernet, but not to Fibre Channel, the new idea would allow client computers to access storage directly, without intervening servers.
A team of researchers led by Hetzler, Richard Freitas, manager of storage attachment at Almaden and Julian Satran of IBM's Haifa Research Laboratory, began work on refining, prototyping and evaluating these concepts two years ago. More recently, they have worked with Cisco and other companies to refine and extend the original idea. The outcome of that collaboration is a proposed new standard — called iSCSI — for running SCSI on top of Ethernet and TCP/IP.
"People have come to think of SAN and Fibre Channel as synonymous," says Hetzler. But both he and Freitas think that the preeminence of Fibre Channel is only temporary and that iSCSI will be able to compete with — and ultimately displace — Fibre Channel.
There are good reasons for thinking so. Ethernet's speed has taken a tenfold jump every three years and is expected to reach 10 gigabits in 2001, before Fibre Channel reaches that milestone. Moreover, the new generation of Ethernet cards are enabling many functions to be performed — as is already the case with Fibre Channel — in hardware, thus reducing the computational burden on the servers and improving the overall performance of Ethernet SANs. And, because Ethernet cards benefit from an economy of scale, they currently cost about half of what the equivalent Fibre Channel hardware does.
People are also influenced by other factors,Êexplains Freitas. "Businesses trust TCP/IP, based on its reliability over the past 30 years. And there is also a concern about interoperability or, rather, the lack of it, which is still plaguing the manufacturers of Fibre Channel equipment."
One thing seems undeniable: just as the 1990s were the decade when networking really came into its own, so the 2000s will be the decade in which storage becomes a hot technology, thanks in large part to the rapid evolution and growth of networked storage.
Dana Mackenzie is a freelance writer living in Santa Cruz, California.
