Operating system interference, caused primarily due to scheduling of daemon processes, and handling of asynchronous events such as interrupts, constitutes “noise” or “jitter” (henceforth referred to as OS jitter) perceived by an application. Various studies have taken place in the recent past that demonstrate the debilitating effects of OS jitter on parallel applications running on large scale HPC systems [1] [8].
The goal of this project is to study OS jitter and its impact on the performance of tightly coupled parallel applications runnning on large HPC clusters. These applications consist of multiple phases of computation where each computation phase is followed by some sort of synchronization across nodes. The work in this project has so far focussed on theoretical modeling of noise, validation of the theoretical model using data from large production HPC clusters and identification of sources of OS jitter on a single node.
Background and Motivation
Jones et al. [2] found that with only 0.31% daemon and OS activity on each CPU of a dedicated 4096 processor IBM SP system, MPI_Allreduce operation degraded more than 400%. The expected average time, using the fact that such reduction operations should scale as the logarithm of the number of processors [4, 5], was calculated to be 710μs, while the observed average time was close to 3000μs. Such collective operations, typical of a large class of parallel applications [5, 6], are most susceptible to a cascading effect, which results when one slow process impedes the progress of all other processes. This is shown in the following figure.
 |
| Collectives in a cluster with no OS Jitter |
 |
| Collectives in a cluster with OS Jitter |
A detailed study of OS jitter and its impact on performance was done by Petrini et al. [3] on the 8192 processor ASCI Q machine. It was observed that the overheads due to OS jitter were mostly in the range 0.5% to 2.5%. However, this jitter had a large impact on the system performance. By reducing the intensity of jitter in the system it was possible to get a factor of 13 improvement in the performance of a micro-benchmark that repeatedly calls barrier with no intervening computation. This also led to a factor 2.21 improvement in the performance of the SAGE benchmark (see Table 3 in [3]) on 7680 processors. Similarly, Kramer and Ryan [7] concluded that the performance variability in EP (of NAS parallel benchmarks) was due to the jitter in systems.
To overcome variabilities in cluster performance and provide generic methods for tuning clusters for sustained high performance, it is essential to understand theoretically, as well as using empirical data, the behavior of production mode clusters. Theoretical modeling can help us predict performance of large clusters in presence of OS jitter. In order to derive an accurate theoretical model, it is imporant to identify the sources of OS jitter and study their impact. As part of this project, we are currently involved in the following efforts:
- Theoretical modeling of OS Jitter in large clusters
- Empirical evaluation of the theoretical model using microbenchmarks in large production clusters
- Identification of sources of OS Jitter on a single node and measuring their impact
- Prediction of jitter impact at large scale using trace driven approaches
- Reduction of jitter through intelligent interrupt routing, user and kernel thread scheduling decisions and scheduling policies
- Synchronization of jitter sources through various co-scheduling techniques
References
[1] D. Tsafrir, Y. Etsion, D. G. Feitelson, and S. Kirkpatrick, “System Noise, OS Clock Ticks, and Fine-grained Parallel Applications,” in ICS, 2005.
[2] T. R. Jones, L. B. Brenner, and J. M. Fier, “Impacts of Operating Systems on the Scalibility of Parallel Applications,” Tech. Rep. UCRL-MI-202629, Lawrence Livermore National Laboratory, Mar. 2003.
[3] F. Petrini, D. J. Kerbyson, and S. Pakin, “The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q,” in ACM/IEEE Conference on Supercomputing (SC’03), (Phoenix, Arizona, USA), Nov. 2003.
[4] A. Moody, J. Fernandez, F. Petrini, and D. K. Panda, “Scalable NIC-based Reduction on Large-scale Clusters,” in ACM/IEEE Conference on Supercomputing (SC’03), (Washington, DC, USA), p. 59, 2003.
[5] F. Petrini, J. Fernandez, E. Frachtenberg, and S. Coll, “Scalable Collective Communication on the ASCI Q Machine,” in 11th Symposium on High Performance Interconnects, (Stanford, California, USA), Aug. 2003.
[6] A. Gupta, A. Tucker, and S. Urushibara, “The Impact of Operating System Scheduling Policies and Synchronization Methods on the Performance of Parallel Applications,” in ACM SIGMETRICS Conference, May 1991.
[7] W. T. C. Kramer and C. Ryan, “Performance Variability of Highly Parallel Architectures,” in International Conference on Computational Science (ICCS 2003), (Melbourne, Australia), Jun. 2003.
[8] P. Beckman, K. Iskra, K. Yoshii, and S. Coghlan, “The Influence of Operating Systems on the Performance of Collective Operations at Extreme Scale,” in IEEE International Conference on Cluster Computing, 2006.
Last updated 13 May 2009