Various projects pertaining to the system infrastructure for streaming include :
The Stream Processing Core : The Stream Processing Core (SPC) provides the scalable, distributed runtime streaming services for System S including efficient stream transport, automatic and application-assisted flow management, failure resiliency, system monitoring and analysis. Appropriate architectural and programming model abstractions allow the SPC to leverage advanced computational and networking hardware dynamically, without affecting applications. The SPC can be logically viewed as shown in the Figure below. Applications are expressed as dataflow graphs consisting of user-written or system-generated stream processing operators called processing elements (PEs), interconnected by data-streams.
Processing elements (PEs) may perform various operations on data streams. Examples include parsing the header of a IP packet, aggregating audio samples into an audio segment or performing speech detection on an audio segment, sampling sensor readings, averaging or joining the readings over a time window of samples, applying spatial, temporal, or frequency filters to extract specific signatures over the audio or video segments and correlating streams of audio news sources. Examples of data streams include packets of market data, audio data, email data, computer generated events, network data packets, or readings from sensors, such as environmental, medical or process sensors. The programming model supported by SPC allows stream interconnections to be computed during runtime, based on declared properties of streams. This enables the stream-mining applications to discover new streams and incorporate them into the processing, dynamically.
The SPADE programming language : SPADE is a language specification and compiler infrastructure for stream processing applications. It comes with a toolkit of commonly used operators such as windowed relational operators and constructs to specify user-defined operators. Stream-oriented applications can be easily programmed declaratively using the SPADE constructs. The accompanying compiler generates optimized code for processing elements (PEs) accepted by the runtime, with knowledge of the underlying hardware infrastructure. Research in SPADE include compiler optimizations for resource allocation and platform-specific code generation, multi-query optimization, exploration of domain-specific operators and algebras, among others.
Dynamic Stream Dataflow Adaptation : Resource management for distributed stream processing systems is complicated by a number of factors – processing elements (PEs) are constrained by their producer-consumer relationships, data and processing rates can be highly bursty, and traditional measures of effectiveness, such as utilization, can be misleading. In this project, we explore application-agnostic as well as application-assisted techniques to adapt stream data flows and application operating points dynamically, to maximize the quality of results obtained. We propose a novel distributed, adaptive control algorithm that maximizes weighted throughput while ensuring stable operation in the face of highly bursty workloads. Our algorithm is designed to meet the challenges of extreme-scale stream processing systems, where overprovisioning is not an option, by making the best use of resources even when the proffered load is greater than available resources. The resource controller, jointly optimizes the input and output rates of the PE and the instantaneous processing rate of a PE, with the express goal of stabilizing the system in the presence of burstiness. Our resource controller uses an adaptive, scalable, distributed optimization technique. Specifically, CPU and flow control for each PE is performed using only the buffer occupancy of that PE and feedback from its downstream PEs and co-located PEs. We use control theoretic analysis techniques to examine the stability and convergence of our algorithms.
Metrics of Effectiveness in Distributed Stream Mining Applications : Stream mining applications perform operations such as classification, filtering, aggregation and correlation over high-volume, unbounded, continuous data such as documents, email, instant messages, transactional data, digital audio, video and image data, network packet traces, and sensor data. These applications process the streaming data in the context of a larger information mining system that tracks information relevant to a large body of long-running, continuous queries on these streaming data sources. Such stream mining applications can be decomposed into a processing graph of networked operators, distributed across several locations. In this research we model these applications as topologies of complex statistical classification processing elements (that may use different underlying algorithms) and develop application relevant end-to-end performance metrics (accuracy of classification) in terms of the throughput and goodput of the individual processing elements. We also develop several distributed algorithms to dynamically reconfigure the operating points of individual classification processing elements to maximize the end-to-end performance metric, while satisfying system imposed resource constraints.
Failure Resiliency : There are various failure-resiliency scenarios unique to stream processing applications. Many stream processing applications can be categorized as partial-fault tolerant (PFT). That is, such applications have the characteristic that they can produce useful output, although of reduced value, even under partial failures. The problem addressed here is the determination of operator placement for PFT applications to prevent, delay, or minimize the impact of failures on the application output quality. Other problems include reliable data streaming in the face of partial failures, dynamic reallocation of operators to nodes and recovery from system failures.
Massive Scalability : One of the basic design principles of the System S architecture is massive scalability. By allowing the application to be specified as a discrete set of processing elements interconnected by data-streams, and by giving the resource manager the flexibility of mapping (and dynamically relocating) one or more of these PEs on to a discrete set of underlying compute resources, the System S data fabric enables massive scalability of applications. We have currently demonstrated a prototype application running on a 100-node cluster and are currently pushing the scalability limits by mapping the data-fabric on to a Bluegene supercomputer.
High performance Stream Data Transport : The SPC data fabric architecture hides the details of the underlying data transport mechanisms from the application API. The application is only concerned with processing stream data objects which it obtains from the data fabric API and hands back to it. The data fabric is responsible for choosing the appropriate communication medium, communication protocol, data encoding and such to get the data to the downstream recipients. A modular transport level API in the data fabric allows the "plug-and-play" of various underlying transport protocols such as TCP, UDP (over 1GigE, 10GigE, Optical), SDP, iWARP (over Infiniband). We are not only experimenting with various such transports (each of which has unique CPU usage, latency and throughput characteristics), but also appropriately modelling them such that the resource manager can choose the appropriate transport based on the application requirements.
System Analytics : Just as applications generate data streams that can be queried, analyzed and mined, system components themselves generate health data and other application-agnostic information that are required for monitoring and predicting the behavior of the system. Additionally, as the underlying system establishes stream connections, validates security and privacy credentials associated with applications, it generates streams of audit data that need to be queried, analyzed and mined. Designing a minimally intrusive, scalable and adaptive infrastructure for system data collection and analysis is the challenge in this project.
Related Publications :
- S. Kashyap, D. S. Turaga, C. Venkatramani, "Towards optimal construction of aggregation trees for mergeable and non-mergeable attributes", submitted to PODS 2008
- N. Bansal, R. Bhagwan, N. Jain, Y. Park, D. S. Turaga, C. Venkaramani, "Towards Optimal Operator Placement in Partial-Fault Tolerant Applications", IEEE Infocom 2008, April, Phoenix, AZ
- F. Fu, D. S. Turaga, O. Verscheure, M. Van der Schaar and L. Amini, "Configuring Competing Classifier Chains in Distributed Stream Mining Systems", IEEE J. Select. Topics in Signal Processing, December 2007.
- F. Fu, D. S. Turaga, O. Verscheure, M. Van der Schaar, and L. Amini,"Configuring networked classifiers in distributed and resource constrained stream processing systems", ICASSP 2007.
- B. Foo, D. Turaga, O. Verscheure, M. van der Schaar and L. Amini,"Resource Constrained Stream Mining: Configuring Trees of Distributed Classifiers", ACAS 2007.
- L. Amini, H. Andrade, R. Bhagwan, F. Eskesen, R. King, P. Selo, Y. Park and C. Venkatramani, "SPC: A Distributed, Scalable Platform for Data Mining", Invited talk, SIGKDD 2006, Workshop on Data Mining Standards, Services, and Platforms, Aug. 20th, 2006, Philadelphia
- L. Amini, N. Jain, A. Sehgal, J. Silber and O. Verscheure, "Adaptive Control of Extreme-scale Stream Processing Systems", Proceedings of the International Conference on Distributed Computing Systems (ICDCS) 2006
- N. Jain, L. Amini, H. Andrade, R. King, Y. Park, P. Selo and C. Venkatramani, "Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core", Proceedings of ACM SIGMOD 2006
- D. S. Turaga, O. Verscheure, U. V. Chaudhari, and L. Amini, "Resource Management for Chained Binary Classifiers" in SysML Workshop (Sigmetrics), Saint-Malo, France, June 2006.
- D. S. Turaga, O. Verscheure, U. V. Chaudhari, and L. Amini, "Resource Management for Networked Classifiers in Distributed Stream Mining Systems" , IEEE ICDM, Hong-Kong, December 2006.