Smart Middleware and Light Ends (SMILE)

Innovation Matters

SMILE is applicable to any business scenario requiring transformations of event streams on the fly to produce derived information that can be acted upon. Applications of this technology span multiple industrial sectors, e.g. monitoring and transforming financial data streams to identify trading opportunities, detecting potential fraud in banking industry for timely alerts, problem determination and IT systems management in enterprises, inventory management in RFID applications, business activity monitoring and business performance management. In the future, such technology could be deployed inter-enterprise and world-wide, e.g., a customer might request a notification on his cell-phone when at least 6 seats are available on a flight he wishes to take with his family.

The SMILE project enriches application integration middleware with distributed stream processing technology, supporting high-level (declarative and separable) specification of stateful stream-oriented mediations, self-optimization algorithms, and fault-tolerance protocols for maximizing the middleware capability and reducing the amount of low-level client programming and the skills level needed to develop and deploy distributed applications.

SMILE is a follow-up to an earlier messaging project at IBM Watson called Gryphon. SMILE combines the highly scalable publish-subscribe paradigm of Gryphon with the relational model and query language pioneered by IBM, to build novel distributed stream processing technology. In the publish-subscribe paradigm, there can be a world-wide network of producers and consumers of information. Producers deliver events to "streams" or" topics"; consumers request results of interest. In SMILE's stateful extension to the publish-subscribe paradigm, consumers can request functions of stream histories, such as joins, aggregations, top-k, as SQL-like continuous queries, e.g., "tell me the average trading price of the 10 highest-volume trading issues", or "alert me whenever two ATM transactions from the same customer appear at different ATMs within an excessively short time window".

SMILE process at development time and at runtime

Research challenges include: compiling the requested derived state (queries) on event streams to a data flow graph; distributing this graph over a network of multiple servers; scheduling and controlling message flow in this network; rigorously defining the semantics for correct delivery; implementing this semantics in the presence of failure; and smoothly integrating this technology with other components, e.g. databases, services, user interfaces.

SMILE addresses these challenges with novel solutions, thus making middleware "smarter:" 1) Consumer specifications are programmed in a high-level declarative language (e.g., SQL, XQuery), consolidated and compiled into a distributed "delivery plan" for incrementally computing and propagating state changes, and executed on a distributed computing engine; 2) a continuous performance optimization service manages assignment of delivery plan components (transforms) to nodes to minimize latency, 3) a rigorous eventual correctness service guarantee, and 4) fault-tolerance mechanisms that exploit monotonicity and determinism to achieve the above guarantee in the presence of server and network failures. It also offers automated deployment and configuration services on different hosting environments such as WAS (WebSphere Application Server), WebSphere Message Broker and plain JVM.

An Enterprise Service Bus is software infrastructure for integrating applications and services, making it easy to dynamically connect, mediate, and control services and their interactions. An ESB routes messages between services, converts transport protocols between requester and service, transforms message formats between requester ansd service and handles business events from disparate sources. Mediation allows the ESB to reconcile incompatible protocols, data formats and interaction patterns of connected services. Most mediations on the Enterprise Service Bus are stateless. Stateful event processing on the bus is handled either by (a) keeping states in a database, incurring a cost each time the state is updated, or (b) utilizing some kind of replication mechanism which involves having two or more engines processing the input streams. In SMILE, we are able to use timestamps to guarantee deterministic execution, thereby permitting state to be recovered after a failure using checkpointing and replaying, which is a much lighter weight technique. This is fundamental to SMILE's eventual correctness service guarantee which is characterized by the safety and liveness properties for accurate state delivery to stream processing users: no false information is ever delivered and all true information will eventually be delivered.

SMILE's impact on IBM and the industry includes providing sophisticated computational services within the Enterprise Service Bus. Currently, transformational services ("mediations") are stateless and the flow graphs are generated manually by developers who need to be aware of all subscribers. SMILE advances this capability because (a) mediation flows are generated automatically in SMILE, (b) individual subscribers need only be aware of their own requirements, and (c) mediations can now hold state.

SMILE lets developers declare subscriptions that correlate and transform data from multiple streams

Related Publications

O. Damani and R. Strom. Smart Middleware and Light Ends for Simplifying Data Integration. IEEE International Conference on Information Reuse and Integration. IEEE, 2006.

Roman Ginis. Performance modeling of pipelined transform flows for distributed stream processing. Tech report. 2004.

Roman Ginis and Rob Strom. A disruption-free mechanism for moving transforms in a distributed stream processing System. Tech report, 2004 .

Roman Ginis and Rob Strom. An autonomic messaging middleware with stateful stream transformation. International Conference on Autonomic Computing. 2004.

Yuhui Jin and Rob Strom. Relational subscription middleware for Internet scale publish-subscribe. DEBS at SIGMOD. 2003.

Robert E. Strom. Fault-tolerance in the SMILE stateful publish-subscribe system. DEBS 2004 - International Workshop on Distributed Event-Based Systems. ACM, BCS, IEE, May 2004.

Rob Strom. Technical challenges and future directions for event-based systems (panel presentation). DEBS . 2005.

Rob Strom. Lazy versus eager message propagation in distributed stream processing with application to top-K queries. Tech report, 2005.

News and Information
IBM looks to make middleware SMILE, by Paul Krill, November 21, 2005, InfoWorld
Wading Through 'Spaghetti Code' The IBM Way, by Clint Boulton, November 18, 2005, Developer
Coming From IBM Researchers, a Reason to SMILE, by Jennifer deJong, August 1, 2005, SD Times
Rate this article

Innovator's corner

Rob Strom Researcher
What is the most exciting potential future use for the work you're doing?
Just as today's Internet allows web clients anywhere in the world to point and click and pull useful data from anywhere else in the world, the future potential of these ideas would be to allow clients anywhere to subscribe to have interesting data derived from raw data events from anywhere else in the world, and pushed to them whenever and wherever they need.

What is the most interesting part of your research?
It's all interesting, but at the present I am excited by an aspect of it that I have been interested in for years: recovery and fault-tolerance without database transactions and without users having to write special code.

What inspired you to go into this field?
I have always been interested in simplifying development of complex software. Distributed, concurrent, and fault-tolerant programming is complex, and my goal and later research has always been to allow developers to write something simple, and use tooling to automatically derive the complex algorithm and hide it from the developer. I previously worked on a language called Hermes that hid both complex data structures and distribution from programmers. In SMILE, we go further, replacing imperative programming with declarative and completely automating persistence and fault-tolerance.

What is your favorite invention of all time?
There are many candidates, but I think that the telephone had a huge impact as interactive communication could take place without physical presence -- this was a big step towards creating a connected world.