Manifold provides representations for data we don’t yet have, and tools for acting upon it. Sometimes this data is something we can compute ourselves, but more often it’s sent to us by something outside our process. And since we don’t control when the data arrives, it’s likely that sometimes it will arrive faster than we can process it.

This means that we not only need to correctly process the data, we need to have a strategy for when we get too much of it. This is discussed in depth in this talk, but the typical strategy is to use backpressure, which is a signal sent to the producer that we can’t handle more messages, and a subsequent message that we now can.

Backpressure is a fundamental property of Java’s threading model, as shown by BlockingQueues, Futures, InputStreams, and others. It’s also a fundamental property of core.async channels, though it uses a completely different execution model built on callbacks and macros. It’s also a fundamental property of Clojure’s lazy sequences, which like Java’s abstractions are blocking, but unlike both Java and core.async relies on pulling data towards the consumer rather than having it pushed.

Unfortunately, while all of these abstractions (or RxJava, or Reactive Streams, or …) can be used to similar effects, they don’t necessarily work well with each other. The practical effect of this is that by choosing one abstraction, we often make the others off-limits. When writing an application, this may be acceptable, if not really desirable. When writing a library or something meant to be reused, though, it’s much worse; only people who have chosen your particular walled garden can use your work.

Manifold provides abstractions that sits at the intersection of all these similar, but incompatible, approaches. It provides an extensible mechanism for coercing unrealized data into a generic form, and piping data from these generic forms into other stream representations.

It has relatively few central ideas:

Manifold is not intended to be as feature-rich as other stream libraries, but rather to be just feature-rich enough to enable library developers to use it as an asynchronous lingua franca. It is, at this moment, fully functional but subject to change. Feedback is welcomed.