10 ms·
From what I understand, Manhattan is based on the ideas from ElephantDB. Unfortunately, development has pretty much stopped on ElephantDB despite the fact a boo
by ffk 13y ago
From what I understand, Manhattan is based on the ideas from ElephantDB. Unfortunately, development has pretty much stopped on ElephantDB despite the fact a book by Nathan Marz is being written about big data that is dependent on it. http://www.manning.com/marz/ http://www.manning.com/marz/
Summingbird (bear with me, I'll tie this in) is also twitter's answer for writing code once and seeing it run on a variety of execution platforms such as hadoop, storm, spark, akka, etc... Not all of these have been built out, but the platform was designed to be a generic framework to support write once execute everywhere.
Summingbird is written to support Manhattan's model as well. The high level idea is to use versioning to determine whether a request is precomputed (batch), computed (realtime) or a hybrid (precomputed + computed). These are expressed as monads with basic functionality present in algebird. One way to bring support to this model to the open source world would be to implement storehaus bindings for elephantdb and to resurrect elephantdb or build a similar service to provide storage similar to Manhattan.
Overall, very early yet promising work in the open source community.
[edit: book is not about elephantdb, but is a critical component. modified wording. Also added link]
- lenn0x 13y agoWhen Manhattan first started, our first use case was this. To support batch (something we talk about in our blog post). Similar to ElephantDB, the biggest difference was we wrote our own storage engine (seadb), instead of using BDBJE. We built the hadoop pipeline service so developers only needed to supply us with sequence files and we did the work for them by watching. Over time, Manhattan evolved into a fully fledged read/write database that is able to support batch + read/write in the same system. Batch is great for some use cases, but sometimes the cost is too expensive when you factor in how much processing power, storage, etc is needed for that model. We support both of course still we want developers to have the freedom to increase their productivity.