6 ms·
I quickly scanned the white-paper on linked on their site [1] So they are doing some interesting things with a virtual memory space using a tree structure of t
by lootsauce 11y ago
I quickly scanned the white-paper on linked on their site [1]
So they are doing some interesting things with a virtual memory space using a tree structure of the data that reflects various relations in the data in a star schema. Those objects are persisted in memory mapped files, mapped into an effectively infinite single shared memory address space in the virtual memory model. They argue that you can then do new more efficient algorithms on these memory mapped files.
Sounds like they are seeing performance we are getting in Google BigQuery with queries against 100s of millions of rows taking less than 10 seconds however they did not show how it scales beyond that.
Another interesting thing is this is designed to sit on top of existing systems to serve not as the source of truth but the layer to do analytics with realtime data, a very compelling approach IMHO.
They liken their data-structure to that of RDD (Resilient Distributed Dataset) used in Spark but unlike RDD it is not temporary and is writeable.
The other interesting thing about this is as far as I understand it, this can be seen as a virtualized implementation of the computing model that is coming with new memory hardware [2] where ram == storage
[1] http://www.slideshare.net/BigObject/big-object-store-in-place-computing-for-interactive-analysis?ref=http://bigobject.io/ http://www.slideshare.net/BigObject/big-object-store-in-plac...
[2] http://www.zdnet.com/article/nanteros-carbon-nanotube-memory-breakthrough/ http://www.zdnet.com/article/nanteros-carbon-nanotube-memory...
"The race for a workable NVRAM is in its final stages. HP's memristor, Crossbar's RRAM and now, Nantero's NRAM, are all technically sound, backed by tens of millions of dollars in R&D, and close to broad market release.”
- mdaniel 11y agoI thought the slideshare.net link was odd, but it seems that is the only non-paywall-ed version of the file. As a friendly FYI to other interested parties, http://bugmenot.com/view/slideshare.net http://bugmenot.com/view/slideshare.net is a mechanism through which one can use the download button, resulting in a much nicer on the eyes PDF.
- chenyuanjen 11y agoHey mdaniel that is an IEEE paper we published.