6 ms·
Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention elimin
by ignoramous 11y ago
Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...
- pneumatics 11y agoWhat is rodb?
- sshykes 11y agoGood question. Is it this [0]? [0] http://researcher.watson.ibm.com/researcher/view_group.php?id=4165 http://researcher.watson.ibm.com/researcher/view_group.php?i...
- csears 11y agoI'm guessing he means a "Real-time Operational Database". This seems to be a generic term for system like a data warehouse that contains current data, instead of just historical data. If you are taking the output of a Spark flow and storing it in Postgres or MongoDB or HBase for applications to query, then those could be considered RODBs. Since this is Amazon, I suspect he is referring to SPICE (or their internal version), which was released last fall as part of AWS's QuickSight BI offering... "SPICE: One of the key ingredients that make QuickSight so powerful is the Super-fast, Parallel, In-memory Calculation Engine (SPICE). SPICE is a new technology built from the ground up by the same team that has also built technologies such as DynamoDB, Amazon Redshift, and Amazon Aurora. SPICE enables QuickSight to scale to many terabytes of analytical data and deliver response time for most visualization queries in milliseconds. When you point QuickSight to a data source, data is automatically ingested into SPICE for optimal analytical query performance. SPICE uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations, machine code generation, and data compression to allow users to run interactive queries on large datasets and get rapid responses." http://www.allthingsdistributed.com/2015/10/amazon-quicksight.html http://www.allthingsdistributed.com/2015/10/amazon-quicksigh...
- sharpy 11y agoRead-only DB. Think of it as read-only memory mapped key/value store.
- ignoramous 11y agoLike rocksdb or lmdb but optimized for reads. Read-only DB.
- scottlocklin 11y agoI'd like to know what rodb is. Columnar DB? Columnar DBs often beat "big data" -I've beaten decent sized spark clusters in one thread of J.
- amzthrow23424 11y agoRead only database. It is a hand optimized/compressed database engine that is used for big "once a day" data sets.
- rspeer 11y agoI'm Googling and finding a few things called "RODB" that don't quite match your descriptions. What in particular is it? Or are you talking about just rolling your own format to dump your data into? I've done that, but I'd still appreciate if I could use something that someone else had put the thought into. (Something like "cdb" by Daniel J. Bernstein, but that's an old 32-bit library that's limited to 4 GB files.)
- deleted 11y ago[deleted]
- ignoramous 11y agorodb is in-house, non-FOSS tech @ Amazon.
- scottlocklin 11y agoThanks, that actually helps, particularly since I sell a competing product (which isn't RO).