2 ms·
Depends a lot on the actual access patterns of your data. Many recent web & mobile apps have a lot of screens where you just want to grab one blob of heterogen
by nostrademons 8y ago
Depends a lot on the actual access patterns of your data.
Many recent web & mobile apps have a lot of screens where you just want to grab one blob of heterogenous data and format it with the UI toolkit of choice. Or if they do display multiple results, it's O(10) rather than O(1000) or O(1M). Chasing pointers is fine for use-cases like this, because you do it once and you have all the information you're looking for.
This is also behind the recent popularity of key/value stores and document databases. If all you need is a key/value lookup, well, just do a key/value lookup and don't pay the overhead of query parsing, query planning, predicate matching, joins, etc. When I was working on Google Search > 50% of features could get by with read-only datasets that supported only key/value lookup. You don't need a database for that, just binary search or hash indexes over a big file.
- barrkel 8y agoOh I agree. Though I think having the rest of a relational DB is really nice should you need more than a simple key-value lookup, and not have to cobble it together through application-side joins and denormalization. The application I work on in my day job does not match the key/value lookup idiom at all. User-defined sorts and filters over user-defined schema, and mass automated operations over data matching certain criteria. If you squint a bit, the app even looks a bit like a database in terms of user actions. And even relational databases (at least row-oriented with primarily disk storage) have their limit here. With increasing volumes of data, it can't keep up. We can't index all the columns, and indexes can't span multiple tables. We increasingly need more denormalization solutions that convert hotter bits of data into e.g. in-memory caches that are faster for ad-hoc sorts and filters. Database first is a decent place to start, though having a first-class event feed for updates would certainly be nice...
- tremon 8y agoDepends a lot on the actual access patterns of your data. Yup. Thing is: with RDBMSs you are in control of both the storage patterns and the access patterns of your data. That's where a large part of the performance benefit comes from. > 50% of features could get by with read-only datasets that supported only key/value lookup Did you implement a storage pattern that was ordered by key (or hash(key) if you used hashing)?
- nostrademons 8y ago"Did you implement a storage pattern that was ordered by key (or hash(key) if you used hashing)?" Well, I didn't implement it, Jeff Dean did, but yes, that's what we used for a lot of features. https://www.quora.com/What-is-an-SSTable-in-Googles-internal-infrastructure https://www.quora.com/What-is-an-SSTable-in-Googles-internal... https://www.igvita.com/2012/02/06/sstable-and-log-structured-storage-leveldb/ https://www.igvita.com/2012/02/06/sstable-and-log-structured...