3 ms·

I am 100% agreeing with the article, with one caveat. Database engines are not just for storing - each is basically a "utility knife" of data retrieval - index

I am 100% agreeing with the article, with one caveat. Database engines are not just for storing - each is basically a "utility knife" of data retrieval - indexing, sorting and filtering are available via (relatively) simple SQL constructs. If your app uses an index right now, ditching the DB will mean re-implementing it manually. It's not hard, but it's extra code. So basically, the DB engine might still be a necessary "library", at least for data retrieval. A middle-of-the-road take on this is e.g. using an in-memory Sqlite instance to perform indexing, etc - seeding it at run-time to help with data searches, but then still not using it for storing persistent information and discarding the data at the end.

jpitz 15y ago
When you're 3-4 orders of magnitude ahead of the game because you've ditched network round trips and disk accesses, indices become less ( not completely un- ) necessary. The impedance mismatch between database and application is a lot of code too.
wpietri 15y ago
Having built a couple of systems like this, I didn't find this to be a big problem. You have to organize your data in memory somehow, and that tended to be along lines that made for fast access. Occasionally I'd have to index something, which meant adding a hashtable here and there. It also allowed me to index and organize in ways that SQL doesn't make available. The area where I still most needed SQL was for ad-hoc querying and reporting. I dreamed of building a relational-to-object mapper, but settled for a) XPath queries against our snapshot files, and b) dumping to an SQL database for reports.
Duff 15y ago
The SQL constructs are great, but the biggest advantage to relational databases is that the engine handles your data consistency issues for you. Consistency isn't just about rolling the datastore back to a specific moment in time -- you have to handle locking, concurrent reads/writes, etc. If you're building a trading platform that handles 6M transactions/second, you have the money to handle this in the application layer and the load to justify the expense. But for many other tasks, you may be wasting money or putting data at risk.
- wpietri 15y ago
  Actually, the NoDB approach that Fowler describes handles consistency much more simply: mutations are executed serially. If all your data is hot in RAM, then changes to the data are very quick. Thus, there's no locking, no concurrent writes, and no need to worry about transaction isolation. If you're a Java user, the Prevayler framework he mentions provides this in a couple thousand lines of code.
- wdrury 15y ago
  I agree. Unless you know you are building something that will start out needing millions of transactions per second, you are more likely over-designing if you are building a bespoke database. Standard tools are useful because you can get to working code fast ... this is why LAMP is still such a powerful framework upon which to build. While it may make sense to consider adding a search indexer (Solr) or key-value cache (Redis), for almost every use case, rewriting data storage is a waste. Also, to paraphrase Ted Dziuba, it probably doesn't matter if your product doesn't scale, because nobody cares, or will ever use it. So I think it is better to get something up and running quickly to see if anyone cares before you bother trying to optimize for the rare case where your product turns out to be the next Twitter.

jpitz 15y ago

When you're 3-4 orders of magnitude ahead of the game because you've ditched network round trips and disk accesses, indices become less ( not completely un- ) necessary. The impedance mismatch between database and application is a lot of code too.

wpietri 15y ago

Having built a couple of systems like this, I didn't find this to be a big problem. You have to organize your data in memory somehow, and that tended to be along lines that made for fast access. Occasionally I'd have to index something, which meant adding a hashtable here and there. It also allowed me to index and organize in ways that SQL doesn't make available. The area where I still most needed SQL was for ad-hoc querying and reporting. I dreamed of building a relational-to-object mapper, but settled for a) XPath queries against our snapshot files, and b) dumping to an SQL database for reports.

Duff 15y ago

The SQL constructs are great, but the biggest advantage to relational databases is that the engine handles your data consistency issues for you. Consistency isn't just about rolling the datastore back to a specific moment in time -- you have to handle locking, concurrent reads/writes, etc. If you're building a trading platform that handles 6M transactions/second, you have the money to handle this in the application layer and the load to justify the expense. But for many other tasks, you may be wasting money or putting data at risk.

wpietri 15y ago
Actually, the NoDB approach that Fowler describes handles consistency much more simply: mutations are executed serially. If all your data is hot in RAM, then changes to the data are very quick. Thus, there's no locking, no concurrent writes, and no need to worry about transaction isolation. If you're a Java user, the Prevayler framework he mentions provides this in a couple thousand lines of code.
wdrury 15y ago
I agree. Unless you know you are building something that will start out needing millions of transactions per second, you are more likely over-designing if you are building a bespoke database. Standard tools are useful because you can get to working code fast ... this is why LAMP is still such a powerful framework upon which to build. While it may make sense to consider adding a search indexer (Solr) or key-value cache (Redis), for almost every use case, rewriting data storage is a waste. Also, to paraphrase Ted Dziuba, it probably doesn't matter if your product doesn't scale, because nobody cares, or will ever use it. So I think it is better to get something up and running quickly to see if anyone cares before you bother trying to optimize for the rare case where your product turns out to be the next Twitter.