6 ms·

> is perilously similar to "writing your own buggy, half-implemented, slow, nonstandard DBMS". People have used files and in-memory data structures just fine f

by jey 19y ago

> is perilously similar to "writing your own buggy, half-implemented, slow, nonstandard DBMS". People have used files and in-memory data structures just fine for a long time. I don't they had bugs in their code owing solely to the fact that RDBMSes hadn't been invented yet. I also don't see how this is slow; it's all in-memory. Why bring a big honking DBMS into the picture when all you wanted was a hash table? Storing data in-memory doesn't amount to a "buggy, half-implemented, slow, nonstandard DBMS" -- it's serving an entirely different set of goals. Storing data in-memory in data structures is how programming is done. If you actually started to make the interface to your data layer as horrid as the interface to most DBMSes ($DIETY help you), you'd definitely end up with a shitty buggy half-implemented DBMS. But if you just want to store a couple hash tables with a sane interface dictated by the software design, not by your DBMS, just store the hash tables! Don't go wrestle with SQL just because that's the buzzword in-vogue. You should measure your DBMS sometime, and consider it in terms of a hash table lookup. It starts looking like a comedy of horrors: first, generate a string in an obtuse language to perform the lookup, and oh wait, don't put a quotation mark in the wrong spot! Got that string generated? Now send it over a socket to a server. Now the server is going to parse your string into an AST, turn the AST into an internal representation, then it's going to guess how to optimize your query (this is also expensive in terms of cycles). When it's done optimizing, we run it to the query evaluator which looks in memory (since your table is so tiny anyway) and pulls out your hash table value, encodes it, and ships it back over the socket to you. I agree and concede that these days you'll fit into the ecosystem better if you do use an RDBMS with no rationale (other than that it's the de facto standard), generate giant SQL strings from objects, and most worryingly from my perspective: deal with the big impedance mismatch between the two paradigms of OO/anything and RDBMS. http://en.wikipedia.org/wiki/Object-relational_impedance_mismatch http://en.wikipedia.org/wiki/Object-relational_impedance_mis... I haven't seen a single ORM implementation that really makes it truly simple. You end up manipulating your structure at a lower-level than you'd like to naturally. This is partly why my next project isn't being done with an RDBMS, and I'm not storing it just in memory because of the volume of data involved. I'm going to be using Erlang and its bundled Mnesia database. There's no impedance mismatch there. The whole thing, including the database interface, works the way Erlang works. I'll throw a party when RDBMSes die.

mechanical_fish 19y ago
> I also don't see how this is slow; it's all in-memory. Are you not writing to disk on every single transaction, then? My bad. And good luck keeping that power cord plugged in. > Why bring a big honking DBMS into the picture when all you wanted was a hash table? I didn't bring that DBMS into the picture. You did, in the second half of your first sentence: "Start simple: store the data in-memory in whatever way makes sense, and just write a log file that contains pickled transactions that can be played back." Indeed, an in-memory cache is really fast. But it's unfair to compare it with an RDBMS, which is handicapped by the need to write every transaction to disk before it can be committed. The hash table is great only up to the point where a stray cosmic ray crashes the server and makes the whole thing disappear, after which you realize that you need a log file. In the general case, writing a transaction log file is a hard problem. If you write a really robust tool for managing that log file - a tool which is efficient at reading and writing even when the number of transactions grows large; one that lets you specify when the log gets written, and how often, and whether the system can be queried during the write, and how long those queries will block, and what they will return; one which allows multiple threads and multiple machines to read and write the data without concurrency problems; one which prevents the in-memory cache from getting out of sync with the filesystem - you will have implemented a substantial portion of MySQL, and probably memcached as well. It is easy to start out designing a fast, simple, non-transactional DB and end up reinventing MySQL. If you don't believe me, ask the folks who invented MySQL! In certain special cases (e.g. Google), rolling your own persistent storage system is a big, big win. You may know, in advance, that your website is one of those cases. If you are correct, you will be a superstar - you will build a relatively untested, nonstandard data storage system with a tiny subset of PostgreSQL's features, but all of that will be worth it because the system will be fast. If you are wrong, you will work for a month or six and then end up installing PostgreSQL anyway. In fact, even if you are right, you will end up installing PostgreSQL after your customer changes the spec at the last minute to require some boring standard feature - like a shopping cart - which any CRUD jockey can build in a day but which your "simple and elegant" database doesn't support because it wasn't in the original spec. So it's no surprise that the "de facto standard" is to build your site around an RDBMS, get the damn thing working, and optimize later - and that, as a result, the typical in-memory data structure ends up being backed by an RDBMS instead of by a "simple" log file. As for your ire at SQL... if you think you're in pain, just imagine how the designers of SQL must have felt back in the 1970s, when string parsing and query planning were tens of thousands of times slower than they are now, an extra database server cost more than a coder's daily salary, and RDBMS software was very, very non-free. It was a dark time. And yet for some reason those guys abandoned their efficient hand-rolled binary databases for SQL. In fact, they did it so fast that Larry Ellison grew richer than the Beatles. Why did they do that, I wonder? It must have been the drugs.

by jey 19y ago

mechanical_fish 19y ago

> I also don't see how this is slow; it's all in-memory. Are you not writing to disk on every single transaction, then? My bad. And good luck keeping that power cord plugged in. > Why bring a big honking DBMS into the picture when all you wanted was a hash table? I didn't bring that DBMS into the picture. You did, in the second half of your first sentence: "Start simple: store the data in-memory in whatever way makes sense, and just write a log file that contains pickled transactions that can be played back." Indeed, an in-memory cache is really fast. But it's unfair to compare it with an RDBMS, which is handicapped by the need to write every transaction to disk before it can be committed. The hash table is great only up to the point where a stray cosmic ray crashes the server and makes the whole thing disappear, after which you realize that you need a log file. In the general case, writing a transaction log file is a hard problem. If you write a really robust tool for managing that log file - a tool which is efficient at reading and writing even when the number of transactions grows large; one that lets you specify when the log gets written, and how often, and whether the system can be queried during the write, and how long those queries will block, and what they will return; one which allows multiple threads and multiple machines to read and write the data without concurrency problems; one which prevents the in-memory cache from getting out of sync with the filesystem - you will have implemented a substantial portion of MySQL, and probably memcached as well. It is easy to start out designing a fast, simple, non-transactional DB and end up reinventing MySQL. If you don't believe me, ask the folks who invented MySQL! In certain special cases (e.g. Google), rolling your own persistent storage system is a big, big win. You may know, in advance, that your website is one of those cases. If you are correct, you will be a superstar - you will build a relatively untested, nonstandard data storage system with a tiny subset of PostgreSQL's features, but all of that will be worth it because the system will be fast. If you are wrong, you will work for a month or six and then end up installing PostgreSQL anyway. In fact, even if you are right, you will end up installing PostgreSQL after your customer changes the spec at the last minute to require some boring standard feature - like a shopping cart - which any CRUD jockey can build in a day but which your "simple and elegant" database doesn't support because it wasn't in the original spec. So it's no surprise that the "de facto standard" is to build your site around an RDBMS, get the damn thing working, and optimize later - and that, as a result, the typical in-memory data structure ends up being backed by an RDBMS instead of by a "simple" log file. As for your ire at SQL... if you think you're in pain, just imagine how the designers of SQL must have felt back in the 1970s, when string parsing and query planning were tens of thousands of times slower than they are now, an extra database server cost more than a coder's daily salary, and RDBMS software was very, very non-free. It was a dark time. And yet for some reason those guys abandoned their efficient hand-rolled binary databases for SQL. In fact, they did it so fast that Larry Ellison grew richer than the Beatles. Why did they do that, I wonder? It must have been the drugs.