9 ms·
I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don't actually fit t
by pavlov 12y ago
I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don't actually fit the shape of their data. Persistence is fundamentally the programmer's responsibility; a magic box behind a socket can't design it for you.
Most applications I've seen wouldn't even need a database, but apparently a lot of programmers are conditioned into believing that writing anything to disk must involve building a database query string and transmitting it over a socket to another process which parses the string, executes it on an interpreter and stuffs the extracted data into a generic 1970s data model that finally gets written to disk in an opaque format from where it can only be retrieved by sending more strings over sockets. This stuff made sense when 1MB was a huge amount of memory, but today it's just not necessary.
- bbcbasic 12y agoHmm. You are not thinking in business terms. You run a software house: do you want your developers reinventing data storage on each application? Or using a fairly decent data storage that is RDBMS. Most of the time RDBMS is a very good choice. Think about the tooling, support, hiring knowledgeable people etc. Lets face it RDBMS are good at the very small (single table, replacing a text file) up to the very large. In the RDBMS/SQL world you have everything from sqllite to save as a single file, to Oracle for your Enterprise app.
- sklogic 12y agoMost of the time a plain text file is good enough. Databases are overrated.
- ams6110 12y agoThis is simply not true, otherwise we would all be using text files. Text files can work in some situations. I disagree that this is "most of the time."
- sklogic 12y agoWe are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.
- duaneb 12y agoThe file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?
- sklogic 12y agoBecause all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.
- superuser2 12y agoIt depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?
- sklogic 12y ago> It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly? > Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own? Parsing?!? In ORMs and DBMSes?!? C'mon, try to parse me some C++ with your Oracle.
- 12y ago
- edraferi 12y agoJWZ wrote a great article about choosing between flat files and a database for Netscape Mail: http://www.jwz.org/doc/mailsum.html http://www.jwz.org/doc/mailsum.html In Netscape 4.0, the new team went both C++ and Database happy, threw away the tightly-tuned mail summary files I had designed, and generally screwed the pooch raw. My code had summary files that were on the order of 2% of the size of the folder they were summarizing, and was blazingly fast in all respects. The 4.0 code had an overhead closer to 30% (last time I checked) and was insanely slow, not to mention extremely fragile: their summary files got corrupted all the time.
- threeseed 12y agoI really question your experience with databases. Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different. I do agree that SQLite is an excellent choice for most small applications.
- bbcbasic 12y ago"I really question your experience with databases. " I have used SQL Server, Oracle, MySQL, SQLite, MS Access(!), Sybase and ... CouchDB! I have optimized queries in SQL Server and Oracle. Most of them commercially (not CouchDB). "Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different." Spot on. Not sure how it relates to my post.
- jbergens 12y agoBut if the storage is a storage library the developers wouldn't be reinventing it every time. And if a new technolgy came along it might be easier to start using it than to say that all applications in an organization must use the same (often relational) db. The latter seems to be very common. And the dba's don't seem to push the use of newer technology much, or look into how much time it would save the developers. Regarding tooling, you are right, it is a bit lacking with the newer database solutions but they seem to be working on it and I think it would help if everyone involved tried to help describe what kind of tooling is needed/wanted. Now a lot of discussions stops with saying the tooling isn't good enough and noone should ever use the new database solutions.
- arielby 12y agoFirst-Order Logic (and therefore SQL, which is based on it) sucks at describing graphs and highly heterogeneous structures. If you're working with these, you should have a better representation.
- rdtsc 12y agoI can kind of see where you're coming from. The problem is this -- if your developers are good enough to understand all the operating system nuances of handling IO -- (how does dirty page writeback works, buffer cache vs page cache, opening file with O_DIRECT, what does append only mode do exactly to the file, how do you use append-only mode effectivelt, what operations are atomic on the file system, how does mmap function work, what about fadvise, etc. etc.), they are probably smart enough to not use the database and write their own storage layer, OR they are also smart enough to read through the fine print and pick a good database. If they don't know those things, they will want to reach and pick a database off the shelf, so to speak. They might pick a bad database. Because they might not know how to read the fine print, how to benchmark, how to benchmark failure, how to not read into the marketing bullshit "But it said Webscale on the front page, how come it ate my data, I don't understand...". Unfortunately quite often databases are treated like black boxes and they are chosen not necessarily becuase of technical merits. Sometimes it is stupid marketing sometimes it is golfware ("CEO spent time playing golf with an Oracle saleman. Salesman lost just enough, but in the end everyone is forced to use Oracle from now on).
- sebcat 12y agoI agree. These days, I tend to approach a persistency implementation by asking myself: * will it be read or write heavy? * by what data/key will I access the persistent data? Most commonly for the type of work that I do, it's write once - read N number of times. What I end up doing then is just to keep the data in-mem in a hash table while writing it async to disk. Concurrent reads hit the hash table while the disk write is being performed. The file(s) are indexed by using a hash of the data itself, multiple ways to to lookups can also be done by using softlinks. Hashing lookup-keys are almost always prefered because of possible locale/security reasons (makes it harder for you to implement an accidental LFI if you only deal with base36) If needed and/or depending on access patterns, I also keep the N most accessed items in a hash table in-mem over time. The API for this usually ends up as two functions, Get and Put. The amount of code for doing this is a couple of hundred lines at most and it's complexity I've found by experience to be preferable to using a db for the type of work that I do. YMMV.
- pavlov 12y agoThat's my usual approach as well. Usually an app's data will easily fit in memory within one process (and if it doesn't, that means the app has become so popular that adding an extra X GB of memory is an insignificant cost). Using the language's native data types and an append-only file for persistence is simple and offers high performance compared to doing all access over a socket.
- Pxtl 12y agoWhile I'd love to be done with passing strings around, still fundamentally the relational model is a great way to store and fetch data, and ACID guarantees are something a my software should provide, soi use tools that provide ACID guarantees to me. I'm down with throwing out SQL, but don't toss the baby with the bathwater. ACID + relational are great!
- jeffdavis 12y agoRDBMS have completely dominated in a way that few other technologies have. Especially for anything that might be described as business data (which is the prime value market for databases). Everything else is relegated to share a small niche. If you count SQLite as an RDBMS, the domination is even more complete. So, my question to you is: why? It can't be some kind of vendor lock-in, because there is no one vendor; there are several commercial vendors and open source projects available to choose from (each individual vendor might lock in their users, but that doesn't explain why they are still all RDBMSs and not something "better"). It can't be legacy, because SQL is as popular as ever; perhaps more so, as it seems that using a normal filesystem to store data is not very common any more. So you need to explain why they are so popular yet so wrong; otherwise it just sounds like a case of "everyone is dumb except me".