7 ms·

The best advice I can give is that you can think of your app and its data store as either: #1 code is of prime importance, data store is simply a "bucket" for

• Dave3of5 5y ago
  Or maybe it's shades of grey in between ...
  • drittich 5y ago
    Wend hence. Thee wilt picketh a side.
• oblio 5y ago
  You've just torn apart most NoSQL databases in a very short comment :-))
  • LAC-Tech 5y ago
    You're familiar with every NoSQL database? Impressive!
    • oblio 5y ago
      There you go, I've edited my previous comment to account for pedantry.
      • LAC-Tech 5y ago
        I guess I should just accept the term that "NoSQL" now pretty much means "Monngo and Dynamo". Because it feels weird to group key-value stores, columnar stores and graph DBs under one term.
        • oblio 5y ago
          Well, I don't want to be super mean to you or NoSQL, but NoSQL itself was a bit of a fad. Columnar stores and graph DBs have always felt more like "real" tech, something promoted for specific use cases. Key-value stores I'm not even going to be denigrating at all since we've had them since forever and they do a great job at their tasks (BerkeleyDB, Memcache, Redis, etc.). They don't deserve being included into fads or niches, they're a very valuable universal resource. At least that's how I see it.
          • LAC-Tech 5y ago
            Yeah but in my mind columnar stores and graph stores are NoSQL databases. Because they're not relational. But the term has shifted underneath me. NoSQL means document database. And - in practice - it means using a document database as the primary store when it's not really the right choice (been there!)
  • neuronic 5y ago
    So this is just an observation and not a judgment. With only 4 years of professional experience I have never worked on a MongoDB project in which MongoDB somehow wasn't an issue. The proposed solutions merely being "improve the indices" or "scale the cluster", often without clearly defining what's going on. So granted I have a very limited experience, it's baffling to me mainly in comparison to the PostgreSQL projects I worked on. They definitely also had problems but these problems were clearly defined even if resolution wasn't quick or easy. It was usually an out-of-date or generally messy schema that was causing issues and folks usually were able to clearly define the schema problems.
    • emptysongglass 5y ago
      I hear this about MongoDB (that it's a plague and you should stay far away) all the time but DynamoDB, which is also NoSQL? One of AWS' finest products; just launching an app on AWS? Use DynamoDB! So on and so forth ad nauseum. Why is DynamoDB the bee's knees but MongoDB is a thing to be despised?
      • oblio 5y ago
        1. One has a better marketing machine than the other? 2. DynamoDB is pushed in slightly more balanced ways than Mongo was, at the start Mongo was supposed to be the second coming of the (database) Messiah. 3. I don't believe DynamoDB has defaults that lose your data.
        • to11mtm 5y ago
          Dynamo seems nice for some very specific use cases, but there are others where it falls short. There's limits on how big your data can be, which is a little annoying if you want to use it to store a couple larger things alongside all your small data. The C# API is absolutely painful IMO, and makes it easy for new developers to use it in a way that you would have been better off grabbing a different technology. I suppose one could argue S3 would be the better choice for large payloads, OTOH I'm a fan of minimizing potential points of failure so if we already are using DynamoDB I'd rather not toss in an additional S3 integration that could break or need maintenance later.
  • cle 5y ago
    NoSQL databases are mostly useful for performance reasons. You can have the most beautifully modeled relational database schema but if it performs/scales outside your tolerance bounds, then it's useless. IMO, this is the main reason we have so many database technologies: they make important performance tradeoffs.
    • oblio 5y ago
      True, but you really, really want to be sure that if you're going the NoSQL route, your manual schema management will be top notch or your data structures will be super simple and the query demands will also be super simple. Most of these high performance use cases actually fit this model well (well, the schema management being awesome is the wildly varying part). But for everything else, there's Mastercard and RDBMS.
      • cle 5y ago
        Yes, you should think about your domain and requirements, and the technical tradeoffs, before picking a DB technology. Don't just run with RDBMS b/c it has better theoretical modeling capabilities, your customers don't care about that. For enterprise stuff with relatively low traffic and high amounts of complex ad-hoc queries, RDBMS is without a doubt the best choice. If you have high traffic web services with strict availability and latency requirements, I would seriously consider avoiding RDBMS's as they tend to be difficult to operate and scale with those requirements, and they let you easily do terrible things in that context (e.g. locking behavior, txid exhaustion, etc.).
    • EMM_386 5y ago
      It isn't that difficult to optimize relational SQL databases. I've dealt with databases with billions of rows in hundreds of tables and the company's initial thought was to flatten the schema due to perf. complaints. We got queries down from 30+ seconds to 5 ms simply by properly indexing, defragmentation, analyzing query plans, SQL Stored Procedures, etc. I see a lot of complaints from developers claiming they have to join a "million row table to million row table" and reports are slow, and this gets blamed on the DB. These should not be slow outside of how much bandwidth is being pushed over the wire, which is often exactly what the problem is. They just didn't see it.
  • skeeter2020 5y ago
    Don't worry; for every misuse of a NoSQL database when they should have used a relational DB there's a solution misusing a relational DB for a queue, event bus, etc. Nature likes a balance!
    • kreeben 5y ago
      Would you remind a dumb-wit like myself why that's a problem, when the API of a relational db is virtually perfect for a queue or event bus? Push, pop, look-but-don't-touch, data available through the most scrutinized and battle-tested query language ever, what else do you need?
      • bshipp 5y ago
        until Postgres added JSON fields I would have retorted that nosql is really handy for scraping purposes, when you just want to grab and store the collected data, which you then parse with another specific program into a relational database later. I've learned this by doing it the opposite way for years. however, website schemas can frequently change, which ruins the database schema if you're immediately parsing into a normalized structure. In other words, it's brittle. noSQL (or JSON fields) allow one to store unstructured data which is more forgiving when a schema changes (I.e. change the spelling of a dictionary key, etc.).
        • oblio 5y ago
          Well, technically you could have done this since forever in an RDBMS. Key value table, have the table be a blob (or just varchar if a string). Then yes, you'd have to parse it yourself, but you'd be doing that anyway with JSON fields, more or less.
          • bshipp 5y ago
            Indeed, you are correct. A lot of these lessons came hard-earned from doing it the wrong way first. I thought I was being such a good little programmer by normalizing my data as soon as possible. All it did was provide me job security (at the cost of headaches) every time the schema changed.
• gpderetta 5y ago
  "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975)
  • gwd 5y ago
    When reviewing patches, I always look at the changes to .h files first, to see how the data structures are modified. Frequently you can predict exactly what the code changes are going to look like as a result.
    • 10000truths 5y ago
      I imagine that this doesn’t work as well with PIMPL code.
  • mannykannot 5y ago
    While I am completely in agreement with the intent of this aphorism, and also what the article says about the relational model, I feel this is (as aphorisms tend to be) something of an exaggeration. I think, for example, that it would be difficult to deduce the rules of cricket from its scoresheets. Tell me your tables' semantics, however, and I think one could do much better. .. I suspect that, when Brooks made that comment, he was probably thinking mostly in terms of programs for entering and reporting on that data. We're doing a lot more now.
    • gpderetta 5y ago
      Well, the scoresheet is not enough of course, you need all the "datastructures" (field design, player number and layout, equipment, etc). edit: but yes, it can be a bit of an exaggeration. Figuring out all the red-black tree rules from just the structure definitions, relations and and invariants is probably a bit of a stretch for most.
      • mannykannot 5y ago
• andy_ppp 5y ago
  Sure, I agree largely with two but writing tests for this logic can get difficult (pgTap looks alright but not amazing) as can making all the database logic scale as you grow to sharding and multiple DBs. I’m not sure you’d want this if you were scaling to hundreds of millions of users but then you have loads of good problems…
• mmcnl 5y ago
  I totally agree. Data lasts forever, code is volatile.
  • dotancohen 5y ago
    Data ages like wine. Code ages like fish.
    • SAI_Peregrinus 5y ago
      Fish can age well, there are some good aged fish sauces used in Japanese cuisine. Milk also ages well (cheese). Some subatomic particles might be a good choice. Pions decay in a few nanoseconds (for + & - charged) or attoseconds (for neutral). But "ages like Π⁰ particles" just doesn't have a nice ring to it.
      • dotancohen 5y ago
        > Fish can age well, there are some good aged fish sauces used in Japanese cuisine. So that would be TeX, the odd exception.
• naasking 5y ago
  > In my personal experience, #2 is a way better way to work. Apps can come and go, but data can last a long time Exactly. Data is far more valuable than code. We still work with data from literally centuries ago. Most programs written even 10 years ago are probably no longer in use. I would be surprised if the oldest code still doing something useful was more than 20 or so years old.
  • kwertyoowiyop 5y ago
    https://www.zdnet.com/article/the-10-oldest-significant-open-source-programs/ https://www.zdnet.com/article/the-10-oldest-significant-open... And that is just open-source.
    • jhgb 5y ago
      How did they miss GNU Maxima?
  • nchi3 5y ago
    > I would be surprised if the oldest code still doing something useful was more than 20 or so years old. ...you can't be serious? Every other company that has been writing code for 20+ years probably has code that is 20+ years old. Just look at banks.
    • astura 5y ago
      I could see how someone would get that impression If they've only worked at companies who only exist solely to light VC cash on fire. It's an incorrect impression, obviously. For >20 year old code, Calc.exe is just one example off the top of my head - https://github.com/Microsoft/calculator https://github.com/Microsoft/calculator The majority of the products I've personally worked on over my 15 year career in software development are still doing useful things and contain code written more than 20 years ago.
      • naasking 5y ago
        Guys, you're missing the point. The scenario being discussed are the set of programs that collect large amounts of data, not about programs like Calc.exe or Apache that don't do anything like that. That's the whole point of asking whether data or code is of prime importance. Banks are probably one of the few examples of very old COBOL code that's still running (although they've been trying to get rid of it for years), which is the exception that proves the rule. Regardless, if you approached bank executives and said, "we've compromised all of your systems and we're going to delete either all of your code or all of your data, you pick", which one do you seriously think they're going to pick?
• forinti 5y ago
  One awful thing I've seen a few times is people who put data into a database in a format meant to make it easier for working with a particular platform, such as serialized Java dates (or even dates as longs). It completely ties you to one implementation and precludes using the data for other things such as reporting and alerts or even ad hoc queries.
  • grogers 5y ago
    You can always convert in the DB when selecting if you want the date. Dates/times in general are kind of horrible in all DBs and programming languages so I don't think using unix seconds (or milliseconds, etc) is obviously wrong. It's kind of like storing strings as plain bytes in the DB, it's not what you'd do by default but it's not completely wrong.
    • bshipp 5y ago
      I tend to agree that dates and times are awful to work with but storing everything as a Unix timestamp still seems to be the easiest way to avoid most of the headaches, in my experience. there's nothing worse than trying to reconcile a database filled with naive dates with no associated time zone information. is it daylight savings? is it in this timezone or did they enter it while traveling and their computer automatically updated the time? The simplest way to minimize these issues, in my experience, is to put the logic for converting local time into UTC in the program and only store the Unix stamps in the DB.
  • AtlasBarfed 5y ago
    So you don't like serialized java dates, which presumably are some standard at least. You don't like longs / epoch. So..... what do you like? Date formatting/conversion/serialization is not standardized across databases. And it's generally hair pulling (at least the Oracle version was last I checked which was a long time ago). Longs or ISO-8601 strings? No DB-specific crud. Yes please.
    • forinti 5y ago
      At least you can compare longs and index them. That's not too bad. But I prefer to just use the native date type; it allows you to use the built-in date functions.