11 ms·

The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://

by nemild 8y ago

The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://www.mongodb.com/customers/guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.slideshare.net/tackers/why-we-chose-mongodb-for-guardiancouk https://www.slideshare.net/tackers/why-we-chose-mongodb-for-... And reupping my previous, three-part series on MongoDB: On MongoDB NoSQL databases were the future. MongoDB was the database for "modern" web engineers and used by countless startups. What happened? https://www.nemil.com/mongo/index.html https://www.nemil.com/mongo/index.html

slaymaker1907 8y ago
People realized a lot of the claimed not only SQL offerings were actually no SQL at all. It turns out it is nice to have those extra features of NoSQL as well as a more traditional RMDS instead of just the NoSQL parts.
- WorldMaker 8y ago
  Also, the RDBMS world is full of some of the oldest scaling experts in the yellow pages. It's interesting how surprised people were that many of the "traditional" RDBMS were able to catch up on some of the scaling support that were the biggest advertised advantages of NoSQL. It's interesting because I feel like the NoSQL world spent a lot of time reinventing the RDBMS from the "opposite direction". For all its faults, SQL is a fascinating language because it mostly ignores low level details of how the database scales, how it operates under the hood. It's not that SQL is intrinsically hard to scale (certainly at the relational algebra roots it shouldn't be hard, in theory), but it certainly leaves a lot of work to anyone building a database engine to figure out what/when/why/how to scale. I feel like a lot of RDBMS' query analyzers/planners resemble things like HBase a lot more than folks realize. It's great that NoSQL realized that sometimes those "low level details" in an SQL engine are useful in their own ways, and have increased the spectrum of performance versus power/flexibility trade-off options. But it shouldn't be that big of a surprise that SQL databases remain competitive in that trade-off space, given under the hood they've had to think about a lot of that stuff over many decades.
pjmlp 8y ago
Fashion driven development and not understanding how to actually make use of SQL.
- bunderbunder 8y ago
  It's amazing how many insurmountable SQL performance problems can be surmounted by putting things in 3rd normal form.
  - michelpp 8y ago
    That, and simply reading the documentation. N+1 problems and denormalization-by-design are a failure to understand SQL.
    - deleted 8y ago
      [deleted]
  - Scarbutt 8y ago
    But denormalization is one strategy to improve the performance of a rdbms.
    - bunderbunder 8y ago
      Standard rules about performance optimization apply. Denormalization can improve performance, but it can just as easily harm it. For code, I think by now we all understand that you should always start with clean, well-factored code, and then optimize only as much as is necessary, which is usually not at all, and always under profiler guidance. It's the same with DBs: You start with a clean, well-normalized schema, and then de-normalize only as much as is necessary, which is usually not at all, and always under profiler guidance. Also, keep in mind that improvements in compiler technology over time mean that the performance tricks of old can be useless or even actively harmful nowadays. This is true of SQL every bit as much as C.
    - trhway 8y ago
      Denormalization as way of dealing with performance issues is like guilottin to cure headache. And more frequently than not the headache is still there even after that. It is true that for some narrow class of analytical workloads 20-25 years ago (behold BW of 199x) the denormalized case performance was better compare with straight non-optimized running of the same queries over normalized schema. Since then, the exponential availability of RAM and huge increase in streaming speed of HDD with stagnating IOPS (the main mistake in analytical workloads on HDD in the last 10-15 years - using nested loop join with indexed lookup into the large facts tables :) have made denormalization obsolete and harmful. If anything, the emergence of SSD and the huge RAMs moved things even further toward and beyond normalization, by making the "super-normalization", i.e. columnar tables, a viable everyday thing.
lmilcin 8y ago
What really happened is people wanted something new but did not want to change the ways they DESIGN their application and processes. There is place for NoSQL but if you are going to use it as if it was SQL then you would be better served by an SQL database. Also, I think MongoDB tried to be everything and failed to be good at anything. It offers neither stellar performance nor scalability and I guess for most projects there is not much advantage over regular SQL database. Certainly nothing to fight over when there is much more technology choices to make.
- brobdingnagians 8y ago
  Just a few thoughts. A lot of little companies like picking hyped tools because they think it will differentiate them and server them well later, but most companies never get to that stage and don't really need what is being offered. Seems much safer to take the tried and true standard solution that has worked for decades and just make your UX outstanding rather than try to put together a "dream team" of new tech...
  - lmilcin 8y ago
    Another thought. Most companies have absolutely no idea how to select the technology. A lot of people that are in position to decide don't have all that much hands on experience with the new technology and so they will decide based on other factors like whether his manager would like or would like not see new technology, what other companies are doing, etc. Another example is "Agile". Everybody is doing it yet I still wait to see a single company that understands what the term means. My current boss is big promoter of "Agile" which in his language is synonym to "Scrum". Yet when asked he has never heard of Pheonix Project, The Goal, theory of constrains or basically any theory at all. So what the people are doing is fighting fires almost 100% of the time with not much project work done for the effort and absolutely no improvements. Yet, because everybody complies to do daily standups and Jira updates we are 100% agile. Don't even get me started on devops...
csytan 8y ago
I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also aimed at younger developers. When was the last time you saw a Postgres rep at a college tech event?
- nemild 8y ago
  I think you'll enjoy the series then, I spent several months investigating and made the same point about JSON and the Javascript-like CLI (plus great Node support, plus savvy marketing). For example: > 10gen's key contributions to databases — and to our industry — was their laser focus on four critical things: onboarding, usability, libraries and support. For startup teams, these were important factors in choosing MongoDB — and a key reason for its powerful word of mouth. Startup Engineers and Our Mistakes with MongoDB https://www.nemil.com/mongo/2.html https://www.nemil.com/mongo/2.html The Marketing Behind MongoDB https://www.nemil.com/mongo/3.html https://www.nemil.com/mongo/3.html
  - RobertKerans 8y ago
    Halfway into part two, this is very good so far. Thank you for the effort you put in (it really does show throughout).
- acjohnson55 8y ago
  > When was the last time you saw a Postgres rep at a college tech event? Is a Postgres rep a thing?
  - WJW 8y ago
    That's the point.
  - twirlip 8y ago
    I remember a hyperbolic readme or other such txt file for Postgres in the far-away long-ago time when everyone was on Slashdot. The author had written one of the most enthusiastic lovenotes to software I'd ever read, and that includes Stephenson's "In The Beginning Was The Commandline." It was a Thomas Wolfe level of ejaculatory keenness. I'd love to read it again if anyone else knows where I can find the file. So, even if there aren't actual Postgres reps, there are most assuredly evangelists.
orblivion 8y ago
I found it a little funny that NoSQL started becoming popular during at least some of the same years that static typing starting becoming popular (again).
- acdha 8y ago
  I've noticed that, too. There's an interesting ping-pong effect where a fair number people have flipped from strong typing at the database layer & dynamic typing at the view layer to the reverse — it seems like someone could write an interesting group psychology paper about how that cycle has repeated over the years.
  - dep_b 8y ago
    I don't think I'm a Luddite but I always had the feeling I just had to sit out the noSQL / JS everywhere storm.
- acjohnson55 8y ago
  I don't see it that way -- I feel like NoSQL's rise was more or less coincident with the adoption of Rails, Django, and Node over Java. The surge in interest in new static languages has mirrored the resurgence of Postgres, right around version 9.4 (and JSONB).
  - orblivion 8y ago
    Could be. I haven't thought too deeply about what happened when.
- bonesss 8y ago
  I think they're both responses to the same challenges: distributed web enabled applications. On the server front that means ever increasing complexity with decoupled microservices and latency issues that play nicely with the classic approaches to those domains (static typing, functional programming). On the data front sites like HN, Reddit, or Facebook need scability more than consistency, and have oodles of 'uninteresting' data that jives nicely with a schemaless document store.
h1d 8y ago
Even large organization like Guardian can have some cheep tech team who rush for such a poor choice? Also dumb for management to let it happen.
kevin_thibedeau 8y ago
> What happened? Text and still imagery based media companies aren't big data providers. NoSql was a bad choice from the start.
dbrgn 8y ago
Nice writeup. I love this quote: "The first few times an engineer sees this kind of hype, they often think it's a structural shift. For engineers later in our career, we’ll often dismiss structural shifts as misplaced hype after getting burned too many times"
brianmcc 8y ago
A great series - thanks! Required reading for anyone preparing to "take a punt" at a technology they've not really done much actual work with.

by nemild 8y ago

slaymaker1907 8y ago

People realized a lot of the claimed not only SQL offerings were actually no SQL at all. It turns out it is nice to have those extra features of NoSQL as well as a more traditional RMDS instead of just the NoSQL parts.

WorldMaker 8y ago
Also, the RDBMS world is full of some of the oldest scaling experts in the yellow pages. It's interesting how surprised people were that many of the "traditional" RDBMS were able to catch up on some of the scaling support that were the biggest advertised advantages of NoSQL. It's interesting because I feel like the NoSQL world spent a lot of time reinventing the RDBMS from the "opposite direction". For all its faults, SQL is a fascinating language because it mostly ignores low level details of how the database scales, how it operates under the hood. It's not that SQL is intrinsically hard to scale (certainly at the relational algebra roots it shouldn't be hard, in theory), but it certainly leaves a lot of work to anyone building a database engine to figure out what/when/why/how to scale. I feel like a lot of RDBMS' query analyzers/planners resemble things like HBase a lot more than folks realize. It's great that NoSQL realized that sometimes those "low level details" in an SQL engine are useful in their own ways, and have increased the spectrum of performance versus power/flexibility trade-off options. But it shouldn't be that big of a surprise that SQL databases remain competitive in that trade-off space, given under the hood they've had to think about a lot of that stuff over many decades.

pjmlp 8y ago

Fashion driven development and not understanding how to actually make use of SQL.

bunderbunder 8y ago
It's amazing how many insurmountable SQL performance problems can be surmounted by putting things in 3rd normal form.
- michelpp 8y ago
  That, and simply reading the documentation. N+1 problems and denormalization-by-design are a failure to understand SQL.
  - deleted 8y ago
    [deleted]
- Scarbutt 8y ago
  But denormalization is one strategy to improve the performance of a rdbms.
  - bunderbunder 8y ago
    Standard rules about performance optimization apply. Denormalization can improve performance, but it can just as easily harm it. For code, I think by now we all understand that you should always start with clean, well-factored code, and then optimize only as much as is necessary, which is usually not at all, and always under profiler guidance. It's the same with DBs: You start with a clean, well-normalized schema, and then de-normalize only as much as is necessary, which is usually not at all, and always under profiler guidance. Also, keep in mind that improvements in compiler technology over time mean that the performance tricks of old can be useless or even actively harmful nowadays. This is true of SQL every bit as much as C.
  - trhway 8y ago
    Denormalization as way of dealing with performance issues is like guilottin to cure headache. And more frequently than not the headache is still there even after that. It is true that for some narrow class of analytical workloads 20-25 years ago (behold BW of 199x) the denormalized case performance was better compare with straight non-optimized running of the same queries over normalized schema. Since then, the exponential availability of RAM and huge increase in streaming speed of HDD with stagnating IOPS (the main mistake in analytical workloads on HDD in the last 10-15 years - using nested loop join with indexed lookup into the large facts tables :) have made denormalization obsolete and harmful. If anything, the emergence of SSD and the huge RAMs moved things even further toward and beyond normalization, by making the "super-normalization", i.e. columnar tables, a viable everyday thing.

lmilcin 8y ago

What really happened is people wanted something new but did not want to change the ways they DESIGN their application and processes. There is place for NoSQL but if you are going to use it as if it was SQL then you would be better served by an SQL database. Also, I think MongoDB tried to be everything and failed to be good at anything. It offers neither stellar performance nor scalability and I guess for most projects there is not much advantage over regular SQL database. Certainly nothing to fight over when there is much more technology choices to make.

brobdingnagians 8y ago
Just a few thoughts. A lot of little companies like picking hyped tools because they think it will differentiate them and server them well later, but most companies never get to that stage and don't really need what is being offered. Seems much safer to take the tried and true standard solution that has worked for decades and just make your UX outstanding rather than try to put together a "dream team" of new tech...
- lmilcin 8y ago
  Another thought. Most companies have absolutely no idea how to select the technology. A lot of people that are in position to decide don't have all that much hands on experience with the new technology and so they will decide based on other factors like whether his manager would like or would like not see new technology, what other companies are doing, etc. Another example is "Agile". Everybody is doing it yet I still wait to see a single company that understands what the term means. My current boss is big promoter of "Agile" which in his language is synonym to "Scrum". Yet when asked he has never heard of Pheonix Project, The Goal, theory of constrains or basically any theory at all. So what the people are doing is fighting fires almost 100% of the time with not much project work done for the effort and absolutely no improvements. Yet, because everybody complies to do daily standups and Jira updates we are 100% agile. Don't even get me started on devops...

csytan 8y ago

I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also aimed at younger developers. When was the last time you saw a Postgres rep at a college tech event?

nemild 8y ago
I think you'll enjoy the series then, I spent several months investigating and made the same point about JSON and the Javascript-like CLI (plus great Node support, plus savvy marketing). For example: > 10gen's key contributions to databases — and to our industry — was their laser focus on four critical things: onboarding, usability, libraries and support. For startup teams, these were important factors in choosing MongoDB — and a key reason for its powerful word of mouth. Startup Engineers and Our Mistakes with MongoDB https://www.nemil.com/mongo/2.html https://www.nemil.com/mongo/2.html The Marketing Behind MongoDB https://www.nemil.com/mongo/3.html https://www.nemil.com/mongo/3.html
- RobertKerans 8y ago
  Halfway into part two, this is very good so far. Thank you for the effort you put in (it really does show throughout).
acjohnson55 8y ago
> When was the last time you saw a Postgres rep at a college tech event? Is a Postgres rep a thing?
- WJW 8y ago
  That's the point.
- twirlip 8y ago
  I remember a hyperbolic readme or other such txt file for Postgres in the far-away long-ago time when everyone was on Slashdot. The author had written one of the most enthusiastic lovenotes to software I'd ever read, and that includes Stephenson's "In The Beginning Was The Commandline." It was a Thomas Wolfe level of ejaculatory keenness. I'd love to read it again if anyone else knows where I can find the file. So, even if there aren't actual Postgres reps, there are most assuredly evangelists.

orblivion 8y ago

I found it a little funny that NoSQL started becoming popular during at least some of the same years that static typing starting becoming popular (again).

acdha 8y ago
I've noticed that, too. There's an interesting ping-pong effect where a fair number people have flipped from strong typing at the database layer & dynamic typing at the view layer to the reverse — it seems like someone could write an interesting group psychology paper about how that cycle has repeated over the years.
- dep_b 8y ago
  I don't think I'm a Luddite but I always had the feeling I just had to sit out the noSQL / JS everywhere storm.
acjohnson55 8y ago
I don't see it that way -- I feel like NoSQL's rise was more or less coincident with the adoption of Rails, Django, and Node over Java. The surge in interest in new static languages has mirrored the resurgence of Postgres, right around version 9.4 (and JSONB).
- orblivion 8y ago
  Could be. I haven't thought too deeply about what happened when.
bonesss 8y ago
I think they're both responses to the same challenges: distributed web enabled applications. On the server front that means ever increasing complexity with decoupled microservices and latency issues that play nicely with the classic approaches to those domains (static typing, functional programming). On the data front sites like HN, Reddit, or Facebook need scability more than consistency, and have oodles of 'uninteresting' data that jives nicely with a schemaless document store.

h1d 8y ago

Even large organization like Guardian can have some cheep tech team who rush for such a poor choice? Also dumb for management to let it happen.

kevin_thibedeau 8y ago

> What happened? Text and still imagery based media companies aren't big data providers. NoSql was a bad choice from the start.

dbrgn 8y ago

Nice writeup. I love this quote: "The first few times an engineer sees this kind of hype, they often think it's a structural shift. For engineers later in our career, we’ll often dismiss structural shifts as misplaced hype after getting burned too many times"

brianmcc 8y ago

A great series - thanks! Required reading for anyone preparing to "take a punt" at a technology they've not really done much actual work with.