28 ms·

Bye Bye Mongo, Hello Postgres

• kstrauser 8y ago
  As a reminder, PostgreSQL is distributable under a license that's basically BSD/MIT: https://www.postgresql.org/about/licence/ https://www.postgresql.org/about/licence/ MongoDB, on the other hand, recently changed their license to an abomination that many people think is no longer Free: https://news.ycombinator.com/item?id=18301116 https://news.ycombinator.com/item?id=18301116
  • rusk 8y ago
    For an enterprise scenario where the costs of remaining outweigh the costs of migrating this seems to me the clearest reason for doing so, and also likely why they skimped on their rationale ... in all likelihood they appeared to be nice juicy whale but turned out to be some form of shark.
  • Thaxll 8y ago
    Because there is no company behind PG, no one is loosing money if FANG get your app for free and build something around it. You should ask Redis, Grafana, Nginx ect ... if they're happy about super large company doing that kind of thing.
    • snaky 8y ago
      There are bunch of companies behind PG actually. From Fujitsu and EnterpriseDB to smaller ones like 2ndQuadrant, PostgresPro, CitusDB and many others.
      • Thaxll 8y ago
        Those companies did not create PG, the company behind Mongo did and does not want Google / AWS / MS to make a service using Mongo for free.
        • snaky 8y ago
          That's great there is no one single company behind PostgreSQL in this sense. The model when many companies around PG do their own niche things based on PG, having their own business models, cooperating and collaborating to make PG better, as it is the base of their business, is rather fruitful and stable in the long term.
        • kstrauser 8y ago
          I'm glad that RedHat didn't feel that way when they were sponsoring kernel development, or that Netscape had panicked and locked down Netscape instead of opening it up, or that Sun had given Java a terrible license. Or Google and Kubernetes. LinkedIn and Kafka. Airbnb and Air Flow. I could keep going like this all day. Mongo made a lot of money building a projects that runs on top of many other projects that were released as Free Software. Now they're upset that other people are building on top of Mongo in the same way.
          • Thaxll 8y ago
            So Redis, HAproxy, MongoDB, ect .. should feel good that nowdays multi-billion $ companies take your product and put more people on it that you do internally, sell it and give you nothing? Explain to me how a startup of 10-20 people can compete against AWS once they grab what you're working on to make an AWS service? Changes Mongo, Redis made to their licence were made to protected against those practices.
            • hnbroseph 8y ago
              thanks for the note about redis... sounds like i should avoid any features that have been locked down (and consider alternatives).
            • pritambaral 8y ago
              1. Redis — according to its creator — has always been, is, and will continue to be FOSS. Redis Labs did not create Redis, nor can they re-license it. 2. HAproxy is still GPLv2. 3. Redis Labs's CCL and MongoDB's SSPL are not open source licenses, but their purveyors sure do like to give off the impression that they are open source. If you want to keep your code proprietary, keep it proprietary. Don't pretend to be open source. If you are not okay with others using your work, even making money off of it, as long they adhere to the rules of the open source licence you used to license your work to them, then don't license your work to them under open source licenses, or don't cry foul when they use it under the terms of the license.
  • kstrauser 8y ago
    Downvoters, would you explain? I didn't think this was a controversial opinion: Mongo is now under a license that is not listed as Open Source, as defined at https://opensource.org/licenses/alphabetical https://opensource.org/licenses/alphabetical If you care about using FOSS - as many companies do - MongoDB is no longer open for consideration.
• tekmaven 8y ago
  I’m not really sure they articulated why they had to switch very clearly. They didn’t like managing Mongo. They said they couldn’t use Mongo’s hosted solution BUT they switch to hosted Postgres. Why not just overcome the limitations preventing them from switching to hosted MongoDB?
  • lykr0n 8y ago
    It looks like they wanted to run Mongo inside their own Amazon account- under their direct control. AWS could do that with Postgres while MongoDB couldn't.
    • flatiron 8y ago
      they talked about NTP. which to me means Mongo on EC2 maybe?
      • lykr0n 8y ago
        That's what I understood they were doing and they didn't want to do that anymore.
        • wftglf 8y ago
          Yep that's what we were doing, and the management software (OpsManager) was also running on EC2 instances. We messed up the VPC configuration so that NTP didn't work on some of the instances - which unsurprisingly broke authentication between OpsManager and the db instances
    • scarface74 8y ago
      That makes me sad. I would love for AWS to have a hosted Mongo solution or some type of NoSQL database besides DynamoDB. Do people use ElasticSearch as a primary data store? In my limited experience implementations don’t treat it as the source of truth.
      • snuxoll 8y ago
• joemccall86 8y ago
  Interesting article. Our stack isn't using mongo, but reading these posts give me good ideas for how others manage their software lifecycle. I like their use of a pre-production environment as a step between QA/UAT and production. This allows for a clean trial run of anything new deployed to production, as well as greatly removing any deployment-day surprises.
  • konschubert 8y ago
    We call it our "canary". :D
• elevenworth 8y ago
  The creator of Envoy had some choice words about Mongo a few days ago. https://twitter.com/mattklein123/status/1074717204224999427 https://twitter.com/mattklein123/status/1074717204224999427 Plenty of people here like to dish on Mongo and the product seems to have been re-architected a few times since I used it seven years ago. By what metrics can we say the product is one worthy of passing a HN smell test? Passing Jepsen was seemingly not enough. https://www.mongodb.com/jepsen https://www.mongodb.com/jepsen
  • jimktrains2 8y ago
    Passing might not be the right way to look at it? > This interpretation hinges on interpreting successful sub-majority writes as not necessarily successful: rather, a successful response is merely a suggestion that the write has probably occurred, or might later occur, or perhaps will occur, be visible to some clients, then un-occur, or perhaps nothing will happen whatsoever. > We note that this remains MongoDB's default level of write safety. - http://jepsen.io/analyses/mongodb-3-6-4 http://jepsen.io/analyses/mongodb-3-6-4 2018-10-23
  • politician 8y ago
    Are you reading the detailed reports prepared by the Jepsen audit team, or the press release? Jepsen audits are public. The summary of the latest MongoDB report [1] follows, "In this Jepsen report, we will verify that MongoDB 3.6.4’s sharded clusters offer comparable safety to non-sharded deployments. We’ll also discuss MongoDB’s new support for causal consistency (CC) in version 3.6.4 and 4.0.0-rc1, and show that sessions prevent anomalies so long as user stick to majority reads and writes. However, with MongoDB’s default consistency levels, CC sessions fail to provide the claimed invariants." [1] http://jepsen.io/analyses/mongodb-3-6-4 http://jepsen.io/analyses/mongodb-3-6-4
  • ken 8y ago
    > the product seems to have been re-architected a few times since I used it seven years ago That's really not what I want to hear about a place I'd be storing data. Was it just a prototype back then? (When did it stop being a prototype?) Would the creators be comfortable using version 1 still? Actually, I think that could be a good test of database stability: would you be willing to run a 5 year old version of your own software?
• KaoruAoiShiho 8y ago
  They have some unique requirements to host within their own VPC. I just use mlab.
  • mikeryan 8y ago
    Define unique? My company does bespoke software development for large enterprises. If they're on AWS having all your services within a VPC you control is something of table stakes for a lot of large company engineering and security teams.
    • thawkins 8y ago
      We are the same, atlas is a non-starter
    • KaoruAoiShiho 8y ago
      Well, guess I'm mostly talking about startups.
      • user5994461 8y ago
        Same in startups. Everything is on AWS and gotta be in a VPC.
• SonnyWortzik 8y ago
  I never got around to using Mongo as my main doc db because it was incredibly hard to find a management tool. I now use json supported functions in SQL Server and do not have the need for a different type of database. SQL Server handles my small 'documents db' implementation with the infrastructure of a RDMS. Win win for me. To me Mongo just got popular by mistake way too early. It's like having a celebrity retweet your post because they liked what they saw at the time, exposing you to the world where everyone now thinks you have something important to say. Not surprisingly, you don't!
  • mey 8y ago
    I really like SQL Server, especially the dev environment, 1st and 3rd party management tools. Not sure if they have improved this, but replication was a limiting factor and the pricing changes a few years back pushed it down my preference list.
  • mmsimanga 8y ago
    I don't have any links to prove my point but I suspect we need to thank Mongo for pushing the traditional SQL databases to support JSON.
    • WorldMaker 8y ago
      I think the fact that the Cosmos DB "document db" API clones the MongoDB API, of all the available options, is a link in that chain.
    • ddebernardy 8y ago
      FWIW and insofar as I can recollect, JSON was supported by Postgres before Mongo was a thing, and Mongo was built based on Postgres.
      • anarazel 8y ago
        No, that's not correct: Initial json support in PG was added in early 2012 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5384a73f98d9829725186a7b65bf4f8adb3cfaf1 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... whereas mongodb was made public in 2009, according to wikipedia: https://en.wikipedia.org/wiki/MongoDB https://en.wikipedia.org/wiki/MongoDB I don't see how Mongo was built based on Postgres, either?
• CyanLite2 8y ago
  Seems like there's one of these articles every week. "Goodbye {Postgres|Mongo|MySQL}, Hello {MySQL|Mongo|Postgres}"
  • bdcravens 8y ago
    When was the last "Goodbye X, Hello Mongo" article?
    • StreamBright 8y ago
      Not sure, but there was Goodbye MySQL, Hello Postgres and Goodbye Postgres, Hello MySQL (same company did it actually) for sure. It is kind of weird why engineers are so obsessed with tools though.
      • bdcravens 8y ago
        > It is kind of weird why engineers are so obsessed with tools though. Would you say the same thing about programming languages? I was actually listening to the Full Stack Radio podcast, and the latest episode talked about the power of moving more into the database. What struck me as a strong idea was that we can't treat databases as equivalent - different database servers have different strengths (for example, some have better JSON support or time evaluation functions). I think it's particularly telling that you seem to be thinking of the database as a "tool", but if you're like most, you fight religious battles over your language of choice. In my experience, the database IS the application (and in many cases, the business) and is a far more important choice than the implementation language, the cloud provider, etc.
        • StreamBright 8y ago
          Look, how I see these things is that we have tools. Do you think that people working on fields would fight over which tractor brand is better? Probably not. Do you thinks that builders would fight over which hammer brand is better. I don't think so. These fights over minor differences are happening mostly in the software industry. Freud wrote a pretty good thesis about it. https://en.wikipedia.org/wiki/Narcissism_of_small_differences https://en.wikipedia.org/wiki/Narcissism_of_small_difference... Moving into databases is not something I understand. I worked the last 10 years as data engineer but I like to move out from databases a lot more than moving into them. :)
• beering 8y ago
  Did they ever publish the rationale for why they used MongoDB in the first place? They mention having 2.3 million content items. If we assume that they have 100 DB entries for each content item, that's still only 230 million DB items. In that case, was it important to run a sharded cluster vs a typical primary-secondary HA setup? (which they ended up switching to)
  • jfb 8y ago
    I wondered about that, yeah. That's just Not That Much Data; too, they're clearly able to sustain multiple-minute downtimes (the blog asserts as much). The original design reeks of over-engineering. Great blogpost, though; well written, which is always a surprise.
  • h1d 8y ago
    Considering they've been using it from 2011 or something, there can't be any good reason but a dumb choice by tech team or management bought Mongo's sales talk without good verification.
  • karmakaze 8y ago
    I was reading the article waiting to get to the part where they handle their shards can scale. It ended up just reading like a migration from one db to another. I'm guessing their original choice for MongoDB was more for the schemaless development flexibility rather than horizontal scaling capabilities. This seems to have worked well enough for them. DynamoDB seems like a natural fit coming from a document store and they did very well not to choose it. I think AWS is due for a managed NewSQL store comparable to Google Cloud Spanner or MS Cosmos DB.
• jpmoyn 8y ago
  We made a similar switch over a year ago and have not looked back since. The migration was tough but once the bandaid is totally off it feels good.
• nemild 8y ago
  The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://www.mongodb.com/customers/guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.slideshare.net/tackers/why-we-chose-mongodb-for-guardiancouk https://www.slideshare.net/tackers/why-we-chose-mongodb-for-... And reupping my previous, three-part series on MongoDB: On MongoDB NoSQL databases were the future. MongoDB was the database for "modern" web engineers and used by countless startups. What happened? https://www.nemil.com/mongo/index.html https://www.nemil.com/mongo/index.html
  • slaymaker1907 8y ago
    People realized a lot of the claimed not only SQL offerings were actually no SQL at all. It turns out it is nice to have those extra features of NoSQL as well as a more traditional RMDS instead of just the NoSQL parts.
    • WorldMaker 8y ago
      Also, the RDBMS world is full of some of the oldest scaling experts in the yellow pages. It's interesting how surprised people were that many of the "traditional" RDBMS were able to catch up on some of the scaling support that were the biggest advertised advantages of NoSQL. It's interesting because I feel like the NoSQL world spent a lot of time reinventing the RDBMS from the "opposite direction". For all its faults, SQL is a fascinating language because it mostly ignores low level details of how the database scales, how it operates under the hood. It's not that SQL is intrinsically hard to scale (certainly at the relational algebra roots it shouldn't be hard, in theory), but it certainly leaves a lot of work to anyone building a database engine to figure out what/when/why/how to scale. I feel like a lot of RDBMS' query analyzers/planners resemble things like HBase a lot more than folks realize. It's great that NoSQL realized that sometimes those "low level details" in an SQL engine are useful in their own ways, and have increased the spectrum of performance versus power/flexibility trade-off options. But it shouldn't be that big of a surprise that SQL databases remain competitive in that trade-off space, given under the hood they've had to think about a lot of that stuff over many decades.
• slaymaker1907 8y ago
  It seems like the use case of a news agency would be better served by storing metadata in a RDMS and using something like HDFS or other file system for the actual articles.
  • KaiserPro 8y ago
    the articles are trivially small. the 2 million articles could just about fit into RAM nowadays.
• jack6e 8y ago
  Specific technology choices aside, this was an incredible write-up of their migration process: thorough, organized, readable prose about a technical topic. It is helpful to read how other teams handle these types of processes in real production systems. Perhaps most refreshing is the description of choices made for the various infrastructure pieces, because it is reasonable and real-world. Blog posts so often describe building a system from scratch where all latest-and-greatest software can be used at each layer. However, here is a more realistic mix of, on the one hand, swapping out DBs for an entirely new (and better) one, but on the other hand finding new tools within their existing primary language to extend the API and proxy. Great read. Well done.
  • harel 8y ago
    Writing is their Core business after all :) I agree though, I read it like a fascinating breaking news story
  • chronotis 8y ago
    This was what was particularly interesting to me - that they went to the effort of writing a purely technical article on the particulars of how parts of their environment operate, and to publish it on their platform even when that's not the sort of content they're known for.
    • viraptor 8y ago
      It's not their main platform though: > Digital Blog > A blog by the Guardian's internal Digital team. We build the Guardian website, mobile apps, Editorial tools, revenue products, support our infrastructure and manage all things tech around the Guardian
      • pdpi 8y ago
        Ish. From what I can tell, the “Digital Blog” seems to be set up as just another column on the platform.
      • vanderZwan 8y ago
• z92 8y ago
  Wanted to read something about the cost impact, and performance impact after migration. But couldn't find that in the article.
  • wftglf 8y ago
    Hi! I'm one of the authors. Good point, we should add something on that. Briefly our mongo db setup was running on 3 r4.xlarge instances which comes to around $5000/year with storage, plus the cost of opsmanager, which I think was around £40,000 including the support contract. In Amazon it's harder to compare directly as the support contract is paid across all of our accounts, but we're spending around $13,000 for a highly available db.r4.xlarge postgres instance. Performance wise, querying without an index is SLOW, basically not worth it - as we end up doing a scan of the entire database. Fortunately we don't need to do this as we can usually rely on the guardian content API for proper searching stuff. The average API (a Scalatra app) response time reaches 150ms at 'peak time'. This isn't a high-performance use case - around 1000 requests/minute at 'peak' time.
• fbonetti 8y ago
  Document DBs are like blockchain projects - overhyped and worse than existing solutions for nearly every use case. Why do ostensibly smart engineers keep falling for this stuff?
  • wongarsu 8y ago
    > Why do ostensibly smart engineers keep falling for this stuff? Review driven design is likely a large contributer. If you know you will be looking for a new job in 2 years, you better get some experience in $latestTrend instead of $provenTechnology.
  • wwweston 8y ago
    For one thing, a good chunk of what happens in the industry isn't particularly well described by the term "engineering," which in my mind describes someone who is well-trained and experience-versed in relevant scientific models for the application domain and the dynamics of relevant "materials" and their interaction that can be composed into a solution. It might be the term "craft" better describes a lot of software development activity. For another, it may be everyone is susceptible to the influence of trends. Even engineers. And the more complex the details of a subdomain of software engineering are, the bigger the tradeoff between becoming someone who can make a true engineering assessment in that niche and developing expertise elsewhere...
    • ownagefool 8y ago
      It's also largely a fact that we're paid better for following hypes, at least in the UK. Thus a smart "engineer" who also cares about their bank balance pretty much has to use everything new to stay relevant. Further, most of the use cases are fairly shallow, so it doesn't actually matter that much, and you'll be paid more to paper over the cracks. We need to change the market, but it doesn't like the strong and stable engineers are getting lead roles to pay more strong and stable engineers to build boring tech, despite that actually being better for most businesses.
  • zzzcpan 8y ago
    They are not overhyped, definitely not as much as old school RDBMS stuff. It's just most engineers can't make good database choices no matter what database they choose. Or more generally they can't make good infrastructure choices, as those are out of their competence and are mostly about things like operations and distributed systems, that take a long time and a lot of experience to get to a level of good decisions.
• diminoten 8y ago
  At this point I'm happy enough with JSONB that, even as someone who wants to try new things, I'm finding it hard to justify trying out MongoDB. My SAs seem cautiously willing to try it, but I won't put them through the work of learning how to manage it unless I can find a clear advantage over Postgres+JSONB.
• gigatexal 8y ago
  My takeaways: 1) MongoDB Atlas would work well here: it’s Mongo hosted on a cloud provider of your choosing managed by the people that make it. 2) a DBA, even a noSQL one*, is worth every penny. (I don’t mean this to put down noSQL DBAs just that it’s good to have someone dedicated to managing them and that a DBA versed in the administration and upkeep of a relational system could retain pretty quickly to help get the most out of a noSQL one)
• RDundon 8y ago
  > “But postgres isn’t a document store!” I hear you cry. Well, no, it isn’t, but it does have a JSONB column type, with support for indexes on fields within the JSON blob. Interesting. I didn't know you could make indexes for things /within/ the JSON.
  • dtech 8y ago
    Yep, you can index any expression on a single row in postgres. The downside is that the optimizer will only use the index if the query contains that exact expression, while it's usually smarter with column indexes. The only limit that I've run into in comparison with NoSQL databases is that you cannot index fields inside arrays within the column (json -> arrayIndex -> field), you need to normalize the array to a different table.
  • cdelsolar 8y ago
    Yes! And a lot of the time querying properly indexed JSONB is faster than Mongo.
  • sitharus 8y ago
    With a GIN index and a JSONB column you can index the entire json document. It's the same index type as Postgres uses for the full text search, but not super efficient. You can also project out specific subsets of the document to an index and Postgres will use them.
    • h1d 8y ago
      I take it this is Postgres specific and MySQL can't index JSON items.
      • sitharus 8y ago
        I'm not familiar with MySQL but the features I outlined are Postgres specific.
      • evanelias 8y ago
        The syntax and terminology are different, but MySQL has supported indexes on values inside of JSON (or any arbitrary expression/function) since MySQL 5.7, released about three years ago. In MySQL, you can define a "generated column" and index it in InnoDB. https://dev.mysql.com/doc/refman/5.7/en/create-table-secondary-indexes.html https://dev.mysql.com/doc/refman/5.7/en/create-table-seconda... contains some JSON-specific examples.
• kokey 8y ago
  I think MongoDB has a place, but that place isn't being the database for a big distributed content store for a web site or something like that. Where I do think it has a place is for some of the in-memory data structures of a node running an application or a service. Applications tend to have a lot of data in various structures that it works with while it's running. Sometimes you need this to be persistent and sometimes you need to be able to scale this across a node. MongoDB is fine for this, so I think of it more like a SQLite replacement.
• Drdrdrq 8y ago
  > Since all our other services are running in AWS, the obvious choice was DynamoDB – Amazon’s NoSQL database offering. Unfortunately at the time Dynamo didn’t support encryption at rest. Whoa, that was close. I really don't see why anyone would choose DynamoDB as a general purpose data store, unless they enjoy wasting countless hours finding ways around the limitations it imposes about how data should be stored and accessed. At least that was my (admittedly limited) experience. Postgres is a much better choice.
  • mr_tristan 8y ago
    NoSQL like DynamoDB or Cassandra does make for good general purpose data stores, but you have to work within the concepts of NoSQL: typically, use 1 table with careful choices in partition and sort keys. It's an epic shift in thinking to go from a schema of 50-60 tables to 1. Almost every dev I've worked with is very new to this. The #1 sign a dev has no business using NoSQL: They chat about how flexible NoSQL is. Yeah, you can add attributes on the fly but I've found Dynamo to require much more careful planning then RDBMSes. Most devs I've met can understand when they need to use an index. But, almost all of them have issues predicting if their changes will lead to unbalanced requests against partitions. Anyhow, since you can even connect the PostgreSQL WAL easily to a log stream like Kafka or Kinesis, I'm not sure why you'd ever start with a NoSQL DB, unless you just had master NoSQL data wonks.
• mrsuprawsm 8y ago
  >“But postgres isn’t a document store!” I hear you cry. Well, no, it isn’t, but it does have a JSONB column type, with support for indexes on fields within the JSON blob. > approximately 2.3m content items. I had a previous project where we did a similar thing (except with HSTORE instead of JSONB) and it exploded rather dramatically (very simple queries took multiple minutes or timed out entirely) after around 30m rows. I hope the Grauniad doesn't run into a similar issue, or at least anticipates it better than we did.
  • scrollaway 8y ago
    2.3m content items is tiny. So is 30m. You're at least 1-2 magnitudes away from something that will start to bother postgres. Anything before that is likely to be an index or IOPS issue.
    • sb8244 8y ago
      On that note, one thing that the article points out is that managing the MongoDB setup was a full time job.....although managing Postgres will probably be as well. It would be easy and understandable to try to make it static and not needing attention, but that is a recipe for bad times. Having issues accessing Xmillion rows seems like something that would be caught by someone focused on performance of the database full-time.
      • scrollaway 8y ago
        RDS is really good at not having to be managed tbh, at the guardian's scale. And when shit does happen they get to be able to throw money at AWS's premium support. I definitely think their move makes a ton of sense there.
    • mrsuprawsm 8y ago
      Even with HSTORE/JSONB columns? I would imagine (gut feeling, no evidence) that the performance of these is worse than more traditional column types.
• InGodsName 8y ago
  While we are here. I know HN hates mongo. But lemme ask this question, what are the things in which mongo is better than postgres?
  • anarazel 8y ago
    Marketing / Evangelism / Outreach. I'm not kidding, and I don't predominantly mean that in a bad way. I think it's something that the postgres community / companies really aren't good at.
    • user5994461 8y ago
      +1 on that. It takes 80% of the work to build the product and 80% to sale it. MongoDB was really good at the later, creating and riding the hype wave.
  • purple_ducks 8y ago
    Subtle data loss...
    • h1d 8y ago
      I'm not trying to cover mongo but I hear stability has improved and random data loss is an old story.
      • pritambaral 8y ago
        I'd rather go with the database that has never blatantly lied, especially about keeping my data safe.
    • StreamBright 8y ago
      The only database we lost data with is Mongo. We had support at the time but even them could not recover. Migrated to Postgres the next month and never looked back.
  • astine 8y ago
• alexandre_m 8y ago
  If anyone is considering doing something similar today, I'd suggest to take a look at the recent FoundationDB Document layer which speaks MongoDB v3 (https://foundationdb.github.io/fdb-document-layer/ https://foundationdb.github.io/fdb-document-layer/). I haven't tried it yet, and there are some limitations at the moment, but I find it extremely promising.
  • grumpydba 8y ago
    I would advise not to use such a baroque technology unless IT is your main activity. Most of the times a boring database is a better fit. Not everybody can be an early adopter, especially on the software which stores what is arguably your most precious asset.
    • alexandre_m 8y ago
      I agree with the boring database, it's most often the good choice. Elaborating on my comment though, it would make sense to consider a drop-in replacement for a technology for which you've already invested yourself, but got bitten enough time to think about moving away to a different model (operational burden, data loss, etc). I'm not sure if "baroque" is a good qualifier for FDB. It's recent in the open source community, but has been running in production for many companies (including Apple) for some years. It's based on sound architecture, compare to many others.
• petilon 8y ago
  Disappointing article. There are no clear reasons mentioned for migrating away from MongoDB. "All these problems" and some issues with Ops Manager are mentioned, where "all these problems" is a couple of outages. As if other database technologies prevent outages. As soon as they experience a couple of outages with their new stack they will migrate to something else, presumably.
  • fastftw 8y ago
    Thank you! I was reading through the comments wondering how no one has seen through the bullshit veil purported by the title, followed by a detailed outline of their migration which I think is the "shiny object" to distract people from an unfounded argument.
    • wftglf 8y ago
      The main advantage of the migration was getting onto a fully managed DB. What we didn't mention was that there were also huge cost implications - savings of ~£40k - through switching to RDS - compared to paying for a mongo support contract.
• deleted 8y ago
  [deleted]
• peterwwillis 8y ago
  My takeaways: 1. Stop. Trying. To. Build. Your. Own. Cloud. A pizza shop doesn't build their own cars to deliver pizzas. 2. There's no such thing as hassle-free anything, unless you are paying someone else to deal with the hassle. Sales teams lie. 3. Justifying an untested idea with "but it's modern technology" is going to backfire. Follow established patterns with good track records. 4. Writing your own in-house behemoth product, of which there are already many kinds available, results in long-term expensive engineering projects necessary to to get around the high costs you didn't know were coming. 5. Don't write business logic, or your primary software product, in a way that talks directly to a database. Just... No. 6. Magic new technologies that remove the problems of old technology also introduce the problems of new technology.
  • indigodaddy 8y ago
    Point 5. I'm not a software/application person... can you expound on how else you would interact with a DB if not directly?
    • dsirola 8y ago
      I'm pretty sure he wants to say that you should abstract your database away so that your business code / domain model doesn't depend on it. It becomes a "plugin" to your application and you can easily switch it just by writing another implementation.
      • kstrauser 8y ago
        I couldn't disagree more, at least in the context of this article. If you have an abstraction layer so high level that app developers can't tell whether they're using MongoDB or PostgreSQL, then they're not able to use any of the advantages of either of those systems. Sure, use an ORM to abstract away the differences between PostgreSQL and MySQL (up until you need to care about them). That's reasonable. But maintaining a magical MaybeSQL layer that's powerful enough to not totally suck is going to totally suck.
        • 8y ago
• wallflower 8y ago
  Kudos to The Guardian for publishing this article on their top-level domain. It looked identical to one of their news articles (including ads). Only the content (excellent writeup) was different. Many tech companies silo their engineering teams writings into a “tech” part of their website. If this wasn’t an ad for The Guardian’s technology team, I think it could be. Putting the engineering team’s content in the same article CMS as the news articles is a small decision that reflects highly on The Guardian’s publishing organization.
  • OJFord 8y ago
    Have you tried to find it from the home page though? It's under `/info`, which un-suffixed redirects to `/about`, which I _can_ find from the home page (it's linked in the _footer_) but I can't find this engineering blog from `/about`, or anywhere else I can get to from `/`. I don't think it's so much a positive choice to use the same system as it is just using (a possibly separate instance of) what they already had, and still squirreling it away in a corner.
    • wftglf 8y ago
      It's the same system, just without any tags on it that mix it in with editorial content. I think if we wanted it published somewhere like the technology section we'd have had to go through a much more rigorous editing process (we probably wouldn't have got away with 3000 words!)
• rb808 8y ago
  I'm not sure why a database is required at all. Surely the main advantage is to support tables and joins. If you just want a key-value store surely S3 or similar would be simplest. Do they search the data? Surely Cassandra is designed for this. Any ideas?
  • scrollaway 8y ago
    Cassandra is more of a DB than Mongo is. Dynamodb is basically the same, and they wanted the hosted version so their approach is reasonable. They definitely want more than a key value store. They at least want date and content indexing, editor, author, etc. I imagine they probably have a few internal requirements as well wrt analytics, tooling and more. S3 doesn't work for all this and another user's suggestion that this is all treatable like static content sounds way off to me. The best you can do is pregenerate some of the HTML.
  • KaiserPro 8y ago
    Cassandra isn't designed for this. Been there, done that. postgres all the way. THe FT use(d) cassandra to store their membership details (6 million rows of largely static, read-only well structured data) The cluster was massive (12+nodes in at least two regions, from memory) slow and was impossible to upgrade reliably. The support from datastax is shite. Backups are not reliable. imports even less so, and you are beta testing the whole system every time you do a point release. CMSs have highly structured data. swallow your pride, map the data and build a proper schema. Its really not that hard. Yes, cassandra has a graph layer, no its really not worth it. Yes it has gremlin, no you shouldn't need it if you've modelled your data correctly.
    • user5994461 8y ago
      What were the issues? You're the first person I hear complain about cassandra for this. Agreed that any database can do that type of data at that volume.
      • KaiserPro 8y ago
        its a 50/50 split between terrible design and horrible support. I've seen cassandra shine when it comes for write optimised loads. Pipe a bucket load of data into gremlin and magic happens. But thats a specific workload, which is pretty rare, and certainly not suited 999:1 read to write ratio. Thats not cassandra's fault, thats the fault of the idiot that chose it, and the boatload of idiots who carried on and added loads of systems that makes it much harder to migrate away. Then we come to support. Datastax is the defacto support provider. They make a lifecycle manager, backup/restore tool, and push a load of patches into the main codebase. But it is shit o Backups fail silently o The only way to make alerts work (ie do an action, rather than create a popup when you log into the ops center) requires work to navigate the API o Its full of CVEs, which are script kiddy-able o migrating data from backups to new clusters was impossible to do without a boat load of manual work, failed 50% of the time o restoring from automated backups was impossible until august. o it couldn't use instance profiles on AWS until august o upgrading to a point release silently breaks backups, _always_ Basically I spent the first half of this year QA very expensive software. There are some very very good support people, but there were some terrible ones as well.
• scrollaway 8y ago
  Fantastic write up article. A few things stood out at me. In no particular order: - Going with Mongo in the first place cost them dearly. CMS are a weird application for schemaless. Not necessarily wrong, but definitely weird. I wonder if they would benefit from moving to more structured schema and I'm willing to bet a lot of the migration complexity comes from that in the first place. - God damn that is a long migration. Holy shit. I know tech isn't their core competency but they do seem to have very competent staff. I've worked in places this glacially slow and I get how it can get this bad but it really strikes me as having no right to be. 10 months to migrate .... from the point where they were ready, until they were done. And somehow integration tests got overlooked during all that. - Close call with dynamodb. Bit of a wtf on waiting for the feature to be implemented for nine months though. I'm sure they have an account manager with aws... I definitely think blocking an internal process on such a fragile externality as a closed process upstream publishing a feature is the wrong move and a red flag. Their migration path would have probably been harder with dynamodb too. - I feel the pain of their troubleshooting issues on the load testing step. I can completely see how this can happen. That said it also raises a few red flags to me. Letting something as simple as the load testing step get complex enough to require weeks of engineering is ... Eh. Have more thoughts but I hate typing on mobile. This is a fantastic write up, I love when non-tech companies publish this sort of stuff. And hurrah for postgres.
  • acjohnson55 8y ago
    > CMS are a weird application for schemaless. That depends. If the CMS is managing documents that have flexible structures that are tree-like, it might not be such a horrible idea to model that structure in a document instead of relationally.
    • mulmen 8y ago
      Document stores might be great for the C but not so much for the MS.
      • sjwright 8y ago
        Or to put that in context of a basic data structure: C = Actual content to be published MS = User accounts, user permissions, user authentication options, user authentication logging, meta change logs (user X removed tag Y at time Z), behaviour logs (user X viewed revision Y of article Z at time A)... and that's just scratching the surface of a very, very basic CMS. I'm being generous and assuming articles, tags, bylines, and attached media (with full change history) is all "C".
• thejerz 8y ago
  Their architectural choices are puzzling to me: 1) Why use Scala to write (a relatively simple) internal CMS? 2) Why use a clustered database for 2 million records? 3) Why write your own proxy? (in Akka, none the less) 4) Why would you migrate articles from Mongo to Postgres using a script that runs overnight in screen? The Guardian is, prima facie, a Wordpress blog. A simpler architecture would be: 1) Any CRUD web framework to build the CMS for reporters to draft their articles (Django, Rails, etc). Any basic RDBMS with read replication will do. Or, ditch the webapp entirely and just make a simple Markdown editor that commits to a git repo, a la Prose or Netlify. 2) When a reporter "publishes" an article, generate HTML for it and push to the CDN network. (I can't easily tell by looking at their HTTP headers, but I assume they're doing this already) Okay, I'm being a little tongue-in-cheek. It's probably not that simple. But, one has to wonder, when you're serving up 100 million static HTML pages a day, if it really has to be this complicated.
  • mattbillenstein 8y ago
    Your architectural choices are puzzling to me - Rails/Django -- I've rescued more bad Django apps than I can count.
    • StreamBright 8y ago
      Exactly, I have moved companies from random web framework + random database to static site generators + CDN with high rate of success too. No point of using Rails/Django like stuff unless you have an extremely good case to, which is certainly not the Guardian use case.
      • ashtonbaker 8y ago
        Genuine question - because I'm currently doing CPR on an old Rails app with Mongo - what do you see as a "good use case" for Rails/Django and similar frameworks?
        • StreamBright 8y ago
• christkv 8y ago
  The only why I saw was this comment This is one of the reasons why building on top of the old API wasn’t an option. There was very little separation of concern in the original API and MongoDB specifics could be found even at the controller level. As a result the task of adding another database type in the existing API was too risky. This seems like the problem was more related to the API layer code quality rather than where the data was stored.
• femto113 8y ago
  > Dynamo didn’t support encryption at rest. This requirement made some sense in a world where a rogue employee might yank your database server out of a rack and walk off with it, but I don't understand why this is still considered relevant in an AWS context. First of all your data is never really "at rest", a huge point of Dynamo is that the data is always available (at least 99.999% of "always" anyway). Second "at rest" where? Literally on a physical disk? Even if there was a single physical disk that held it I'm willing to bet that even if you were in the right AWS data center and had support of a willing AWS employee you couldn't find that disk and even if you could there'd be no way to remove it. I'll guess its literally millions of times more likely for you to fail to deprecate some AWS API key that's sitting in clear text on a laptop you sold as surplus than anybody gets their hands on your at rest Dynamo data.
  • 0x264 8y ago
    GDPR and Cyber Essentials
  • adrianpike 8y ago
    At least for those of us in the healthcare space, HIPAA compliance demands it. The regulations and the attack risk aren't always in alignment unfortunately.
• idbentley 8y ago
  I really don't get this as an indictment of MongoDB, or their OpsManager product really. They used the version of OpsManager that doesn't manage the deployment - is specifically not a deployment manager. Mongo does offer a managed version of this software, which the author mentions - with a justification for why they couldn't use that offering. However, I think this was the main mistake that The Guardian made. As the author notes: "Database management is important and hard – and we’d rather not be doing it ourselves." They underestimated the complexity of managing database infrastructure. If they had been attempting to set up and manage a large scale redundant PostgreSQL system, they would have spent an enormous engineering effort to do so as well. Using a fully managed solution - like PostgreSQL on RDS from the beginning would have saved them time. Comparing such a fully managed solution to an unmanaged one is an inappropriate comparison. Full disclosure - I used to work at MongoDB. I have my biases and feelings w.r.t the product & company. In this case I felt that this article didn't actually represent the problem or it's source very accurately.
  • beokop 8y ago
    I do like the article but it sounds like they’re in over their heads, this whole (very risky) project could have been avoided if they just brought in someone that knew what they were doing. > Clocks are important – don’t lock down your VPC so much that NTP stops working. > Automatically generating database indexes on application startup is probably a bad idea. > Database management is important and hard – and we’d rather not be doing it ourselves. This is true of any database infrastructure with redundancy/scalability requirements. What they did was take a technical problem and solve it by buying an off the shelf solution. Which is fine, of course, but I’m a bit surprised by the reaction here on HN.
  • wftglf 8y ago
    Fair criticisms! It's true if we'd used Mongo Atlas or something similar it would likely have been a different story - often the MongoDB support spent half the time on the phone trying to work out what version of mongo, opsmanager etc. we were running. Re criticism of OpsManager - I think this is fair, given the sheer number of hoops we had to jump through to get a functioning OpsManager system running in AWS - no provided cloudformation, AMIs etc. £40,000 a year felt like a lot for a system that took 2 or more weeks of dev time to install/upgrade. The authentication schema thing was a bit of a pain as well, though we were going from a very nearly EOL version of Mongo (2.4 I think).
• kailuowang 8y ago
  >Since migrating to AWS we’ve had two significant outages due to database problems, each preventing publication on theguardian.com for at least an hour. This happened to us numerous times before and after we migrated to their Atlas platform. We are a major streaming company with more than 120M registered user. Disclosure, we are moving away from MongoDB as well.
• indigodaddy 8y ago
  Why would you delete the MongoDB's so soon after the migration? What if things look great for a few hours/days, but then things go horribly wrong? I suppose they still had Mongo backups too? Just unsure why you wouldn't leave those around for awhile.
  • kstrauser 8y ago
    Guessing: - They'd been running the two systems in parallel and comparing their outputs for months. By the time the pulled the plug, they were confident that PostgreSQL was working fine. - They had written systems to copy data from the MonogoDB-backed service to the one running on PostgreSQL. Once they started treating PostgreSQL as the official store of record, they'd be faced with either writing another system to mirror PostgreSQL back to MongoDB or committing to lots of double entry. Either of those sound painful.
  • manigandham 8y ago
    The whole migration took almost a year, looks like they've had enough testing.
• mulmen 8y ago
  Standard disclaimer in database discussions involving NoSQL: SQL is not RDBMS is not ACID is not NoSQL. When comparing NoSQL to something you should really compare it to RDBMS which is what most people conflate with "SQL". This conceptual understanding gap is IMO what led to the huge growth and misuse of "NoSQL" for a decade.
• abhishektwr 8y ago
  Good read. Although I don’t quite understand why every few years media companies keep rewriting their content management layer which generally has no impact on company’s top line or bottom line whatsoever. How can these type of investments are justified when all you are doing is recreating exactly same APIs with same data. Am I missing something here?
  • manigandham 8y ago
    Media companies do not have the best technical leadership or talent, leading to high turnover with junior devs who build with the latest hype to build resumes.
  • magduf 8y ago
    Yes, you're missing something pretty big. It's not just media companies rewriting something significant every few years; everyone in the tech industry does it. There's a very good reason for it: it provides interesting and high-paying work for tech employees. Who wants to go to a company, set up a great system that doesn't need any more work except maintenance, and then just settle down into maintenance mode? No one who wants a rising career. "Maintained system using $oldTech" doesn't look good on a developer's resume, whereas "migrated critical system to $newTech" does. And it's not just the developers: managers who oversee big teams of developers working on a big new project have more prestige and are paid more than managers who just oversee some boring already-done project that's in maintenance mode. Now, in this case, there were probably very good genuine reasons for making a big change (I'm no fan of MongoDB), but many times if you look closely with a critical eye, you'll probably find that people padding their resume and making up a reason for their job to exist is the real reason something is being done.
    • zozbot123 8y ago
      > interesting and high-paying work Busy-work is never interesting, and if it's "high-paying" someone is getting fleeced (and will do what they can so as not to get fleeced in the future). At some point, the constant rewriting will have to stop. In contrast, maintenance work on a well-engineered critical system can be quite interesting in its own right (and it's not like the "maintenance" doesn't involve writing some new stuff on occasion. Nothing is ever truly "done").
• tixocloud 8y ago
  Really appreciated the detail on the post as it provides a lot of great insight into how other tech teams handle migration. Thanks!
• manigandham 8y ago
  Why did they actually migrate databases? The real issue is database operations, not the database itself. The data model is simple so anything would've worked fine, and they already have a working system. The proper solution is to onboard the proper talent and tools or outsource it all. MongoDB does have fully-managed offerings that will automatically migrate and run your database, across cloud-providers if you need, and would've cost less than all the time and effort spent on this. This is just another example of poor technical competency at media publishers.
• MagicPropmaker 8y ago
  But MongoDB is web scale! https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
• Thaxll 8y ago
  It's not a great article tbh, it's well written but it shows the clear lack of knowledge running a backend. The title should be "we didn't know what we were doing so we switched to a managed DB" I mean yeah who knew that blocking NTP therefore time drifting would break everything... For those criticizing MongoDB, Fortnite generates $3B/year and runs on MongoDB, you should tell them it's a mistake and that they should use PG instead.
  • snacktaster 8y ago
    I'm sure mongodb is fine and powers a lot of high-profile sites and services. But I've never had a good experience running it or administering it. But I'm also one of the the people that still prefers to run MySQL over PostreSQL just because the tooling is still far superior
    • philliphaydon 8y ago
      Can you give an example of superior tooling or tooling that isn’t good in PostgreSQL compared to MySQL. Serious question. I’m curious cos I mostly use PostgreSQL.
      • Thaxll 8y ago
        All the Percona tool are for MySQL and there is nothing comparable for PG afaik. https://www.percona.com/doc/percona-monitoring-and-management/index.html https://www.percona.com/doc/percona-monitoring-and-managemen... https://www.percona.com/software/database-tools/percona-toolkit https://www.percona.com/software/database-tools/percona-tool...
        • philliphaydon 8y ago
          It looks like Percona has partial support for PostgreSQL. It's not as detailed as MySQL. https://www.percona.com/software/database-tools/percona-monitoring-and-management https://www.percona.com/software/database-tools/percona-moni...
  • 8y ago
• jasondc 8y ago
  If you think MongoDB isn't doing well, just look at their financials and stock price, they're on fire: https://finance.yahoo.com/quote/MDB?p=MDB https://finance.yahoo.com/quote/MDB?p=MDB Anecdotes like this are just that, anecdotes, there are no numbers in this article to show a trend away from Mongo, Mongo is actually continuing to gain adoption. See https://db-engines.com/en/ranking https://db-engines.com/en/ranking
  • hu3 8y ago
    Maybe I'm wrong but your link tells me stock price took a hit today: > -8.82%% As of 11:56AM EST. Market open. 1 month chart still looks green so this might just be a correction.
• fogetti 8y ago
  I find this specially troubling just like abandoning the Edge browser development. Steering away from Mongo just for the sole reason it's not in AWS sounds like a bad advice in general. Also it sounds like a false dichotomy. Implies there are no other options but it didn't even mention MongoDB Atlas for example.
• kureikain 8y ago
  The article is very well written. Their team do leverage great tooling for thing such as Gor. I love Gor. I used it + Mongo OpsLog when migrate from AWS -> DigitalOcean and eventually have to migrate back to AWS because DigitalOcean cannot deliver good performance as AWS and they throtle CPU(lots of steal CPU). OpsManager is great for team that don't have dedicated devops I think and have great dashboard/visualization. However, run your own MongoDB is very easy. Not like Postgres(unless you used RDS). However, when using RDS, you still have downtime when upgrading db, it still have a small amount of time the standby in MultiAZ is promoted to master, DNS is updated, and during that yourcurrent primary is not writeable or even not available. MongoDB is way easier to operator. You add/delete/node on fly and client auto discover network topology. Plus https://docs.mongodb.com/manual/administration/production-notes/ https://docs.mongodb.com/manual/administration/production-no... this links give great info to tune it: thing like run on XFS, mount with notatime options...puts opslog on high iops volume etc... Peformance isn't a factor to pick your database much nowsaday. They looks great on the benchmark. However, try for yourself before pick one on your workload. Postgres does has its own ward. Pick a database based on how well your team confident with it, how does the database help you move fast enough or deliver business value. Don't follow the hype or silly benchmark.
• deleted 8y ago
  [deleted]
• craftoman 8y ago
  Coming next: Bye Bye JavaScript, welcome Golang and bye bye Angular welcome React. More catchy titles incoming.
• ramsundhar20 8y ago
  I think Cassandra would be better choice
• ramsundhar20 8y ago
  I think Cassandra would make a better fit and scales huge.
• myth2018 8y ago
  Great article. Such case studies are way more significant contributions to software engineering as a field than the big pile of whitelabel/fanboy articles we see around highlighting advantages of some fancy stuff.
• _10x_dev 8y ago
  “This is one of the reasons why building on top of the old API wasn’t an option. There was very little separation of concern in the original API and MongoDB specifics could be found even at the controller level. As a result the task of adding another database type in the existing API was too risky.“ This bit sounds like a cautionary tale about the importance of a layer and well structured API. There’s no reason why details about your data store should leak into an API Controller.
• aprdm 8y ago
  Would be interesting to share the different in ops cost (year support paid aside), for example, do you need a beefier server now? more servers? less? size on disk? IOPS of servers? also, any different in your metrics from an end user perspective? e.g: time to publish an article.
• rasikjain 8y ago
  Very well written article on data migration. Going to process and details. This will help anyone who is also considering such a move. I guess we will see a shift in direction towards Postgres in 2019.
• codecrusade 8y ago
  2 million contents. lol
• brianmcc 8y ago
  I enjoyed the write up. I love these kinds of semi-technical, semi-story-telling, pieces. I am saddened by many of the comments here though which equate to: "never try anything until you know everything" - sorry but that's just not realistic and it's unfair to the people who - commendably - contribute to these write ups and hold their hands up to mistakes-made, decisions that went badly with hindsight, etc. Bigger picture: the Guardian appears to be thriving, and is succeeding based on the efforts of the tech team here. So if you read this and come away with a sense of "failure" you're probably missing something important. It's OK to try new things and fail - just get back up and keep on trying, and use the new wisdom you build!
• conanthe 8y ago
  I couldn't care less what's this propaganda rag uses to store their fake news.
• mscasts 8y ago
  I worked on a large site that used MongoDB. We hundreds of thousands of rows in one specific collection that were used as a log. We couldn't even query it without the whole database crashing. We moved that specific collection to MySQL, no problem and there was virtually no change in the data structure. Both were of course indexed.
  • mr_toad 8y ago
    You can query hundreds of thousands of rows in a text file on a command line in a raspberry pi with grep and wc.
• Fiahil 8y ago
  Was is most surprising (well, not really), is the apparent immaturity of Scala tooling (Ammonite), and Akka-based projects like Akka-HTTP and Akka-Stream. If you can get into very cryptic issues when using these project, it makes you reconsider the actual value you get from using a functional, type safe and on-paper concurrent language. Their proxy is one example where Rust will give the same capabilities as Scala and will be more robust, in the future, if it's not already the case today.