20 ms·

Bye Bye Mongo, Hello Postgres (2018)

198 points by mathattack 6y ago

KMnO4 6y ago
So if I understand this correctly, they didn’t have the correct understanding/skills to manage a Mongo DB, so they switched to one they were more familiar with. I wonder what this ultimately cost. Migrating from a document model to a relational DB requires code change in pretty much every single location it’s used. That’s a lot of engineer time to develop, test, and do the data migration.
- afavour 6y ago
  > they didn’t have the correct understanding/skills to manage a Mongo DB According to the post they paid Mongo a lot of money for support and they didn’t have the correct understanding/skills either. It sounds to me like the core problem is their requirement to run Mongo on their own AWS stack rather than Mongo’s, and while Mongo claimed to support it seems like they didn’t really. No doubt this was a huge migration but from my personal perspective going from Mongo to Postgres feels like a move in the right direction in the long term. For one you have a lot more options for support!
- jpz 6y ago
  If you hire a room full of developers at high salary and nobody is capable of acquiring "the correct understanding/skills" then maybe it's not such a straightforward technology. It is reasonably straightforward to deal with relational databases in comparison, this is a well-known space.
- znpy 6y ago
  > So if I understand this correctly Clearly you don't. They did have the correct understanding/skills to manage a Mongo DB, and on top of that they bough the MongoDB-Inc-recommended management tools and the MongoDB-Inc-recommended support plan. Basically they were doing everything by the book, as advertised and advised by MongoDB itself. Yet they were having problems, so much so that even MongoDB Inc's own support engineers weren't able to help. After a great lenght of discomfort they evaluated that MongoDB was not a good fit for them, and moved to PostgreSQL.
- znpy 6y ago
  > I wonder what this ultimately cost. > That’s a lot of engineer time to develop, test, and do the data migration. Downtime is more expensive. EDIT: Also, the article says that they spent at least two months a year planning and executiting database upgrades, since the OpsManager tool from MongoDB Inc wasn't able to help.
  - sokoloff 6y ago
    Two months a year planning and executing upgrades isn't that outrageous-sounding to me. We did that for years on a commercially supported RDBMS (MS-SQL Enterprise in our case). It wasn't the whole engineering organization, but the overall "scale existing capabilities" effort was a 3K-10K hour project every year depending on how extensive the work was.
    - znpy 6y ago
      Well it kinda is if you bought a product that supposedly lets you do upgrades easily, from the vendor itself, and also have the support package from the vendor.
- deleted 6y ago
  [deleted]
paulryanrogers 6y ago
> OpsManager didn’t really deliver on its promise of hassle-free database management. I'm a Pgsql fan yet the months of work that followed don't seem justified. I wonder if they're still happy with it given how document oriented journalism appears to be from the outside.
- arcturus17 6y ago
  In my experience many domains that seem document-oriented suddenly reveal themselves as relation-oriented as soon as you start building.
  - nevi-me 6y ago
    This goes both ways to an extent. Any data that you can model in Mongo, can be modelled in a SQL database. There's often the argument that documents aren't structured, but in many cases, there's structure if you categorise them well. If one is storing truly randomly-structured records in a document-database, I would question why, as they could store the documents as files instead. The other way is that you could store documents that fit a relational structure, into a document-store. It depends on use-cases, and those use-cases exist and are valid. We can store KV structures in a SQL database, but we're seeing projects using Cassandra and co.
    - arcturus17 6y ago
      You can indeed model relational domains using document stores, but soon you'll start having keys everywhere to glue the different documents together. This may not be that big of an issue if you have a few different types of entities and you're not keeping that many relationships between them. But you're still going to face problems when you need to change the relationships, an issue which is trivial to resolve in a relational DB. If there are more than a few relationships and they're even just moderately unstable, you've probably made a critical mess of the application.
    - acdha 6y ago
      > We can store KV structures in a SQL database, but we're seeing projects using Cassandra and co. I think a key difference here is illustrative: you typically only see that when the problem is well understood to need the characteristics of those specialized KV stores and were willing to pay the costs of working with those trade-offs. In contrast, Mongo was overwhelmingly favored people who were speculating about problems they might have without much experience supporting that call. Usually they never actually came close to that level, likely because they were spending their time on the much harder problem of trying to build SQL database semantics into their application instead of working on their business. The key lesson I drew is the powerful value of sticking with proven tools unless you know you can’t. Reading about things FAANG work is cool but people cost themselves a lot by not asking how many orders of magnitude separate their workload and team size from yours.
throw_m239339 6y ago
previous discussion https://news.ycombinator.com/item?id=18717168 https://news.ycombinator.com/item?id=18717168 When is Mongo ever a good fit? so I've heard stuctured logs, but they can be shoved in a dedicated RDBMS themselves, or just a file system. Unstructured data? But PG now has supports for XML or even JSON. I've heard it's also easier to administrate and scale, which I'm sure it is but then why some businesses do go back to MySQL or PG if Mongo is easier?
- mathattack 6y ago
  Does it take hold purely because it’s easy to install and the company has a hyper-aggressive salesforce?
  - golergka 6y ago
    There's a lot of beginner developers who have somehow not yet ran into problems that a proper relational OLTP would prevent, and love the initial ease of development that Mongo provides for small projects. Some of them go freelance as web developers and complete quite a lot of small projects for small clients without seeing any problems.
  - jlouis 6y ago
    MongoDB is really easy to set up and the library API is really good. This provides the affordance in the start of a project where you don't have to think that much about system load from users. You don't have to think that much about your relational model, so you can postpone all kinds of nasty questions about data integrity and multiplicity early on. This enables a team to move fast and break things. The switch to something like Postgres comes later in the project where you know your requirements better and you need a system where you have more control over your data. You might even have a person responsible for maintaining your data for you on the payroll. Personally, I usually just grab Postgres these days and use `json` columns for the documents and loosely fitting data, then drag out things into a more relational model as we go.
    - richajak 6y ago
lexx 6y ago
I started doing a demo rewrite of my app in Postrgres from Mongodb. The reality is that even though Portgres seems more stable etc, making queries in Mongodb is much easier and fun. Postgres JSON operators are cryptic and made my head hurt every time I wanted to accomplish something more complex. To express it differently, if mongo and postgres where identical operationally, I would choose mongo every day. More friendly and fun. But there are so many horror stories that I am still not sure that it is the right choice for a safe production environment.
- kfk 6y ago
  I don’t know I was exited to use json for a couple of months but then I needed the same data for analytics and I quickly missed sql.
  - szundi 6y ago
    Some Mongo pro please answer something to this, as without enough experience this is the biggest issue for me in my head.
    - winrid 6y ago
      Mongo's aggregation pipeline is pretty straightforward, so not sure what they're looking for.
- deleted 6y ago
  [deleted]
- tbrock 6y ago
  Today it’s just fine. I think a decade ago it was pretty risky but databases take 10 years to do properly.
- ryanschneider 6y ago
  Are there tools for auto converting mongo queries to postgres’ query format? The mongo drivers are Apache licensed so in theory someone could use them as the basis for a driver that speaks to postgres instead, right? I see tools like that showing up in my search results but since I have no experience with them I won’t link to any, if anyone here does have such experience sharing their results would be appreciated.
makkesk8 6y ago
One big drawback of postgres when updating data in a jsonb column is that it needs to rewrite everything even when you update the value of a single key, this gets very expensive if you need to update many rows. As far as I can tell, Mongodb handles this better. Other than this I also prefer postgres despite this limitation.
- jabberwcky 6y ago
  Mongo stores objects pretty much the same as PG here, a single key update requires a rewrite of the object. Maybe Mongo has some fancy algorithms to avoid full decode/reencode in some cases, but in principle the physical data formats are almost identical
  - isbvhodnvemrwvn 6y ago
    In a tiny bit more detail - mongo uses BSON, which is effectively TLV encoded json. I'm not sure how storage of entire collections is done though. https://en.wikipedia.org/wiki/BSON https://en.wikipedia.org/wiki/BSON
    - rualca 6y ago
      > (...) BSON, which is effectively TLV encoded json. For those who, like me, never heard of TLV prior to this post, it's type-length-value encoding. https://en.wikipedia.org/wiki/Type-length-value https://en.wikipedia.org/wiki/Type-length-value
      - swyx 6y ago
        thanks for the link, you weren't alone
- karsinkk 6y ago
  Disclaimer : I'm a Software Engineer at Oracle In Oracle 21c and the Autonomous JSON Database, we introduced a new Native JSON Datatype with it's own Binary Storage format called OSON[1]. With OSON, you can perform partial updates on a json document. The added benefit is that this results in significantly lesser redo log size. [1] https://blogs.oracle.com/jsondb/osonformat https://blogs.oracle.com/jsondb/osonformat
Jeema101 6y ago
The takeway IMO is that the reason their transition was so seamless is because they took the time to do it 'the right way' - that is: set up the new system in parallel, duplicating the reads and writes, and verifying that things looked good every step of the way. That gives you not only a long window to test and validate, but also the option to rollback if things suddenly don't look so good with the new system. It seems like people sometimes get impatient and want to do a 'big switchover' with insufficient production testing and no real rollback plan, but in my experience that's almost never a good idea.
- lma21 6y ago
  Indeed, though that must have cost them twice as much to do the migration with the two systems being live in parallel. Some choose to avoid the extra cost.
  - tyre 6y ago
    Cost is not only measured in dollars for servers. The cost of doing a massive, irreversible migration that doesn’t work seamlessly the first time is significant. And often more significant in engineering time, lost revenue from services going offline, trust both from external users and internal teams watching the shitshow, and possibly data loss or backwards-incompatible changes than those server dollars. It’s funny that one consistent theme of the best engineers I’ve worked with is less their “amazing 10x ideas” than the number of “sounds like a good idea if you’ve never done it” ideas that they shut down. Huge migrations are one.
    - acdha 6y ago
      I had an example of that recently: we’ve been buying grocery boxes from Hungry Harvest. They announced a new website was coming a couple of weeks ago and had the usually puffery about how it’d be better in vague ways (the old one was not noticeably bad). Then the site was down longer than expected. Then everyone was told their order customizations were lost. Now it’s so bad that they’re skipping an entire week’s delivery for all of their customers. I don’t know how they got there but that’s expensive, disrupts their supply chain, and I suspect they’d cheerfully pay four times as much not to have to slog through this mess.
awinter-py 6y ago
> Composer, and its database, originally started their lives in Guardian Cloud – a data centre in the basement of our office near Kings Cross yes
colesantiago 6y ago
you would think that the hate that mongodb (and mongodb inc) gets right now that the stock should be shorted to oblivion by now. long overdue.
- Nextgrid 6y ago
  I don't think the hate on MongoDB or NoSQL itself is justified. The hate should be directed at developers or CTOs that go "all in" on it because of hype or lack of skill and the illusion that not having to deal with schemas is somehow a silver bullet. NoSQL databases are great for storing arbitrary JSON-like structures, retrieving them by ID and occasional queries/aggregations on the inner fields of those structures. NoSQL is absolutely not suitable for most business logic and the lack of a schema is actually a major drawback. It feels like a solution (because inserting invalid data will succeed on NoSQL compared to a conventional DB which will reject it based on constraint violations, missing fields or mismatched column types) but in reality you're just kicking the problem down the road and it will come back to bite you because your application now has to be able to deal with this inconsistent data (and in most cases that isn't accounted for with dealing with NoSQL, and the result is predictable). I've been on a project where the main database was MongoDB and while it worked fine for the most part, we'd get exceptions when the application tries to read some records (most likely from earlier on in the business' lifetime) that had missing keys and would predictably explode. This isn't the fault of MongoDB by itself - the whole point of it is to be able to deal with unstructured data - but the fact that someone chose to use it while they actually needed structured data and constraints to ensure only valid data is inserted in the DB, which traditional relational databases do provide. Of course, it's easier to blame MongoDB rather than admit "we've been stupid and shouldn't have chosen the wrong tool for the job".
  - pydry 6y ago
    I hate on Mongodb partly because of the super aggressive salesforce that tries to sweet talk less-technical managers and grab internal "champions". It's a poster boy for making a shoddy product and then trying to paper over the deficiencies with sales and marketing. Part of this marketing includes creating a super easy, slick set up for beginners and just burying the grenades for beginner users to discover later when they're already locked in. NoSQL is indeed appropriate sometimes, but Mongo never is.
436363636 6y ago
The sheer number of "why we switched from Mongo to Postgres" blog posts is astounding, since all or most of the reasons typically cited would have been just as valid for not using in Mongo in the first place, all the way back to 2010. These blog posts amount to a bunch of people not only admitting incompetence, but apparently lacking even the self-awareness to realize that that's what they're doing.
AzzieElbab 6y ago
All newspapers sites should use Cassandra on the backend. It is columnar ...

198 points by mathattack 6y ago

So if I understand this correctly, they didn’t have the correct understanding/skills to manage a Mongo DB, so they switched to one they were more familiar with. I wonder what this ultimately cost. Migrating from a document model to a relational DB requires code change in pretty much every single location it’s used. That’s a lot of engineer time to develop, test, and do the data migration.

afavour 6y ago
> they didn’t have the correct understanding/skills to manage a Mongo DB According to the post they paid Mongo a lot of money for support and they didn’t have the correct understanding/skills either. It sounds to me like the core problem is their requirement to run Mongo on their own AWS stack rather than Mongo’s, and while Mongo claimed to support it seems like they didn’t really. No doubt this was a huge migration but from my personal perspective going from Mongo to Postgres feels like a move in the right direction in the long term. For one you have a lot more options for support!
jpz 6y ago
If you hire a room full of developers at high salary and nobody is capable of acquiring "the correct understanding/skills" then maybe it's not such a straightforward technology. It is reasonably straightforward to deal with relational databases in comparison, this is a well-known space.
znpy 6y ago
> So if I understand this correctly Clearly you don't. They did have the correct understanding/skills to manage a Mongo DB, and on top of that they bough the MongoDB-Inc-recommended management tools and the MongoDB-Inc-recommended support plan. Basically they were doing everything by the book, as advertised and advised by MongoDB itself. Yet they were having problems, so much so that even MongoDB Inc's own support engineers weren't able to help. After a great lenght of discomfort they evaluated that MongoDB was not a good fit for them, and moved to PostgreSQL.
znpy 6y ago
> I wonder what this ultimately cost. > That’s a lot of engineer time to develop, test, and do the data migration. Downtime is more expensive. EDIT: Also, the article says that they spent at least two months a year planning and executiting database upgrades, since the OpsManager tool from MongoDB Inc wasn't able to help.
- sokoloff 6y ago
  Two months a year planning and executing upgrades isn't that outrageous-sounding to me. We did that for years on a commercially supported RDBMS (MS-SQL Enterprise in our case). It wasn't the whole engineering organization, but the overall "scale existing capabilities" effort was a 3K-10K hour project every year depending on how extensive the work was.
  - znpy 6y ago
    Well it kinda is if you bought a product that supposedly lets you do upgrades easily, from the vendor itself, and also have the support package from the vendor.
deleted 6y ago
[deleted]

paulryanrogers 6y ago

> OpsManager didn’t really deliver on its promise of hassle-free database management. I'm a Pgsql fan yet the months of work that followed don't seem justified. I wonder if they're still happy with it given how document oriented journalism appears to be from the outside.

arcturus17 6y ago
In my experience many domains that seem document-oriented suddenly reveal themselves as relation-oriented as soon as you start building.
- nevi-me 6y ago
  This goes both ways to an extent. Any data that you can model in Mongo, can be modelled in a SQL database. There's often the argument that documents aren't structured, but in many cases, there's structure if you categorise them well. If one is storing truly randomly-structured records in a document-database, I would question why, as they could store the documents as files instead. The other way is that you could store documents that fit a relational structure, into a document-store. It depends on use-cases, and those use-cases exist and are valid. We can store KV structures in a SQL database, but we're seeing projects using Cassandra and co.
  - arcturus17 6y ago
    You can indeed model relational domains using document stores, but soon you'll start having keys everywhere to glue the different documents together. This may not be that big of an issue if you have a few different types of entities and you're not keeping that many relationships between them. But you're still going to face problems when you need to change the relationships, an issue which is trivial to resolve in a relational DB. If there are more than a few relationships and they're even just moderately unstable, you've probably made a critical mess of the application.
  - acdha 6y ago
    > We can store KV structures in a SQL database, but we're seeing projects using Cassandra and co. I think a key difference here is illustrative: you typically only see that when the problem is well understood to need the characteristics of those specialized KV stores and were willing to pay the costs of working with those trade-offs. In contrast, Mongo was overwhelmingly favored people who were speculating about problems they might have without much experience supporting that call. Usually they never actually came close to that level, likely because they were spending their time on the much harder problem of trying to build SQL database semantics into their application instead of working on their business. The key lesson I drew is the powerful value of sticking with proven tools unless you know you can’t. Reading about things FAANG work is cool but people cost themselves a lot by not asking how many orders of magnitude separate their workload and team size from yours.

throw_m239339 6y ago

previous discussion https://news.ycombinator.com/item?id=18717168 https://news.ycombinator.com/item?id=18717168 When is Mongo ever a good fit? so I've heard stuctured logs, but they can be shoved in a dedicated RDBMS themselves, or just a file system. Unstructured data? But PG now has supports for XML or even JSON. I've heard it's also easier to administrate and scale, which I'm sure it is but then why some businesses do go back to MySQL or PG if Mongo is easier?

mathattack 6y ago
Does it take hold purely because it’s easy to install and the company has a hyper-aggressive salesforce?
- golergka 6y ago
  There's a lot of beginner developers who have somehow not yet ran into problems that a proper relational OLTP would prevent, and love the initial ease of development that Mongo provides for small projects. Some of them go freelance as web developers and complete quite a lot of small projects for small clients without seeing any problems.
- jlouis 6y ago
  MongoDB is really easy to set up and the library API is really good. This provides the affordance in the start of a project where you don't have to think that much about system load from users. You don't have to think that much about your relational model, so you can postpone all kinds of nasty questions about data integrity and multiplicity early on. This enables a team to move fast and break things. The switch to something like Postgres comes later in the project where you know your requirements better and you need a system where you have more control over your data. You might even have a person responsible for maintaining your data for you on the payroll. Personally, I usually just grab Postgres these days and use `json` columns for the documents and loosely fitting data, then drag out things into a more relational model as we go.
  - richajak 6y ago

lexx 6y ago

I started doing a demo rewrite of my app in Postrgres from Mongodb. The reality is that even though Portgres seems more stable etc, making queries in Mongodb is much easier and fun. Postgres JSON operators are cryptic and made my head hurt every time I wanted to accomplish something more complex. To express it differently, if mongo and postgres where identical operationally, I would choose mongo every day. More friendly and fun. But there are so many horror stories that I am still not sure that it is the right choice for a safe production environment.

kfk 6y ago
I don’t know I was exited to use json for a couple of months but then I needed the same data for analytics and I quickly missed sql.
- szundi 6y ago
  Some Mongo pro please answer something to this, as without enough experience this is the biggest issue for me in my head.
  - winrid 6y ago
    Mongo's aggregation pipeline is pretty straightforward, so not sure what they're looking for.
deleted 6y ago
[deleted]
tbrock 6y ago
Today it’s just fine. I think a decade ago it was pretty risky but databases take 10 years to do properly.
ryanschneider 6y ago
Are there tools for auto converting mongo queries to postgres’ query format? The mongo drivers are Apache licensed so in theory someone could use them as the basis for a driver that speaks to postgres instead, right? I see tools like that showing up in my search results but since I have no experience with them I won’t link to any, if anyone here does have such experience sharing their results would be appreciated.

makkesk8 6y ago

One big drawback of postgres when updating data in a jsonb column is that it needs to rewrite everything even when you update the value of a single key, this gets very expensive if you need to update many rows. As far as I can tell, Mongodb handles this better. Other than this I also prefer postgres despite this limitation.

jabberwcky 6y ago
Mongo stores objects pretty much the same as PG here, a single key update requires a rewrite of the object. Maybe Mongo has some fancy algorithms to avoid full decode/reencode in some cases, but in principle the physical data formats are almost identical
- isbvhodnvemrwvn 6y ago
  In a tiny bit more detail - mongo uses BSON, which is effectively TLV encoded json. I'm not sure how storage of entire collections is done though. https://en.wikipedia.org/wiki/BSON https://en.wikipedia.org/wiki/BSON
  - rualca 6y ago
    > (...) BSON, which is effectively TLV encoded json. For those who, like me, never heard of TLV prior to this post, it's type-length-value encoding. https://en.wikipedia.org/wiki/Type-length-value https://en.wikipedia.org/wiki/Type-length-value
    - swyx 6y ago
      thanks for the link, you weren't alone
karsinkk 6y ago
Disclaimer : I'm a Software Engineer at Oracle In Oracle 21c and the Autonomous JSON Database, we introduced a new Native JSON Datatype with it's own Binary Storage format called OSON[1]. With OSON, you can perform partial updates on a json document. The added benefit is that this results in significantly lesser redo log size. [1] https://blogs.oracle.com/jsondb/osonformat https://blogs.oracle.com/jsondb/osonformat

Jeema101 6y ago

The takeway IMO is that the reason their transition was so seamless is because they took the time to do it 'the right way' - that is: set up the new system in parallel, duplicating the reads and writes, and verifying that things looked good every step of the way. That gives you not only a long window to test and validate, but also the option to rollback if things suddenly don't look so good with the new system. It seems like people sometimes get impatient and want to do a 'big switchover' with insufficient production testing and no real rollback plan, but in my experience that's almost never a good idea.

lma21 6y ago
Indeed, though that must have cost them twice as much to do the migration with the two systems being live in parallel. Some choose to avoid the extra cost.
- tyre 6y ago
  Cost is not only measured in dollars for servers. The cost of doing a massive, irreversible migration that doesn’t work seamlessly the first time is significant. And often more significant in engineering time, lost revenue from services going offline, trust both from external users and internal teams watching the shitshow, and possibly data loss or backwards-incompatible changes than those server dollars. It’s funny that one consistent theme of the best engineers I’ve worked with is less their “amazing 10x ideas” than the number of “sounds like a good idea if you’ve never done it” ideas that they shut down. Huge migrations are one.
  - acdha 6y ago
    I had an example of that recently: we’ve been buying grocery boxes from Hungry Harvest. They announced a new website was coming a couple of weeks ago and had the usually puffery about how it’d be better in vague ways (the old one was not noticeably bad). Then the site was down longer than expected. Then everyone was told their order customizations were lost. Now it’s so bad that they’re skipping an entire week’s delivery for all of their customers. I don’t know how they got there but that’s expensive, disrupts their supply chain, and I suspect they’d cheerfully pay four times as much not to have to slog through this mess.

awinter-py 6y ago

> Composer, and its database, originally started their lives in Guardian Cloud – a data centre in the basement of our office near Kings Cross yes

colesantiago 6y ago

you would think that the hate that mongodb (and mongodb inc) gets right now that the stock should be shorted to oblivion by now. long overdue.

Nextgrid 6y ago
I don't think the hate on MongoDB or NoSQL itself is justified. The hate should be directed at developers or CTOs that go "all in" on it because of hype or lack of skill and the illusion that not having to deal with schemas is somehow a silver bullet. NoSQL databases are great for storing arbitrary JSON-like structures, retrieving them by ID and occasional queries/aggregations on the inner fields of those structures. NoSQL is absolutely not suitable for most business logic and the lack of a schema is actually a major drawback. It feels like a solution (because inserting invalid data will succeed on NoSQL compared to a conventional DB which will reject it based on constraint violations, missing fields or mismatched column types) but in reality you're just kicking the problem down the road and it will come back to bite you because your application now has to be able to deal with this inconsistent data (and in most cases that isn't accounted for with dealing with NoSQL, and the result is predictable). I've been on a project where the main database was MongoDB and while it worked fine for the most part, we'd get exceptions when the application tries to read some records (most likely from earlier on in the business' lifetime) that had missing keys and would predictably explode. This isn't the fault of MongoDB by itself - the whole point of it is to be able to deal with unstructured data - but the fact that someone chose to use it while they actually needed structured data and constraints to ensure only valid data is inserted in the DB, which traditional relational databases do provide. Of course, it's easier to blame MongoDB rather than admit "we've been stupid and shouldn't have chosen the wrong tool for the job".
- pydry 6y ago
  I hate on Mongodb partly because of the super aggressive salesforce that tries to sweet talk less-technical managers and grab internal "champions". It's a poster boy for making a shoddy product and then trying to paper over the deficiencies with sales and marketing. Part of this marketing includes creating a super easy, slick set up for beginners and just burying the grenades for beginner users to discover later when they're already locked in. NoSQL is indeed appropriate sometimes, but Mongo never is.

436363636 6y ago

The sheer number of "why we switched from Mongo to Postgres" blog posts is astounding, since all or most of the reasons typically cited would have been just as valid for not using in Mongo in the first place, all the way back to 2010. These blog posts amount to a bunch of people not only admitting incompetence, but apparently lacking even the self-awareness to realize that that's what they're doing.

AzzieElbab 6y ago

All newspapers sites should use Cassandra on the backend. It is columnar ...