3 ms·

They were using it like a NoSQL database back in 2010, just one year after MongoDB started. So there were no options. In 2022, there are so many more mature, r

They were using it like a NoSQL database back in 2010, just one year after MongoDB started. So there were no options. In 2022, there are so many more mature, reliable, battle-tested NoSQL options that could have solved their problem more elegantly. But even today, if I were to build Reddit as a startup, I'd start with a Postgres database and go as far as I can. Postgres allows me to use relations when I want to enforce data integrity and JSONB when I just want a key/value store.

lmm 4y ago
> But even today, if I were to build Reddit as a startup, I'd start with a Postgres database and go as far as I can. Postgres allows me to use relations when I want to enforce data integrity and JSONB when I just want a key/value store. This is accepted wisdom on HN but I don't see it: - True HA is very difficult. Even just automatic failover takes a lot of manual sysadmin effort - You can't autoscale, and while scaling is a nice problem to have, having to not just fundamentally rearchitect all your code when you need to shard, but also probably migrate all your data, is probably the worst version of it. - Even if you're not using transactions, you still pay the price for them. For example your writes won't commit until all your secondary indices have been updated. - If you're actually using JSONB then driver/mapper/etc. support is very limited. E.g. it's still quite fiddly to have an object where one field is a set of strings / enum values and have a standard ORM store that in a column rather than creating a join table.
- naniwaduni 4y ago
  Does a startup in Reddit's niche demand HA?
  - lmm 4y ago
    I would think that if you want to create a habit where people check you frequently, avoiding downtime would be important. But IDK.
    - antupis 4y ago
      you probably would have some CDN front of it so some database unavailability would not impact to browsing that much.
    - sophacles 4y ago
      Reddit has gone through many periods of regular unplanned downtime as they hit various scale issues over the years. It's certainly something to avoid, but not necessarily something that's catastrophic - like most decisions there are tradeoffs.
    - rakoo 4y ago
      For something like reddit, who cares about consistency ? Under massive load no user cares about knowing the exact number of votes, all the replies to a comment, etc... It's also perfectly fine to refuse some writes if the load is too high to bear.
- senttoschool 4y ago
  If I were to start building Reddit from scratch without knowing if it's going to be successful or not, I'd rather iterate quick on features rather trying to spend time scaling something that will most likely fail before reaching that kind of scale. >True HA is very difficult. Even just automatic failover takes a lot of manual sysadmin effort Nearly all Postgres cloud providers such as RDS, CloudSQL, Upcloud, provide multi-region nodes and automatic failover. Also, true HA databases are very expensive and have their own drawbacks. Not really designed for a Reddit-style startup. >You can't autoscale, and while scaling is a nice problem to have, having to not just fundamentally rearchitect all your code when you need to shard, but also probably migrate all your data, is probably the worst version of it. Hence why I said "as far as I can". I can keep upgrading the instance, creating material views, caching responses, etc. Lastly, the most important reason is because I know Postgres and I can get started today. I don't want to learn a highly scalable, new type of database that I might need 5 years down the road.
  - lmm 4y ago
    > Nearly all Postgres cloud providers such as RDS, CloudSQL, Upcloud, provide multi-region nodes and automatic failover. From memory you have to specify a maintenance window and failover is not instant. And relying on your cloud provider comes with its own drawbacks. > Also, true HA databases are very expensive and have their own drawbacks. Not really designed for a Reddit-style startup. Plenty of free HA datastores out there, and if we're assuming it'll be managed by the cloud provider anyway then that's a wash. > Lastly, the most important reason is because I know Postgres and I can get started today. I don't want to learn a highly scalable, new type of database that I might need 5 years down the road. That's a good argument for using what you know; by the same token I'd use Cassandra. Postgres is a very complex system with lots of legacy and various edge cases.
deleted 4y ago
[deleted]
senttoschool 4y ago
On a side note, I find it a bit ironic how NoSQL was all the rage back in the last decade or so but in 2022, NoSQL DBs are racing to add SQL querying back to key value stores. It turns out that SQL is crucial to non-app-developers such as business analysts, data scientists. Trying to setup an ETL to pull data from your MongoDB datastore to a Postgres DB so the analysts could generate reports is such a waste of time and resources. Or worse, the analysts have to request devs to write Javascript code to query the data they need. For this reason alone, I will always start with a DB that supports SQL out of the box.
- deleted 4y ago
  [deleted]
- ralusek 4y ago
  People didn't leave RDBMS for NoSQL because they wanted to get rid of RDBMS features, they did it because they couldn't scale their applications on single instances anymore. So they had to give up transactions and foreign keys and schemas, etc, because those concepts just didn't exist across instances. Not because they wanted to, but because they had to. The entire database world has been working ever since trying to get back those RDBMS features into their horizontally scalable solutions, it's just taken a while because the problems are exponentially harder in a network. It's not like people are using eventual consistency and a loss of referential integrity because they prefer it.
  - senttoschool 4y ago
    >People didn't leave RDBMS for NoSQL because they wanted to get rid of RDBMS features, they did it because they couldn't scale their applications on single instances anymore. I agree that this should have been the reason for choosing something like MongoDB over MySQL/Postgres. In reality, I saw a lot about the companies switching/starting with because so their devs didn't have to learn SQL and they can just stay in Javascript/whatever language they were using.
    - Tainnor 4y ago
Misdicorl 4y ago
> So there were no options. Hbase was a pretty solid NoSQL option by 2010, no?

lmm 4y ago

> But even today, if I were to build Reddit as a startup, I'd start with a Postgres database and go as far as I can. Postgres allows me to use relations when I want to enforce data integrity and JSONB when I just want a key/value store. This is accepted wisdom on HN but I don't see it: - True HA is very difficult. Even just automatic failover takes a lot of manual sysadmin effort - You can't autoscale, and while scaling is a nice problem to have, having to not just fundamentally rearchitect all your code when you need to shard, but also probably migrate all your data, is probably the worst version of it. - Even if you're not using transactions, you still pay the price for them. For example your writes won't commit until all your secondary indices have been updated. - If you're actually using JSONB then driver/mapper/etc. support is very limited. E.g. it's still quite fiddly to have an object where one field is a set of strings / enum values and have a standard ORM store that in a column rather than creating a join table.

naniwaduni 4y ago
Does a startup in Reddit's niche demand HA?
- lmm 4y ago
  I would think that if you want to create a habit where people check you frequently, avoiding downtime would be important. But IDK.
  - antupis 4y ago
    you probably would have some CDN front of it so some database unavailability would not impact to browsing that much.
  - sophacles 4y ago
    Reddit has gone through many periods of regular unplanned downtime as they hit various scale issues over the years. It's certainly something to avoid, but not necessarily something that's catastrophic - like most decisions there are tradeoffs.
  - rakoo 4y ago
    For something like reddit, who cares about consistency ? Under massive load no user cares about knowing the exact number of votes, all the replies to a comment, etc... It's also perfectly fine to refuse some writes if the load is too high to bear.
senttoschool 4y ago
If I were to start building Reddit from scratch without knowing if it's going to be successful or not, I'd rather iterate quick on features rather trying to spend time scaling something that will most likely fail before reaching that kind of scale. >True HA is very difficult. Even just automatic failover takes a lot of manual sysadmin effort Nearly all Postgres cloud providers such as RDS, CloudSQL, Upcloud, provide multi-region nodes and automatic failover. Also, true HA databases are very expensive and have their own drawbacks. Not really designed for a Reddit-style startup. >You can't autoscale, and while scaling is a nice problem to have, having to not just fundamentally rearchitect all your code when you need to shard, but also probably migrate all your data, is probably the worst version of it. Hence why I said "as far as I can". I can keep upgrading the instance, creating material views, caching responses, etc. Lastly, the most important reason is because I know Postgres and I can get started today. I don't want to learn a highly scalable, new type of database that I might need 5 years down the road.
- lmm 4y ago
  > Nearly all Postgres cloud providers such as RDS, CloudSQL, Upcloud, provide multi-region nodes and automatic failover. From memory you have to specify a maintenance window and failover is not instant. And relying on your cloud provider comes with its own drawbacks. > Also, true HA databases are very expensive and have their own drawbacks. Not really designed for a Reddit-style startup. Plenty of free HA datastores out there, and if we're assuming it'll be managed by the cloud provider anyway then that's a wash. > Lastly, the most important reason is because I know Postgres and I can get started today. I don't want to learn a highly scalable, new type of database that I might need 5 years down the road. That's a good argument for using what you know; by the same token I'd use Cassandra. Postgres is a very complex system with lots of legacy and various edge cases.

deleted 4y ago

[deleted]

senttoschool 4y ago

On a side note, I find it a bit ironic how NoSQL was all the rage back in the last decade or so but in 2022, NoSQL DBs are racing to add SQL querying back to key value stores. It turns out that SQL is crucial to non-app-developers such as business analysts, data scientists. Trying to setup an ETL to pull data from your MongoDB datastore to a Postgres DB so the analysts could generate reports is such a waste of time and resources. Or worse, the analysts have to request devs to write Javascript code to query the data they need. For this reason alone, I will always start with a DB that supports SQL out of the box.

deleted 4y ago
[deleted]
ralusek 4y ago
People didn't leave RDBMS for NoSQL because they wanted to get rid of RDBMS features, they did it because they couldn't scale their applications on single instances anymore. So they had to give up transactions and foreign keys and schemas, etc, because those concepts just didn't exist across instances. Not because they wanted to, but because they had to. The entire database world has been working ever since trying to get back those RDBMS features into their horizontally scalable solutions, it's just taken a while because the problems are exponentially harder in a network. It's not like people are using eventual consistency and a loss of referential integrity because they prefer it.
- senttoschool 4y ago
  >People didn't leave RDBMS for NoSQL because they wanted to get rid of RDBMS features, they did it because they couldn't scale their applications on single instances anymore. I agree that this should have been the reason for choosing something like MongoDB over MySQL/Postgres. In reality, I saw a lot about the companies switching/starting with because so their devs didn't have to learn SQL and they can just stay in Javascript/whatever language they were using.
  - Tainnor 4y ago

Misdicorl 4y ago

> So there were no options. Hbase was a pretty solid NoSQL option by 2010, no?