4 ms·

Shocking to see a company like Pinterest has built multiple in house data stores. Talking about a maintenance burden...

vkazanov 2y ago
My company's data function, established 14 years ago, has a bunch of custom stores, as well as data-related tools. Back then data use cases were not really well-known, we had to build things from scratch a lot. It is a very different landscape these days!
lopkeny12ko 2y ago
This is my first reaction as well. How in the world do you end up with close to 10 different databases in production? And furthermore, most of which are totally proprietary? Just use Postgres...
- aeyes 2y ago
  Easy: These companies have a promotion process which require you to generate "impact" to get promoted. Using existing but boring technology which solves the problem in an efficient manner apparently doesn't show off how smart you are. This is not a snarky comment from an outsider, I very much work at such a company. It's ridiculous. Edit: But I must admit that most of this was probably developed 10 years ago when the ecosystem was much more limited. At least they now came to the conclusion that it's time to move on. I would not have recommended Postgres 10 years ago and even today there is no way it would work at the scale of Pinterest.
  - openplatypus 2y ago
    So sad because so true. Seeing working technology replaced because someone doesn't like it, get accolades for it, only to face the same limitations as old tech is just depressing.
    - aeyes 2y ago
      You forgot the next step: The people who developed it move on to other companies and nobody wants to touch it.
  - ted_dunning 2y ago
    HBase is coming up on twenty years old. In 2008, scaling was really hard. And the senior engineers at that point had been badly burned by trying to scale during the 2000-2005 period. The comment about servers of the time being less powerful than smart phones of today is spot on.
- doctorpangloss 2y ago
  Your application wants to use Postgres. Kubernetes, which it runs on, wants etcd. Keycloak wants to use Postgres and Redis for sessions. Plausible Analytics wants to use Clickhouse and Postgres. Matomo specifically wants to use MySQL. Lago Billing wants to use Postgres, Redis and Clickhouse. Prometheus uses TSDB, but okay, it’s “just” files and a WAL. Wait a minute, so is Postgres… Jaeger wants to use Cassandra or Elastic. Thanos wants S3 APIs. So maybe you use Minio. Okay. That’s 2024: in production, you will wind up with 7 different databases. At least.
- dehrmann 2y ago
  Not sure if I can get you to 10, but different DBs work at different scales for different workloads. There's relational, key-value, caching, time series, data warehouse, and search index.
- llm_trw 2y ago
  >Just use Postgres Now try running Postgres on vintage hardware from the period and you quickly see why we did what we did.
  - lopkeny12ko 2y ago
    You would be surprised at how far a well-tuned and optimized Postgres instance will get you. I've worked with so many teams who chase the newest and shiniest database, deploying hundreds of nodes, and spending hundreds of man-hours staffing a sysadmin team to maintain it, only to eventually replace it with an old fashioned Postgres box. You just need an experienced DBA. Edit to add: many claims of "Postgres doesn't work at our scale" are frankly BS, and in large part due to an inexperienced ops team. Take a look at the top post on frontpage right now, for example: https://news.ycombinator.com/item?id=40372296 https://news.ycombinator.com/item?id=40372296
    - llm_trw 2y ago
      > You would be surprised at how far a well-tuned and optimized Postgres instance will get you. I wouldn't be because I was hired after a team of very experienced DBAs saw their databases melt when trying to run analytics queries on the production database in 2012 for a telco. We then moved to HBase so analysts could figure out what parts of the network needed extra towers installed for peak demand with 15 second intervals. You can do this today because the database sits on a machine that is between 10 to 100 times as capable as what we had back then and the number of people using cellphones hasn't increased substantially. Again, don't look at this technology by what you can do today, but by what you could do back then.
- cortesoft 2y ago
  They are operating at a scale where you can’t “just use Postgres”
zerkten 2y ago
If you rewind 10 years to a time before waves of big tech layoffs and the market for engineers was very much in favor, you'll find your answer. The explosive growth in hiring coupled with FOMO-driven risk taking resulted in many projects which were essentially resumeware for engineers. This environment permitted people to escape strategic thinking and any consequences. Imagine having to deal with multiple data stores for your daily development but then go on-call for a week and have to deal with twice as many plus the inevitable lack of runbooks etc.? I personally have a big tech experience with the same data store proliferation in an organization with pretty solid RDBMS use. In the last 3-4 years we've been undoing the damage and migrating data. Toil for engineers is lower and uptimes are better.
- llm_trw 2y ago
  At the time there were _no_ tools that scaled to what people needed so they had to build their own. It's easy to look back and think we were idiots for doing it that way but you need to remember that the average server in 2014 has the power of the average smartphone today.
zenbowman 2y ago
This was the norm about a decade ago. When I was at Hulu, we built our own analytics platform on top of Hadoop, we almost certainly wouldn't have done that today with the amount of off-the-shelf stuff available. Even things like protobuf or Avro weren't as broadly adopted (>80%) at the time, many many companies at the time persisted stuff using JSON or other text formats (which in retrospect was very dumb, but it was very normal for a while).

My company's data function, established 14 years ago, has a bunch of custom stores, as well as data-related tools. Back then data use cases were not really well-known, we had to build things from scratch a lot. It is a very different landscape these days!

lopkeny12ko 2y ago

This is my first reaction as well. How in the world do you end up with close to 10 different databases in production? And furthermore, most of which are totally proprietary? Just use Postgres...

aeyes 2y ago
Easy: These companies have a promotion process which require you to generate "impact" to get promoted. Using existing but boring technology which solves the problem in an efficient manner apparently doesn't show off how smart you are. This is not a snarky comment from an outsider, I very much work at such a company. It's ridiculous. Edit: But I must admit that most of this was probably developed 10 years ago when the ecosystem was much more limited. At least they now came to the conclusion that it's time to move on. I would not have recommended Postgres 10 years ago and even today there is no way it would work at the scale of Pinterest.
- openplatypus 2y ago
  So sad because so true. Seeing working technology replaced because someone doesn't like it, get accolades for it, only to face the same limitations as old tech is just depressing.
  - aeyes 2y ago
    You forgot the next step: The people who developed it move on to other companies and nobody wants to touch it.
- ted_dunning 2y ago
  HBase is coming up on twenty years old. In 2008, scaling was really hard. And the senior engineers at that point had been badly burned by trying to scale during the 2000-2005 period. The comment about servers of the time being less powerful than smart phones of today is spot on.
doctorpangloss 2y ago
Your application wants to use Postgres. Kubernetes, which it runs on, wants etcd. Keycloak wants to use Postgres and Redis for sessions. Plausible Analytics wants to use Clickhouse and Postgres. Matomo specifically wants to use MySQL. Lago Billing wants to use Postgres, Redis and Clickhouse. Prometheus uses TSDB, but okay, it’s “just” files and a WAL. Wait a minute, so is Postgres… Jaeger wants to use Cassandra or Elastic. Thanos wants S3 APIs. So maybe you use Minio. Okay. That’s 2024: in production, you will wind up with 7 different databases. At least.
dehrmann 2y ago
Not sure if I can get you to 10, but different DBs work at different scales for different workloads. There's relational, key-value, caching, time series, data warehouse, and search index.
llm_trw 2y ago
>Just use Postgres Now try running Postgres on vintage hardware from the period and you quickly see why we did what we did.
- lopkeny12ko 2y ago
  You would be surprised at how far a well-tuned and optimized Postgres instance will get you. I've worked with so many teams who chase the newest and shiniest database, deploying hundreds of nodes, and spending hundreds of man-hours staffing a sysadmin team to maintain it, only to eventually replace it with an old fashioned Postgres box. You just need an experienced DBA. Edit to add: many claims of "Postgres doesn't work at our scale" are frankly BS, and in large part due to an inexperienced ops team. Take a look at the top post on frontpage right now, for example: https://news.ycombinator.com/item?id=40372296 https://news.ycombinator.com/item?id=40372296
  - llm_trw 2y ago
    > You would be surprised at how far a well-tuned and optimized Postgres instance will get you. I wouldn't be because I was hired after a team of very experienced DBAs saw their databases melt when trying to run analytics queries on the production database in 2012 for a telco. We then moved to HBase so analysts could figure out what parts of the network needed extra towers installed for peak demand with 15 second intervals. You can do this today because the database sits on a machine that is between 10 to 100 times as capable as what we had back then and the number of people using cellphones hasn't increased substantially. Again, don't look at this technology by what you can do today, but by what you could do back then.
cortesoft 2y ago
They are operating at a scale where you can’t “just use Postgres”

zerkten 2y ago

If you rewind 10 years to a time before waves of big tech layoffs and the market for engineers was very much in favor, you'll find your answer. The explosive growth in hiring coupled with FOMO-driven risk taking resulted in many projects which were essentially resumeware for engineers. This environment permitted people to escape strategic thinking and any consequences. Imagine having to deal with multiple data stores for your daily development but then go on-call for a week and have to deal with twice as many plus the inevitable lack of runbooks etc.? I personally have a big tech experience with the same data store proliferation in an organization with pretty solid RDBMS use. In the last 3-4 years we've been undoing the damage and migrating data. Toil for engineers is lower and uptimes are better.

llm_trw 2y ago
At the time there were _no_ tools that scaled to what people needed so they had to build their own. It's easy to look back and think we were idiots for doing it that way but you need to remember that the average server in 2014 has the power of the average smartphone today.

zenbowman 2y ago

This was the norm about a decade ago. When I was at Hulu, we built our own analytics platform on top of Hadoop, we almost certainly wouldn't have done that today with the amount of off-the-shelf stuff available. Even things like protobuf or Avro weren't as broadly adopted (>80%) at the time, many many companies at the time persisted stuff using JSON or other text formats (which in retrospect was very dumb, but it was very normal for a while).