5 ms·

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immed

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

_3u10 4y ago
No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…
- wikfwikf 4y ago
  There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
  - pmoriarty 4y ago
    Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!). Not tens of thousands at any one organization supporting postgres and clickhouse. No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
    - rvnx 4y ago
      This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
      - _3u10 4y ago
        You’ve obviously never used FredDB
- bigiain 4y ago
  The Postgres engineers are not responsible for running their code on a shared platform supporting a billion users.
  - iso1631 4y ago
    Neither is twitter. Twitter isn't Facebook or Google. It's not even Reddit or Pinterest.
    - badwolf 4y ago
      All those tweets with pictures and video that also get quoted and embedded in news articles that also need to be seen by people without an account.
- itsoktocry 4y ago
  >Where’s the 1000s of engineers for Postgres? You're comparing Postgres to Twitter? If some of the people making comments like this actually work in tech, then yeah, maybe there is a lot of bloat to be cut.
  - 1500100900 4y ago
    Postgres actually does have thousands of engineers but it's not easy to notice because they all share the same name. https://www.citusdata.com/blog/2017/04/20/analyzing-postgresql-email-archives/ https://www.citusdata.com/blog/2017/04/20/analyzing-postgres...
- lightbendover 4y ago
  I am entirely missing how Clickhouse solves the problem Twitter is solving out of the box, do you care to explain? Sub-question: how does Clickhouse provide monetization for a product?
  - _3u10 4y ago
    If Twitter was solving a problem it wouldn’t be in the situation it is. It provides a house for your clicks, then you sell the clicks from your house. In twitters case I would recommend they use checkhouse which will help them monetize their checks. Tumblr is already way ahead of them.
donkeyd 4y ago
It's bloat to devs who just want to write code. You're kinda confirming the point I was trying to make. Anyone on that list performs an important role at a company of their scale. I'm currently the first and sole architect on a product that was built by only devs. I really know why I exist.

_3u10 4y ago

No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…

wikfwikf 4y ago
There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.
- pmoriarty 4y ago
  Tens of thousands across all the organizations where postgres and clickhouse are used, maybe (though even then probably not!). Not tens of thousands at any one organization supporting postgres and clickhouse. No single organization needs even hundreds of people to support these apps. You just need a good architecture and a handful of dba's, developers, and sysadmins... maybe.. depending on your scale. At many smaller orgs you can probably get away with one.
  - rvnx 4y ago
    This 100%. You don't need 10'000 engineers to run billions of queries. It's not like the SREs are running the queries themselves. Your job as an SRE is to design and run computer systems, not to become the computing platform yourself.
    - _3u10 4y ago
      You’ve obviously never used FredDB
bigiain 4y ago
The Postgres engineers are not responsible for running their code on a shared platform supporting a billion users.
- iso1631 4y ago
  Neither is twitter. Twitter isn't Facebook or Google. It's not even Reddit or Pinterest.
  - badwolf 4y ago
    All those tweets with pictures and video that also get quoted and embedded in news articles that also need to be seen by people without an account.
itsoktocry 4y ago
>Where’s the 1000s of engineers for Postgres? You're comparing Postgres to Twitter? If some of the people making comments like this actually work in tech, then yeah, maybe there is a lot of bloat to be cut.
- 1500100900 4y ago
  Postgres actually does have thousands of engineers but it's not easy to notice because they all share the same name. https://www.citusdata.com/blog/2017/04/20/analyzing-postgresql-email-archives/ https://www.citusdata.com/blog/2017/04/20/analyzing-postgres...
lightbendover 4y ago
I am entirely missing how Clickhouse solves the problem Twitter is solving out of the box, do you care to explain? Sub-question: how does Clickhouse provide monetization for a product?
- _3u10 4y ago
  If Twitter was solving a problem it wouldn’t be in the situation it is. It provides a house for your clicks, then you sell the clicks from your house. In twitters case I would recommend they use checkhouse which will help them monetize their checks. Tumblr is already way ahead of them.

donkeyd 4y ago

It's bloat to devs who just want to write code. You're kinda confirming the point I was trying to make. Anyone on that list performs an important role at a company of their scale. I'm currently the first and sole architect on a product that was built by only devs. I really know why I exist.