10 ms·

“just put Twitter in the cloud” we tried that- $300M monthly

63 points by ren_engineer 4y ago

funstuff007 4y ago
I guess this makes FTX's $250K a month spend on AWS downright thrifty.
- nunez 4y ago
  there are banks that also spend millions per month on AWS
  - funstuff007 4y ago
    probably the same banks to say no to vendors who cannot provide on-prem solutions. but, but, but it's ok if you're AWS.
    - nunez 4y ago
      or if you say "on-prem is on the roadmap"
mikkergp 4y ago
Often people will say “AWS is so much bigger than us, they have to accommodate us / what could we know that they don’t” is that AWS is still building for the median customer (+/-). It doesn’t make financial sense to optimize for the extremes and extremes can be extreme (per the tweet thread). If you’re a reasonably technical company there’s a good chance something about your main workloads will challenge AWS.
- PaulsWallet 4y ago
  I wouldn't call my company reasonably technical and we frequently hit barriers in AWS. Twitter has no chance on AWS.
- yrgulation 4y ago
  deployments alone are a challenge. there are so many race conditions it’s sometimes impossible to do it reliably.
Ptchd 4y ago
The cloud is for peace of mind (in some ways)... not for saving money
yrgulation 4y ago
“in the cloud” - is it a desktop app now? or is the world cloud synonymous with aws, and aws only?
andrewstuart 4y ago
I can’t understand how cloud means “simple and cheap” to so many people and companies. In fact it’s the absolute opposite. There’s tons of much, much more powerful and cheap servers you can get online than those of the major clouds, often with little or no traffic fees. Most companies are too lazy to even investigate the choices.
- gsatic 4y ago
  More than laziness, it's too complex a problem for most decision makers. Environment is ever changing and extremely competitive. Most people don't handle such problems well and few get the training or have the support. It's pretty much like crypto. Most people are sold shit they have no capacity to understand. So the story writes itself. These are info asymmetry problems and society and business hasn't been dealing with them at such scales for too long. So solutions are a work in progress.
- papito 4y ago
  The cloud is great when you are starting out, but at scale you are going to hemorrhage money. This is why it's dangerous to tie yourself to all those cloud "toys" and then rapidly expand. If I just want to have my own little home page, a micro EC2 instance will do just fine - I don't want to go to a colocation for that. Once you are at scale, you need to get out of the cloud. I agree with DHH on this. https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0 https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...
the8bit 4y ago
m6.12xl are 48 vCPU systems. 25 million cores is a hilariously huge amount of compute to use to run a site of twitter size, which I guess does point to how much tech debt they have ignored over time.
- throwaway2729 4y ago
  They should have probably written it in Assembly.
- crazygringo 4y ago
  By "twitter size" are you implying it's... small? Twitter is the 4th most visited site in the world. https://en.wikipedia.org/wiki/List_of_most_visited_websites https://en.wikipedia.org/wiki/List_of_most_visited_websites
  - steve1977 4y ago
    But it’s basically just simple data moving, it should be rather trivial in terms of compute power needed.
    - mazurnification 4y ago
      There is nothing simple in data moving. Problem is that it can be abstracted away so you do not have to worry about it. Out of sight out of mind. Example - majority of the CPU area is devoted to data moving problem (cache, branch prediction, out of order execution, register renaming). The same is happening on other levels of the stack (OS, network and so on).
    - deleted 4y ago
      [deleted]
- lolbserious 4y ago
  It sounds high for sure. But they do serve videos, which possibly they have to transcode or otherwise process on import (?) They probably also have some filters for pictures to remove porn and stuff like that. This can amount to serious CPU if you have millions of users uploading that stuff daily. A pure text-based Twitter would be much smaller.
karmakaze 4y ago
Developing for the cloud makes sense when you're developing features as fast as you can leveraging cloud services to make it all easier. When you have a relatively stable feature set and scaling massively it's better to use private datacenters.
funstuff007 4y ago
Snapchat pays Google cloud $400MM a year. Using these cost metrics, are Twitter's compute needs 9X those of Snapchat? maybe so. seems reasonable to me. And then perhaps, this person is saying that cloud (Google or otherwise) is not the way to go for Snapchat https://www.vox.com/2017/3/1/14661126/snap-snapchat-ipo-spending-2-billion-google-cloud https://www.vox.com/2017/3/1/14661126/snap-snapchat-ipo-spen...
trh0awayman 4y ago
Isn't this sorta what Oxide Computer is trying to solve? Where you can have your cloud and eat it too?
larksimian 4y ago
300m/month to deliver text, some photos and a bit of video to (admittedly tons of) people sounds insane. Then again based on the architecture sketch Elon posted https://mobile.twitter.com/elonmusk/status/1593899029531803649 https://mobile.twitter.com/elonmusk/status/15938990295318036... most of the compute is being used by ... the feed recommender AI crap that works awfully anyway? Insanely bad job programming this product.
tim333 4y ago
>500,000 m6gd.12xlarge systems And I think HN still runs on one server? 500,000 for twitter seems a tad inefficient
- aaomidi 4y ago
  Did you compare HN with Twitter?
  - jscheel 4y ago
    lol, right?
  - tim333 4y ago
    I guess kinda. Googling HN has about 300k daily active users, Twitter about 250m so 800x approx.
    - FartyMcFarter 4y ago
      Do you really think the number of users tells the whole story? The websites are very different.

63 points by ren_engineer 4y ago

funstuff007 4y ago

I guess this makes FTX's $250K a month spend on AWS downright thrifty.

nunez 4y ago
there are banks that also spend millions per month on AWS
- funstuff007 4y ago
  probably the same banks to say no to vendors who cannot provide on-prem solutions. but, but, but it's ok if you're AWS.
  - nunez 4y ago
    or if you say "on-prem is on the roadmap"

mikkergp 4y ago

Often people will say “AWS is so much bigger than us, they have to accommodate us / what could we know that they don’t” is that AWS is still building for the median customer (+/-). It doesn’t make financial sense to optimize for the extremes and extremes can be extreme (per the tweet thread). If you’re a reasonably technical company there’s a good chance something about your main workloads will challenge AWS.

PaulsWallet 4y ago
I wouldn't call my company reasonably technical and we frequently hit barriers in AWS. Twitter has no chance on AWS.
yrgulation 4y ago
deployments alone are a challenge. there are so many race conditions it’s sometimes impossible to do it reliably.

Ptchd 4y ago

The cloud is for peace of mind (in some ways)... not for saving money

yrgulation 4y ago

“in the cloud” - is it a desktop app now? or is the world cloud synonymous with aws, and aws only?

andrewstuart 4y ago

I can’t understand how cloud means “simple and cheap” to so many people and companies. In fact it’s the absolute opposite. There’s tons of much, much more powerful and cheap servers you can get online than those of the major clouds, often with little or no traffic fees. Most companies are too lazy to even investigate the choices.

gsatic 4y ago
More than laziness, it's too complex a problem for most decision makers. Environment is ever changing and extremely competitive. Most people don't handle such problems well and few get the training or have the support. It's pretty much like crypto. Most people are sold shit they have no capacity to understand. So the story writes itself. These are info asymmetry problems and society and business hasn't been dealing with them at such scales for too long. So solutions are a work in progress.
papito 4y ago
The cloud is great when you are starting out, but at scale you are going to hemorrhage money. This is why it's dangerous to tie yourself to all those cloud "toys" and then rapidly expand. If I just want to have my own little home page, a micro EC2 instance will do just fine - I don't want to go to a colocation for that. Once you are at scale, you need to get out of the cloud. I agree with DHH on this. https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0 https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...

the8bit 4y ago

m6.12xl are 48 vCPU systems. 25 million cores is a hilariously huge amount of compute to use to run a site of twitter size, which I guess does point to how much tech debt they have ignored over time.

throwaway2729 4y ago
They should have probably written it in Assembly.
crazygringo 4y ago
By "twitter size" are you implying it's... small? Twitter is the 4th most visited site in the world. https://en.wikipedia.org/wiki/List_of_most_visited_websites https://en.wikipedia.org/wiki/List_of_most_visited_websites
- steve1977 4y ago
  But it’s basically just simple data moving, it should be rather trivial in terms of compute power needed.
  - mazurnification 4y ago
    There is nothing simple in data moving. Problem is that it can be abstracted away so you do not have to worry about it. Out of sight out of mind. Example - majority of the CPU area is devoted to data moving problem (cache, branch prediction, out of order execution, register renaming). The same is happening on other levels of the stack (OS, network and so on).
  - deleted 4y ago
    [deleted]
lolbserious 4y ago
It sounds high for sure. But they do serve videos, which possibly they have to transcode or otherwise process on import (?) They probably also have some filters for pictures to remove porn and stuff like that. This can amount to serious CPU if you have millions of users uploading that stuff daily. A pure text-based Twitter would be much smaller.

karmakaze 4y ago

Developing for the cloud makes sense when you're developing features as fast as you can leveraging cloud services to make it all easier. When you have a relatively stable feature set and scaling massively it's better to use private datacenters.

funstuff007 4y ago

Snapchat pays Google cloud $400MM a year. Using these cost metrics, are Twitter's compute needs 9X those of Snapchat? maybe so. seems reasonable to me. And then perhaps, this person is saying that cloud (Google or otherwise) is not the way to go for Snapchat https://www.vox.com/2017/3/1/14661126/snap-snapchat-ipo-spending-2-billion-google-cloud https://www.vox.com/2017/3/1/14661126/snap-snapchat-ipo-spen...

trh0awayman 4y ago

Isn't this sorta what Oxide Computer is trying to solve? Where you can have your cloud and eat it too?

larksimian 4y ago

300m/month to deliver text, some photos and a bit of video to (admittedly tons of) people sounds insane. Then again based on the architecture sketch Elon posted https://mobile.twitter.com/elonmusk/status/1593899029531803649 https://mobile.twitter.com/elonmusk/status/15938990295318036... most of the compute is being used by ... the feed recommender AI crap that works awfully anyway? Insanely bad job programming this product.

tim333 4y ago

>500,000 m6gd.12xlarge systems And I think HN still runs on one server? 500,000 for twitter seems a tad inefficient

aaomidi 4y ago
Did you compare HN with Twitter?
- jscheel 4y ago
  lol, right?
- tim333 4y ago
  I guess kinda. Googling HN has about 300k daily active users, Twitter about 250m so 800x approx.
  - FartyMcFarter 4y ago
    Do you really think the number of users tells the whole story? The websites are very different.