6 ms·

2010: "At any moment, Justin Bieber uses 3% of our infrastructure. Racks of servers are dedicated to him" https://gizmodo.com/5632095/justin-bieber-has-dedicat

2010: "At any moment, Justin Bieber uses 3% of our infrastructure. Racks of servers are dedicated to him" https://gizmodo.com/5632095/justin-bieber-has-dedicated-servers-at-twitter https://gizmodo.com/5632095/justin-bieber-has-dedicated-serv...

fizx 8y ago
Bieber-related conversation might reasonably have been 3%. Updating his friend count was a constant source of lock contention on a particular mysql. There were never explicitly dedicated databases/servers, unless you count the hotspot caused by whatever shard he hashed into.
- lwansbrough 8y ago
  You would think a HyperLogLog would be approximately good enough when follower counts reach tens of millions.
  - fizx 8y ago
    Hyperloglog was only just invented, and hadn't really made its way into the popular developer consciousness. Anyhow, the solution is easier than that (implement lock striping, i.e. https://stackoverflow.com/questions/16151606/need-simple-explanation-how-lock-striping-works-with-concurrenthashmap https://stackoverflow.com/questions/16151606/need-simple-exp...), but I was reorg'd away before my code hit production, so I'm not sure what happened there.
    - lorenzhs 8y ago
      Approximate counters are much much older than HyperLogLog. Here’s something from 1978: https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.pdf https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.... Of course this solves a different problem because HyperLogLog is an, uh, interesting way to count followers (why would you need to do a count-distinct query? You can’t follow someone multiple times). In any case, the Flajolet Martin sketch dates back to 1985 and solves the same problem as HyperLogLog: https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algorithm https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
      - aflam 8y ago
        Here is nice presentation covering those: (link: https://www.cs.princeton.edu/~rs/talks/AC11-Cardinality.pdf https://www.cs.princeton.edu/~rs/talks/AC11-Cardinality.pdf)
- tzs 8y ago
  Did they actually need to know Bieber's friend count both accurately and in real time? I would expect that for top Twitter accounts like his, you could get a very good approximation by logging events that increment or decrement the friend count, and using the rates those events are occurring and the last known exact friend count to forecast the current friend count. That would not be exact, but I bet you could get it close enough that users would not see anything off about it.
  - mrgordon 8y ago
    Yes they made Storm for this
  - segmondy 8y ago
    Ha, I always joke that Twitter is a good example of why one should avoid distributed systems. Every often, you see it showing the wrong number of friends, tweets, likes, etc. I don't care, I suppose some people care about such metrics.
    - scarface74 8y ago
      That's the essence of the CAP theorem. You can have two of three - consistency, availability, and partition tolerance. Sometimes it makes sense to choose availability over consistency.
freeflight 8y ago
Clicking that was already worth it for the lead picture, best laugh I had today, thank you!
loopdoend 8y ago
A couple years ago, Bieber tweeted out a Twilio phone number for people to text. Luckily it happened on a Friday night, or it would have caused widespread outages. Celebrity is an edge case that you have to be prepared for.
- mentat 8y ago
  https://mashable.com/2015/10/30/justin-bieber-hotline-bling/#tL1BMuwFoGq0 https://mashable.com/2015/10/30/justin-bieber-hotline-bling/... - It was an interesting experience.
firebones 8y ago
But, but, but...why does Twitter have so many engineers? I could write Twitter in a weekend! --95% of anti-TWTR posters circa 2010-2016.
- spondyl 8y ago
  I think this post is relevant here: https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/
- snovv_crash 8y ago
  Before bring acquired, WhatsApp had what, 30 employees? How did they do it? I know they used custom BSD servers so that a single box could keep close to 1M TCP connections open. I'm sure with a fixed target to aim for and all scope known upfront a small crack team of devs could do something similar for Twitter.
  - akhilcacharya 8y ago
    The beauty of Erlang/OTP!
    - evanweaver 8y ago
      Especially if you completely rewrite Mnesia.
  - zmb_ 8y ago
    One-to-one vs. many-to-many messaging. The amount of work you need to do to deliver a WhatsApp message is constant and small -- just route the message to a single recipient's mailbox. The amount of work Twitter has to do to deliver a message grows as a function of followers. One celebrity tweeting another celebrity means you have to deliver the message to the mailboxes of the followers of both -- millions of times more work than WhatsApp per message. In addition, Twitter persists all the messages while WhatsApp doesn't.
    - segmondy 8y ago
liquidgecka 8y ago
That was a period when the only people talking outside the company were the ones not trying to fix things inside of the company. Twitter never had dedicated machines for individual users. Its just not how the infra ever worked (at least until I left). Requests landed on random boxes behind load balancers, those boxes talked to pools of memcache or mysql, etc. At no point was there ever "racks" or "machines" dedicated to specific individuals like the article claims. That being said, some users created crazy hot shards when specific tweets went absolutely madhouse, especially Beiber and those like him. Random twitter internals tidbit: We had a unit of measurement called a "MJ". Its the number of tweets per second that we had when the rumors of Michael Jacksons death were circling. It basically overloaded the system and had us running around on fire. It was 465 tweets a second. Within a year we had crossed a line where we never were below that number again. Hence "we are at about 12 MJ's" was jokingly used to compare "hair on fire" to every day a couple of years later. =)
- evanweaver 8y ago
  Sorry, we just kept the Bieber box hidden from you in the "special closet". It was a SPARCstation 20 and we were afraid if ops found it they would shut it down.
  - liquidgecka 8y ago
    You know.. I found a special box (PowerMac Pro) in the "special closet" that had been moved like 5 times. It was Blane's old desktop and it had the twitter codebase checked out from like ~2007 era? Kind of nifty seeing just how much had changed since that got tucked away.
- deleted 8y ago
  [deleted]

fizx 8y ago

Bieber-related conversation might reasonably have been 3%. Updating his friend count was a constant source of lock contention on a particular mysql. There were never explicitly dedicated databases/servers, unless you count the hotspot caused by whatever shard he hashed into.

lwansbrough 8y ago
You would think a HyperLogLog would be approximately good enough when follower counts reach tens of millions.
- fizx 8y ago
  Hyperloglog was only just invented, and hadn't really made its way into the popular developer consciousness. Anyhow, the solution is easier than that (implement lock striping, i.e. https://stackoverflow.com/questions/16151606/need-simple-explanation-how-lock-striping-works-with-concurrenthashmap https://stackoverflow.com/questions/16151606/need-simple-exp...), but I was reorg'd away before my code hit production, so I'm not sure what happened there.
  - lorenzhs 8y ago
    Approximate counters are much much older than HyperLogLog. Here’s something from 1978: https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.pdf https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.... Of course this solves a different problem because HyperLogLog is an, uh, interesting way to count followers (why would you need to do a count-distinct query? You can’t follow someone multiple times). In any case, the Flajolet Martin sketch dates back to 1985 and solves the same problem as HyperLogLog: https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algorithm https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
    - aflam 8y ago
      Here is nice presentation covering those: (link: https://www.cs.princeton.edu/~rs/talks/AC11-Cardinality.pdf https://www.cs.princeton.edu/~rs/talks/AC11-Cardinality.pdf)
tzs 8y ago
Did they actually need to know Bieber's friend count both accurately and in real time? I would expect that for top Twitter accounts like his, you could get a very good approximation by logging events that increment or decrement the friend count, and using the rates those events are occurring and the last known exact friend count to forecast the current friend count. That would not be exact, but I bet you could get it close enough that users would not see anything off about it.
- mrgordon 8y ago
  Yes they made Storm for this
- segmondy 8y ago
  Ha, I always joke that Twitter is a good example of why one should avoid distributed systems. Every often, you see it showing the wrong number of friends, tweets, likes, etc. I don't care, I suppose some people care about such metrics.
  - scarface74 8y ago
    That's the essence of the CAP theorem. You can have two of three - consistency, availability, and partition tolerance. Sometimes it makes sense to choose availability over consistency.

freeflight 8y ago

Clicking that was already worth it for the lead picture, best laugh I had today, thank you!

loopdoend 8y ago

A couple years ago, Bieber tweeted out a Twilio phone number for people to text. Luckily it happened on a Friday night, or it would have caused widespread outages. Celebrity is an edge case that you have to be prepared for.

mentat 8y ago
https://mashable.com/2015/10/30/justin-bieber-hotline-bling/#tL1BMuwFoGq0 https://mashable.com/2015/10/30/justin-bieber-hotline-bling/... - It was an interesting experience.

firebones 8y ago

But, but, but...why does Twitter have so many engineers? I could write Twitter in a weekend! --95% of anti-TWTR posters circa 2010-2016.

spondyl 8y ago
I think this post is relevant here: https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/
snovv_crash 8y ago
Before bring acquired, WhatsApp had what, 30 employees? How did they do it? I know they used custom BSD servers so that a single box could keep close to 1M TCP connections open. I'm sure with a fixed target to aim for and all scope known upfront a small crack team of devs could do something similar for Twitter.
- akhilcacharya 8y ago
  The beauty of Erlang/OTP!
  - evanweaver 8y ago
    Especially if you completely rewrite Mnesia.
- zmb_ 8y ago
  One-to-one vs. many-to-many messaging. The amount of work you need to do to deliver a WhatsApp message is constant and small -- just route the message to a single recipient's mailbox. The amount of work Twitter has to do to deliver a message grows as a function of followers. One celebrity tweeting another celebrity means you have to deliver the message to the mailboxes of the followers of both -- millions of times more work than WhatsApp per message. In addition, Twitter persists all the messages while WhatsApp doesn't.
  - segmondy 8y ago

liquidgecka 8y ago

That was a period when the only people talking outside the company were the ones not trying to fix things inside of the company. Twitter never had dedicated machines for individual users. Its just not how the infra ever worked (at least until I left). Requests landed on random boxes behind load balancers, those boxes talked to pools of memcache or mysql, etc. At no point was there ever "racks" or "machines" dedicated to specific individuals like the article claims. That being said, some users created crazy hot shards when specific tweets went absolutely madhouse, especially Beiber and those like him. Random twitter internals tidbit: We had a unit of measurement called a "MJ". Its the number of tweets per second that we had when the rumors of Michael Jacksons death were circling. It basically overloaded the system and had us running around on fire. It was 465 tweets a second. Within a year we had crossed a line where we never were below that number again. Hence "we are at about 12 MJ's" was jokingly used to compare "hair on fire" to every day a couple of years later. =)

evanweaver 8y ago
Sorry, we just kept the Bieber box hidden from you in the "special closet". It was a SPARCstation 20 and we were afraid if ops found it they would shut it down.
- liquidgecka 8y ago
  You know.. I found a special box (PowerMac Pro) in the "special closet" that had been moved like 5 times. It was Blane's old desktop and it had the twitter codebase checked out from like ~2007 era? Kind of nifty seeing just how much had changed since that got tucked away.
deleted 8y ago
[deleted]