4 ms·

Anatomy of a Crushing (2011)

69 points by gsylvie 10y ago

mirceal 10y ago
Given how events played out, I think this went exceptionally well. Also, nowadays with the cloud and co. it's easy to forget / fully appreciate how much complexity is abstracted away for you.
gsylvie 10y ago
My favorite line: > In my defense, heavy write loads seemed like the last thing Pinboard would ever face. It was my experience that people approached an online purchase of six dollars with the same deliberation and thoughtfulness they might bring to bear when buying a new car.
- pjc50 10y ago
  This is sadly true and a kind of "bikeshedding". It's why virtually all mobile games transitioned from paid-upfront to the current model of dangling progress in front of the user and offering to pay to relieve the artificial obstacle. (My favourite line is "This is especially true in the world of Rails and other frameworks, where there is a tendency to treat one's app like a high-level character in a role-playing game, equipping it with epic gems, sinatras, capistranos, and other mithril armor into a mighty "application stack"". )
  - vinceguidry 10y ago
    > "This is especially true in the world of Rails and other frameworks, where there is a tendency to treat one's app like a high-level character in a role-playing game, equipping it with epic gems, sinatras, capistranos, and other mithril armor into a mighty "application stack"". The whole reason that ecosystem of gems exists is specifically to alleviate the problem of servicing your app in extreme fear. If you can actually use Capistrano effectively, deployment becomes a solved problem. Mithril armor is actually a great way of putting it.
  - dilap 10y ago
    > It's why virtually all mobile games transitioned from paid-upfront to the current model of dangling progress in front of the user and offering to pay to relieve the artificial obstacle. Yes, that, but also because people will pay a lot more for IAP, over and over again. (Total amounts paid for super-engaged users will run to 10s of thousands of $, far beyond what you can even charge for a single app.) (And on the flip side, the developer has to continue to improve and maintain the game to keep the $ coming.) Much of the time, in my experience, when you pay to download an unknown app, it'll end up being poorly implemented in some way, and never made better. It's like paying $2 for a cup of coffee, but 95% of the time, the coffee turns out to be dirty water.
    - DougN7 10y ago
      Exactly. I'd pay to try many more apps if I could refund the crap. The best would be a 1 hour free trial. If I don't uninstall after one hour, bill me. If I do uninstall, don't. Apps could cost 10x what they do now if that was possible because I'd see the value, rather than gambling like today.
- gsylvie 10y ago
  Interesting how the $6 helped him survive the initial onslaught of new customers, since it would have probably been 10x more writes without this $6 throttle in place. One of his tweets from that same day contains my other favourite line: > there are worse things than being DDOS'ed by people trying to give you money (Source: http://delong.typepad.com/sdj/2010/12/the-small-exodus-from-delicious-to-pinboard.html http://delong.typepad.com/sdj/2010/12/the-small-exodus-from-... )
felixgallo 10y ago
Fundamentally, Yahoo is a charity dedicated to the stealth funding of misanthropic single-founder Renaissance twitter accounts.
- jolux 10y ago
  this is hilarious. don't know why you're being down voted ;)
luckystartup 10y ago
The main takeaway for me is that dedicated hardware is a nightmare when you have large traffic spikes. You could save a ton of stress by running this on a cloud platform with autoscaling, such as AWS or Heroku. You can always spin up a reserved instances on a long-term contract when you hit a certain baseline level of demand. And you should never consider dedicated servers or colocating until you hit that point. AWS or Heroku could have easily handled the "freemium" traffic that would have crushed these servers, instead of only accepting paid users. And it sounds like that might have been a far more lucrative opportunity.
- pjc50 10y ago
  If your app is designed to autoscale, and you're prepared to have Amazon charge you an arbitrary amount for the privilege, and your cashflow through the payments processor is faster than you have to pay AWS. (How do people scale cloud DBs? Presumably you can't just keep increasing the write speed into a single DB?)
  - luckystartup 10y ago
    Autoscaling is free to set up on AWS [1]. All web server code should be able to scale, and there's nothing special about autoscaling. The easiest (and default) way is to just spin up servers based on CPU usage. For Heroku, you can use a third-party service that costs $10 per month [2]. If you use Amazon RDS, it's pretty easy to scale your database both vertically and horizontally [3]. If you can use Amazon DynamoDB [4], scaling is something you will never, ever have to worry about. Not even when you have millions of daily users. [1] https://aws.amazon.com/autoscaling/ https://aws.amazon.com/autoscaling/ [2] https://www.hirefire.io/ https://www.hirefire.io/ [3] https://aws.amazon.com/blogs/database/scaling-your-amazon-rds-instance-vertically-and-horizontally/ https://aws.amazon.com/blogs/database/scaling-your-amazon-rd... [4] https://aws.amazon.com/dynamodb/ https://aws.amazon.com/dynamodb/
    - chousuke 10y ago
      Eh, you "never, ever" have to worry about DynamoDB scaling in a perfect world where marketing is literally true. If you read Amazon's documentation, there's quite a lot that you need to pay attention to if you want DynamoDB to actually perform and scale well for your application. You don't get scalability for free. Tradeoffs and effort are always required. In Amazon's case you can usually throw money at the problem, but even that has its limits.
Mithaldu 10y ago
-
- pjc50 10y ago
  You've got them confused with Pinterest.
  - Mithaldu 10y ago
    Well damn, so i did. Thanks for pointing out my mistake. :)

69 points by gsylvie 10y ago

Given how events played out, I think this went exceptionally well. Also, nowadays with the cloud and co. it's easy to forget / fully appreciate how much complexity is abstracted away for you.

gsylvie 10y ago

My favorite line: > In my defense, heavy write loads seemed like the last thing Pinboard would ever face. It was my experience that people approached an online purchase of six dollars with the same deliberation and thoughtfulness they might bring to bear when buying a new car.

pjc50 10y ago
This is sadly true and a kind of "bikeshedding". It's why virtually all mobile games transitioned from paid-upfront to the current model of dangling progress in front of the user and offering to pay to relieve the artificial obstacle. (My favourite line is "This is especially true in the world of Rails and other frameworks, where there is a tendency to treat one's app like a high-level character in a role-playing game, equipping it with epic gems, sinatras, capistranos, and other mithril armor into a mighty "application stack"". )
- vinceguidry 10y ago
  > "This is especially true in the world of Rails and other frameworks, where there is a tendency to treat one's app like a high-level character in a role-playing game, equipping it with epic gems, sinatras, capistranos, and other mithril armor into a mighty "application stack"". The whole reason that ecosystem of gems exists is specifically to alleviate the problem of servicing your app in extreme fear. If you can actually use Capistrano effectively, deployment becomes a solved problem. Mithril armor is actually a great way of putting it.
- dilap 10y ago
  > It's why virtually all mobile games transitioned from paid-upfront to the current model of dangling progress in front of the user and offering to pay to relieve the artificial obstacle. Yes, that, but also because people will pay a lot more for IAP, over and over again. (Total amounts paid for super-engaged users will run to 10s of thousands of $, far beyond what you can even charge for a single app.) (And on the flip side, the developer has to continue to improve and maintain the game to keep the $ coming.) Much of the time, in my experience, when you pay to download an unknown app, it'll end up being poorly implemented in some way, and never made better. It's like paying $2 for a cup of coffee, but 95% of the time, the coffee turns out to be dirty water.
  - DougN7 10y ago
    Exactly. I'd pay to try many more apps if I could refund the crap. The best would be a 1 hour free trial. If I don't uninstall after one hour, bill me. If I do uninstall, don't. Apps could cost 10x what they do now if that was possible because I'd see the value, rather than gambling like today.
gsylvie 10y ago
Interesting how the $6 helped him survive the initial onslaught of new customers, since it would have probably been 10x more writes without this $6 throttle in place. One of his tweets from that same day contains my other favourite line: > there are worse things than being DDOS'ed by people trying to give you money (Source: http://delong.typepad.com/sdj/2010/12/the-small-exodus-from-delicious-to-pinboard.html http://delong.typepad.com/sdj/2010/12/the-small-exodus-from-... )

felixgallo 10y ago

Fundamentally, Yahoo is a charity dedicated to the stealth funding of misanthropic single-founder Renaissance twitter accounts.

jolux 10y ago
this is hilarious. don't know why you're being down voted ;)

luckystartup 10y ago

The main takeaway for me is that dedicated hardware is a nightmare when you have large traffic spikes. You could save a ton of stress by running this on a cloud platform with autoscaling, such as AWS or Heroku. You can always spin up a reserved instances on a long-term contract when you hit a certain baseline level of demand. And you should never consider dedicated servers or colocating until you hit that point. AWS or Heroku could have easily handled the "freemium" traffic that would have crushed these servers, instead of only accepting paid users. And it sounds like that might have been a far more lucrative opportunity.

pjc50 10y ago
If your app is designed to autoscale, and you're prepared to have Amazon charge you an arbitrary amount for the privilege, and your cashflow through the payments processor is faster than you have to pay AWS. (How do people scale cloud DBs? Presumably you can't just keep increasing the write speed into a single DB?)
- luckystartup 10y ago
  Autoscaling is free to set up on AWS [1]. All web server code should be able to scale, and there's nothing special about autoscaling. The easiest (and default) way is to just spin up servers based on CPU usage. For Heroku, you can use a third-party service that costs $10 per month [2]. If you use Amazon RDS, it's pretty easy to scale your database both vertically and horizontally [3]. If you can use Amazon DynamoDB [4], scaling is something you will never, ever have to worry about. Not even when you have millions of daily users. [1] https://aws.amazon.com/autoscaling/ https://aws.amazon.com/autoscaling/ [2] https://www.hirefire.io/ https://www.hirefire.io/ [3] https://aws.amazon.com/blogs/database/scaling-your-amazon-rds-instance-vertically-and-horizontally/ https://aws.amazon.com/blogs/database/scaling-your-amazon-rd... [4] https://aws.amazon.com/dynamodb/ https://aws.amazon.com/dynamodb/
  - chousuke 10y ago
    Eh, you "never, ever" have to worry about DynamoDB scaling in a perfect world where marketing is literally true. If you read Amazon's documentation, there's quite a lot that you need to pay attention to if you want DynamoDB to actually perform and scale well for your application. You don't get scalability for free. Tradeoffs and effort are always required. In Amazon's case you can usually throw money at the problem, but even that has its limits.

Mithaldu 10y ago

pjc50 10y ago
You've got them confused with Pinterest.
- Mithaldu 10y ago
  Well damn, so i did. Thanks for pointing out my mistake. :)