7 ms·

This isn't shocking - Twitter was notorious for being held together with Scotch tape technically. Honestly this hands-on approach is an impressive example of d

This isn't shocking - Twitter was notorious for being held together with Scotch tape technically. Honestly this hands-on approach is an impressive example of doing things that don't scale.

giancarlostoro 8y ago
I found it amusing that Twitter was Rails' biggest advertisement. Everyone wanted to use Rails but Twitter turned into a franken app with different stacks to keep it running
- phaedryx 8y ago
  Twitter was Rails' worst advertisement. They used Rails as a scapegoat to hide their bad tech. I still hear things like "Rails can't scale; remember Twitter?"
  - joering2 8y ago
    At least 3 large companies I know my friends developers work at, over the last 5 years switched from Rails to PHP. All told me same story "after twitter, noone wants to work or touch rails anymore"
    - bdcravens 8y ago
      Meanwhile all 3 large companies probably host their code on Github :-)
      - treahauet 8y ago
        And even if they don’t, many if not most of their dependencies do.
    - dasil003 8y ago
      What will they do when they find out phpBB and WordPress are written in PHP?
  - jensvdh 8y ago
    Airbnb is still on rails.. Probably the largest rails app out there for now.
    - Jagat 8y ago
      Except that I'd bet Airbnb's read qps requirement is less than 1% of Twitter's. Write qps would be even smaller.
      - MBCook 8y ago
        True, but on an absolute scale AirBNB is still very big.
        ksec 8y ago
        By RPS, I imagined Shopify is largest. But then from what I understand, Shopify operate in such a way that every shop is its own app. i.e there is 100s of thousands of Basecamp / AirBnb running on the same code base and every shop is somewhat isolated. Purely in terms of App in think the largest would be Cookpad. AirBnB never shared their numbers so I don't know. https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-...
        segmondy 8y ago
        I don't think so. Seriously think about it. How many active users does AirBnB have? How many put their home up for rent, and how many rent in a year? I reckon you get more tweets in a day than AirBnB rentals in a year. That's how big Twitter is or small AirBnB is. :D
        MBCook 8y ago
        Twitter was bigger but they don’t run on Rails anymore where I gather AirBNB does.
      - myth_drannon 8y ago
        I would imagine Shopify on Black Friday is under heavier load.
  - lfxyz 8y ago
    But Rails is nice to work with and whispers 99% of websites will never need to reach Twitter-scale.
  - fanpuns 8y ago
    I had the same thought when I read this. I wasn't into Rails back then (or development) so I don't have a sense of context for what the framework was like at that point in time, but the more articles I read about Twitter in the early days, the more of a sense I get that maybe they didn't write the best code.
- scarface74 8y ago
  From 2008: Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources. When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers. Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them. http://www.businessinsider.com/2008/5/why-can-t-twitter-scale-blaine-cook-tries-to-explain http://www.businessinsider.com/2008/5/why-can-t-twitter-scal...
  - sulam 8y ago
    Yes, well, that's a nice idea in theory. In practice, you could get over 10x (sometimes 100x) the rps off a box running the new, JVM-based services vs their Rails-equivalents. Orders of magnitude probably matter a little less when you're well-funded and have hundreds of servers, but when you are thinking about trying to go public and your bottom line is being scrutinized and you have 10's of thousands of servers, it starts to matter.
    - scarface74 8y ago
      That's exactly what he said. but a well-capitalized company can certainly afford them But these days when you don't have to buy servers and make a long term capital commitment and you can use something like AWS, if you have a scalable but not efficient architecture and you have the faith of the investors, you can get enough servers to get you over the hump temporarily, slowly start replacing the most performance sensitive part of your architecture and then scale down. Look at what HN darling Dropbox did, they bootstrapped on AWS, got big and then when the time was right, they moved to a cheaper architecture - off of AWS and built their own infrastructure.
      - 8y ago
- evanweaver 8y ago
  Rails, and the way it used MySQL/Postgres, was designed for building CMSs. Very amenable to CDN caching to scale. Twitter was a realtime messaging platform (fundamentally not edge-cacheable) that evolved from that CMS foundation. So the reason for the difficult evolution should be clear. It's not really a coincidence that before Twitter I worked at CNET on Urbanbaby.com which was also a realtime, threaded, short message web chat implemented in Rails. Anyway the point is: use my new project/company https://fauna.com/ https://fauna.com/ to scale your operational data. :-)
  - Operyl 8y ago
    I take it from a quick glance: you either own/work for Fauna and it’s a completely proprietary product? If so .. I’ll pass, I like building things atop stuff I can control. :)
    - evanweaver 8y ago
      Yes, it's not open source (for now). Managed cloud and on-premises edition. We will have a free download soon.
      - Operyl 8y ago
        Might I suggest disclosing it when you post about it? Clicking through your profile revealed it, but it seems like it could read off better for you in the future. :) EDIT: either it was edited above, or I’m blind. I think it was the earlier?
        evanweaver 8y ago
        Yeah I fixed it. Thanks.
truncate 8y ago
Just wondering how is engineering stack at Twitter now? Is it still bandaging things like everyone is talking about here (in Rails days) or they have matured to good engineering practices. Interested, as I was considering applying for engineering job there.
- myth_drannon 8y ago
  If you check their Github account it looks like they built a lot of interesting Scala tech.
- sulam 8y ago
  The fail whale image was retired over 4 years ago now because it almost never showed up anymore. That should tell you enough.
- evanweaver 8y ago
  It's fine. Very Scala/Java/Mesos heavy. Big company integration problems now instead of startup scaling problems. They still use some of the distributed storage we built almost a decade ago though.
tonystubblebine 8y ago
Oh good. This is where I get to chime in. Twitter wouldn’t have existed with out Rails. It was a side project inside of a failing startup. It took Jack and Florian like two weeks to have a solid version up that we could all use and love. That’s two weeks of designing and then having Rails make it super easy to prototype. I was in the management meetings and my read was that if they’d wasted any time fighting with technology or gold plating infrastructure then we would have moved on. So I do see Twitter as a huge Rails success story. To use the Rails term, Rails was the scaffolding. Replacing any part of Rails later isn’t a failure of Rails.

This isn't shocking - Twitter was notorious for being held together with Scotch tape technically. Honestly this hands-on approach is an impressive example of doing things that don't scale.

giancarlostoro 8y ago

I found it amusing that Twitter was Rails' biggest advertisement. Everyone wanted to use Rails but Twitter turned into a franken app with different stacks to keep it running

phaedryx 8y ago
Twitter was Rails' worst advertisement. They used Rails as a scapegoat to hide their bad tech. I still hear things like "Rails can't scale; remember Twitter?"
- joering2 8y ago
  At least 3 large companies I know my friends developers work at, over the last 5 years switched from Rails to PHP. All told me same story "after twitter, noone wants to work or touch rails anymore"
  - bdcravens 8y ago
    Meanwhile all 3 large companies probably host their code on Github :-)
    - treahauet 8y ago
      And even if they don’t, many if not most of their dependencies do.
  - dasil003 8y ago
    What will they do when they find out phpBB and WordPress are written in PHP?
- jensvdh 8y ago
  Airbnb is still on rails.. Probably the largest rails app out there for now.
  - Jagat 8y ago
    Except that I'd bet Airbnb's read qps requirement is less than 1% of Twitter's. Write qps would be even smaller.
    - MBCook 8y ago
      True, but on an absolute scale AirBNB is still very big.
      - ksec 8y ago
        By RPS, I imagined Shopify is largest. But then from what I understand, Shopify operate in such a way that every shop is its own app. i.e there is 100s of thousands of Basecamp / AirBnb running on the same code base and every shop is somewhat isolated. Purely in terms of App in think the largest would be Cookpad. AirBnB never shared their numbers so I don't know. https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-largest-rails-monolith https://speakerdeck.com/a_matsuda/the-recipe-for-the-worlds-...
      - segmondy 8y ago
        I don't think so. Seriously think about it. How many active users does AirBnB have? How many put their home up for rent, and how many rent in a year? I reckon you get more tweets in a day than AirBnB rentals in a year. That's how big Twitter is or small AirBnB is. :D
        MBCook 8y ago
        Twitter was bigger but they don’t run on Rails anymore where I gather AirBNB does.
    - myth_drannon 8y ago
      I would imagine Shopify on Black Friday is under heavier load.
- lfxyz 8y ago
  But Rails is nice to work with and whispers 99% of websites will never need to reach Twitter-scale.
- fanpuns 8y ago
  I had the same thought when I read this. I wasn't into Rails back then (or development) so I don't have a sense of context for what the framework was like at that point in time, but the more articles I read about Twitter in the early days, the more of a sense I get that maybe they didn't write the best code.
scarface74 8y ago
From 2008: Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources. When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers. Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them. http://www.businessinsider.com/2008/5/why-can-t-twitter-scale-blaine-cook-tries-to-explain http://www.businessinsider.com/2008/5/why-can-t-twitter-scal...
- sulam 8y ago
  Yes, well, that's a nice idea in theory. In practice, you could get over 10x (sometimes 100x) the rps off a box running the new, JVM-based services vs their Rails-equivalents. Orders of magnitude probably matter a little less when you're well-funded and have hundreds of servers, but when you are thinking about trying to go public and your bottom line is being scrutinized and you have 10's of thousands of servers, it starts to matter.
  - scarface74 8y ago
    That's exactly what he said. but a well-capitalized company can certainly afford them But these days when you don't have to buy servers and make a long term capital commitment and you can use something like AWS, if you have a scalable but not efficient architecture and you have the faith of the investors, you can get enough servers to get you over the hump temporarily, slowly start replacing the most performance sensitive part of your architecture and then scale down. Look at what HN darling Dropbox did, they bootstrapped on AWS, got big and then when the time was right, they moved to a cheaper architecture - off of AWS and built their own infrastructure.
    - 8y ago
evanweaver 8y ago
Rails, and the way it used MySQL/Postgres, was designed for building CMSs. Very amenable to CDN caching to scale. Twitter was a realtime messaging platform (fundamentally not edge-cacheable) that evolved from that CMS foundation. So the reason for the difficult evolution should be clear. It's not really a coincidence that before Twitter I worked at CNET on Urbanbaby.com which was also a realtime, threaded, short message web chat implemented in Rails. Anyway the point is: use my new project/company https://fauna.com/ https://fauna.com/ to scale your operational data. :-)
- Operyl 8y ago
  I take it from a quick glance: you either own/work for Fauna and it’s a completely proprietary product? If so .. I’ll pass, I like building things atop stuff I can control. :)
  - evanweaver 8y ago
    Yes, it's not open source (for now). Managed cloud and on-premises edition. We will have a free download soon.
    - Operyl 8y ago
      Might I suggest disclosing it when you post about it? Clicking through your profile revealed it, but it seems like it could read off better for you in the future. :) EDIT: either it was edited above, or I’m blind. I think it was the earlier?
      - evanweaver 8y ago
        Yeah I fixed it. Thanks.

truncate 8y ago

Just wondering how is engineering stack at Twitter now? Is it still bandaging things like everyone is talking about here (in Rails days) or they have matured to good engineering practices. Interested, as I was considering applying for engineering job there.

myth_drannon 8y ago
If you check their Github account it looks like they built a lot of interesting Scala tech.
sulam 8y ago
The fail whale image was retired over 4 years ago now because it almost never showed up anymore. That should tell you enough.
evanweaver 8y ago
It's fine. Very Scala/Java/Mesos heavy. Big company integration problems now instead of startup scaling problems. They still use some of the distributed storage we built almost a decade ago though.

tonystubblebine 8y ago

Oh good. This is where I get to chime in. Twitter wouldn’t have existed with out Rails. It was a side project inside of a failing startup. It took Jack and Florian like two weeks to have a solid version up that we could all use and love. That’s two weeks of designing and then having Rails make it super easy to prototype. I was in the management meetings and my read was that if they’d wasted any time fighting with technology or gold plating infrastructure then we would have moved on. So I do see Twitter as a huge Rails success story. To use the Rails term, Rails was the scaffolding. Replacing any part of Rails later isn’t a failure of Rails.