5 ms·

No, you have no idea what your scale problems are going to be (if you ever have them). That is because if you get lucky and your application scales, it (and the

by diego 13y ago

No, you have no idea what your scale problems are going to be (if you ever have them). That is because if you get lucky and your application scales, it (and the world) will change significantly from what it is today. Let me tell you a story: in 1998 at Inktomi (look it up) we had a distributed search engine running on Sun hardware. We could not have anticipated that we'd need to migrate to Linux/PC because Sun's prices would make it impractical for us to continue scaling using their hardware. It took us three years to make the switch, and that's one of the reasons we lost to Google. Had we started two years later (when Gigabit ethernet became available for no-brand PC hardware), then we would have built the entire thing on Linux to begin with. "It really doesn't take a lot of time to plan ahead." Have you ever experienced the growth of a startup to see your infrastructure cost soar to five, six, seven figures per month? Two hours will get you as far as "one day we'll probably need to replace MySQL by something else." What you don't know is what that something else will be. Too many writes per second? Reads? Need for geographical distribution? A schema that changes all the time because you need features and fields you never imagined? Will you need to break up your store into a service-oriented architecture with different types of components? Will you run your own datacenter, or will you be on AWS? What will be the maturity level of different pieces of software in two years? I hope you get the point.

peterwwillis 13y ago
Do you really expect me to buy the idea that your company failed because Gigabit was too expensive? Even if 100Mbit was far cheaper, there are plenty of workarounds to cheaply increase throughput. I assume by no-brand you mean custom built, and also assuming you mean the cheapest available, in which case even one gigabit interface may have been difficult, seeing as 32-bit 33mhz bus capacity is barely above gigabit speed. In any case, the money you saved on Sun gear could have built you a sizeable PC cluster and even with several 100Mbit interfaces would have been more powerful and cheaper. Really I think it wasn't built on Linux because Sun was the more robust, stable platform. But I could be crazy. While i'm being crazy, I should point out all the other things you mentioned can be planned for. Anybody who's even read about high-performance systems design should be able to account for too many reads/writes! Geographical distribution is simple math: at some point, there are too many upstream clients for one downstream server, capacity fills, latency goes through the roof. A DBA knows all about schema woes. I thought service-oriented architecture was basic CS stuff? (I don't know, I never went to school) AWS didn't exist at the time. And the maturity level of your software in two years will, obviously, be two years more mature. All of these problems are what someone with no experience or training will run into. But there should be enough material out there now that anyone can read up enough to account for all these operational and design problems, and more. But if your argument is that start-up people shouldn't have to know it because hey, I haven't got time to figure out how to do it right because I have to ship right now, I don't buy that for a minute. There's a paper that some guy wrote years ago that goes over in great detail every single operational scaling issue you can think of, and it's free. I don't remember where it is. But it should be required reading for anyone who ever works on a network of more than two servers. As an aside: was it really cost that prohibited you from porting to Linux? This article[1] from 2000 states "adding Linux to the Traffic Server's already impressive phalanx of operating systems, including Solaris, Windows 2000, HP-UX, DEC, Irix and others, shows that Inktomi is dedicated to open-source standards, similar to the way IBM Corp. has readily embraced the technology for its eServers." And this HN thread[2] has a guy claiming that in 1996 "we used Intel because with Sun servers you paid an extreme markup for unnecessary reliability". However, it did take him 4 years to move to Linux. (?!) A lot of other interesting comments on that thread. [1] http://www.internetnews.com/bus-news/article.php/526691/Inktomi+to+Embrace+Linux+With+Latest+Traffic+Server.htm http://www.internetnews.com/bus-news/article.php/526691/Inkt... [2] https://news.ycombinator.com/item?id=3924609 https://news.ycombinator.com/item?id=3924609
- deleted 13y ago
  [deleted]
- krenoten 13y ago
  > Geographical distribution is simple math HAHAHAHAHAHAHA. > And the maturity level of your software in two years will, obviously, be two years more mature. HAHAHAHAHAHAHA.
  - peterwwillis 13y ago
    You don't get token karma on here for being an ass. This isn't reddit.
    - sbarre 13y ago
      If this isn't Reddit, then why are you telling diego that you know more about his personal life experiences than he does? That's a pretty Reddit thing to do..
- asdfjjjjjj 13y ago
  Hi Peter, another ex-inktomi/ex-yahoo guy here. I worked on this infrastruture much later than Diego. Traffic Server is not a significant part of the Inktomi environment -- you are looking at the wrong thing. Diego is describing the search engine itself, which ran on Myrinet at that time. It did not run on 100baseT ethernet. Myrinet was costly and difficult to operate, but necessary as the clusters performed an immense amount of network i/o. It is also extremely non-trivial to replace your entire network fabric alongside new serving hardware and a new OS platform. These are not independent web servers, these are clustered systems which all speak peer to peer during the process of serving a search result. This is very different from running a few thousand web servers. Even once migrated to gigE and linux, I watched the network topology evolve several times as the serving footprint doubled and doubled. I assure you, there is no single collection of "every single operational scaling issue you can think of," because some systems have very different architectures and scale demands -- often driven by costs unique to their situation.
  - 13y ago

by diego 13y ago

peterwwillis 13y ago

Do you really expect me to buy the idea that your company failed because Gigabit was too expensive? Even if 100Mbit was far cheaper, there are plenty of workarounds to cheaply increase throughput. I assume by no-brand you mean custom built, and also assuming you mean the cheapest available, in which case even one gigabit interface may have been difficult, seeing as 32-bit 33mhz bus capacity is barely above gigabit speed. In any case, the money you saved on Sun gear could have built you a sizeable PC cluster and even with several 100Mbit interfaces would have been more powerful and cheaper. Really I think it wasn't built on Linux because Sun was the more robust, stable platform. But I could be crazy. While i'm being crazy, I should point out all the other things you mentioned can be planned for. Anybody who's even read about high-performance systems design should be able to account for too many reads/writes! Geographical distribution is simple math: at some point, there are too many upstream clients for one downstream server, capacity fills, latency goes through the roof. A DBA knows all about schema woes. I thought service-oriented architecture was basic CS stuff? (I don't know, I never went to school) AWS didn't exist at the time. And the maturity level of your software in two years will, obviously, be two years more mature. All of these problems are what someone with no experience or training will run into. But there should be enough material out there now that anyone can read up enough to account for all these operational and design problems, and more. But if your argument is that start-up people shouldn't have to know it because hey, I haven't got time to figure out how to do it right because I have to ship right now, I don't buy that for a minute. There's a paper that some guy wrote years ago that goes over in great detail every single operational scaling issue you can think of, and it's free. I don't remember where it is. But it should be required reading for anyone who ever works on a network of more than two servers. As an aside: was it really cost that prohibited you from porting to Linux? This article[1] from 2000 states "adding Linux to the Traffic Server's already impressive phalanx of operating systems, including Solaris, Windows 2000, HP-UX, DEC, Irix and others, shows that Inktomi is dedicated to open-source standards, similar to the way IBM Corp. has readily embraced the technology for its eServers." And this HN thread[2] has a guy claiming that in 1996 "we used Intel because with Sun servers you paid an extreme markup for unnecessary reliability". However, it did take him 4 years to move to Linux. (?!) A lot of other interesting comments on that thread. [1] http://www.internetnews.com/bus-news/article.php/526691/Inktomi+to+Embrace+Linux+With+Latest+Traffic+Server.htm http://www.internetnews.com/bus-news/article.php/526691/Inkt... [2] https://news.ycombinator.com/item?id=3924609 https://news.ycombinator.com/item?id=3924609

deleted 13y ago
[deleted]
krenoten 13y ago
> Geographical distribution is simple math HAHAHAHAHAHAHA. > And the maturity level of your software in two years will, obviously, be two years more mature. HAHAHAHAHAHAHA.
- peterwwillis 13y ago
  You don't get token karma on here for being an ass. This isn't reddit.
  - sbarre 13y ago
    If this isn't Reddit, then why are you telling diego that you know more about his personal life experiences than he does? That's a pretty Reddit thing to do..
asdfjjjjjj 13y ago
Hi Peter, another ex-inktomi/ex-yahoo guy here. I worked on this infrastruture much later than Diego. Traffic Server is not a significant part of the Inktomi environment -- you are looking at the wrong thing. Diego is describing the search engine itself, which ran on Myrinet at that time. It did not run on 100baseT ethernet. Myrinet was costly and difficult to operate, but necessary as the clusters performed an immense amount of network i/o. It is also extremely non-trivial to replace your entire network fabric alongside new serving hardware and a new OS platform. These are not independent web servers, these are clustered systems which all speak peer to peer during the process of serving a search result. This is very different from running a few thousand web servers. Even once migrated to gigE and linux, I watched the network topology evolve several times as the serving footprint doubled and doubled. I assure you, there is no single collection of "every single operational scaling issue you can think of," because some systems have very different architectures and scale demands -- often driven by costs unique to their situation.
- 13y ago