4 ms·

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The a

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We purchased systems from Sun because of the quality of the HW and the stability of Solaris. Now a stack of cheap linux boxes with the right design (IP CLOS, BGP fabric, load balancing and containers) who cares if a switch fails or a rack goes out? Most issues today are caused my humans and bugs from software upgrades for new code.

brianwawok 9y ago
Kind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.
- ProAm 9y ago
  Sounds like an argument against micro services?
  - jameskegel 9y ago
    I'd give a dollar to hear how you arrived at this conclusion.
  - stouset 9y ago
    Only if you don't stop to consider the alternative.
    - ProAm 9y ago
      "The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."
- Xeoncross 9y ago
  I think you both have a point. The tools are mostly all here now (some in primitive forms), but no one person on earth knows how to use them all. From netfilter to kubernetes to sharded redis instances to nginx RTMP to embeded golang structs and git commit hooks. It takes an amazing team to cover all the bases.
- cat199 9y ago
  You are thinking small scale, and public web. The fact that VAXClusters were already going out of fashion by that time, and the fact of IBM's parallel sysplex existing by that time already negate your point. previous $employer had built a globally distributed multisite c&c processing system twice by then and were in the process of revamping it for a 3rd.. 1st on mainframes & remote serial compute devices in the 80s, then on unix workstations in the early-mid 90s. They were by far not alone in dealing with this level of complexity for the time.
- myrandomcomment 9y ago
  My ISP? 5 racks in 32 cities as POP. Each POP had 10 Sun Netra T1 1RU boxes behind SLB. All those were a service cache to offload the main DC which had...hum... ~100 or so Sun E450 (4x480Mhz Usparc3, 4GB RAM, 20x4GB or 9GB UltraSCSI3 drives, 2x1G NICs). There was a cluster filesystem also. We had huge databases, etc. This is just not that hard today.
pvg 9y ago
In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.
- myrandomcomment 9y ago
  It was not a criticism so much as "well, it will be interesting to see what went wrong." My bet is a human.
  - deleted 9y ago
    [deleted]
- cat199 9y ago
  Is there, fundamentally? Hotmail, Altavista, Yahoo Mail, eBay were all a thing then.. and it wasn't like noone was using them at the time either.. which isn't to say everything hasn't advanced, but in my opinion this isn't really a 'difference in kind'..
  - myrandomcomment 9y ago
    Agree. Hotmail IIRC had 9K or so servers. Yahoo had even more. I think doing this at scale is simpler today because the networking is simpler, i.e., IP fabrics and VMs and containers. If you are building something today and want to use Layer2 I question your santiy.
  - pvg 9y ago
    Yeah I think there is. This is what wikipedia says Altavista "As of 1998, it used 20 multi-processor machines using DEC's 64-bit Alpha processor. Together, the back-end machines had 130 GB of RAM and 500 GB of hard disk drive space, and received 13 million queries every day" There are probably more read-only queries to gists per day than that.
- hinkley 9y ago
rco8786 9y ago
> fairly simple mmhmm
hinkley 9y ago
The future is here, it's just unevenly distributed. Someone is doing all of the kinds of work necessary to verify that the servers are up, being useful, and nominally working the way the previous deployment did, but not everyone has the tools yet to do that (especially the last one). We need a compete and complementary tool chain that everyone can use, and so far a lot of these are still business differentiators (i.e., proprietary)

brianwawok 9y ago

Kind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

ProAm 9y ago
Sounds like an argument against micro services?
- jameskegel 9y ago
  I'd give a dollar to hear how you arrived at this conclusion.
- stouset 9y ago
  Only if you don't stop to consider the alternative.
  - ProAm 9y ago
    "The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."
Xeoncross 9y ago
I think you both have a point. The tools are mostly all here now (some in primitive forms), but no one person on earth knows how to use them all. From netfilter to kubernetes to sharded redis instances to nginx RTMP to embeded golang structs and git commit hooks. It takes an amazing team to cover all the bases.
cat199 9y ago
You are thinking small scale, and public web. The fact that VAXClusters were already going out of fashion by that time, and the fact of IBM's parallel sysplex existing by that time already negate your point. previous $employer had built a globally distributed multisite c&c processing system twice by then and were in the process of revamping it for a 3rd.. 1st on mainframes & remote serial compute devices in the 80s, then on unix workstations in the early-mid 90s. They were by far not alone in dealing with this level of complexity for the time.
myrandomcomment 9y ago
My ISP? 5 racks in 32 cities as POP. Each POP had 10 Sun Netra T1 1RU boxes behind SLB. All those were a service cache to offload the main DC which had...hum... ~100 or so Sun E450 (4x480Mhz Usparc3, 4GB RAM, 20x4GB or 9GB UltraSCSI3 drives, 2x1G NICs). There was a cluster filesystem also. We had huge databases, etc. This is just not that hard today.

pvg 9y ago

In 1998 There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later. There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

myrandomcomment 9y ago
It was not a criticism so much as "well, it will be interesting to see what went wrong." My bet is a human.
- deleted 9y ago
  [deleted]
cat199 9y ago
Is there, fundamentally? Hotmail, Altavista, Yahoo Mail, eBay were all a thing then.. and it wasn't like noone was using them at the time either.. which isn't to say everything hasn't advanced, but in my opinion this isn't really a 'difference in kind'..
- myrandomcomment 9y ago
  Agree. Hotmail IIRC had 9K or so servers. Yahoo had even more. I think doing this at scale is simpler today because the networking is simpler, i.e., IP fabrics and VMs and containers. If you are building something today and want to use Layer2 I question your santiy.
- pvg 9y ago
  Yeah I think there is. This is what wikipedia says Altavista "As of 1998, it used 20 multi-processor machines using DEC's 64-bit Alpha processor. Together, the back-end machines had 130 GB of RAM and 500 GB of hard disk drive space, and received 13 million queries every day" There are probably more read-only queries to gists per day than that.
hinkley 9y ago

rco8786 9y ago

> fairly simple mmhmm

hinkley 9y ago

The future is here, it's just unevenly distributed. Someone is doing all of the kinds of work necessary to verify that the servers are up, being useful, and nominally working the way the previous deployment did, but not everyone has the tools yet to do that (especially the last one). We need a compete and complementary tool chain that everyone can use, and so far a lot of these are still business differentiators (i.e., proprietary)