4 ms·

I can barely follow along with this, it's very technical. I can't imagine how Kyle Vogt acquired the necessary knowledge to make this work. Example: > The poin

I can barely follow along with this, it's very technical. I can't imagine how Kyle Vogt acquired the necessary knowledge to make this work. Example: > The point of having multiple datacenters is not for redundancy, it's to be as close as possible to all the major peering exchanges. They picked the best locations in the country so they would have access to the largest number of peers. This is the kind of thing where I would have to hire some kind of network engineering expert, and he just figured this stuff out and made it work? I can't fathom other people's intelligence sometime.

arcticfox 7y ago
Pretty exceptional indeed. Also impressive that he was able to grow from founder-stage tech to that scale, since they're largely different problems. Especially back in 2010. I feel like I'd have a much better shot of being able to figure out that scale these days than a decade ago. (If I spent my free time studying and not watching Age of Empires 2 on Justin.tv/Twitch).
drwl 7y ago
Justin Kan goes a bit into Kyle's accelerated learning - https://www.youtube.com/watch?v=YzyatiQrQlQ https://www.youtube.com/watch?v=YzyatiQrQlQ
kd5bjo 7y ago
> I can’t imagine how Kyle Vogt acquired the necessary knowledge to make this work. By this point in history, it wasn’t just him anymore and we’d done a few rounds of improvements already out of necessity. As I recall, he got us up and running at PAIX based mostly on research, but most of the other data centers were built out by a network engineer(1) we hired away from YouTube. While he was working on the network engineering and keeping the original system afloat, I did a lot of the software work for the system described here. (1) Name withheld out of courtesy
- kvogt 7y ago
  True. Kd5bjo is an incredible engineer.
  - kd5bjo 7y ago
    I really should thank you for the opportunity— not many people would let a new college hire spend months on a green-field rewrite of something so core to the company.
- scurvy 7y ago
  CC? Dude's on webcasts with GigaOM. He was in customer sales roles with a VAR. He's not shy.
smcl 7y ago
Don't be so hard on yourself, it's pretty common to read blog posts like this and come away with the idea that a super smart person took one look at the lay of the land and leapt directly from problem -> solution in one neat step. What you don't see is the people the talked to about the problem, their back-and-forth spitballing ideas, the various googling to see if there's a standard approach ... and most importantly you don't see any failed attempts. Bear in mind that I don't think this is some deliberate attempt to appear superhuman, I think it's just accidental
jedberg 7y ago
He leveraged the YCombinator network to absorb a lot of information quickly. For example, I taught him basic networking (routers, switches, multicast/anycast, AS numbers, etc). I shared my 10 years of knowledge with him in a single two hour session because he's a genius, and then he ran from there, vastly exceeding my knowledge. I was there because Emmet asked Steve and Steve asked me to go over, and I was happy to help. I'm sure I wasn't the only one.
- wslack 7y ago
  This is the talk I want to hear sometime.
- karambir 7y ago
  Like other sibling, I would be very much interested in a talk like this. Teaching networking with real world examples and explaining 2-3 large scale architectures w.r.t networking. Maybe a long video(or small series) and a follow-up on twitch for Q/A. Would even pay for this.
- nickpsecurity 7y ago
  That's pretty nice. Quick look at articles show Emmet was the CEO. Which Steve is this? Huffman at Reddit?
  - jedberg 7y ago
    Yes Steve Huffman
joefourier 7y ago
I'm sure if you tried building your own livestreaming or VOD service, you'd come up with similar solutions and insights. Peering problems are fairly obvious - put up a gigabit server in Germany and try livestreaming high bitrate video to a highspeed connection in San Francisco (or vice-versa), and watch as you run into problems despite having more than enough theoretical bandwidth. When your users start to complain, you tend to develop the domain knowledge necessary to solve their problems pretty quickly.
throwaway9876a 7y ago
> I can’t imagine how Kyle Vogt acquired the necessary knowledge to make this work. I've worked on projects with Kyle, and he often goes into bulldozer mode. It is no surprise to me that Kyle could "learn" all he needed in order to get something like this set up (or at least learn enough to orchestrate a small group in constructing it). Kyle is, by all means, a "force of nature" as YC tends to define it. The downside to Kyle's optimism is that he often has very little concern for the humanity of others. He can set up decent optics around his actions and decisions in the wake of what many might consider failures, but he has consistently abused those who try to give him good-faith constructive feedback and often brought co-workers to tears. This is all well-documented at least through the past 4-5 years. (Kyle does actually explicitly ask for "direct" feedback btw. He just is only capable of handing the feedback on a periodic, weekly or monthly basis). A key lesson of this article (and in glacials post above) is a testament to what can be achieved very quickly if technical debt is of minor concern. Kyle's key strength is in building a proof of concept that supports rapid iteration. This point appears to be something the Justin / Twitch teams did very well. A secondly lesson is in getting alignment among diverse engineers. Think about how the team might have debated the architecture presented. Think about how some of the choices might rub people the wrong way. Finally, Kyle is a unique character in several ways but is not alone in possessing a transient "bulldozer" mentality. If you see yourself having the same pattern of behavior, get help before others get hurt. There are a variety of mitigations that can help, but they need explicit participation.
scurvy 7y ago
If that was actually the case, why the f did they have a boatload of gear in 200 Paul? There's almost no peering exchange there whatsoever (until SFMIX about 3 years ago). Can think of a lot better connected places in the Bay.
- kd5bjo 7y ago
  It’s one of the reasons we moved out of there. Moving day was an ... interesting experience: lots of planning to minimize downtime, and everything that was actually planned went relatively well. Unfortunately, what we thought was a 90% plan turned out to be more like 50%. Several people pulled all-nighters on that one. At the time PAIX had a reverse-billing setup: the more data you transferred, the cheaper your connection charge was; we managed to get all the way into the cheapest billing tier within the first billing cycle which was basically unheard-of at the time.
  - scurvy 7y ago
    You didn't move out of 200 Paul until long after you were acquired though. I always wondered why you were there, as there are better connected facilities and those that had cheaper power. Was TelX a hedge or something?
    - kd5bjo 7y ago
      I was long gone by the time the acquisition came along; looks like I was remembering the move into 200 Paul from somewhere downtown (Spear, maybe). The primary motivation was running out of space, but it also coincided with our first forays into peering. I never went deep into that side of things, so I can’t say for certain what the particular arrangements were.

arcticfox 7y ago

Pretty exceptional indeed. Also impressive that he was able to grow from founder-stage tech to that scale, since they're largely different problems. Especially back in 2010. I feel like I'd have a much better shot of being able to figure out that scale these days than a decade ago. (If I spent my free time studying and not watching Age of Empires 2 on Justin.tv/Twitch).

drwl 7y ago

Justin Kan goes a bit into Kyle's accelerated learning - https://www.youtube.com/watch?v=YzyatiQrQlQ https://www.youtube.com/watch?v=YzyatiQrQlQ

kd5bjo 7y ago

> I can’t imagine how Kyle Vogt acquired the necessary knowledge to make this work. By this point in history, it wasn’t just him anymore and we’d done a few rounds of improvements already out of necessity. As I recall, he got us up and running at PAIX based mostly on research, but most of the other data centers were built out by a network engineer(1) we hired away from YouTube. While he was working on the network engineering and keeping the original system afloat, I did a lot of the software work for the system described here. (1) Name withheld out of courtesy

kvogt 7y ago
True. Kd5bjo is an incredible engineer.
- kd5bjo 7y ago
  I really should thank you for the opportunity— not many people would let a new college hire spend months on a green-field rewrite of something so core to the company.
scurvy 7y ago
CC? Dude's on webcasts with GigaOM. He was in customer sales roles with a VAR. He's not shy.

smcl 7y ago

Don't be so hard on yourself, it's pretty common to read blog posts like this and come away with the idea that a super smart person took one look at the lay of the land and leapt directly from problem -> solution in one neat step. What you don't see is the people the talked to about the problem, their back-and-forth spitballing ideas, the various googling to see if there's a standard approach ... and most importantly you don't see any failed attempts. Bear in mind that I don't think this is some deliberate attempt to appear superhuman, I think it's just accidental

jedberg 7y ago

He leveraged the YCombinator network to absorb a lot of information quickly. For example, I taught him basic networking (routers, switches, multicast/anycast, AS numbers, etc). I shared my 10 years of knowledge with him in a single two hour session because he's a genius, and then he ran from there, vastly exceeding my knowledge. I was there because Emmet asked Steve and Steve asked me to go over, and I was happy to help. I'm sure I wasn't the only one.

wslack 7y ago
This is the talk I want to hear sometime.
karambir 7y ago
Like other sibling, I would be very much interested in a talk like this. Teaching networking with real world examples and explaining 2-3 large scale architectures w.r.t networking. Maybe a long video(or small series) and a follow-up on twitch for Q/A. Would even pay for this.
nickpsecurity 7y ago
That's pretty nice. Quick look at articles show Emmet was the CEO. Which Steve is this? Huffman at Reddit?
- jedberg 7y ago
  Yes Steve Huffman

joefourier 7y ago

I'm sure if you tried building your own livestreaming or VOD service, you'd come up with similar solutions and insights. Peering problems are fairly obvious - put up a gigabit server in Germany and try livestreaming high bitrate video to a highspeed connection in San Francisco (or vice-versa), and watch as you run into problems despite having more than enough theoretical bandwidth. When your users start to complain, you tend to develop the domain knowledge necessary to solve their problems pretty quickly.

throwaway9876a 7y ago

> I can’t imagine how Kyle Vogt acquired the necessary knowledge to make this work. I've worked on projects with Kyle, and he often goes into bulldozer mode. It is no surprise to me that Kyle could "learn" all he needed in order to get something like this set up (or at least learn enough to orchestrate a small group in constructing it). Kyle is, by all means, a "force of nature" as YC tends to define it. The downside to Kyle's optimism is that he often has very little concern for the humanity of others. He can set up decent optics around his actions and decisions in the wake of what many might consider failures, but he has consistently abused those who try to give him good-faith constructive feedback and often brought co-workers to tears. This is all well-documented at least through the past 4-5 years. (Kyle does actually explicitly ask for "direct" feedback btw. He just is only capable of handing the feedback on a periodic, weekly or monthly basis). A key lesson of this article (and in glacials post above) is a testament to what can be achieved very quickly if technical debt is of minor concern. Kyle's key strength is in building a proof of concept that supports rapid iteration. This point appears to be something the Justin / Twitch teams did very well. A secondly lesson is in getting alignment among diverse engineers. Think about how the team might have debated the architecture presented. Think about how some of the choices might rub people the wrong way. Finally, Kyle is a unique character in several ways but is not alone in possessing a transient "bulldozer" mentality. If you see yourself having the same pattern of behavior, get help before others get hurt. There are a variety of mitigations that can help, but they need explicit participation.

scurvy 7y ago

If that was actually the case, why the f did they have a boatload of gear in 200 Paul? There's almost no peering exchange there whatsoever (until SFMIX about 3 years ago). Can think of a lot better connected places in the Bay.

kd5bjo 7y ago
It’s one of the reasons we moved out of there. Moving day was an ... interesting experience: lots of planning to minimize downtime, and everything that was actually planned went relatively well. Unfortunately, what we thought was a 90% plan turned out to be more like 50%. Several people pulled all-nighters on that one. At the time PAIX had a reverse-billing setup: the more data you transferred, the cheaper your connection charge was; we managed to get all the way into the cheapest billing tier within the first billing cycle which was basically unheard-of at the time.
- scurvy 7y ago
  You didn't move out of 200 Paul until long after you were acquired though. I always wondered why you were there, as there are better connected facilities and those that had cheaper power. Was TelX a hedge or something?
  - kd5bjo 7y ago
    I was long gone by the time the acquisition came along; looks like I was remembering the move into 200 Paul from somewhere downtown (Spear, maybe). The primary motivation was running out of space, but it also coincided with our first forays into peering. I never went deep into that side of things, so I can’t say for certain what the particular arrangements were.