5 ms·

One of the biggest things I've had to unlearn as an SRE leaving google is this: RPC traffic is not free, fast, and reliable (So long as you don't go cross-datac

One of the biggest things I've had to unlearn as an SRE leaving google is this: RPC traffic is not free, fast, and reliable (So long as you don't go cross-datacenter). For most companies it is expensive and slow. Facebook's networks are still designed for early-2000s era topologies and their newer topologies won't fix that; They've still got way too little top-of-rack bandwidth to the other racks nearby. Microsoft hasn't even caught on yet, and is still designing for bigger and bigger monolithic servers. I can't tell what Amazon is doing, but they seem to have the idea with ELBs at multiple layers.

balakk 11y ago
Try this link for what MS is doing : https://www.youtube.com/watch?v=RffHFIhg5Sc https://www.youtube.com/watch?v=RffHFIhg5Sc
- GauntletWizard 11y ago
  Microsoft has ambitions, and then they hire bottom-feeders to mismanage it to hell. No, that's not at all what they're doing.
  - richmarr 11y ago
    > ... then they hire bottom-feeders... I don't work for Microsoft, but I'm kinda disgusted by that comment. By all means criticise their work, but calling them derogatory names is deeply deeply uncool.
    - GauntletWizard 11y ago
      I'm sorry, when I learn a non-derogatory phrase of criticism, I'll use it. Unfortunately, worthless hacks like yourself have have redefined every judgement as being offensive, so I'm left calling you a spineless fuckwad til you learn to tell the difference.
      - DanBC 11y ago
        Come on. You know that's not welcome here.
tux1968 11y ago
> RPC traffic is expensive and slow, not free, fast, reliable If i've understood your point, you should have rather said this is what you've learned, not unlearned.
- GauntletWizard 11y ago
  You are correct; I've edited my comment to (hopefully) make it a little clearer. Google RPCs are fast. The RPC trace depth of many requests is >20 in miliseconds. Google RPCs are free - Nobody budgets for intradatacenter traffic. Google RPCs are reliable - Most teams hold themselves to a SLA of 4 9s, as measured by their customers, and many see >5 as the usual state of affairs.
  - maaku 11y ago
    Is it opposite day? You seem to mean <20ms right?
    - deleted 11y ago
      [deleted]
    - morpher 11y ago
      I parsed his comment as "a request that results in 20 nested RPCs completes in a few ms".
    - allenz 11y ago
      I think he means that over 20 remote procedure calls can be made within milliseconds, which is game-changing.
  - justicezyx 11y ago
    rpc isn't free, you just did not encounter the situations that it matters...
InclinedPlane 11y ago
This is what continues to shock me about the tech industry. Google started revolutionizing the way data centers work more than 15 years ago, and it's a huge reason why they are able to execute on ambitious projects so well and serve up content that has a ton of compute behind it yet do so incredibly fast yet also incredibly cheaply. That cuts right to the bottom line, yet nobody else has copied google yet. I think it's a huge indication of just how immature the entire industry truly is. It's difficult enough to get anything done these days, let alone get something exceptional done well. We're only just a little bit out of the deep depths of the "software crisis" where the ability to run a successful software project was a hit or miss thing. But we're still in the software crisis shallows, and it'll be a long time before we make land fall.
- threeseed 11y ago
  > nobody else has copied google yet Why would they ? Almost nobody is operating at Google's scale and interconnects like Infiniband are rarely saturated to the point that exotic solutions are needed. Fast RPC is nice and all but (a) not that many companies are using microservices and (b) not that many are scaling out to the billions of requests/second. And the 20 or so companies who are operating at Google's scale are hardly floundering around blind. Facebook has done very interesting work in the infrastructure space, LinkedIn is amazing on the core software side and Netflix has pioneered the use of the cloud.
  - mikecb 11y ago
    I think this is short-sighted. Everyone who operates with any portion of their stack on any cloud provider is "operating at Google's scale" since even a small service is affected both by the infrastructure on which it is run and all of the other services run on that infrastructure. If you're running in a cloud, you need to plan and develop like you're a product team at facebook or google, and cloud providers need to act like they are providing infrastructure for a cohesive system, not a bunch of individual pieces.
    - threeseed 11y ago
thrownaway2424 11y ago
It would be nice if the SREs (and SWEs) still at Google would also unlearn these incorrect facts. RPCs fail, bandwidth costs money.

balakk 11y ago

Try this link for what MS is doing : https://www.youtube.com/watch?v=RffHFIhg5Sc https://www.youtube.com/watch?v=RffHFIhg5Sc

GauntletWizard 11y ago
Microsoft has ambitions, and then they hire bottom-feeders to mismanage it to hell. No, that's not at all what they're doing.
- richmarr 11y ago
  > ... then they hire bottom-feeders... I don't work for Microsoft, but I'm kinda disgusted by that comment. By all means criticise their work, but calling them derogatory names is deeply deeply uncool.
  - GauntletWizard 11y ago
    I'm sorry, when I learn a non-derogatory phrase of criticism, I'll use it. Unfortunately, worthless hacks like yourself have have redefined every judgement as being offensive, so I'm left calling you a spineless fuckwad til you learn to tell the difference.
    - DanBC 11y ago
      Come on. You know that's not welcome here.

tux1968 11y ago

> RPC traffic is expensive and slow, not free, fast, reliable If i've understood your point, you should have rather said this is what you've learned, not unlearned.

GauntletWizard 11y ago
You are correct; I've edited my comment to (hopefully) make it a little clearer. Google RPCs are fast. The RPC trace depth of many requests is >20 in miliseconds. Google RPCs are free - Nobody budgets for intradatacenter traffic. Google RPCs are reliable - Most teams hold themselves to a SLA of 4 9s, as measured by their customers, and many see >5 as the usual state of affairs.
- maaku 11y ago
  Is it opposite day? You seem to mean <20ms right?
  - deleted 11y ago
    [deleted]
  - morpher 11y ago
    I parsed his comment as "a request that results in 20 nested RPCs completes in a few ms".
  - allenz 11y ago
    I think he means that over 20 remote procedure calls can be made within milliseconds, which is game-changing.
- justicezyx 11y ago
  rpc isn't free, you just did not encounter the situations that it matters...

InclinedPlane 11y ago

This is what continues to shock me about the tech industry. Google started revolutionizing the way data centers work more than 15 years ago, and it's a huge reason why they are able to execute on ambitious projects so well and serve up content that has a ton of compute behind it yet do so incredibly fast yet also incredibly cheaply. That cuts right to the bottom line, yet nobody else has copied google yet. I think it's a huge indication of just how immature the entire industry truly is. It's difficult enough to get anything done these days, let alone get something exceptional done well. We're only just a little bit out of the deep depths of the "software crisis" where the ability to run a successful software project was a hit or miss thing. But we're still in the software crisis shallows, and it'll be a long time before we make land fall.

threeseed 11y ago
> nobody else has copied google yet Why would they ? Almost nobody is operating at Google's scale and interconnects like Infiniband are rarely saturated to the point that exotic solutions are needed. Fast RPC is nice and all but (a) not that many companies are using microservices and (b) not that many are scaling out to the billions of requests/second. And the 20 or so companies who are operating at Google's scale are hardly floundering around blind. Facebook has done very interesting work in the infrastructure space, LinkedIn is amazing on the core software side and Netflix has pioneered the use of the cloud.
- mikecb 11y ago
  I think this is short-sighted. Everyone who operates with any portion of their stack on any cloud provider is "operating at Google's scale" since even a small service is affected both by the infrastructure on which it is run and all of the other services run on that infrastructure. If you're running in a cloud, you need to plan and develop like you're a product team at facebook or google, and cloud providers need to act like they are providing infrastructure for a cohesive system, not a bunch of individual pieces.
  - threeseed 11y ago

thrownaway2424 11y ago

It would be nice if the SREs (and SWEs) still at Google would also unlearn these incorrect facts. RPCs fail, bandwidth costs money.