5 ms·

> With Prequal we saw dramatic improvements across all metrics [for youtube], including reductions of 2x in tail latency, 5-10x in tail RIF, 10-20% in tail memo

> With Prequal we saw dramatic improvements across all metrics [for youtube], including reductions of 2x in tail latency, 5-10x in tail RIF, 10-20% in tail memory usage, 2x in tail CPU utilization, and a near-elimination of errors due to load imbalance. In addition to meeting SLOs, this also allowed us to significantly raise the utilization targets that govern how much traffic we are willing to send to each datacenter, thereby saving significant resources. This feels like one of those "my company realized savings greater than my entire career's expected compensation."

up2isomorphism 2y ago
The biggest YouTube tail latency is “skip after x seconds”. And I don’t think this paper helps.
toenail 2y ago
The person chose to be employed instead of starting their own business. Less risk, less reward.
- eikenberry 2y ago
  This is pretty much the only option for a person to have any sort of ownership... at least in the US where co-ops and the like are extremely rare w/ little legal help.
- FooBarWidget 2y ago
  I disagree with toenail being downvoted. In Europe, employees enjoy a great deal of stability. In Netherlands, it's nigh-impossible to fire someone: you have to fill in 10 pages of justification paperwork and have an independent government agency review and approve it. If someone has a long-term illness then you have to pay 70% of their salary for up to 2 years, even when they do no work at all. Most people don't want to be entrepreneur: they want clear instructions and stability. When you try to give stock to employees, the tax authorities raise an eyebrow: why would you give stock to employees when they enjoy none of your risks? It makes no sense, so we'll treat it as a form of salary, so we'll tax you 52% on the stock's paper value. At the end of the day, what's left for the entrepreneur? You enjoy all the risk, but you don't get to have a paid 2 year sick leave. Even sympathy for your hard work can be hard to get. The potential of money is all you have. Things are different in the US of course, where people can be fired the next minute without reason. That looks like just borderline abuse to me. But from a European perspective, the above comment does not deserve downvoting at all.
  - supermatt 2y ago
    > why would you give stock to employees when they enjoy none of your risks? It makes no sense, so we'll treat it as a form of salary, so we'll tax you 52% on the stock's paper value. The Netherlands is the only country in the EU that taxes on unrealised gains.
    - sabellito 2y ago
      Not sure if you mean just for stock options, but more countries/regions have some form wealth tax on unrealised gains.
      - supermatt 2y ago
        Just stock, but I forgot that Spain also have stock included in wealth tax for people with over 700k assets. I think it’s just NL and ES though - I dont think any other EU countries tax stocks in this way?
        sabellito 2y ago
        I thought that were more, but a quick search only revealed Norway, apart from Spain that you've already mentioned. Also worth noting that in the Madrid region they don't have the wealth tax.
        koe123 2y ago
        Also Denmark I believe
  - huijzer 2y ago
    What you get as an employee is certainty that your compensation will only increase in the single digits per year and that the pension fund will compound a part of your income in single digits as well (pension funds typically underperform the S&P; even in bad years). So either 2 years of “stable income” or the chance of much higher compounding rate. As I see it, employment is a nice backup if being self-employed doesn’t work out.
  - 2y ago
- danielmarkbruce 2y ago
  Honestly this guy probably got paid so much that the reward will be higher than many "successful" startup outcomes.
rmbyrro 2y ago
which should make the employee proud, and I'm sure Google is compensating them very, very well. let's also not forget that the people involved didn't create this in a vacuum. it cost Google a LOT more than their compensation to make it possible for them to even start working on this project, let alone carrying it forward to completion. people underestimate how hard and expensive it is to manage a company in a way that allows its employees to do a good job.
- kqr 2y ago
  > let's also not forget that the people involved didn't create this in a vacuum. it cost Google a LOT more than their compensation to make it possible for them to even start working on this project ...let's also not forget that Google didn't manufacture this opportunity in a vacuum. It cost the rest of society a lot more than their revenue to make it possible for them to employ people who can spend their entire days working on software.
  - rmbyrro 2y ago
    Didn't it cost society to raise us to adulthood so that we can work for Google and other businesses? How many people have sacrificed their time and attention, directly or indirectly, to our benefit, since we were born? Yet, nobody's saying our salaries aren't deserved or the merit of our own efforts.
    - daelon 2y ago
      I think many people are explicitly saying that.
- velcrovan 2y ago
  > it cost Google a LOT more than their compensation to make it possible for them to even start working on this project The actual "not a vacuum" context here is an environment that has been basically printing money for google for the last twenty years. It did not "cost" them anything. It's fine to acknowledge that the people who built google, however well paid they are, are creating vastly more value than they are personally receiving.
vasco 2y ago
The question you should ask is for how much less others would accept to do it. How much it saves isn't how to price some work, you price work based on the people available that can do it. If the fire department puts out a fire in your house you don't pay them the cost of the building. You don't give your life to a doctor, etc. That way of thinking is weird.
- greggyb 2y ago
  It's not weird, but incomplete. It is broadly captured in the Economics concept of "willingness to pay", loosely the maximum price someone or some firm would be willing to pay for something of benefit to them. This contrasts with "willingness to accept", loosely the minimum compensation someone or some firm would accept to produce a good or service (or accept some negative thing). Neither of these is sufficient to determine the price of something precisely, but, in aggregate, these concepts bound the market price for some good or service.
  - vasco 2y ago
    I agree I wasn't super precise, I thought it was enough to show that pricing work equal or close to equal to value produced is unlikely as long as there's many people available to do it. In my example of the fire department if nobody is really coming and you have no insurance or other way to save stuff you would indeed pay a lot. From what I read these were the dynamics in Roman times.
    - mrec 2y ago
      If you're thinking of Marcus Crassus, the dynamic there was that he'd offer to buy your burning property from you at a steep discount, and would only put the fire out if you agreed.
  - kqr 2y ago
    > "willingness to pay" [and] "willingness to accept" Often abbreviated somewhat sloppily into demand and supply.
jldugger 2y ago
Pretty common really. Huge web properties like this have just massive room for cost optimizations, and AFAICT, is largely bottlenecked by management capacity. On the other hand, it's also pretty common to emphasize pet metrics like tails, which are pretty much by definition a minority of the cost. We give them the benefit of the doubt because a) google and b) peer review publication, but thats not usually a heurestic available to decision makers.
- pradn 2y ago
  Optimizing tail latency isn't about saving fleet cost. It's about reducing the chance of one tail latency event ruining a page view, when a page view incurs dozens of backend requests. The more requests you have, the higher the chance one of them hits a tail. So the overall latency a user sees is largely dependent on a) number of requests b) tail latency of each event. This method improves the tail latency for ALL supported services, in a generic way. That's multiplicative impact across all services, from a user perspective. Presumably, the number of requests is harder to reduce if they're all required for the business.
  - tedd4u 2y ago
    The last sentence of the abstract indicates the goal is to be able to run at higher utilization without sacrificing latency. The outcome of that is you need less hardware for a given load which for Google undoubtedly means massive cost savings and higher profitability. > Prequal has dramatically decreased tail latency, error rates, and resource > use, enabling YouTube and other production systems at Google to run at much > higher utilization.
  - jldugger 2y ago
    > Optimizing tail latency isn't about saving fleet cost. Indirectly, it is. As the quote I replied to suggests, in order to combat tail latency services often run with surplus capacity. This is just a fundamental tradeoff between the two variables mentioned. So, by improving the LB algo, they (and anyone, really) can reduce the surplus needed to meet any specific SLO.

up2isomorphism 2y ago

The biggest YouTube tail latency is “skip after x seconds”. And I don’t think this paper helps.

toenail 2y ago

The person chose to be employed instead of starting their own business. Less risk, less reward.

eikenberry 2y ago
This is pretty much the only option for a person to have any sort of ownership... at least in the US where co-ops and the like are extremely rare w/ little legal help.
FooBarWidget 2y ago
I disagree with toenail being downvoted. In Europe, employees enjoy a great deal of stability. In Netherlands, it's nigh-impossible to fire someone: you have to fill in 10 pages of justification paperwork and have an independent government agency review and approve it. If someone has a long-term illness then you have to pay 70% of their salary for up to 2 years, even when they do no work at all. Most people don't want to be entrepreneur: they want clear instructions and stability. When you try to give stock to employees, the tax authorities raise an eyebrow: why would you give stock to employees when they enjoy none of your risks? It makes no sense, so we'll treat it as a form of salary, so we'll tax you 52% on the stock's paper value. At the end of the day, what's left for the entrepreneur? You enjoy all the risk, but you don't get to have a paid 2 year sick leave. Even sympathy for your hard work can be hard to get. The potential of money is all you have. Things are different in the US of course, where people can be fired the next minute without reason. That looks like just borderline abuse to me. But from a European perspective, the above comment does not deserve downvoting at all.
- supermatt 2y ago
  > why would you give stock to employees when they enjoy none of your risks? It makes no sense, so we'll treat it as a form of salary, so we'll tax you 52% on the stock's paper value. The Netherlands is the only country in the EU that taxes on unrealised gains.
  - sabellito 2y ago
    Not sure if you mean just for stock options, but more countries/regions have some form wealth tax on unrealised gains.
    - supermatt 2y ago
      Just stock, but I forgot that Spain also have stock included in wealth tax for people with over 700k assets. I think it’s just NL and ES though - I dont think any other EU countries tax stocks in this way?
      - sabellito 2y ago
        I thought that were more, but a quick search only revealed Norway, apart from Spain that you've already mentioned. Also worth noting that in the Madrid region they don't have the wealth tax.
      - koe123 2y ago
        Also Denmark I believe
- huijzer 2y ago
  What you get as an employee is certainty that your compensation will only increase in the single digits per year and that the pension fund will compound a part of your income in single digits as well (pension funds typically underperform the S&P; even in bad years). So either 2 years of “stable income” or the chance of much higher compounding rate. As I see it, employment is a nice backup if being self-employed doesn’t work out.
- 2y ago
danielmarkbruce 2y ago
Honestly this guy probably got paid so much that the reward will be higher than many "successful" startup outcomes.

rmbyrro 2y ago

which should make the employee proud, and I'm sure Google is compensating them very, very well. let's also not forget that the people involved didn't create this in a vacuum. it cost Google a LOT more than their compensation to make it possible for them to even start working on this project, let alone carrying it forward to completion. people underestimate how hard and expensive it is to manage a company in a way that allows its employees to do a good job.

kqr 2y ago
> let's also not forget that the people involved didn't create this in a vacuum. it cost Google a LOT more than their compensation to make it possible for them to even start working on this project ...let's also not forget that Google didn't manufacture this opportunity in a vacuum. It cost the rest of society a lot more than their revenue to make it possible for them to employ people who can spend their entire days working on software.
- rmbyrro 2y ago
  Didn't it cost society to raise us to adulthood so that we can work for Google and other businesses? How many people have sacrificed their time and attention, directly or indirectly, to our benefit, since we were born? Yet, nobody's saying our salaries aren't deserved or the merit of our own efforts.
  - daelon 2y ago
    I think many people are explicitly saying that.
velcrovan 2y ago
> it cost Google a LOT more than their compensation to make it possible for them to even start working on this project The actual "not a vacuum" context here is an environment that has been basically printing money for google for the last twenty years. It did not "cost" them anything. It's fine to acknowledge that the people who built google, however well paid they are, are creating vastly more value than they are personally receiving.

vasco 2y ago

The question you should ask is for how much less others would accept to do it. How much it saves isn't how to price some work, you price work based on the people available that can do it. If the fire department puts out a fire in your house you don't pay them the cost of the building. You don't give your life to a doctor, etc. That way of thinking is weird.

greggyb 2y ago
It's not weird, but incomplete. It is broadly captured in the Economics concept of "willingness to pay", loosely the maximum price someone or some firm would be willing to pay for something of benefit to them. This contrasts with "willingness to accept", loosely the minimum compensation someone or some firm would accept to produce a good or service (or accept some negative thing). Neither of these is sufficient to determine the price of something precisely, but, in aggregate, these concepts bound the market price for some good or service.
- vasco 2y ago
  I agree I wasn't super precise, I thought it was enough to show that pricing work equal or close to equal to value produced is unlikely as long as there's many people available to do it. In my example of the fire department if nobody is really coming and you have no insurance or other way to save stuff you would indeed pay a lot. From what I read these were the dynamics in Roman times.
  - mrec 2y ago
    If you're thinking of Marcus Crassus, the dynamic there was that he'd offer to buy your burning property from you at a steep discount, and would only put the fire out if you agreed.
- kqr 2y ago
  > "willingness to pay" [and] "willingness to accept" Often abbreviated somewhat sloppily into demand and supply.

jldugger 2y ago

Pretty common really. Huge web properties like this have just massive room for cost optimizations, and AFAICT, is largely bottlenecked by management capacity. On the other hand, it's also pretty common to emphasize pet metrics like tails, which are pretty much by definition a minority of the cost. We give them the benefit of the doubt because a) google and b) peer review publication, but thats not usually a heurestic available to decision makers.

pradn 2y ago
Optimizing tail latency isn't about saving fleet cost. It's about reducing the chance of one tail latency event ruining a page view, when a page view incurs dozens of backend requests. The more requests you have, the higher the chance one of them hits a tail. So the overall latency a user sees is largely dependent on a) number of requests b) tail latency of each event. This method improves the tail latency for ALL supported services, in a generic way. That's multiplicative impact across all services, from a user perspective. Presumably, the number of requests is harder to reduce if they're all required for the business.
- tedd4u 2y ago
  The last sentence of the abstract indicates the goal is to be able to run at higher utilization without sacrificing latency. The outcome of that is you need less hardware for a given load which for Google undoubtedly means massive cost savings and higher profitability. > Prequal has dramatically decreased tail latency, error rates, and resource > use, enabling YouTube and other production systems at Google to run at much > higher utilization.
- jldugger 2y ago
  > Optimizing tail latency isn't about saving fleet cost. Indirectly, it is. As the quote I replied to suggests, in order to combat tail latency services often run with surplus capacity. This is just a fundamental tradeoff between the two variables mentioned. So, by improving the LB algo, they (and anyone, really) can reduce the surplus needed to meet any specific SLO.