7 ms·
This is an extremely important article to read for professionals of all sorts, not just freelancers. It's an issue that I struggle with nearly daily. I can't c
by jiggawatts 3y ago
This is an extremely important article to read for professionals of all sorts, not just freelancers.
It's an issue that I struggle with nearly daily. I can't convince customers to pay me what I'm worth, because they can't judge my true value. Meanwhile, they'll happily pay Deloitte or Accenture 3x as much as what I charge, and then take the misguided ramblings of a junior-sold-as-senior at face value, not knowing the difference.
> I’ve noticed that in nearly every skill tree I’ve climbed, there comes a point where you begin to notice certain nuances of skill that novices not only don’t notice, but that you cannot articulate using the commonly accepted vocabulary of the skill.
This happened to me just over the last month or so. I was trying to explain to a CTO how horrifically wrong it is to plumb their entire cloud network through just a pair of virtual machines for "security". I just could not make him understand that 5 milliseconds of additional latency in the era where microseconds are an eternity is a disaster for a veritable smorgasbord of upcoming projects. It just. Would. Not. Sink. In.
To some people, "micro" and "milli" sound the same. Latency just doesn't compute as a thing to be concerned about.
I've spent my career performance tuning, but then how do I explain this to people who think a 60-second timeout is a performance problem best solved by buying another million dollars worth of kit?
The article jumps between martial arts and the world of finance, so I'll jump to another example: AI playing games.
When I watched AlphaZero play chess, I was just a tiny bit impressed. I watched it beat the Go grandmaster and I was not very impressed. Then I watched it play Dota 2 -- in which I've sunk several thousand hours -- and holy crap. Wow! It was suddenly terrifying how good it was, because I could judge the nuance of what it was doing.
- marginalia_nu 3y ago> To some people, "micro" and "milli" sound the same To be fair, it almost is. Both units are used to talk about times of about the same size. Using both in the same sentence is like mixing meters and feet. It's much clearer to say e.g. 1500us vs 3us; or 1.5ms vs 0.003ms. Don't mix units when making a comparison.
- chaxor 3y agoIn the era of Electron, most things take minutes anyway, so milliseconds is an upgrade.
- maccard 3y agoThe reason things take minutes is because people ignore order of magnitude diffences when they're insignificant, (it doesn't matter if X takes 10ms instead of 0.1ms), and then over time someone builds on that and makes the same assumption (it doesn't matter if it takes 100ms, when it could take 1ms, followed by it doesn't matter if it takes 10s). Avoiding these order of magnitude differencdz early on is the best way to ensure you don't end up with a SPA that takes 15 seconds to populate a list with 20 items in it.
- nooby12345 3y agoA factor of 1000 actually
- marginalia_nu 3y agoHmm, I'm clearly not awake enough to SI. But the point still stands. Don't mix units.
- jiggawatts 3y agoThe lack of expertise here is confirming the point of the article. I'm not mixing units. Milliseconds and microseconds are both measured in units of seconds. The milli- and micro- prefixes are orders of magnitude. They're scales, not units. When a scientist starts talking about measurements accurate to attoseconds, I sit up and take notice. The number in front doesn't make a difference. It's the "atto" that raises my eyebrows. Back to computing: I had a debate here with someone who claimed that 128-bit floats are required for some chip layout codes. I questioned that, because a 32-bit number can subdivide the side-length of a chip down to sub-atomic resolution, and a 64-bit float can model the wiggles of the hadrons making up the atomic nuclei! To some people, 128-bit sounds "twice as good as 64-bit". To experts, it's... umm... a sign you've made a mistake.
- marginalia_nu 3y agoWe can absolutely go into a long side-discussion about units and how most imperial units are also clearly just a constant-factor away from SI units and nobody would argue they are the same unit, but I don't think any of that actually matters. It's semantics. When you are an expert, nearly by definition the average person you're talking to is less well read on the subject than you, the expert. Because of this, it's extremely important to be able to communicate with clarity and patience. Your expertise doesn't matter if you can't make yourself understood, and it's not reasonable to expect people around you to share your expertise, so you need to adapt the message. Communication in general is easier the fewer concepts you invoke. In almost all situations it's better to be mostly correct and well understood than to be perfectly correct and not understood at all. Even if you can expect most people to have an understanding of SI units, far fewer will have an intuition for them (because they aren't working with them daily). Everyone has a basic intuition for big numbers though, so if you want to say "this is bad", show them a big number; the bigger the better. It can still be correct and make a well founded argument, but presentation matters a lot more than you'd think for convincing someone. In increasing order of likelihood it will convince anyone: The new solution adds a 15ms latency, whereas our usual requests have a 2us latency. The new solution adds a 15ms latency, whereas our usual requests have 0.002ms latency. The new solution is a factor 7.5e3 slower than the existing solution, which has a 2 microsecond latency. The new solution is 7,500 times slower than the existing solution, which has a 2 microsecond latency. The new solution is 75,000% slower than the existing solution, which has a 2 microsecond latency. What is said is basically identical in each version, but the presentation puts an increasing emphasis on just how big a difference this is by literally showing a big number.
- Attrecomet 3y agoThis is beautiful, thank you Edit: aww, ninja edit takes away all the fun :/
- marginalia_nu 3y ago> Edit: aww, ninja edit takes away all the fun :/ You know I have a degree in theoretical physics so I really should get this stuff right, but before I've had my morning coffee all bets are off :P
- hef19898 3y agoWell, the difference is the whole point in fields like, e.g., high frequency trading
- verve_rat 3y agoI whole heartedly disagree. Micrometers and millimetres are very, very different and that should be obvious to anyone that uses grams and kgs, or meters and km everyday. Microseconds and milliseconds are obviously different orders of magnitude, that's sort of the whole point of SI prefixes. That CTO is just doesn't sound qualified for the job.
- hardware2win 3y ago>. I just could not make him understand that 5 milliseconds of additional latency in the era where microseconds are an eternity Context matters, not everywhere 5ms more makes difference.
- bryanrasmussen 3y agoI felt the context was really far down the stack, so services that lots of other services would be depending on was how I read it. The places I want fast, high volume and never failing cause so much depends on a red wheelbarrow.
- jiggawatts 3y agoAt the core of a data centre, between VMs, it always makes a difference. An enormous one.
- nmcela 3y agoAs a gamedev, that sentence nearly gave me a heart attack. :) I get where you are coming from, but the amount of work you can do in 5ms is mindblowing.
- ZephyrBlu 3y agoIt sounds like you were just telling the CTO "this is wrong" instead of offering an alternative solution? If that's the case it's not surprising it didn't sink in. You're just giving him another problem. > I've spent my career performance tuning, but then how do I explain this to people who think a 60-second timeout is a performance problem best solved by buying another million dollars worth of kit? I'm surprised you've spent your career doing this and you still haven't figured out how to get people to listen to you. Making your work legible to others feels as important as the work itself.
- jiggawatts 3y agoFor reference, I did offer alternative solutions. > I'm surprised you've spent your career doing this and you still haven't figured out how to get people to listen to you. This is the point of the article, which you appear to have missed. There's a point where it becomes essentially impossible to articulate the differences to the lay person. Or for that matter, someone actually quite skilled, but not a "master". It's not a lack of skill on the part of the expert, it's the lack of skill on the part of everybody else.
- thesumofall 3y agoBut is this really the point of the article? The article argues that it is difficult to identify expertise. However, the situation you describe is one where (if I read you correctly) not your expertise was questioned but your recommendation. If there wasn’t any tangible difference between the CTO’s and your suggested course of action on a metric that he as a customer cared about, then maybe his proposal was indeed the better one? And yes, probably he didn’t care about the exact latencies in the system, but he would probably care about customer retention, system availability, etc.?
- jiggawatts 3y agoI'm finding this entire line of discussion incredibly curious because not only does it confirm my original argument and the that of the article, but it hammers the point home. Everything you've just said is gibberish to a master. It's like making a straight-faced argument that setting the speed limit to 1 mph on a motorway might make perfect sense, and the traffic engineer is actually at fault for not making it clear that this is nonsense. I get that you don't see the ridiculousness of 50 microseconds vs 5 milliseconds, but that's the point. You don't see it, and you start making random arguments untethered from reality. The fact that you are blind to it, unlike being blind to 1 mph vs 65 mph is the entire point. You may have driven thousands of hours. You have don't have thousands of hours of network performance tuning expertise. Both of us wouldn't be able to even begin to understand the complaints Ayrton Senna might make about how the steering is too "crunchy" at the extremes or that the brake strength imbalance is shifting the car about. There's levels of expertise, and if two people are too far apart, meaningful communication between them becomes impossible.
- dumpsterdiver 3y agoIf two nearly identical wallets (very minor, but noticeable differences) are placed on a counter and one is significantly more expensive... how would you expect those wallets to sell? I would expect both of them to sell out. The people looking for a higher quality product will immediately go for the more expensive product, and the remaining wallets would sell to people who just need a wallet.
- joshuaissac 3y ago> The people looking for a higher quality product will immediately go for the more expensive product It is more complicated than that. There is no guarantee that the more expensive wallet is also the higher-quality product. The people looking for a higher-quality product can be classified into those who can measure the quality well (perhaps by observing those minor differences), and those who have to use less effective proxies of quality (like price).
- detourdog 3y agoThis day and age we can make our own inflation. We are all free to pay as much or as little for wallet as we would like. So we all shop by price, quality, or availability.
- ZephyrBlu 3y ago> When I watched AlphaZero play chess, I was just a tiny bit impressed. I watched it beat the Go grandmaster and I was not very impressed. Then I watched it play Dota 2 -- in which I've sunk several thousand hours -- and holy crap. Wow! It was suddenly terrifying how good it was, because I could judge the nuance of what it was doing Funny, my game of choice is StarCraft 2 and when I watched AlphaStar beat StarCraft 2 pros I was significantly less impressed than when I watched AlphaGo beat Lee Sedol at Go.
- jiggawatts 3y agoYou should be more impressed knowing that that AlphaStar was handicapped. Its clicking was rate-limited and its inputs were delayed by something like 150 milliseconds to make it "fair" for the humans. If you remove those limits, it would absolutely wipe the floor with the twitching corpse of any human player. I saw it play Warcraft 2 without the limiters, and it was just ridiculous. It would dance the units around and keep using them with 1 hp left for many minutes and then heal them back up juuuuust enough to take exactly one hit from an enemy.
- whateveracct 3y agoBut not having human reaction time in a real-time game isn't interesting. And the strategies that arise from that aren't either. If anything, human-level I/O constraints should make the AI more impressive because it's actually playing the same game.
- ZephyrBlu 3y agoThe handicapping was very ineffective. AlphaStar had 250-300+ APM and it was basically 100% effective APM. Pros have 300-400+ APM, but their effective APM is much lower due to spam patterns and checking production among other things. Almost everyone in the community seemed to agree AlphaStar was very unfair. Things like a 3 way surround and performing blink micro on individual Stalkers on opposite sides of the screen are completely inhuman. This is also what made the Dota 2 bots feel somewhat disappointing. Of course perfect mechanical skill beats humans. Watching the SF mid 1v1 matches illustrated this very clearly to me. The AI chipped the player down with perfect AAs, then finished them off with perfect distance razes. The bots weren't 100% mechanical skill, but it was a large enough proportion that it was uninteresting.
- ianpurton 3y ago> Meanwhile, they'll happily pay Deloitte or Accenture 3x as much as what I charge, and then take the misguided ramblings of a junior-sold-as-senior at face value, not knowing the difference. This is so true. It's also very difficult to stop as the consultancy gets the project and the client doesn't get to interview the people who actually turn up.
- 3abiton 3y ago> Meanwhile, they'll happily pay Deloitte or Accenture 3x as much as what I charge, and then take the misguided ramblings of a junior-sold-as-senior at face value, not knowing the difference. The main selling point is that big 4/8/16 have standard process, that does not really work, but gives the impression of "high quality" to clients, who are not technical.
- VoodooJuJu 3y ago>I’ve noticed that in nearly every skill tree I’ve climbed, there comes a point where you begin to notice certain nuances of skill that novices not only don’t notice, but that you cannot articulate using the commonly accepted vocabulary of the skill. I've encountered this, and what works is re-framing it in terms they'll understand. Things that are universally understood are: time, money, and competition. In this case, the "micro" and "milli" pitch is technical minutiae and is not what I'm referring to when I say "time". These are more effective frames: >If we do this, it will save/cost us months/years/dollars... >If we do this, the performance hit will knock out our competitive advantage, which we'll have to make up for it with something else, which will cost time/money... >This thing is insufficient for our customer's needs, which will cause our customers to seek out competitors, which will cause us to lose money... >This thing is too complex and will require additional training and is error-prone, which will cost us time and money... In addition to those, for every decision you refute, recommend a superior alternative, and again, frame it in terms of time and money. Now, this won't be a one-sided thing, you'll likely get into a conversation, and a lot of "why's" will try to extract that technical minutiae answer from you, which you might want to mention at some point in the exchange, but don't give it up easily, and do not emphasize it. Make sure you continue to emphasize the more convincing frames: time and money.
- giantg2 3y ago"take the misguided ramblings of a junior-sold-as-senior at face value, not knowing the difference" Based on my experiences, I'm throughly convinced that most of the time the consultants will tell management what they want to hear. If management doesn't like the answer then they go to someone else. How likely it is that they will accept an unwanted opinion from a consultant is solely based on prestige.
- rvba 3y agoOne thing are presentations, other thing is actuall stuff build by 'juniors sold as seniors'. Read all the horror stories of shitty software or ERP implementations. Often it is the consultant's fault - since they are technically weak.
- darkerside 3y agoIsn't it possible that you may not understand the CTO's priorities? Security is clearly essential to him, and performance is essential to you. I don't doubt there are other ways to satisfy both criteria, but frankly, it just doesn't sound like the CTO cared about those milliseconds.
- jiggawatts 3y agoThat came up too. The security aspect of the product does less than nothing: not only does it fail to encrypt data end-to-end, it uses public ip addresses for both the control and data plane. Those public endpoints had an unauthenticated RCE vulnerability recently. It’s a staggeringly bad product and it blew my mind now it was impossible to convince anyone of this. We used pictures, for crying out loud. We did all the things everyone here has been helpfully suggesting. Talking about business consequences (dire), costs (enormous), risks (massive), etc… The response was: “the sales guy from the vendor assured me it’s all fine. This is just your opinion.” The causal chain of a->b->c was just not sinking in. The CTO he could not judge anyone’s expertise not just mine — he also couldn’t judge the lack of expertise by the vendor and their sales team. So he chose based on who has the nicer suit, the shiniest smile, etc…
- xyzelement 3y ago// I just could not make him understand that 5 milliseconds of additional latency in the era where microseconds are an eternity is a disaster for a veritable smorgasbord of upcoming projects. It just. Would. Not. Sink. In. Does this company do something where this latency would be noticeable? Can you give an example of business impact that would result from it? If you can, why don't you? If you can't, then sounds like it doesn't matter. The CTO didn't get to be the CTO by optimizing the irrelevant.
- cgeier 3y agoIdeally, estimate the business values in local currency (and be able to defend those estimates).
- xyzelement 3y agoI agree that's ideal but probably wouldn't expect a latency optimization expert to understand my business well enough to do that and that's fine. But I -would- expect them to say something on the level of: "this is the kinda latency that will prevent you from ever moving to a cloud desktop" or "at this level of latency, you are going to suffer X second delay every time you interact with a complex web app like Jira" or "you are an HFT firm and I know that your competitors are at about 1% of your latency." Then let your client decide if this is something that matters to them. Quite different than "wow the CTO is too dumb to understand my genius"
- heyoni 3y agoThat was my exact reaction too. There’s a reason python and other “slow” languages are dominating. Very few industries require that kind speed, and even fewer require it at the cost of simplifying code or architecture.
- girvo 3y agoIn my experience the CTO got to be CTO because they’re friends with the CEO, the board, etc, rather than their own technical ability being superior than the seniors/leads under them. And that goes for me too, when I was CTO. It’s a position, it doesn’t have any inherent guarantee of ability to avoid optimising the irrelevant: some will, some won’t.
- thrw4wy-d2 3y ago> Then I watched it play Dota 2 -- in which I've sunk several thousand hours -- and holy crap Fascinating. I’ve never professionally worked in what it sounds like you do, so I was moderately agreeing with you until this. I’ve played Dota likely as long as you in addition to playing at a semi pro level (placing well in a couple official pro qualifying tournaments as well as playing pubs with the players the AI played against) and I couldn’t disagree more. It was impressive, but it’s main advantage came from its inhuman reaction times. They added some buffer but still didn’t account for 5 players working in perfect unison. That kind of coordination is an enormous advantage in a game like Dota. I left roughly feeling like I do now about AI - very impressive, but way over hyped and only sold to people that don’t truly get Dota.
- danielmarkbruce 3y agoBranding is real.
- msp26 3y agoIt's interesting that you bring up Dota because I find that it's a ready example of the vocab point failing. When you listen to pros talk, they misuse words and concepts all the time but it doesn't matter because the other pros listening get the 'feel' of what they're saying. (I'm not a pro or anything, just high rank, but to use myself as an example) I try to make an effort to use the correct words when explaining things to friends. But even I find myself saying things like 'just look at the [lane] creeps' to explain what I want my laning partner to do and then I realise how unhelpful it is. Because when they see the creeps they don't understand what I see.
- Corrado 3y agoI have a couple of similar experiences. I've spent >1,500 hours playing Team Fortress 2 and can recognize things that other gamer friends just can't see. I'm still not good at TF2, but I can recognize great playing. Years ago I was active in Tae Kwon Do and achieved black-belt status. One time, during a sparing match with my teacher, I was astounded to be able to spot a hole in his defense. In fact, I was so distracted by my perception that I completely froze and he scored a point. I like to think that my seeing that hole gave me a glimpse into what it might be like to be good at TDK. I still think about that experience to this day.