5 ms·

Oh God. Pilots use LGTM too.

by agloe_dreams 4y ago

dclowd9901 4y ago
Honestly, I’m glad they do that much. God forbid they just blindly accept the data.
- agloe_dreams 4y ago
  That has, historically been my issue with LGTM. It is ultimately a meaningless rubber stamp. Many people get a large PR and give it a quick scroll, type LGTM and move on. If I had it my way, every PR with over 100 LOC diff should require two PR code comments for a reviewer to approve it.
  - dclowd9901 4y ago
    I see it more as “this result is commensurate with my experience,” the problem with it, of course, being reliant on the pilot’s experience. But otherwise we’re just talking about a “sniff test” on a number. LGTM sucks in software because software inherently is a lot more complex than a number.
- tialaramex 4y ago
  Indeed. Designing software so that it helps humans to eyeball stuff where humans eyeballing stuff might help is really good. For example in that incident in Ireland where crew erroneously told their jet plane it was very, very cold, and then (cold air is dense, less power needed) they barely had enough thrust to take off before they ran entirely out of runway - that was partly caused (Swiss cheese model) by the pilots being issued with software that didn't make it easy for them to cross check the facts they can eyeball. Is it certain they'd have realised if their software did show this information - like the software issued to similar crews with other outfits? No, but that's one more opportunity for everybody to not die, and the whole point of the Swiss cheese model is that lots of things have to go wrong or else you're fine. One of the other potential mitigations for that incident illustrates where human eyeballs don't help. In principle a human pilot could realise they aren't accelerating enough. In practice humans are lousy at estimating acceleration to the tolerance needed so this isn't effective unless you use the same strip so often that your brain learns exactly how things ought to look. However thanks to GPS a machine can measure acceleration very well, and it can know how long your chosen runway is and what its take off speed is - so it can estimate if you're accelerating enough and the latest models can announce "Caution: Acceleration" early in the take off roll, before V1, which means the crew can abort and check why they didn't have the intended acceleration - regardless of the cause.
dentemple 4y ago
What we need to do is package LGTM as an OpenAI-based automated service...
papruapap 4y ago
Let's Gamble, Try Merging
- junon 4y ago
  This is a life changing comment for me. Thank you.
- bobleeswagger 4y ago
  Let's Get That Money
- ravenstine 4y ago
  Move fast, break things, bruh.
deleted 4y ago
[deleted]
mcherm 4y ago
Notice that they are not using it as the only means for verifying system. They are using it as an additional check that supplements the extensive regression test suite. The test suite failed in this case because the bug occurred only under high load (after this, they added a new test case that simulates high load). The human verification that was in place in production as an additional safety check operated as intended and detected the bug.
hyperthesis 4y ago
looks good to me
dotancohen 4y ago
Let's Get This Moving
sfmike 4y ago
devils advocate, what if No AI will ever reach a point of being perfect without a human LGTM verification

Oh God. Pilots use LGTM too.

dclowd9901 4y ago

Honestly, I’m glad they do that much. God forbid they just blindly accept the data.

agloe_dreams 4y ago
That has, historically been my issue with LGTM. It is ultimately a meaningless rubber stamp. Many people get a large PR and give it a quick scroll, type LGTM and move on. If I had it my way, every PR with over 100 LOC diff should require two PR code comments for a reviewer to approve it.
- dclowd9901 4y ago
  I see it more as “this result is commensurate with my experience,” the problem with it, of course, being reliant on the pilot’s experience. But otherwise we’re just talking about a “sniff test” on a number. LGTM sucks in software because software inherently is a lot more complex than a number.
tialaramex 4y ago
Indeed. Designing software so that it helps humans to eyeball stuff where humans eyeballing stuff might help is really good. For example in that incident in Ireland where crew erroneously told their jet plane it was very, very cold, and then (cold air is dense, less power needed) they barely had enough thrust to take off before they ran entirely out of runway - that was partly caused (Swiss cheese model) by the pilots being issued with software that didn't make it easy for them to cross check the facts they can eyeball. Is it certain they'd have realised if their software did show this information - like the software issued to similar crews with other outfits? No, but that's one more opportunity for everybody to not die, and the whole point of the Swiss cheese model is that lots of things have to go wrong or else you're fine. One of the other potential mitigations for that incident illustrates where human eyeballs don't help. In principle a human pilot could realise they aren't accelerating enough. In practice humans are lousy at estimating acceleration to the tolerance needed so this isn't effective unless you use the same strip so often that your brain learns exactly how things ought to look. However thanks to GPS a machine can measure acceleration very well, and it can know how long your chosen runway is and what its take off speed is - so it can estimate if you're accelerating enough and the latest models can announce "Caution: Acceleration" early in the take off roll, before V1, which means the crew can abort and check why they didn't have the intended acceleration - regardless of the cause.

dentemple 4y ago

What we need to do is package LGTM as an OpenAI-based automated service...

papruapap 4y ago

Let's Gamble, Try Merging

junon 4y ago
This is a life changing comment for me. Thank you.
bobleeswagger 4y ago
Let's Get That Money
ravenstine 4y ago
Move fast, break things, bruh.

deleted 4y ago

[deleted]

mcherm 4y ago

Notice that they are not using it as the only means for verifying system. They are using it as an additional check that supplements the extensive regression test suite. The test suite failed in this case because the bug occurred only under high load (after this, they added a new test case that simulates high load). The human verification that was in place in production as an additional safety check operated as intended and detected the bug.

hyperthesis 4y ago

looks good to me

dotancohen 4y ago

Let's Get This Moving

sfmike 4y ago

devils advocate, what if No AI will ever reach a point of being perfect without a human LGTM verification