46 ms·

The burden you are placing is too high here. Do you demand controlled trials for everything you do or else you refuse to use it or accept that other people migh

by johnfn 9mo ago

The burden you are placing is too high here. Do you demand controlled trials for everything you do or else you refuse to use it or accept that other people might see productivity gains? Do you demand studies showing that static typing is productive? Syntax highlighting? IDEs or Vim? Unit testing? Whatever language you use? Obviously not? It would be absurd to walk into a thread about Rust and say “Rust doesn’t increase your productivity and unless you can produce a study proving it does then your own personal anecdotes are worthless.” Why the increased demand for rigor when it comes to AI specifically?

llmslave2 9mo ago
Do you just believe everything everybody says? No quantifiable data required, as long as someone somewhere says it it must be true? One of the reasons software is in decline is because it's all vibes, nobody has much interest in conducting research to find anything out. It doesn't have to be some double blinded peer reviewed meta analysis, the bar can still be low, it just should be higher than "I feel like"...
- nfw2 9mo ago
  here are some https://resources.github.com/learn/pathways/copilot/essentials/measuring-the-impact-of-github-copilot/ https://resources.github.com/learn/pathways/copilot/essentia... https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic https://www.anthropic.com/research/how-ai-is-transforming-wo... https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier https://www.mckinsey.com/capabilities/tech-and-ai/our-insigh...
  - llmslave2 9mo ago
    They're all marketing slop lol. Go look at their methodology. Absolutely shite.
    - nfw2 9mo ago
      This is what you claimed the bar was "it just should be higher than 'I feel like'" Now you are moving it because your statement is provably false. Your criticism of it is based on vibes. What specifically is wrong with the methodologies? One of them broke randomly developers into two groups, one with access to ai and one without, timed them to complete the same task, and compared the results. That seems fine? Any measurement of performance in a lab environment comes with caveats, but since real world accounts you dismiss as vibes, that seems like the best you can do.
      - llmslave2 9mo ago
        I'm sorry but I'm not going to take "research" about Claude seriously from Anthropic, the company who makes and sells Claude. I'm also not going to do that for Copilot from Microsoft, the company who makes and sells Copilot.
- johnfn 9mo ago
  You don't seem to have answered my questions - you are just reiterating your own point (which I already responded to). Again I ask you - do you have studies to prove that syntax highlighting is useful or are you just using it because of vibes? Do you have research showing that writing in your language of choice is faster than Assembly?
  - llmslave2 9mo ago
    I actually prefer no syntax highlighting, and I certainly wouldn't make any claims about it being useful. But something being "useful" is often personal - I find IDEs useful, others find Vim useful, maybe one is better or worse than the other or maybe we're all different and our brains function in different ways and that explains the difference. With assembly versus say, Go for writing a web server? That's trivially observable, good luck arguing against that one.
    - nfw2 9mo ago
      That's the whole point. The sky is blue is trivially observable. Any claim that someone has disproven something that is trivially observable should be met with skepticism. If you have something that needs to be done, and an agent goes and does the whole thing for you without mistakes, it is trivially observable that that is useful. That is the definition of usefulness.
      - llmslave2 9mo ago
        But useful in the context of these debates isn't that it solves any single problem for someone. Nobody is arguing that LLM's have zero utility. So I don't really see what your point is?
AstroBen 9mo ago
Typically I hear how other people are doing things and I test it out for myself. Just like I'm doing with AI Actually IDEs vs vim are a perfect analogy because they both have the ability to feel like they're helping a tonne, and at the end of the work day neither group outperforms the other I'm not standing on the sidelines criticizing this stuff. I'm using it. I'm growing more and more skeptical because it's not noticably helping me deliver features faster At this point I'm at "okay record a video and show me these 3x gains you're seeing because I'm not experiencing the same thing" The increased demand for rigor is because my experience isn't matching what others say I can see a 25% bump in productivity being realistic if I learn where it works well. There are people claiming 3-10x. It sounds ridiculous
- bluGill 9mo ago
  I canzt see a 25% jump in productivity because writting code isn't even 25% of what I do. Even if it was infitiely fast I still can't get that high.
  - bonesss 9mo ago
    Given a 25% hypothetical boost: there are categories of errors vibe testing vibed code will bring in, we know humans suck at critical reading. On the support timeline of an Enterprise product that’s gonna lead to one or more true issues. At what point is an ‘extra’ 25% coding overhead worth it to ensure a sane human reasonably concerned about criminal consequences for impropriety read all code when making it, and every change around it? To prevent public embarrassment that can and will chase off customers? To have someone to fire and sue if need be? [Anecdotally, the inflection point was finding tests updated to short circuit through mildly obfuscated code (introduced after several reviews). Paired with a working system developed with TDD, that mistake only becomes obvious when the system stops working but the tests don’t. I wrote it, I ran the agents, I read it, I approved it, but was looking for code quality not intentional sabotage/trickery… lesson learned.] From a team lead perspective in an Enterprise space, using 25% more time on coding to save insane amounts of aggressive and easy to flubb review and categories of errors sounds like a smart play. CYA up front, take the pain up front.
shimman 9mo ago
I honestly wish we had studies that truly answered these Qs. Modern programming has been a cargo cult for a good 20 years now.
- nfw2 9mo ago
  People who think syntax highlighting is useful are a cargo cult?

by johnfn 9mo ago

llmslave2 9mo ago

Do you just believe everything everybody says? No quantifiable data required, as long as someone somewhere says it it must be true? One of the reasons software is in decline is because it's all vibes, nobody has much interest in conducting research to find anything out. It doesn't have to be some double blinded peer reviewed meta analysis, the bar can still be low, it just should be higher than "I feel like"...

nfw2 9mo ago
here are some https://resources.github.com/learn/pathways/copilot/essentials/measuring-the-impact-of-github-copilot/ https://resources.github.com/learn/pathways/copilot/essentia... https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic https://www.anthropic.com/research/how-ai-is-transforming-wo... https://www.mckinsey.com/capabilities/tech-and-ai/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier https://www.mckinsey.com/capabilities/tech-and-ai/our-insigh...
- llmslave2 9mo ago
  They're all marketing slop lol. Go look at their methodology. Absolutely shite.
  - nfw2 9mo ago
    This is what you claimed the bar was "it just should be higher than 'I feel like'" Now you are moving it because your statement is provably false. Your criticism of it is based on vibes. What specifically is wrong with the methodologies? One of them broke randomly developers into two groups, one with access to ai and one without, timed them to complete the same task, and compared the results. That seems fine? Any measurement of performance in a lab environment comes with caveats, but since real world accounts you dismiss as vibes, that seems like the best you can do.
    - llmslave2 9mo ago
      I'm sorry but I'm not going to take "research" about Claude seriously from Anthropic, the company who makes and sells Claude. I'm also not going to do that for Copilot from Microsoft, the company who makes and sells Copilot.
johnfn 9mo ago
You don't seem to have answered my questions - you are just reiterating your own point (which I already responded to). Again I ask you - do you have studies to prove that syntax highlighting is useful or are you just using it because of vibes? Do you have research showing that writing in your language of choice is faster than Assembly?
- llmslave2 9mo ago
  I actually prefer no syntax highlighting, and I certainly wouldn't make any claims about it being useful. But something being "useful" is often personal - I find IDEs useful, others find Vim useful, maybe one is better or worse than the other or maybe we're all different and our brains function in different ways and that explains the difference. With assembly versus say, Go for writing a web server? That's trivially observable, good luck arguing against that one.
  - nfw2 9mo ago
    That's the whole point. The sky is blue is trivially observable. Any claim that someone has disproven something that is trivially observable should be met with skepticism. If you have something that needs to be done, and an agent goes and does the whole thing for you without mistakes, it is trivially observable that that is useful. That is the definition of usefulness.
    - llmslave2 9mo ago
      But useful in the context of these debates isn't that it solves any single problem for someone. Nobody is arguing that LLM's have zero utility. So I don't really see what your point is?

AstroBen 9mo ago

Typically I hear how other people are doing things and I test it out for myself. Just like I'm doing with AI Actually IDEs vs vim are a perfect analogy because they both have the ability to feel like they're helping a tonne, and at the end of the work day neither group outperforms the other I'm not standing on the sidelines criticizing this stuff. I'm using it. I'm growing more and more skeptical because it's not noticably helping me deliver features faster At this point I'm at "okay record a video and show me these 3x gains you're seeing because I'm not experiencing the same thing" The increased demand for rigor is because my experience isn't matching what others say I can see a 25% bump in productivity being realistic if I learn where it works well. There are people claiming 3-10x. It sounds ridiculous

bluGill 9mo ago
I canzt see a 25% jump in productivity because writting code isn't even 25% of what I do. Even if it was infitiely fast I still can't get that high.
- bonesss 9mo ago
  Given a 25% hypothetical boost: there are categories of errors vibe testing vibed code will bring in, we know humans suck at critical reading. On the support timeline of an Enterprise product that’s gonna lead to one or more true issues. At what point is an ‘extra’ 25% coding overhead worth it to ensure a sane human reasonably concerned about criminal consequences for impropriety read all code when making it, and every change around it? To prevent public embarrassment that can and will chase off customers? To have someone to fire and sue if need be? [Anecdotally, the inflection point was finding tests updated to short circuit through mildly obfuscated code (introduced after several reviews). Paired with a working system developed with TDD, that mistake only becomes obvious when the system stops working but the tests don’t. I wrote it, I ran the agents, I read it, I approved it, but was looking for code quality not intentional sabotage/trickery… lesson learned.] From a team lead perspective in an Enterprise space, using 25% more time on coding to save insane amounts of aggressive and easy to flubb review and categories of errors sounds like a smart play. CYA up front, take the pain up front.

shimman 9mo ago

I honestly wish we had studies that truly answered these Qs. Modern programming has been a cargo cult for a good 20 years now.

nfw2 9mo ago
People who think syntax highlighting is useful are a cargo cult?