5 ms·

We've just gotten through a big second generation of available models, but at least for the projects I'm on, I still feel just as far away as ever as being able

We've just gotten through a big second generation of available models, but at least for the projects I'm on, I still feel just as far away as ever as being able to trust responses. I spent a few hours this weekend working on a side project that involved setting up a graphql server with a few basic query resolvers and field resolvers, and my experience with 4o, R1, o3-mini-high were all akin to arguing with too-confident junior engineers that were confused, without realizing it, about what they thought they knew. And this was basic stuff about simple graphql resolvers. I did have my first experience of o3-mini-high (finally, after much arguing) being able to answer something that R1 couldn't, though. It's weird, because it's still wildly useful and my ideas for side projects are definitely more expansive than they used to be. And yet, I'm really far from having any fear of replacement. Almost none of the answers I'm getting are truly nailing the experience of teaching me something new, while also having perfect accuracy. (I'm on ChatGPT+, not pro.)

ilrwbwrkhv 2y ago
Same. I think the tools have gotten a little better with agent mode and all that but it still doesn't get anything even slightly complicated for an API it hasn't seen before. Like it hasn't gotten more intelligent since maybe Gpt 3.5. I quite like them as somebody who has a huge problem with onset procrastination. I immediately ask them to write something and criticize them and write it myself because whatever it comes up with is so damn stupid.
- cma 2y ago
  Have you tried putting the API in context? Gemini has 2M context, where original 3.5 had like 4K and not great with it. While some large apis may not fit, not many that it hasn't seen before fit that bill. For updated large APIs you can put in the changelog but more work to gather all the detailed API changes and changed documentation on their own.
sunpazed 2y ago
Even though I have a reasonable intuition and understanding of how a LLM works, I am still awe-struck each and every time I use one. The fact that I have a junior developer at my convenience is a huge efficiency gain. I’ve been able to automate the rudimentary elements and focus my time on the stuff that counts.
aerhardt 2y ago
The other day o1 hallucinated on a question of some Salesforce object properties that are clearly laid out in the docs. It took me a while to disentangle it because it was very persuasive. I concur with you though, I still find the tech tremendously useful.

ilrwbwrkhv 2y ago

Same. I think the tools have gotten a little better with agent mode and all that but it still doesn't get anything even slightly complicated for an API it hasn't seen before. Like it hasn't gotten more intelligent since maybe Gpt 3.5. I quite like them as somebody who has a huge problem with onset procrastination. I immediately ask them to write something and criticize them and write it myself because whatever it comes up with is so damn stupid.

cma 2y ago
Have you tried putting the API in context? Gemini has 2M context, where original 3.5 had like 4K and not great with it. While some large apis may not fit, not many that it hasn't seen before fit that bill. For updated large APIs you can put in the changelog but more work to gather all the detailed API changes and changed documentation on their own.

sunpazed 2y ago

Even though I have a reasonable intuition and understanding of how a LLM works, I am still awe-struck each and every time I use one. The fact that I have a junior developer at my convenience is a huge efficiency gain. I’ve been able to automate the rudimentary elements and focus my time on the stuff that counts.

aerhardt 2y ago

The other day o1 hallucinated on a question of some Salesforce object properties that are clearly laid out in the docs. It took me a while to disentangle it because it was very persuasive. I concur with you though, I still find the tech tremendously useful.