3 ms·

Hard limit: - Plenty of API hallucination happening on cutting edge Spark (4.0.0+) functionality, especially PySpark. Spark bares some blame here for broken

by datsci_est_2015 7mo ago

Hard limit: - Plenty of API hallucination happening on cutting edge Spark (4.0.0+) functionality, especially PySpark. Spark bares some blame here for broken and incomplete documentation. Takes a human in the loop to realize that the documentation is misleading or wrong or missing. Soft limit: - API design. I’ve found that, unless specifically steered towards “good” API design (highly subjective), agents tend to just add another endpoint / function to satisfy the exact task at hand, with total disregard to how the rest of the API looks. (Pretty much exactly what a junior engineer would do…)

michaelmior 7mo ago
On this note, one thing I've found Codex to do is worry more than necessary about breaking changes for internal APIs. Maybe a bit more prompting would fix this, but I found even when iteratively implementing larger new features, it worries about breaking APIs that aren't used by anything but the new code yet.
lloydatkinson 7mo ago
That last part about it acting like a junior matches my experience very well. I'm using LLM's for refactoring, adding repetitive blocks of code, etc. Unless I'm very clear at all times it will write code like the most annoying stubborn junior you've ever worked with. Nothing is sacred, everything can be abbreviated, shortened, made more confusing, made less readable, and concepts like readability or naming conventions are not even considered. It also adds superfluous nonsense comments that don't explain the "why".
rstuart4133 7mo ago
> “good” API design (highly subjective) A good API does two things. Firstly it DRYs the code out. Many APIs start life doing only that, as a collection of routines you get tired of writing over and over again. Secondly, the functions are designed in a way that reduces the need to share information. Typically they do that by hiding a whole pile of details in their implementation so knowledge of those details is all in one place rather than scattered across a code base. A term often used for APIs that don't do that well is "leaky", or we say it's a "leaky abstraction". Perhaps a good API has other more subjective attributes but it must have those two. LLMs suck at both. You can see that in the comments here, when people say they write verbose code. It's verbose because the LLM didn't go looking for duplicated functionality - if it needed it, it just put the code where it was focused on at the time. If they are bad at DRY then I need a better superlative to describe how they fare at respecting the isolation boundaries that underpin good module design. As far as I can tell, they have no idea about the concept or how to implement it. Let loose, they are like a bull in a china shop, breaking one boundary after another.

michaelmior 7mo ago

On this note, one thing I've found Codex to do is worry more than necessary about breaking changes for internal APIs. Maybe a bit more prompting would fix this, but I found even when iteratively implementing larger new features, it worries about breaking APIs that aren't used by anything but the new code yet.

lloydatkinson 7mo ago

That last part about it acting like a junior matches my experience very well. I'm using LLM's for refactoring, adding repetitive blocks of code, etc. Unless I'm very clear at all times it will write code like the most annoying stubborn junior you've ever worked with. Nothing is sacred, everything can be abbreviated, shortened, made more confusing, made less readable, and concepts like readability or naming conventions are not even considered. It also adds superfluous nonsense comments that don't explain the "why".

rstuart4133 7mo ago

> “good” API design (highly subjective) A good API does two things. Firstly it DRYs the code out. Many APIs start life doing only that, as a collection of routines you get tired of writing over and over again. Secondly, the functions are designed in a way that reduces the need to share information. Typically they do that by hiding a whole pile of details in their implementation so knowledge of those details is all in one place rather than scattered across a code base. A term often used for APIs that don't do that well is "leaky", or we say it's a "leaky abstraction". Perhaps a good API has other more subjective attributes but it must have those two. LLMs suck at both. You can see that in the comments here, when people say they write verbose code. It's verbose because the LLM didn't go looking for duplicated functionality - if it needed it, it just put the code where it was focused on at the time. If they are bad at DRY then I need a better superlative to describe how they fare at respecting the isolation boundaries that underpin good module design. As far as I can tell, they have no idea about the concept or how to implement it. Let loose, they are like a bull in a china shop, breaking one boundary after another.