8 ms·

The premise might possibly be true, but as an actually seasoned Python developer, I've taken a look at one file: https://github.com/dx-tooling/platform-problem-

The premise might possibly be true, but as an actually seasoned Python developer, I've taken a look at one file: https://github.com/dx-tooling/platform-problem-monitoring-core/blob/main/src/platform_problem_monitoring_core/utils.py https://github.com/dx-tooling/platform-problem-monitoring-co... All of it smells of a (lousy) junior software engineer: from configuring root logger at the top, module level (which relies on module import caching not to be reapplied), over not using a stdlib config file parser and building one themselves, to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there... In a nutshell, if the rest of it is like this, it simply sucks.

abid786 2y ago
Doesn’t load_json throw if the file doesn’t exist?
- isoprophlex 2y ago
  Yes but then why do the check in the first place
ilrwbwrkhv 2y ago
Yup this tracks with what I have seen as well. Most devs who use this daily are usually junior devs or javascript devs who both write sloppy questionable code.
rybosome 2y ago
Ok - not wrong at all. Now take that feedback and put it in a prompt back to the LLM. They’re very good at honing bad code into good code with good feedback. And when you can describe good code faster than you can write it - for instance it uses a library you’re not intimately familiar with - this kind of coding can be enormously productive.
- necovek 2y ago
  I do plan on experimenting with the latest versions of coding assistants, but last I tried them (6 months ago), none could satisfy all of the requirements at the same time. Perhaps there is simply too much crappy Python code around that they were trained on as Python is frequently used for "scripting". Perhaps the field has moved on and I need to try again. But looking at this, it would still be faster for me to type this out myself than go through multiple rounds of reviews and prompts. Really, a senior has not reviewed this, no matter their language (raciness throughout, not just this file).
  - deleted 2y ago
    [deleted]
- imiric 2y ago
  > They’re very good at honing bad code into good code with good feedback. And they're very bad at keeping other code good across iterations. So you might find that while they might've fixed the specific thing you asked for—in the best case scenario, assuming no hallucinations and such—they inadvertently broke something else. So this quickly becomes a game of whack-a-mole, at which point it's safer, quicker, and easier to fix it yourself. IME the chance of this happening is directly proportional to the length of the context.
  - bongodongobob 2y ago
    This typically happens when you run the chat too long. When it gives you a new codebase, fire up a new chat so the old stuff doesn't poison the context window.
    - no_wizard 2y ago
      Why isn’t it smart enough to recognize new contexts that aren’t related to old ones?
      - bongodongobob 2y ago
        I don't know, I didn't invent transformers. I do however know how to work with them.
    - achierius 2y ago
      But it rarely gives me a totally-new codebase unless I'm working on a very small project -- so I have to choose between ditching its understanding of some parts (e.g. "don't introduce this bug here, please") and avoiding confusion with others.
  - deleted 2y ago
    [deleted]
- aunty_helen 2y ago
  Nah. This isn’t true. Every time you hit enter you’re not just getting a jr dev, you’re getting a randomly selected jr dev. So, how did I end up with a logging.py, config.py, config in __init__.py and main.py? Well I prompted for it to fix the logging setup to use a specific format. I use cursor, it can spit out code at an amazing rate and reduced the amount of docs I need to read to get something done. But after its second attempt at something you need to jump in and do it yourself and most likely debug what was written.
  - skydhash 2y ago
    Are you reading a whole encyclopedia each time you assigned to a task? The one thing about learning is that it compounds. You get faster the longer you use a specific technology. So unless you use a different platform for each task, I don't think you have to read that much documentation (understanding them is another matter).
    - achierius 2y ago
      This is an important distinction though. LLMs don't have any persistent 'state': they have their activations, their context, and that's it. They only know what's pre-trained, and what's in their context. Now, their ability to do in-context learning is impressive, but you're fundamentally still stuck with the deviations and, eventually, forgetting that characterizes these guys -- while a human, while less quick on the uptake, will nevertheless 'bake in' the lessons in a way that LLMs currently cannot. In some ways this is even more impressive -- every prompt you make, your LLM is in effect re-reading (and re-comprehending) your whole codebase, from scratch!
- BikiniPrince 2y ago
  I’ve found AI tools extremely helpful in getting me up to speed with a library or defining an internal override not exposed by the help. However, if I’m not explicit in how to solve a problem the result looks like the bad code it’s been ingesting.
- barrell 2y ago
  I would not say it is “very good” at that. Maybe it’s “capable,” but my (ample) experience has been the opposite. I have found the more exact I describe a solution, the less likely it is to succeed. And the more of a solution it has come up with, the less likely it is to change its mind about things. Every since ~4o models, there seems to be a pretty decent chance that you ask it to change something specific and it says it will and it spits out line for line identical code to what you just asked it to change. I have had some really cool success with AI finding optimizations in my code, but only when specifically asked, and even then I just read the response as theory and go write it myself, often in 1-15% the LoC as the LLM
mjr00 2y ago
I "love" this part: def ensure_dir_exists(path: str) -> None: """ Ensure a directory exists. Args: path: Directory path """ An extremely useful and insightful comment. Then you look where it's actually used, # Ensure the directory exists and is writable ensure_dir_exists(work_dir) work_path = Path(work_dir) if not work_path.exists() or not os.access(work_dir, os.W_OK): ... so like, the entire function and its call (and its needlessly verbose comment) could be removed because the existence of the directory is being checked anyway by pathlib. This might not matter here because it's a small, trivial example, but if you have 10, 50, 100, 500 developers working on a codebase, and they're all thoughtlessly slinging code like this in, you're going to have a dumpster fire soon enough. I honestly think "vibe coding" is the best use case for AI coding, because at least then you're fully aware the code is throwaway shit and don't pretend otherwise. edit: and actually looking deeper, `ensure_dir_exists` actually makes the directory, except it's already been made before the function is called so... sigh. Code reviews are going to be pretty tedious in the coming years, aren't they?
milicat 2y ago
The more I browse through this, the more I agree. I feel like one could delete almost all comments from that project without losing any information – which means, at least the variable naming is (probably?) sensible. Then again, I don't know the application domain. Also… def _save_current_date_time(current_date_time_file: str, current_date_time: str) -> None: with Path(current_date_time_file).open("w") as f: f.write(current_date_time) there is a lot of obviously useful abstraction being missed, wasting lines of code that will all need to be maintained. The scary thing is: I have seen professional human developers write worse code.
- ramesh31 2y ago
  >The scary thing is: I have seen professional human developers write worse code. This is kind of the rub of it all. If the code works, passes all relevant tests, is reasonably maintainable, and can be fitted into the system correctly with a well defined interface, does it really matter? I mean at that point its kind of like looking at the output of a bytecode compiler and being like "wow what a mess". And it's not like they can't write code up to your stylistic standards, it's just literally a matter of prompting for that.
  - mjr00 2y ago
    > If the code works, passes all relevant tests, is reasonably maintainable, and can be fitted into the system correctly with a well defined interface, does it really matter? You're not wrong here, but there's a big difference in programming one-off tooling or prototype MVPs and programming things that need to be maintained for years and years. We did this song and dance pretty recently with dynamic typing. Developers thought it was so much more productive to use dynamically typed languages, because it is in the initial phases. Then years went by, those small, quick-to-make dynamic codebases ended up becoming unmaintainable monstrosities, and those developers who hyped up dynamic typing invented Python/PHP type hinting and Flow for JavaScript, later moving to TypeScript entirely. Nowadays nobody seriously recommends building long-lived systems in untyped languages, but they are still very useful for one-off scripting and more interactive/exploratory work where correctness is less important, i.e. Jupyter notebooks. I wouldn't be surprised to see the same pattern happen with low-supervision AI code; it's great for popping out the first MVP, but because it generates poor code, the gung-ho junior devs who think they're getting 10x productivity gains will wisen up and realize the value of spending an hour thinking about proper levels of abstraction instead of YOLO'ing the first thing the AI spits out when they want to build a system that's going to be worked on by multiple developers for multiple years.
nottorp 2y ago
Here's a rl example from today: I asked $random_llm to give me code to recursively scan a directory and give me a list of file names relative to the top directory scanned and their sizes. It gave me working code. On my test data directory it needed ... 6.8 seconds. After 5 min of eliminating obvious inefficiencies the new code needed ... 1.4 seconds. And i didn't even read the docs for the used functions yet, just changed what seemed to generate too many filesystem calls for each file.
- bongodongobob 2y ago
  Nice, sounds like it saved you some time.
  - nottorp 2y ago
    You "AI" enthusiasts always try to find a positive spin :) What if I had trusted the code? It was working after all. I'm guessing that if i asked for string manipulation code it would have done something worth posting on accidentally quadratic.
    - noisy_boy 2y ago
      Depends on how toxic the culture is in your workplace. This could have been an opportunity to "work" on another JIRA task showing 600% improvement over AI generated code.
      - nottorp 2y ago
        I'll write that down for reference in case I do ever join an organization like that in the future, thanks. 600% improvement is worth what, 3 days of billable work if it lasts 5 minutes?
        noisy_boy 2y ago
        Series of such "improvements" could be fame and fortune in your team/group/vertical. In such places, the guy who toots the loudest wins the most.
layoric 2y ago
Also somewhat strangely, I've found Python output has remained bad, especially for me with dataframe tasks/data analysis. For remembering matplotlib syntax I still find most of them pretty good, but for handling datagframes, very bad and extremely counter productive. Saying that, for typed languages like TypeScript and C#, they have gotten very good. I suspect this might be related to the semantic information can be found in typed languages, and hard to follow unstructured blobs like dataframes, and there for, not well repeated by LLMs.
- datadrivenangel 2y ago
  Spark especially is brutal for some reason. Even databrick's AI is bad at spark, which is very funny. It's probably because spark is so backwards compatible with pandas, but not fully.
gerdesj 2y ago
My current favourite LLM wankery example is this beauty: https://blog.fahadusman.com/proxmox-replacing-failed-drive-in-zfs-pool/ https://blog.fahadusman.com/proxmox-replacing-failed-drive-i... Note how it has invented the faster parameter for the zpool command. It is possible that the blog writer hallucinated a faster parameter themselves without needing a LLM - who knows. I think all developers should add a faster parameter to all commands to make them run faster. Perhaps a LLM could create the faster code. I predict an increase of man page reading, and better quality documentation at authoritative sources. We will also improve our skills at finding auth sources of docs. My uBlacklist is getting quite long.
- Henchman21 2y ago
  What makes you think this was created by an LLM? I suspect they might actually have a pool named faster -- I know I've named pools similarly in the past. This is why I now name my pools after characters from the Matrix, as is tradition.
  - taurath 2y ago
    This really gets to an acceleration of enshittification. If you can't tell its an LLM, and there's nobody to verify the information, humanity is architecting errors and mindfucks into everything. All of the markers of what is trustworthy has been coopted by untrustworthy machines, so all of the way's we'd previously differentiated actors have stopped working. It feels like we're just losing truth as rapidly as LLMs can generate mistakes. We've built a scoundrels paradise. How useful is a library of knowledge when n% of the information is suspect? We're all about to find out.
    - Henchman21 2y ago
      You know, things looked off to me, but thinking it was the output of an LLM just didn't seem obvious -- even though that was the claim! I feel ill-equipped to deal with this, and as the enshittification has progressed I've found myself using "the web" less and less. At this point, I'm not sure there's much left I value on the web. I wish the enshittification wasn't seemingly pervasive in life.
byproxy 2y ago
As an actually unseasoned Python developer, would you be so kind as to explain why the problems you see are problems and their alternatives? Particularly the first two you note.
- saila 2y ago
  The call to logging.basicConfig happens at import time, which could cause issues in certain scenarios. For a one-off script, it's probably fine, but for a production app, you'd probably want to set up logging during app startup from whatever your main entry point is. The Python standard library has a configparser module, which should be used instead of custom code. It's safer and easier than manual parsing. The standard library also has a tomllib module, which would be an even better option IMO.
  - cinntaile 2y ago
    Regarding your first paragraph, we still don't understand what the issue actually is.
    - necovek 2y ago
      Logging configuration is done at import time for "utils" module. Imagine code like this: main.py: import logging logging.basicConfig(...) logging.info("foo") # uses above config if __name__ == "__main__": import utils # your config is overridden with the one in utils logging.info("bar") # uses utils configuration ... Or two "commands", one importing utils and another not: they would non-obviously use different logging configuration. It gets even crazier: you could import utils to set the configuration, override it, but a second import would not re-set it, as module imports are cached. Basically, don't do it and no unexpected, confusing behaviour anywhere.
      - bumblehean 2y ago
        As a non Python developer, what would be the use-case(s) for importing a module inside of the main function instead of importing it at the top of main.py with the others?
Perizors 2y ago
How do you properly configure a logger in application like that?
- rcfox 2y ago
  Usually you would do it in your main function, or a code path starting from there. Executing code with non-local side effects during import is generally frowned upon. Maybe it's fine for a project-local module that won't be shared, but it's a bad habit and can make it had to track down.
- necovek 2y ago
  Just imagine a callsite that configured a logger in another way, and then imports the utils module for a single function: its configuration getting overridden by the one in utils. There are plenty of ways to structure code so this does not happen, but simply "do not do anything at the top module level" will ensure you don't hit these issues.
NewsaHackO 2y ago
>to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there... Explain the issue with load_json to me more. From my reading it checks if the file exists, then raises an error if it does not. How is that carrying on as if the file is certainly there?
- selcuka 2y ago
  There is a small amount of time between the `if` and the `with` where another process can delete the file, hence causing a race condition. Attempting to open the file and catching any exceptions raised is generally safer.
  - NewsaHackO 2y ago
    OK, that does make sense. Thanks!
  - taberiand 2y ago
    Won't it throw the same "FileNotFound" exception in that case? The issue being bothering to check if it exists in the first place I suppose.
    - selcuka 2y ago
      Yes, but it won't log the error as it is clearly the intention of the first check.
nunez 2y ago
Makes sense given that so much of the training data for so many of these tools are trained on hello world examples where this kind of configuration is okay. Not like this will matter in a world where there are no juniors to replace aged-out seniors because AI was "good enough"...
johnfn 2y ago
Not all code needs to be written at a high level of quality. A good deal of code just needs to work. Shell scripts, one-offs, linter rules, etc.
- jayd16 2y ago
  It'll be really interesting to see if the tech advances fast enough that future AI can deal with the tech debt of present day AI or if we'll see a generational die off of apps/companies.
  - bdhcuidbebe 2y ago
    I expect some of the big companies that went all in on relying on AI to fall in the coming years. It will take some time tho, as decision makers will struggle to make up reasons why why noone on the payroll is able to fix production.
- Aperocky 2y ago
  Having seen my fair share of those, they tend to work either until they don't, or you need to somehow change it.
- jjice 2y ago
  You’re objectively correct in a business context, which is what most software is for. For me, seeing AI slop code more and more is just sad from a craft perspective. Software that’s well designed and architected is a pleasure to read and write, even if a lower quality version would get the job done. I’m watching one of the things I love most in the world become more automated and having the craftsmanship stripped out of it. That’s a bit over dramatic from me, but it’s been sad to watch.
  - deergomoo 2y ago
    I feel exactly the same way, it’s profoundly depressing.
  - hjnilsson 2y ago
cess11 2y ago
wrap_long_lines shares those characteristics: https://github.com/dx-tooling/platform-problem-monitoring-core/blob/main/src/platform_problem_monitoring_core/step10_send_email_report.py https://github.com/dx-tooling/platform-problem-monitoring-co... Where things are placed in the project seems rather ad hoc too. Put everything in the same place kind of architecture. A better strategy might be to separate out the I and the O of IO. Maybe someone wants SMS or group chat notifications later on, instead of shifting the numbers in filenames step11_ onwards one could then add a directory in the O part and hook it into an actual application core.
- thwarted 2y ago
  > instead of shifting the numbers in filenames step11_ onwards There are idioms used when programming in BASIC on how to number the lines so you don't end up renumbering them all the time to make an internal change. It's interesting that such idioms are potentially applicable here also.
dheera 2y ago
I disagree, I think it's absolutely astounding that they've gotten this good in such a short time, and I think we'll get better models in the near future. By the way, prompting models properly helps a lot for generating good code. They get lazy if you don't explicitly ask for well-written code (or put that in the system prompt). It also helps immensely to have two contexts, one that generates the code and one that reviews it (and has a different system prompt).
- henrikschroder 2y ago
  > They get lazy if you don't explicitly ask for well-written code (or put that in the system prompt). This is insane on so many levels.
  - globnomulous 2y ago
    Computer, enhance 15 to 23.
ManuelKiessling 2y ago
Thanks for looking into it. While I would have hoped for a better result, I'm not surprised. In this particular case, I really didn't care about the code at all; I cared about the end result at runtime, that is, can I create a working, stable solution that solves my problem, in a tech stack I'm not familiar with? (While still taking care of well-structured requirements and guard rails — not to guarantee a specific level of code quality per se, but to ensure that the AI works towards my goals without the need to intervene as much as possible). I will spin up another session where I ask it to improve the implementation, and report back.
- necovek 2y ago
  I'd definitely be curious to see if another session provides higher quality code — good luck, and thanks for taking this amicably!
  - ManuelKiessling 2y ago
    I did another session with the sole focus being on code quality improvements. The commit with all changes that Cursor/claude-3.7-sonnet(thinking) has done is at https://github.com/dx-tooling/platform-problem-monitoring-core/commit/33eca1b9c893983c1fa17e4f116095fd10deb123 https://github.com/dx-tooling/platform-problem-monitoring-co.... As you can see, I've fed your feeback verbatim: I have received the following feedback regarding this codebase: "The premise might possibly be true, but as an actually seasoned Python developer, I've taken a look at one file: @utils.py. All of it smells of a (lousy) junior software engineer: from configuring root logger at the top, module level (which relies on module import caching not to be reapplied), over not using a stdlib config file parser and building one themselves, to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there..." I therefore ask you to thoroughly improve the code quality of the implementation in @src while staying in line with the requirements from @REQUIREMENTS.md, and while ensuring that the Quality Tools (see @makefile) won't fail. Also, make sure that the tests in folder @tests don't break. See file @pyproject.toml for the general project setup. There is already a virtualenv at @venv. You can watch a screen recording of the resulting Agent session at https://www.youtube.com/watch?v=zUSm1_NFKpA https://www.youtube.com/watch?v=zUSm1_NFKpA — I think it's an interesting watch because it nicely shows how the tool-based guard rails help the AI to keep on track and reach a "green" state eventually.
theteapot 2y ago
> to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there... It's not a race. It's just redundant. If the file does not exist at the time you actually try to access it you get the same error with slightly better error message.
- necovek 2y ago
  There is a log message that won't be output in that case: whether getting a full, "native" FileNotFound exception is better is beside the point, since the goal of the code was obviously to print a custom error message. And it's trivial to achieve the desired effect sanely: try: with open(...) ... except FileNotFound: logger.error(...) raise It'd even be fewer lines of code.
  - theteapot 2y ago
    Or even fewer by doing it in a global exception handler instead of every time you try to open a file, since all your doing is piping the error though logger.
gessha 2y ago
> This is especially noteworthy because I don’t actually know Python. > However, my broad understanding of software architecture, engineering best practices, system operations, and what makes for excellent software projects made this development process remarkably smooth. If the seniors are going to write this sort of Python code and then talk about how knowledge and experience made it smooth or whatever, might as well hire a junior and let them learn through trials and tribulations.
raxxorraxor 2y ago
In my opinion this isn't even too relevant. I am no python expert but I believe defining a logger at the top for the average one file python script is perfectly adequate or even very sensible in many scenarios. Depends on what you expect the code to do. Ok, the file is named utils.py... Worse by far is still the ability of AI to really integrate different problems and combine them into a solution. And it also seems to depend on the language. In my opinion especially Python and JS results are often very mixxed while other languages with presumably a smaller training set might even fare better. JS seems often fine with asynchronous operation like that file check however. Perhaps really vetting a training set would improve AIs, but it would be quite work intensive to build something like that. That would require a lot of senior devs, which is hard to come by. And then they need to agree on code quality, which might be impossible.
- necovek 2y ago
  This is a logging setup being done top-level in an auxiliary module "utils": you might import it into one command and not another, and end up surprised why is one getting the logging setup and the other isn't. Or you might attempt to configure it and the import would override it. As for getting a lot of code that was vetted by senior engineers, that's not so hard: you just have to pay for it. Basically, any company could — for a price — consider sharing their codebase for training.
globnomulous 2y ago
Thanks for doing the footwork. These TED talk blog posts always stink of phony-baloney nonsense.
inerte 2y ago
100%! But the alternative would be the tool doesn't get built because the author doesn't know enough Python to even produce crappy code, or doesn't have the money to hire an awesome Python coder to do that for them.
- necovek 2y ago
  If you check elsewhere in this thread, the author decided on Python to test out AI capabilities — they could have built it quickly in a language of their choice. I am sure I could have built it quickly in Python to a higher standard of quality. Perhaps they wouldn't have built it because they did not set the time aside for it, like they did for this experiment (+ the blog post).
spoonfeeder006 2y ago
Perhaps that partly because 90% of the training data used to teach LLMs to code is made by junior engineers?
tracker1 2y ago
I can say it isn't any better for JS/Node/Deno/Bun projects that I've seen or tried. About the only case it's been helpful (GitHub CoPilot) is in creating boilerplate .sql files for schema creation, and in that it became kind of auto-complete on overdrive. It still made basic missteps though.

abid786 2y ago

Doesn’t load_json throw if the file doesn’t exist?

isoprophlex 2y ago
Yes but then why do the check in the first place

ilrwbwrkhv 2y ago

Yup this tracks with what I have seen as well. Most devs who use this daily are usually junior devs or javascript devs who both write sloppy questionable code.

rybosome 2y ago

Ok - not wrong at all. Now take that feedback and put it in a prompt back to the LLM. They’re very good at honing bad code into good code with good feedback. And when you can describe good code faster than you can write it - for instance it uses a library you’re not intimately familiar with - this kind of coding can be enormously productive.

necovek 2y ago
I do plan on experimenting with the latest versions of coding assistants, but last I tried them (6 months ago), none could satisfy all of the requirements at the same time. Perhaps there is simply too much crappy Python code around that they were trained on as Python is frequently used for "scripting". Perhaps the field has moved on and I need to try again. But looking at this, it would still be faster for me to type this out myself than go through multiple rounds of reviews and prompts. Really, a senior has not reviewed this, no matter their language (raciness throughout, not just this file).
- deleted 2y ago
  [deleted]
imiric 2y ago
> They’re very good at honing bad code into good code with good feedback. And they're very bad at keeping other code good across iterations. So you might find that while they might've fixed the specific thing you asked for—in the best case scenario, assuming no hallucinations and such—they inadvertently broke something else. So this quickly becomes a game of whack-a-mole, at which point it's safer, quicker, and easier to fix it yourself. IME the chance of this happening is directly proportional to the length of the context.
- bongodongobob 2y ago
  This typically happens when you run the chat too long. When it gives you a new codebase, fire up a new chat so the old stuff doesn't poison the context window.
  - no_wizard 2y ago
    Why isn’t it smart enough to recognize new contexts that aren’t related to old ones?
    - bongodongobob 2y ago
      I don't know, I didn't invent transformers. I do however know how to work with them.
  - achierius 2y ago
    But it rarely gives me a totally-new codebase unless I'm working on a very small project -- so I have to choose between ditching its understanding of some parts (e.g. "don't introduce this bug here, please") and avoiding confusion with others.
- deleted 2y ago
  [deleted]
aunty_helen 2y ago
Nah. This isn’t true. Every time you hit enter you’re not just getting a jr dev, you’re getting a randomly selected jr dev. So, how did I end up with a logging.py, config.py, config in __init__.py and main.py? Well I prompted for it to fix the logging setup to use a specific format. I use cursor, it can spit out code at an amazing rate and reduced the amount of docs I need to read to get something done. But after its second attempt at something you need to jump in and do it yourself and most likely debug what was written.
- skydhash 2y ago
  Are you reading a whole encyclopedia each time you assigned to a task? The one thing about learning is that it compounds. You get faster the longer you use a specific technology. So unless you use a different platform for each task, I don't think you have to read that much documentation (understanding them is another matter).
  - achierius 2y ago
    This is an important distinction though. LLMs don't have any persistent 'state': they have their activations, their context, and that's it. They only know what's pre-trained, and what's in their context. Now, their ability to do in-context learning is impressive, but you're fundamentally still stuck with the deviations and, eventually, forgetting that characterizes these guys -- while a human, while less quick on the uptake, will nevertheless 'bake in' the lessons in a way that LLMs currently cannot. In some ways this is even more impressive -- every prompt you make, your LLM is in effect re-reading (and re-comprehending) your whole codebase, from scratch!
BikiniPrince 2y ago
I’ve found AI tools extremely helpful in getting me up to speed with a library or defining an internal override not exposed by the help. However, if I’m not explicit in how to solve a problem the result looks like the bad code it’s been ingesting.
barrell 2y ago
I would not say it is “very good” at that. Maybe it’s “capable,” but my (ample) experience has been the opposite. I have found the more exact I describe a solution, the less likely it is to succeed. And the more of a solution it has come up with, the less likely it is to change its mind about things. Every since ~4o models, there seems to be a pretty decent chance that you ask it to change something specific and it says it will and it spits out line for line identical code to what you just asked it to change. I have had some really cool success with AI finding optimizations in my code, but only when specifically asked, and even then I just read the response as theory and go write it myself, often in 1-15% the LoC as the LLM

mjr00 2y ago

I "love" this part: def ensure_dir_exists(path: str) -> None: """ Ensure a directory exists. Args: path: Directory path """ An extremely useful and insightful comment. Then you look where it's actually used, # Ensure the directory exists and is writable ensure_dir_exists(work_dir) work_path = Path(work_dir) if not work_path.exists() or not os.access(work_dir, os.W_OK): ... so like, the entire function and its call (and its needlessly verbose comment) could be removed because the existence of the directory is being checked anyway by pathlib. This might not matter here because it's a small, trivial example, but if you have 10, 50, 100, 500 developers working on a codebase, and they're all thoughtlessly slinging code like this in, you're going to have a dumpster fire soon enough. I honestly think "vibe coding" is the best use case for AI coding, because at least then you're fully aware the code is throwaway shit and don't pretend otherwise. edit: and actually looking deeper, `ensure_dir_exists` actually makes the directory, except it's already been made before the function is called so... sigh. Code reviews are going to be pretty tedious in the coming years, aren't they?

milicat 2y ago

The more I browse through this, the more I agree. I feel like one could delete almost all comments from that project without losing any information – which means, at least the variable naming is (probably?) sensible. Then again, I don't know the application domain. Also… def _save_current_date_time(current_date_time_file: str, current_date_time: str) -> None: with Path(current_date_time_file).open("w") as f: f.write(current_date_time) there is a lot of obviously useful abstraction being missed, wasting lines of code that will all need to be maintained. The scary thing is: I have seen professional human developers write worse code.

ramesh31 2y ago
>The scary thing is: I have seen professional human developers write worse code. This is kind of the rub of it all. If the code works, passes all relevant tests, is reasonably maintainable, and can be fitted into the system correctly with a well defined interface, does it really matter? I mean at that point its kind of like looking at the output of a bytecode compiler and being like "wow what a mess". And it's not like they can't write code up to your stylistic standards, it's just literally a matter of prompting for that.
- mjr00 2y ago
  > If the code works, passes all relevant tests, is reasonably maintainable, and can be fitted into the system correctly with a well defined interface, does it really matter? You're not wrong here, but there's a big difference in programming one-off tooling or prototype MVPs and programming things that need to be maintained for years and years. We did this song and dance pretty recently with dynamic typing. Developers thought it was so much more productive to use dynamically typed languages, because it is in the initial phases. Then years went by, those small, quick-to-make dynamic codebases ended up becoming unmaintainable monstrosities, and those developers who hyped up dynamic typing invented Python/PHP type hinting and Flow for JavaScript, later moving to TypeScript entirely. Nowadays nobody seriously recommends building long-lived systems in untyped languages, but they are still very useful for one-off scripting and more interactive/exploratory work where correctness is less important, i.e. Jupyter notebooks. I wouldn't be surprised to see the same pattern happen with low-supervision AI code; it's great for popping out the first MVP, but because it generates poor code, the gung-ho junior devs who think they're getting 10x productivity gains will wisen up and realize the value of spending an hour thinking about proper levels of abstraction instead of YOLO'ing the first thing the AI spits out when they want to build a system that's going to be worked on by multiple developers for multiple years.

nottorp 2y ago

Here's a rl example from today: I asked $random_llm to give me code to recursively scan a directory and give me a list of file names relative to the top directory scanned and their sizes. It gave me working code. On my test data directory it needed ... 6.8 seconds. After 5 min of eliminating obvious inefficiencies the new code needed ... 1.4 seconds. And i didn't even read the docs for the used functions yet, just changed what seemed to generate too many filesystem calls for each file.

bongodongobob 2y ago
Nice, sounds like it saved you some time.
- nottorp 2y ago
  You "AI" enthusiasts always try to find a positive spin :) What if I had trusted the code? It was working after all. I'm guessing that if i asked for string manipulation code it would have done something worth posting on accidentally quadratic.
  - noisy_boy 2y ago
    Depends on how toxic the culture is in your workplace. This could have been an opportunity to "work" on another JIRA task showing 600% improvement over AI generated code.
    - nottorp 2y ago
      I'll write that down for reference in case I do ever join an organization like that in the future, thanks. 600% improvement is worth what, 3 days of billable work if it lasts 5 minutes?
      - noisy_boy 2y ago
        Series of such "improvements" could be fame and fortune in your team/group/vertical. In such places, the guy who toots the loudest wins the most.

layoric 2y ago

Also somewhat strangely, I've found Python output has remained bad, especially for me with dataframe tasks/data analysis. For remembering matplotlib syntax I still find most of them pretty good, but for handling datagframes, very bad and extremely counter productive. Saying that, for typed languages like TypeScript and C#, they have gotten very good. I suspect this might be related to the semantic information can be found in typed languages, and hard to follow unstructured blobs like dataframes, and there for, not well repeated by LLMs.

datadrivenangel 2y ago
Spark especially is brutal for some reason. Even databrick's AI is bad at spark, which is very funny. It's probably because spark is so backwards compatible with pandas, but not fully.

gerdesj 2y ago

My current favourite LLM wankery example is this beauty: https://blog.fahadusman.com/proxmox-replacing-failed-drive-in-zfs-pool/ https://blog.fahadusman.com/proxmox-replacing-failed-drive-i... Note how it has invented the faster parameter for the zpool command. It is possible that the blog writer hallucinated a faster parameter themselves without needing a LLM - who knows. I think all developers should add a faster parameter to all commands to make them run faster. Perhaps a LLM could create the faster code. I predict an increase of man page reading, and better quality documentation at authoritative sources. We will also improve our skills at finding auth sources of docs. My uBlacklist is getting quite long.

Henchman21 2y ago
What makes you think this was created by an LLM? I suspect they might actually have a pool named faster -- I know I've named pools similarly in the past. This is why I now name my pools after characters from the Matrix, as is tradition.
- taurath 2y ago
  This really gets to an acceleration of enshittification. If you can't tell its an LLM, and there's nobody to verify the information, humanity is architecting errors and mindfucks into everything. All of the markers of what is trustworthy has been coopted by untrustworthy machines, so all of the way's we'd previously differentiated actors have stopped working. It feels like we're just losing truth as rapidly as LLMs can generate mistakes. We've built a scoundrels paradise. How useful is a library of knowledge when n% of the information is suspect? We're all about to find out.
  - Henchman21 2y ago
    You know, things looked off to me, but thinking it was the output of an LLM just didn't seem obvious -- even though that was the claim! I feel ill-equipped to deal with this, and as the enshittification has progressed I've found myself using "the web" less and less. At this point, I'm not sure there's much left I value on the web. I wish the enshittification wasn't seemingly pervasive in life.

byproxy 2y ago

As an actually unseasoned Python developer, would you be so kind as to explain why the problems you see are problems and their alternatives? Particularly the first two you note.

saila 2y ago
The call to logging.basicConfig happens at import time, which could cause issues in certain scenarios. For a one-off script, it's probably fine, but for a production app, you'd probably want to set up logging during app startup from whatever your main entry point is. The Python standard library has a configparser module, which should be used instead of custom code. It's safer and easier than manual parsing. The standard library also has a tomllib module, which would be an even better option IMO.
- cinntaile 2y ago
  Regarding your first paragraph, we still don't understand what the issue actually is.
  - necovek 2y ago
    Logging configuration is done at import time for "utils" module. Imagine code like this: main.py: import logging logging.basicConfig(...) logging.info("foo") # uses above config if __name__ == "__main__": import utils # your config is overridden with the one in utils logging.info("bar") # uses utils configuration ... Or two "commands", one importing utils and another not: they would non-obviously use different logging configuration. It gets even crazier: you could import utils to set the configuration, override it, but a second import would not re-set it, as module imports are cached. Basically, don't do it and no unexpected, confusing behaviour anywhere.
    - bumblehean 2y ago
      As a non Python developer, what would be the use-case(s) for importing a module inside of the main function instead of importing it at the top of main.py with the others?

Perizors 2y ago

How do you properly configure a logger in application like that?

rcfox 2y ago
Usually you would do it in your main function, or a code path starting from there. Executing code with non-local side effects during import is generally frowned upon. Maybe it's fine for a project-local module that won't be shared, but it's a bad habit and can make it had to track down.
necovek 2y ago
Just imagine a callsite that configured a logger in another way, and then imports the utils module for a single function: its configuration getting overridden by the one in utils. There are plenty of ways to structure code so this does not happen, but simply "do not do anything at the top module level" will ensure you don't hit these issues.

NewsaHackO 2y ago

>to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there... Explain the issue with load_json to me more. From my reading it checks if the file exists, then raises an error if it does not. How is that carrying on as if the file is certainly there?

selcuka 2y ago
There is a small amount of time between the `if` and the `with` where another process can delete the file, hence causing a race condition. Attempting to open the file and catching any exceptions raised is generally safer.
- NewsaHackO 2y ago
  OK, that does make sense. Thanks!
- taberiand 2y ago
  Won't it throw the same "FileNotFound" exception in that case? The issue being bothering to check if it exists in the first place I suppose.
  - selcuka 2y ago
    Yes, but it won't log the error as it is clearly the intention of the first check.

nunez 2y ago

Makes sense given that so much of the training data for so many of these tools are trained on hello world examples where this kind of configuration is okay. Not like this will matter in a world where there are no juniors to replace aged-out seniors because AI was "good enough"...

johnfn 2y ago

Not all code needs to be written at a high level of quality. A good deal of code just needs to work. Shell scripts, one-offs, linter rules, etc.

jayd16 2y ago
It'll be really interesting to see if the tech advances fast enough that future AI can deal with the tech debt of present day AI or if we'll see a generational die off of apps/companies.
- bdhcuidbebe 2y ago
  I expect some of the big companies that went all in on relying on AI to fall in the coming years. It will take some time tho, as decision makers will struggle to make up reasons why why noone on the payroll is able to fix production.
Aperocky 2y ago
Having seen my fair share of those, they tend to work either until they don't, or you need to somehow change it.
jjice 2y ago
You’re objectively correct in a business context, which is what most software is for. For me, seeing AI slop code more and more is just sad from a craft perspective. Software that’s well designed and architected is a pleasure to read and write, even if a lower quality version would get the job done. I’m watching one of the things I love most in the world become more automated and having the craftsmanship stripped out of it. That’s a bit over dramatic from me, but it’s been sad to watch.
- deergomoo 2y ago
  I feel exactly the same way, it’s profoundly depressing.
- hjnilsson 2y ago

cess11 2y ago

wrap_long_lines shares those characteristics: https://github.com/dx-tooling/platform-problem-monitoring-core/blob/main/src/platform_problem_monitoring_core/step10_send_email_report.py https://github.com/dx-tooling/platform-problem-monitoring-co... Where things are placed in the project seems rather ad hoc too. Put everything in the same place kind of architecture. A better strategy might be to separate out the I and the O of IO. Maybe someone wants SMS or group chat notifications later on, instead of shifting the numbers in filenames step11_ onwards one could then add a directory in the O part and hook it into an actual application core.

thwarted 2y ago
> instead of shifting the numbers in filenames step11_ onwards There are idioms used when programming in BASIC on how to number the lines so you don't end up renumbering them all the time to make an internal change. It's interesting that such idioms are potentially applicable here also.

dheera 2y ago

I disagree, I think it's absolutely astounding that they've gotten this good in such a short time, and I think we'll get better models in the near future. By the way, prompting models properly helps a lot for generating good code. They get lazy if you don't explicitly ask for well-written code (or put that in the system prompt). It also helps immensely to have two contexts, one that generates the code and one that reviews it (and has a different system prompt).

henrikschroder 2y ago
> They get lazy if you don't explicitly ask for well-written code (or put that in the system prompt). This is insane on so many levels.
- globnomulous 2y ago
  Computer, enhance 15 to 23.

ManuelKiessling 2y ago

Thanks for looking into it. While I would have hoped for a better result, I'm not surprised. In this particular case, I really didn't care about the code at all; I cared about the end result at runtime, that is, can I create a working, stable solution that solves my problem, in a tech stack I'm not familiar with? (While still taking care of well-structured requirements and guard rails — not to guarantee a specific level of code quality per se, but to ensure that the AI works towards my goals without the need to intervene as much as possible). I will spin up another session where I ask it to improve the implementation, and report back.

necovek 2y ago
I'd definitely be curious to see if another session provides higher quality code — good luck, and thanks for taking this amicably!
- ManuelKiessling 2y ago
  I did another session with the sole focus being on code quality improvements. The commit with all changes that Cursor/claude-3.7-sonnet(thinking) has done is at https://github.com/dx-tooling/platform-problem-monitoring-core/commit/33eca1b9c893983c1fa17e4f116095fd10deb123 https://github.com/dx-tooling/platform-problem-monitoring-co.... As you can see, I've fed your feeback verbatim: I have received the following feedback regarding this codebase: "The premise might possibly be true, but as an actually seasoned Python developer, I've taken a look at one file: @utils.py. All of it smells of a (lousy) junior software engineer: from configuring root logger at the top, module level (which relies on module import caching not to be reapplied), over not using a stdlib config file parser and building one themselves, to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there..." I therefore ask you to thoroughly improve the code quality of the implementation in @src while staying in line with the requirements from @REQUIREMENTS.md, and while ensuring that the Quality Tools (see @makefile) won't fail. Also, make sure that the tests in folder @tests don't break. See file @pyproject.toml for the general project setup. There is already a virtualenv at @venv. You can watch a screen recording of the resulting Agent session at https://www.youtube.com/watch?v=zUSm1_NFKpA https://www.youtube.com/watch?v=zUSm1_NFKpA — I think it's an interesting watch because it nicely shows how the tool-based guard rails help the AI to keep on track and reach a "green" state eventually.

theteapot 2y ago

> to a raciness in load_json where it's checked for file existence with an if and then carrying on as if the file is certainly there... It's not a race. It's just redundant. If the file does not exist at the time you actually try to access it you get the same error with slightly better error message.

necovek 2y ago
There is a log message that won't be output in that case: whether getting a full, "native" FileNotFound exception is better is beside the point, since the goal of the code was obviously to print a custom error message. And it's trivial to achieve the desired effect sanely: try: with open(...) ... except FileNotFound: logger.error(...) raise It'd even be fewer lines of code.
- theteapot 2y ago
  Or even fewer by doing it in a global exception handler instead of every time you try to open a file, since all your doing is piping the error though logger.

gessha 2y ago

> This is especially noteworthy because I don’t actually know Python. > However, my broad understanding of software architecture, engineering best practices, system operations, and what makes for excellent software projects made this development process remarkably smooth. If the seniors are going to write this sort of Python code and then talk about how knowledge and experience made it smooth or whatever, might as well hire a junior and let them learn through trials and tribulations.

raxxorraxor 2y ago

In my opinion this isn't even too relevant. I am no python expert but I believe defining a logger at the top for the average one file python script is perfectly adequate or even very sensible in many scenarios. Depends on what you expect the code to do. Ok, the file is named utils.py... Worse by far is still the ability of AI to really integrate different problems and combine them into a solution. And it also seems to depend on the language. In my opinion especially Python and JS results are often very mixxed while other languages with presumably a smaller training set might even fare better. JS seems often fine with asynchronous operation like that file check however. Perhaps really vetting a training set would improve AIs, but it would be quite work intensive to build something like that. That would require a lot of senior devs, which is hard to come by. And then they need to agree on code quality, which might be impossible.

necovek 2y ago
This is a logging setup being done top-level in an auxiliary module "utils": you might import it into one command and not another, and end up surprised why is one getting the logging setup and the other isn't. Or you might attempt to configure it and the import would override it. As for getting a lot of code that was vetted by senior engineers, that's not so hard: you just have to pay for it. Basically, any company could — for a price — consider sharing their codebase for training.

globnomulous 2y ago

Thanks for doing the footwork. These TED talk blog posts always stink of phony-baloney nonsense.

inerte 2y ago

100%! But the alternative would be the tool doesn't get built because the author doesn't know enough Python to even produce crappy code, or doesn't have the money to hire an awesome Python coder to do that for them.

necovek 2y ago
If you check elsewhere in this thread, the author decided on Python to test out AI capabilities — they could have built it quickly in a language of their choice. I am sure I could have built it quickly in Python to a higher standard of quality. Perhaps they wouldn't have built it because they did not set the time aside for it, like they did for this experiment (+ the blog post).

spoonfeeder006 2y ago

Perhaps that partly because 90% of the training data used to teach LLMs to code is made by junior engineers?

tracker1 2y ago

I can say it isn't any better for JS/Node/Deno/Bun projects that I've seen or tried. About the only case it's been helpful (GitHub CoPilot) is in creating boilerplate .sql files for schema creation, and in that it became kind of auto-complete on overdrive. It still made basic missteps though.