12 ms·

Society's Technical Debt and Software's Gutenberg Moment

63 points by tim_sw 4y ago

seizethecheese 4y ago
Interesting article, but it all rests on the assumption that AI can produce quality software consistently. It only does a very weak job of proving this claim.
- coldtea 4y ago
  >Interesting article, but it all rests on the assumption that AI can produce quality software consistently Doesn't have to. It just has to make it easier for a developer to do, say, 1/5th the work to get the same results he gets today doing it alone.
yobert 4y ago
I always think of technical debt being a case of too much software, not too little.
- tough 4y ago
  thats right, you can't owe what you haven't bought yet
bottlepalm 4y ago
The amount of tech debt at my last job was soul crushing. Trying to keep your head above water as the bugs keep coming in and features keep getting requested. Trying to balance the needs of 10 different departments all asking for priority. And this is with a team of 50 software engineers. We’re already using AI, but still probably need an order of magnitude improvement to make a real impact. I’m sure it’s coming. I’m hopeful AI can help all of us really tackle legacy software - rebuild it to be simpler, more secure, well tested and easier to maintain and iterate. I kind of view companies and governments as super intelligences unto themselves. Capable of doing things individuals are not. That is what AI will be competing with and/or augmenting.
- jiggawatts 4y ago
  As an external consultant every time I see this scenario it’s always a group that is too busy digging with shovels to learn how to operate the backhoe. Literally just last week I was showing several dev teams how to cut days out of their troubleshooting time by using an APM that can capture debug snapshots from production. Combine with source indexing during builds and code versioned with the Git commit id and you can jump to the line of code with the issue in about a minute. You’ll see the stack variables and everything. This is possible even in a distributed microservices monstrosity. It takes like a day to wire this up. “We’re too busy for that today, there’s an issue in production!”
  - plugin-baby 4y ago
    What’s an APM? Bingle are not helping.
    - adrienthebo 4y ago
      Application performance monitoring (https://en.wikipedia.org/wiki/Application_performance_management?wprov=sfla1 https://en.wikipedia.org/wiki/Application_performance_manage...)
    - JudsonStevens 4y ago
      APM generally stands for Application Performance Monitoring - an example is DataDogs APM module[1]. 1 - https://www.datadoghq.com/product/apm/ https://www.datadoghq.com/product/apm/
  - bottlepalm 4y ago
    The millions of lines of tech debt combined with the pace our company moves and volumes (high tech manufacturing) would blow your mind. It’s thunder dome.
  - brundolf 4y ago
    Yeah. I think many or most large established codebases/software stacks could be an order of magnitude or two simpler (making them an order of magnitude or two easier to iterate on, in terms of both time and number of devs needed) But the reality we've discovered is that there's a constant gravitational force pulling most dev orgs to keep increasing complexity, forever, much faster than the actual product increases in complexity. We can avoid this at the individual level, maybe the team level, but at this point it feels unavoidable at the company or industry level
29athrowaway 4y ago
Spoiler: It is not worth a read. 99% of the code that you can generate via ChatGPT can be already be found in the form of an open source library, already packaged and ready to use. The number of libraries to solve various problems continues to grow each day. Many of the questions you can ask to ChatGPT, if not already present in a project documentation, have been already asked on Stack Overflow, mailing lists, etc. and indexed by search engines. Typing a well formulated question in a search engine may likely take you to a valid answer these days. When taking that into consideration, that leaves us with a very different narrative. LLMs can make a difference, but software engineers have been already reusing solutions to save effort for many decades now. There are already programming languages that are close to natural language such as SQL and BASIC. Did the release of those mean the end of software engineering as an occupation? no. Each time programming gets easier, the result is often more programmers. Finally, there are many expert developers on freelancing sites such as Upwork that you can hire, are "generally intelligent" beyond ChatGPT, and perhaps will cost you less than paying for OpenAI tokens. Still, organizations choose to hire full-time software engineers.
- johnthebaptist 4y ago
  Can’t argue with this genius quadrant though: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F680eccbd-f941-4ebe-b0fc-0de978d8c313_1600x1103.png https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr...
  - deleted 4y ago
    [deleted]
  - 29athrowaway 4y ago
    An expert system can answer many of the questions a doctor can. Yet people choose to go to the doctor. Many people are already using automated tax preparation tools. Yet tax preparers are still needed. There are already customer support chatbots. Yet people often want to talk to a representative. A smarter versions of those will probably be game changers but for high-stakes situations you will still want to deal with a human expert. If you have an infected limb that is at risk of being amputated, will you go see DoctorGPT? Hell no.
fabianhjr 4y ago
I disagree on the effect, ChatGPT / CoPilot is making it cheaper to produce more lines of code as a solution instead of requiring better understood and tested abstractions. I believe this will cause an increase on the trend of technical debt. There are several talks/papers/blogs on lines of code being a cost/liability and not a product/asset. (this blog post seems to assume the inverse)
- ChatGTP 4y ago
  The best code is code they doesn’t exist, the best part of my job is deleting code. Maybe the bots will start to get better at deleting, but generating more code is not necessarily a good thing.
- Bjorkbat 4y ago
  Yeah, I was about to say the same thing. The use of technical debt is a bit “funny” in that they’re using it to refer to a deficit of software, whereas it’s more commonly used to refer to less-than-ideal code that will need to be fixed later. For that reason, I could see AI-generated code leading to more tech debt since its output is, as the authors put it, “cheap enough to waste”. People might get the idea to just have it crank out AI code snippets with no real bearing on what they actually do or no real care to refactor said snippets for reuse / readability until their house of cards collapses.
  - fabianhjr 4y ago
    According to github, 46% of new code of their organization clients is copilot written. https://github.blog/2023-02-14-github-copilot-for-business-is-now-available/ https://github.blog/2023-02-14-github-copilot-for-business-i... So we get about 2-fold increase on code to debug and fix later.
    - candiddevmike 4y ago
      That's a problem for next quarter though If you can't create the code to begin with, what hope could you possibly have of supporting it?
keithwinstein 4y ago
I may be missing something obvious here, and I'm not a Python expert, but isn't this "get rid of all the emojis, except for one I like" code snippet completely wrong (despite the article's assertions that "it works fine," "the LLM used a neat hack," etc.)? It seems to always just erase the file, because it declares `keep_emojis` to be a list of one element (`keep_emojis = ['¯\_(ツ)_/¯']`) and then goes USV-by-USV through the file to check if the encoded/decoded coded char is "in" keep_emojis. And there's basically no way that any single char can be. So the output is always empty. If we wrote it like `keep_emojis = '¯\_(ツ)_/¯'` instead, then it would be a little closer, but in that case the program ends up preserving all non-ASCII characters (including all emojis), because these characters become the empty string after emoji.encode('ascii', 'ignore') and are therefore always "in" the keep_emojis string, so they never get replaced. The explanation also doesn't make any sense, where it writes: "In the above code, the assumption is made that any ASCII character that cannot be encoded using UTF-8 is an emoji." That's silly, because every ASCII character can be encoded in UTF-8. It also doesn't correspond to what the code is doing -- what it's really doing (I think) is testing for USVs that can't be encoded in ASCII, but then throwing the char away no matter what. And, of course, the vast majority of non-ASCII coded characters in Unicode are not emoji, so this is not really a good way to satisfy the user's request. The roundtrip through ASCII seems like a bad idea -- if the user wants you to preserve some emojis while getting rid of most emojis, converting to ASCII (where there are no emojis) doesn't seem like it will be helpful. Of course it's amazing that ChatGPT is already good enough to produce something close enough to fool venture capitalists into trusting it without trying the code, and these techniques will doubtless continue to improve.
- thundergolfer 4y ago
  I don’t think you’re missing something. The code looks busted, wouldn’t work at all.
- posnet 4y ago
  This actually gets at the heart of the problem with the current batch of LLMs. They have no concept of truth, and so will gladly generate tokens that are "likely", but are at best clearly wrong, or worse look correct to a non-expert, but are subtly wrong. With software at least you can run it and inspect the behavior for flaws. Or have the results be validated by an actual programmer for correctness. And hopefully the developer using the LLM understands the limitations, and understands how it can hallucinate incorrect but convincing results. However even software engineers I know, who are aware of all of the above issues, will ask ChatGPT questions about fields they aren't familiar with and view the results as authoritative. It's the same effect as when you read a news article about a topic you're familiar with and can immediately see flaws, shortcuts and exaggerations; but then you turn the page to a topic you aren't an expert it and say to yourself "what an interesting article". The example from the article is even worse, since the author asked for an explanation of the code, and it hallucinated a clear and convincing but completely wrong explanation. Again to a programmer it is obvious, but to a layperson it's actively deceptive. I could easily imagine a scenario where some asks ChatGPT, what is the best combination of chemicals to clean mold from a ceramic surface, and it responding with "Bleach and Vinegar are the perfect way to clean a ceramic surface." and following up with an explanation like "Vinegar de-greases the surface allowing the the disinfectant power of the bleach to penetrate into the mold". Which all sounds reasonable, unless you know beforehand that those chemicals mixed produces toxic chlorine gas. Now that was definitely a contrived example, but unless you know and constantly question the output of an LLM you could easily be misled. Especially if you are asking about topics outside of it's training data or with minimal training examples. Maybe with enough training data or some combination of transformers with reinforcement learning that has a "truth" metric, hallucinating completely incorrect information can be reduced to an acceptable level. But at this point it seems intractable.
neilv 4y ago
I don't understand why those examples of generated Python code and explanation were held up uncritically. I know ChatGPT's statistical plagiarizing can do better than this.
rapiz 4y ago
I stopped reading this after realizing that most of the article depends on an assumption that is unlikely to happens, which is that AI can produce high quality and complex software. One important aspect of software is that it's not much easier to read than rewrite the code. Currently AI generated code needs much reading and checking, and some rounds of chats to iterate. I am even not quite happy to read humans' code, nevertheless AI's. Only when one day AI can code that needs little human intervention will the article mean something to me. But is that ever possible? The crux is that a software bug is not in the code, it's the mismatch between the code and our mental model. Programming language is a precise way for us to present the mental model. Without a program, how do we tell AI precisely of our mental model? By natural language we can never do that. Even we can do that, we're only repeating a program in a extremely unconvinent way. I do agree that AI can help developers spend less time on boilerplate and creeping in documentations. It can greatly improve productivity of the software industry.
Animats 4y ago
AI is definitely coming from some jobs previously considered creative. See [1], on HN today. I expect it to come for many web-related jobs in the near future. Large language models by themselves are basically cut and paste programmers. But that's good enough for some routine work. This is just getting started. Large language models coupled to more structured systems are going to be very powerful agents. [1] https://news.ycombinator.com/item?id=35308498 https://news.ycombinator.com/item?id=35308498
thisisthenewme 4y ago
https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F680eccbd-f941-4ebe-b0fc-0de978d8c313_1600x1103.png https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... That graph places software engineers in the top-right quadrant, suggesting that they are predictable and "grammatical". However, I believe there should be a distinction between writing code and engineering. I would not classify software engineering as entirely predictable. For instance, while you could use a script to handle emojis with ChatGPT's help, it would be challenging to use the same strategy for a solution involving large systems that have multiple failure points and span multiple regions. Moreover, even if such a system were successful, multiple redundancies would need to be in place to prevent large-scale disruptions caused by downstream failures. On the other hand, the chart places executives and investors on the 'safer' side. However, I think that with the real-time information available to AI, investing as a 'job' could be at a higher risk of being replaced. The same could be true for many executive positions (assuming that we are comparing merit-based positions to those based on nepotism).
- brigadier132 4y ago
  Agreed. All these people that think their "soft" jobs are safe. Nope yours will be the first to go. The last to go will be the ones that require actual physical presence.
  - pclmulqdq 4y ago
    Also, it's looking more and more like jobs that involve hard math and logic will be sticking around for a long time in the new AI regime. I have tried to get chatgpt, copilot, and llama to write lock-free stacks and other examples of simple but difficult code, and it has not gone well. "Soft skill" jobs like executives and managers are definitely on the chopping block first.
    - anon7725 4y ago
      > "Soft skill" jobs like executives and managers are definitely on the chopping block first. Wouldn’t that be some delicious irony? Never gonna happen but it would be quite funny if it did.
stefan_ 4y ago
What ASCII emojis? This guy doesn't have the first clue. The generated code is garbage. The explanation is nonsense. "Gutenberg Moment", where you are found out as the gargantuan fraud you are? This is like a tick above the fucking whitepapers that crypto frauds used to have to sell their coin.
blibble 4y ago
that code is a very long winded way of truncating a file
bbor 4y ago
I hope this isn't too informal for HN, but... commenting here to mark this momentous instant, where the software engineers were warned and dismissed it as hype. However, let's take a step back and consider a few basic arguments that this article makes - I suspect most here would only disagree with the very last. Firstly, the demand for quality software is far greater than the current supply. If there were suddenly twice the number of mid-level software engineers, there would still be more than enough work for them. Secondly, software engineering is a process that can be refined and created almost entirely without the interference of non-deterministic real-world systems like roads, weather, or courts. This makes it an ideal field for automation and AI to play a larger role. Thirdly, a sufficiently intelligent computer could exponentially increase its own efficacy in performing purely-digital tasks -- basically an [intelligence explosion](https://www.lesswrong.com/tag/intelligence-explosion https://www.lesswrong.com/tag/intelligence-explosion) but much softer and much much more achievable (how smart is a mid-level SWE, really?). Finally, LLMs that get a ~100 on an IQ test are enough to start that cycle. Perhaps you're super duper convinced that the last point is wrong. Perhaps you have strongly-held convictions about explainability, symbolic reasoning, higher level thinking, etc. But if you really sit down and think, what are the chances you're wrong? What are the chances that we get another leap or two in the next 1-5 years like we just got with DL transformers? If you're feeling excited and anxious about the implications of this, you're not alone. I've found that it's a difficult topic to discuss with those close to me, especially if they're not familiar with the latest developments in AI. If you have thoughts on how to use our SWE experience to navigate this exciting but uncertain landscape while maintaining a sense of self-preservation and helping others as much as possible, I'd be interested in hearing your ideas.
- anon7725 4y ago
  > If you have thoughts on how to use our SWE experience to navigate this exciting but uncertain landscape while maintaining a sense of self-preservation and helping others as much as possible, I'd be interested in hearing your ideas. Sell magazine subscriptions door to door?
- qsort 4y ago
  The last point is neither right nor wrong, it's fanfiction. The second point is wrong. You can automatically synthesize code from a specification (and you could already do so before LLMs, arguably even better with SMT solvers), but who is going to write the specification in the first place, buddy? That's the problem that's never been solved, and it will never be solved because it can't be solved. It is not easier to write a specification in English than it is to write it in C++. It's like saying that math would become easier if we took away the burden of using mathematical notation. Not only it wouldn't get any easier, it would get significantly more difficult! Lies are dangerous, because if you say too many of them, you start believing your own. "AI is like a human you can talk to", they said. Now they act as though that's true, and they talk to the AI, without realizing it's the same faithful mechanical slave it has always been, perfectly capable of carrying out nonsensical instructions to their nonsensical results. And the instructions are necessarily nonsensical, because they insist on using an interface too wide, where saying something precise is all but impossible.
mikewarot 4y ago
Conflating technical debt with financial terms is an interesting tact, but I feel it is a bad tool for analysis. Their own chart shows a declining cost of software. I feel that complexity (not the lack of refactoring away technical debt) is a far better proxy for analysis. The complexity inherent in software development reached a minimum in the era of Visual Basic and Borland's Delphi, and has rapidly increased since then in the abandoning of the Windows Desktop and the Win32 UI as a standard and fairly reliable platform for applications. It was possible to build and deploy applications in a rapid enough fashion that agile just happened without prompting in many one-man development projects. In fact, many domain experts were able to craft their own reliable and usable (to them) applications that remain in use to this day. That world existed, and has largely been abandoned. I suspect what AI tooling will enable in aggregate (if we're lucky) is a return to the levels of productivity we previously had, despite the multiple new layers of interfaces and abstractions inherent in the non-Win32-GUI world we now inhabit. Thus, I feel if you want to see how AI/LLMs are going to effect the economy and our profession, look to the effects of the Win32 desktop on software in terms of size and scale of disruption. History doesn't repeat, but it does rhyme. -- EDIT/APPEND -- in an era of "free" debt, where the interest is effectively zero, management learns to spend money like crazy, and let the debt stack up, because it is the efficient thing to do When that management mindset is mis-applied to understanding when the programmers say "we've got tech debt we've got to pay off".... they incorrectly assume it has no cost in interest, and can be ignored, as with money debt... except it does have interest.... loan shark levels of interest
drewcoo 4y ago
A very tortured, not-quite-explained redefinition of "technical debt." Graphs of complete imagination, not based on any data at all. Comparisons of apples and exaggerated notions of oranges. Ungrounded, magical claims about LLMs. This is unhinged fantasy.
kanyethegreat 4y ago
All these people commenting on the positive relationship between amount of code and technical debt. Generative AI can refactor code too (making it smaller)
LightFog 4y ago
Another piece where a plausible looking but completely nonsensical code generation output is uncritically presented as correct. At least it is a quick way to see either the author has no idea about software development or doesn’t care about facts in their arguments.
anchochilis 4y ago
You know the old saying "familiarity breeds contempt"? I believe the opposite is true as well. People who don't code think coding is typing. People who don't manage think it is as simple as forwarding emails from executives to subordinates. I myself have often wondered if what doctors do couldn't be automated. You have the primary care physicians who seem to run on a loop of "listen to symptoms" - "order tests" - "prescribe medication" - "refer to specialist". And the specialists themselves seem to follow their own loops. Many surgeons for example, perform the exact same procedure day in/day out for decades. Surely well-trained robots, not prone to fatigue or loss of dexterity as they age, could produce superior outcomes. But they haven't been replaced yet, just like I haven't, so I assume there is more going on there than what looks from the outside like repetitive manual labor.
ravagat 4y ago
Good article. Some key points: > Again, programming is a good example of a predictable domain, one created to produce the same outputs given the same inputs. If it doesn’t do that, that’s 99.9999% likely to be on you, not the language. Other domains are much less predictable, like equity investing, or psychiatry, or maybe, meteorology. > Entrepreneur and publisher Tim O’Reilly has a nice phrase that is applicable at this point. He argues investors and entrepreneurs should “create more value than you capture.” The technology industry started out that way, but in recent years it has too often gone for the quick win, usually by running gambits from the financial services playbook.

63 points by tim_sw 4y ago

Interesting article, but it all rests on the assumption that AI can produce quality software consistently. It only does a very weak job of proving this claim.

coldtea 4y ago
>Interesting article, but it all rests on the assumption that AI can produce quality software consistently Doesn't have to. It just has to make it easier for a developer to do, say, 1/5th the work to get the same results he gets today doing it alone.

yobert 4y ago

I always think of technical debt being a case of too much software, not too little.

tough 4y ago
thats right, you can't owe what you haven't bought yet

bottlepalm 4y ago

The amount of tech debt at my last job was soul crushing. Trying to keep your head above water as the bugs keep coming in and features keep getting requested. Trying to balance the needs of 10 different departments all asking for priority. And this is with a team of 50 software engineers. We’re already using AI, but still probably need an order of magnitude improvement to make a real impact. I’m sure it’s coming. I’m hopeful AI can help all of us really tackle legacy software - rebuild it to be simpler, more secure, well tested and easier to maintain and iterate. I kind of view companies and governments as super intelligences unto themselves. Capable of doing things individuals are not. That is what AI will be competing with and/or augmenting.

jiggawatts 4y ago
As an external consultant every time I see this scenario it’s always a group that is too busy digging with shovels to learn how to operate the backhoe. Literally just last week I was showing several dev teams how to cut days out of their troubleshooting time by using an APM that can capture debug snapshots from production. Combine with source indexing during builds and code versioned with the Git commit id and you can jump to the line of code with the issue in about a minute. You’ll see the stack variables and everything. This is possible even in a distributed microservices monstrosity. It takes like a day to wire this up. “We’re too busy for that today, there’s an issue in production!”
- plugin-baby 4y ago
  What’s an APM? Bingle are not helping.
  - adrienthebo 4y ago
    Application performance monitoring (https://en.wikipedia.org/wiki/Application_performance_management?wprov=sfla1 https://en.wikipedia.org/wiki/Application_performance_manage...)
  - JudsonStevens 4y ago
    APM generally stands for Application Performance Monitoring - an example is DataDogs APM module[1]. 1 - https://www.datadoghq.com/product/apm/ https://www.datadoghq.com/product/apm/
- bottlepalm 4y ago
  The millions of lines of tech debt combined with the pace our company moves and volumes (high tech manufacturing) would blow your mind. It’s thunder dome.
- brundolf 4y ago
  Yeah. I think many or most large established codebases/software stacks could be an order of magnitude or two simpler (making them an order of magnitude or two easier to iterate on, in terms of both time and number of devs needed) But the reality we've discovered is that there's a constant gravitational force pulling most dev orgs to keep increasing complexity, forever, much faster than the actual product increases in complexity. We can avoid this at the individual level, maybe the team level, but at this point it feels unavoidable at the company or industry level

29athrowaway 4y ago

Spoiler: It is not worth a read. 99% of the code that you can generate via ChatGPT can be already be found in the form of an open source library, already packaged and ready to use. The number of libraries to solve various problems continues to grow each day. Many of the questions you can ask to ChatGPT, if not already present in a project documentation, have been already asked on Stack Overflow, mailing lists, etc. and indexed by search engines. Typing a well formulated question in a search engine may likely take you to a valid answer these days. When taking that into consideration, that leaves us with a very different narrative. LLMs can make a difference, but software engineers have been already reusing solutions to save effort for many decades now. There are already programming languages that are close to natural language such as SQL and BASIC. Did the release of those mean the end of software engineering as an occupation? no. Each time programming gets easier, the result is often more programmers. Finally, there are many expert developers on freelancing sites such as Upwork that you can hire, are "generally intelligent" beyond ChatGPT, and perhaps will cost you less than paying for OpenAI tokens. Still, organizations choose to hire full-time software engineers.

johnthebaptist 4y ago
Can’t argue with this genius quadrant though: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F680eccbd-f941-4ebe-b0fc-0de978d8c313_1600x1103.png https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr...
- deleted 4y ago
  [deleted]
- 29athrowaway 4y ago
  An expert system can answer many of the questions a doctor can. Yet people choose to go to the doctor. Many people are already using automated tax preparation tools. Yet tax preparers are still needed. There are already customer support chatbots. Yet people often want to talk to a representative. A smarter versions of those will probably be game changers but for high-stakes situations you will still want to deal with a human expert. If you have an infected limb that is at risk of being amputated, will you go see DoctorGPT? Hell no.

fabianhjr 4y ago

I disagree on the effect, ChatGPT / CoPilot is making it cheaper to produce more lines of code as a solution instead of requiring better understood and tested abstractions. I believe this will cause an increase on the trend of technical debt. There are several talks/papers/blogs on lines of code being a cost/liability and not a product/asset. (this blog post seems to assume the inverse)

ChatGTP 4y ago
The best code is code they doesn’t exist, the best part of my job is deleting code. Maybe the bots will start to get better at deleting, but generating more code is not necessarily a good thing.
Bjorkbat 4y ago
Yeah, I was about to say the same thing. The use of technical debt is a bit “funny” in that they’re using it to refer to a deficit of software, whereas it’s more commonly used to refer to less-than-ideal code that will need to be fixed later. For that reason, I could see AI-generated code leading to more tech debt since its output is, as the authors put it, “cheap enough to waste”. People might get the idea to just have it crank out AI code snippets with no real bearing on what they actually do or no real care to refactor said snippets for reuse / readability until their house of cards collapses.
- fabianhjr 4y ago
  According to github, 46% of new code of their organization clients is copilot written. https://github.blog/2023-02-14-github-copilot-for-business-is-now-available/ https://github.blog/2023-02-14-github-copilot-for-business-i... So we get about 2-fold increase on code to debug and fix later.
  - candiddevmike 4y ago
    That's a problem for next quarter though If you can't create the code to begin with, what hope could you possibly have of supporting it?

keithwinstein 4y ago

I may be missing something obvious here, and I'm not a Python expert, but isn't this "get rid of all the emojis, except for one I like" code snippet completely wrong (despite the article's assertions that "it works fine," "the LLM used a neat hack," etc.)? It seems to always just erase the file, because it declares `keep_emojis` to be a list of one element (`keep_emojis = ['¯\_(ツ)_/¯']`) and then goes USV-by-USV through the file to check if the encoded/decoded coded char is "in" keep_emojis. And there's basically no way that any single char can be. So the output is always empty. If we wrote it like `keep_emojis = '¯\_(ツ)_/¯'` instead, then it would be a little closer, but in that case the program ends up preserving all non-ASCII characters (including all emojis), because these characters become the empty string after emoji.encode('ascii', 'ignore') and are therefore always "in" the keep_emojis string, so they never get replaced. The explanation also doesn't make any sense, where it writes: "In the above code, the assumption is made that any ASCII character that cannot be encoded using UTF-8 is an emoji." That's silly, because every ASCII character can be encoded in UTF-8. It also doesn't correspond to what the code is doing -- what it's really doing (I think) is testing for USVs that can't be encoded in ASCII, but then throwing the char away no matter what. And, of course, the vast majority of non-ASCII coded characters in Unicode are not emoji, so this is not really a good way to satisfy the user's request. The roundtrip through ASCII seems like a bad idea -- if the user wants you to preserve some emojis while getting rid of most emojis, converting to ASCII (where there are no emojis) doesn't seem like it will be helpful. Of course it's amazing that ChatGPT is already good enough to produce something close enough to fool venture capitalists into trusting it without trying the code, and these techniques will doubtless continue to improve.

thundergolfer 4y ago
I don’t think you’re missing something. The code looks busted, wouldn’t work at all.
posnet 4y ago
This actually gets at the heart of the problem with the current batch of LLMs. They have no concept of truth, and so will gladly generate tokens that are "likely", but are at best clearly wrong, or worse look correct to a non-expert, but are subtly wrong. With software at least you can run it and inspect the behavior for flaws. Or have the results be validated by an actual programmer for correctness. And hopefully the developer using the LLM understands the limitations, and understands how it can hallucinate incorrect but convincing results. However even software engineers I know, who are aware of all of the above issues, will ask ChatGPT questions about fields they aren't familiar with and view the results as authoritative. It's the same effect as when you read a news article about a topic you're familiar with and can immediately see flaws, shortcuts and exaggerations; but then you turn the page to a topic you aren't an expert it and say to yourself "what an interesting article". The example from the article is even worse, since the author asked for an explanation of the code, and it hallucinated a clear and convincing but completely wrong explanation. Again to a programmer it is obvious, but to a layperson it's actively deceptive. I could easily imagine a scenario where some asks ChatGPT, what is the best combination of chemicals to clean mold from a ceramic surface, and it responding with "Bleach and Vinegar are the perfect way to clean a ceramic surface." and following up with an explanation like "Vinegar de-greases the surface allowing the the disinfectant power of the bleach to penetrate into the mold". Which all sounds reasonable, unless you know beforehand that those chemicals mixed produces toxic chlorine gas. Now that was definitely a contrived example, but unless you know and constantly question the output of an LLM you could easily be misled. Especially if you are asking about topics outside of it's training data or with minimal training examples. Maybe with enough training data or some combination of transformers with reinforcement learning that has a "truth" metric, hallucinating completely incorrect information can be reduced to an acceptable level. But at this point it seems intractable.

neilv 4y ago

I don't understand why those examples of generated Python code and explanation were held up uncritically. I know ChatGPT's statistical plagiarizing can do better than this.

rapiz 4y ago

I stopped reading this after realizing that most of the article depends on an assumption that is unlikely to happens, which is that AI can produce high quality and complex software. One important aspect of software is that it's not much easier to read than rewrite the code. Currently AI generated code needs much reading and checking, and some rounds of chats to iterate. I am even not quite happy to read humans' code, nevertheless AI's. Only when one day AI can code that needs little human intervention will the article mean something to me. But is that ever possible? The crux is that a software bug is not in the code, it's the mismatch between the code and our mental model. Programming language is a precise way for us to present the mental model. Without a program, how do we tell AI precisely of our mental model? By natural language we can never do that. Even we can do that, we're only repeating a program in a extremely unconvinent way. I do agree that AI can help developers spend less time on boilerplate and creeping in documentations. It can greatly improve productivity of the software industry.

Animats 4y ago

AI is definitely coming from some jobs previously considered creative. See [1], on HN today. I expect it to come for many web-related jobs in the near future. Large language models by themselves are basically cut and paste programmers. But that's good enough for some routine work. This is just getting started. Large language models coupled to more structured systems are going to be very powerful agents. [1] https://news.ycombinator.com/item?id=35308498 https://news.ycombinator.com/item?id=35308498

thisisthenewme 4y ago

https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F680eccbd-f941-4ebe-b0fc-0de978d8c313_1600x1103.png https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... That graph places software engineers in the top-right quadrant, suggesting that they are predictable and "grammatical". However, I believe there should be a distinction between writing code and engineering. I would not classify software engineering as entirely predictable. For instance, while you could use a script to handle emojis with ChatGPT's help, it would be challenging to use the same strategy for a solution involving large systems that have multiple failure points and span multiple regions. Moreover, even if such a system were successful, multiple redundancies would need to be in place to prevent large-scale disruptions caused by downstream failures. On the other hand, the chart places executives and investors on the 'safer' side. However, I think that with the real-time information available to AI, investing as a 'job' could be at a higher risk of being replaced. The same could be true for many executive positions (assuming that we are comparing merit-based positions to those based on nepotism).

brigadier132 4y ago
Agreed. All these people that think their "soft" jobs are safe. Nope yours will be the first to go. The last to go will be the ones that require actual physical presence.
- pclmulqdq 4y ago
  Also, it's looking more and more like jobs that involve hard math and logic will be sticking around for a long time in the new AI regime. I have tried to get chatgpt, copilot, and llama to write lock-free stacks and other examples of simple but difficult code, and it has not gone well. "Soft skill" jobs like executives and managers are definitely on the chopping block first.
  - anon7725 4y ago
    > "Soft skill" jobs like executives and managers are definitely on the chopping block first. Wouldn’t that be some delicious irony? Never gonna happen but it would be quite funny if it did.

stefan_ 4y ago

What ASCII emojis? This guy doesn't have the first clue. The generated code is garbage. The explanation is nonsense. "Gutenberg Moment", where you are found out as the gargantuan fraud you are? This is like a tick above the fucking whitepapers that crypto frauds used to have to sell their coin.

blibble 4y ago

that code is a very long winded way of truncating a file

bbor 4y ago

I hope this isn't too informal for HN, but... commenting here to mark this momentous instant, where the software engineers were warned and dismissed it as hype. However, let's take a step back and consider a few basic arguments that this article makes - I suspect most here would only disagree with the very last. Firstly, the demand for quality software is far greater than the current supply. If there were suddenly twice the number of mid-level software engineers, there would still be more than enough work for them. Secondly, software engineering is a process that can be refined and created almost entirely without the interference of non-deterministic real-world systems like roads, weather, or courts. This makes it an ideal field for automation and AI to play a larger role. Thirdly, a sufficiently intelligent computer could exponentially increase its own efficacy in performing purely-digital tasks -- basically an [intelligence explosion](https://www.lesswrong.com/tag/intelligence-explosion https://www.lesswrong.com/tag/intelligence-explosion) but much softer and much much more achievable (how smart is a mid-level SWE, really?). Finally, LLMs that get a ~100 on an IQ test are enough to start that cycle. Perhaps you're super duper convinced that the last point is wrong. Perhaps you have strongly-held convictions about explainability, symbolic reasoning, higher level thinking, etc. But if you really sit down and think, what are the chances you're wrong? What are the chances that we get another leap or two in the next 1-5 years like we just got with DL transformers? If you're feeling excited and anxious about the implications of this, you're not alone. I've found that it's a difficult topic to discuss with those close to me, especially if they're not familiar with the latest developments in AI. If you have thoughts on how to use our SWE experience to navigate this exciting but uncertain landscape while maintaining a sense of self-preservation and helping others as much as possible, I'd be interested in hearing your ideas.

anon7725 4y ago
> If you have thoughts on how to use our SWE experience to navigate this exciting but uncertain landscape while maintaining a sense of self-preservation and helping others as much as possible, I'd be interested in hearing your ideas. Sell magazine subscriptions door to door?
qsort 4y ago
The last point is neither right nor wrong, it's fanfiction. The second point is wrong. You can automatically synthesize code from a specification (and you could already do so before LLMs, arguably even better with SMT solvers), but who is going to write the specification in the first place, buddy? That's the problem that's never been solved, and it will never be solved because it can't be solved. It is not easier to write a specification in English than it is to write it in C++. It's like saying that math would become easier if we took away the burden of using mathematical notation. Not only it wouldn't get any easier, it would get significantly more difficult! Lies are dangerous, because if you say too many of them, you start believing your own. "AI is like a human you can talk to", they said. Now they act as though that's true, and they talk to the AI, without realizing it's the same faithful mechanical slave it has always been, perfectly capable of carrying out nonsensical instructions to their nonsensical results. And the instructions are necessarily nonsensical, because they insist on using an interface too wide, where saying something precise is all but impossible.

mikewarot 4y ago

Conflating technical debt with financial terms is an interesting tact, but I feel it is a bad tool for analysis. Their own chart shows a declining cost of software. I feel that complexity (not the lack of refactoring away technical debt) is a far better proxy for analysis. The complexity inherent in software development reached a minimum in the era of Visual Basic and Borland's Delphi, and has rapidly increased since then in the abandoning of the Windows Desktop and the Win32 UI as a standard and fairly reliable platform for applications. It was possible to build and deploy applications in a rapid enough fashion that agile just happened without prompting in many one-man development projects. In fact, many domain experts were able to craft their own reliable and usable (to them) applications that remain in use to this day. That world existed, and has largely been abandoned. I suspect what AI tooling will enable in aggregate (if we're lucky) is a return to the levels of productivity we previously had, despite the multiple new layers of interfaces and abstractions inherent in the non-Win32-GUI world we now inhabit. Thus, I feel if you want to see how AI/LLMs are going to effect the economy and our profession, look to the effects of the Win32 desktop on software in terms of size and scale of disruption. History doesn't repeat, but it does rhyme. -- EDIT/APPEND -- in an era of "free" debt, where the interest is effectively zero, management learns to spend money like crazy, and let the debt stack up, because it is the efficient thing to do When that management mindset is mis-applied to understanding when the programmers say "we've got tech debt we've got to pay off".... they incorrectly assume it has no cost in interest, and can be ignored, as with money debt... except it does have interest.... loan shark levels of interest

drewcoo 4y ago

A very tortured, not-quite-explained redefinition of "technical debt." Graphs of complete imagination, not based on any data at all. Comparisons of apples and exaggerated notions of oranges. Ungrounded, magical claims about LLMs. This is unhinged fantasy.

kanyethegreat 4y ago

All these people commenting on the positive relationship between amount of code and technical debt. Generative AI can refactor code too (making it smaller)

LightFog 4y ago

Another piece where a plausible looking but completely nonsensical code generation output is uncritically presented as correct. At least it is a quick way to see either the author has no idea about software development or doesn’t care about facts in their arguments.

anchochilis 4y ago

You know the old saying "familiarity breeds contempt"? I believe the opposite is true as well. People who don't code think coding is typing. People who don't manage think it is as simple as forwarding emails from executives to subordinates. I myself have often wondered if what doctors do couldn't be automated. You have the primary care physicians who seem to run on a loop of "listen to symptoms" - "order tests" - "prescribe medication" - "refer to specialist". And the specialists themselves seem to follow their own loops. Many surgeons for example, perform the exact same procedure day in/day out for decades. Surely well-trained robots, not prone to fatigue or loss of dexterity as they age, could produce superior outcomes. But they haven't been replaced yet, just like I haven't, so I assume there is more going on there than what looks from the outside like repetitive manual labor.

ravagat 4y ago

Good article. Some key points: > Again, programming is a good example of a predictable domain, one created to produce the same outputs given the same inputs. If it doesn’t do that, that’s 99.9999% likely to be on you, not the language. Other domains are much less predictable, like equity investing, or psychiatry, or maybe, meteorology. > Entrepreneur and publisher Tim O’Reilly has a nice phrase that is applicable at this point. He argues investors and entrepreneurs should “create more value than you capture.” The technology industry started out that way, but in recent years it has too often gone for the quick win, usually by running gambits from the financial services playbook.