8 ms·
Funny thing - I've been using ChatGPT as a programming assistant. My experience with it is that the more knowledgeable I am at the domain, the more useful it is
by surgical_fire 4y ago
Funny thing - I've been using ChatGPT as a programming assistant. My experience with it is that the more knowledgeable I am at the domain, the more useful it is.
It often generates code that won't compile or won't run correctly, but that sort of goes in the correct direction, enough that I can spot the blunders and refine my questions. Quite often I even learn a few new things when using it to explore libraries and frameworks I don't know that well but in languages that I am very comfortable with.
Now, for example, I am a lousy front end developer. Not really my area of expertise. I can do minor maintenance on an existing code base, but that's about it. ChatGPT has been more a hindrance than helpful in that particular domain, because I can't easily spot the flaws in its output, and I don't know in which directions to properly refine my questions.
- pixl97 4y agoMy friends in the gaming industry have the complains they can't get new junior and associate members on their team for development. Instead companies want to throw more tooling at them to make the senior developers more productive. With how bad companies are at providing on the job training now, what does the future look like with 100x developers versus newbies that can't talk to AI?
- surgical_fire 4y agoWell, I fully expect juniors in a few years to be somewhat familiar with AI assistants such as ChatGPT, much like they come with some familiarity with a programming language, an IDE, web search, etc. I see LLM models not as the end of society, just as new powerful productivity tools.
- pixl97 4y ago>I see LLM models not as the end of society, This is because you're used to normal technological development scaling. Unfortunately with AI we are unsure at this point if our old paradigms still apply. What happens with society really depends on where we are on the technology growth curve. If LLMs plateau for a while and we don't see significant growth in their abilities, then we'll just see pretty massive technological disruption and reshifting of human priorities, kind of like when the car or plane showed up. On the other hand, if we pop the AGI rabbit out of the top hat in short order it is going to surface deep fundamental problems that humans world wide have been ignoring because of the status quo has not demanded such rapid change.
- surgical_fire 4y agoThat's because you're used to normal technological development hype. Every few years a shiny new toy enters the hype cycle, only to fall out of favor as things stop advancing as people expected. LLMs are impressive at natural language processing (which is a damn hard problem to solve), and can both parse and generate natural language at a satisfactory level. It's an interesting facet of AI, but it also has some glaring limitations for wider use.
- klibertp 4y ago> Every few years a shiny new toy enters the hype cycle I'd agree, but it's not good to forget that every few decades a new shiny toy shines so brightly that it razes two whole cities to the ground in seconds.
- pixl97 4y agoIt's almost as if you didn't parse most of my reply. Again, if we are close to AGI every single one of your past ideas on how technology should behave is outdated and done for. I didn't say we were going to get AGI, and I said it's completely possible we are going plateau out. But if we create general intelligence it will be as if we have created fire, or the wheel, it will present a profound and drastic change in technological capabilities that it will reshape all of humanity.
- 4y ago
- 908B64B197 4y agoThe gaming industry is probably the worse place to look at. It's constantly trying to farm work abroad and looks at every possible way to cut costs. And it's dominated by young, naïve devs and artists who don't understand the market and are "passionate" about the product.
- couchand 4y agoThat all makes it sound like it would be a leading indicator for the industry as a whole.
- Tiktaalik 4y agoIf we can get these sort of tools custom trained on proprietary custom video game engines I feel they could be enormously useful in job training and getting new jr members of the team up and running quickly. Things like "how do I make a debug cheat command to increase player speed" would be a simple question that would send a jr programmer spending time asking around and poking through many areas of code (how do I make a debug command? oh It's imgui? how do I do that? what controls player speed?) but if an AI could provide a working example immediately that would be a huge time saver that would accelerate the new programmers' understanding of the codebase.
- Xorlev 4y agoOr it would stunt their growth. Those sorts of tasks force you to poke around and learn, rather than having the answer handed to you by a LLM or senior engineer.
- Method-X 4y agoThis has been exactly my experience as well.
- nlh 4y agoI've had a similar experience (but slightly different). The biggest innovation that ChatGPT has brought to me in my programming/workflow tasks is the comfort and ease in which it creates shell scripts. That's an area that's always been just past my comfort zone ("Create a shell script to solve this problem? Now I have two problems..."). But all of a sudden, I feel like a master at shell scripting! Things that have always been manual-and-slightly-annoying are now scripted away into automation world, and it feels like I have a newfound superpower :) This past week has been a workflow productivity superweek.
- rrgok 4y agoI can agree to that. I've been writing a lot small cli utilities than before, with argument parsers, help message ecc and in relatively small span of time. It is really useful and very fast.
- nomel 4y agoA large time saving I’ve recently discovered is documenting functions. Paste the function in, tell it to write a docstring/whatever, and it’s 95% there, most of the time.
- throw__away7391 4y agoShell scripting and JavaScript are strengths, any kind of server-side language like Scala or C# it really falls apart quite quickly. I think that integrating into existing APIs is the Achilles' heel, GPT-4 doesn't seem to do any better here either. It makes me wonder if someday we might not get better results completely eschewing libraries altogether and letting the transformers generate literally everything--perhaps recreating similar client libraries over and over and perhaps based on specific existing human-authored libraries, but each one unique to the service, e.g. no external dependencies at all.
- manmal 4y agoI think it will be quite the opposite. Generating a lot of code means generating a lot of bugs. LMMs should get the (battle tested) libs they have to use as part of the prompt, and they should just use those APIs.
- goldemerald 4y agoThis article is not about ChatGPT, but GPT-4. I've found the latter to be significantly better at being a coding assistant in all areas: the code actually compiles and often works out of the box. Even better than chat, I can paste it's generated code and simply respond with any errors given and it will fix it accordingly, something I had lots of trouble using chat to do. GPT-4 is that much better and it makes sense to me that the author is lamenting now and not 4 months ago.
- Gunnerhead 4y agoBut in ChatGPT you can toggle the model between GPT3 and 4 in the premium version.
- surgical_fire 4y agoI'm using ChatGPT-4. For coding, at least for me, it's marginally better than 3.5. In fact, I sometimes prefer reverting back to 3.5 to get speedier output. I am finding ChatGPT-4 much better for other tasks that have less strictness requirements. Such as parsing through a large text and summarizing it for me, and for creative tasks.
- michelledepeil 4y agoI spent some time thinking about this when the GPTs started becoming useful, and I'm glad to see my semi-prediction come true (related toot: https://fosstodon.org/@samuelnorbury/109799880269029379 https://fosstodon.org/@samuelnorbury/109799880269029379). Specifically: LLMs are great when you know what to ask for, and sort of know what the answer should be - call it a boilerplate machine - but bad if you're a novice, in which case meta-analysis of a query is needed to give a useful answer to a question.
- kenjackson 4y agoWhat does "novice" mean in this context? For example, I recently used chatGPT to explore file systems in the Linux kernel. I'm a novice in this field (but know Linux decently well, and the general field of file systems pretty well), but I could pretty easily tell the information it was giving me was correct.
- michelledepeil 4y agoWhen you don't have enough background knowledge to even verify whether the answer is plausible. Or when you don't know what question to ask. It may very well be that newer AIs and models even manage to get that right more often, though. Or that a canny prompter will start by asking which question to subsequently ask. As a side note, you're not a novice if you know the general field of file systems pretty well, IMHO.
- Mezzie 4y agoIt's also really good to use as a way to explore the dusty old corners of your own brain. I'm writing something in vanilla JS for reasons and it's been a while since I've written that, but I can tell pretty easily if what it generates will run/be suitable and where it needs to be tweaked. Basically I use it as a way to induce Deja Vu. Or for things where, to quote Potter Stewart: "I'll know it when I see it."
- phkahler 4y agoSince you are actually using it to write code I have a question. Would it be fair to characterize its programming ability as "writing imperative code" or say creating macros and glue code? Or can it do higher level stuff like choosing data structures and algorithms?
- surgical_fire 4y agoGood question. It's important to understand that it doesn't know anything. It gives approximate answers based on statistical relevance. For example, it has been noticed that ChatGPT can answer leetcode interview questions. That doesn't surprise me in the least, chances are that it contained the algorithm associated to slight variations of the question in its training data. To generate boilerplate code it has been pretty good. Those are pretty repetitive tasks, so it is also expected. Using it to explore the capabilities of frameworks, libraries and services has been great for the most part, but it sometimes tells me things that don't exist or just plain doesn't work. The funniest one was when it told me I could use an annotation on a method to do something, but the annotation didn't exist. When I highlighted it to ChatGPT, it started to get in a loop with variations of code that didn't work, claiming the previous answer was valid for version 3.0. The library doesn't have a version 3.0. it doesn't even have a 2.0. Getting it to code is a wild ride. Sometimes it gets real close to give me something workable, sometimes it gives me stuff that will never work. However, it is awesome at telling me what code is doing when I paste a snippet.
- pharmakom 4y agoYeah... it's closer to text autocomplete than a human SWE
- surgical_fire 4y agoThis is an oversimplification that understate its abilities. It can be a powerful productivity tool. My best analogy is "search engine on steroids"
- wouldbecouldbe 4y agoIf you use 4 it almost always compiles, if you feed it the error will often explain why and fix it for you.
- robomartin 4y ago> My experience with it is that the more knowledgeable I am at the domain, the more useful it is. Exactly right. I made this argument in an unrelated thread: I gave an example of testing ChatGPT by asking it to write CRC calculation code in ARM assembler. The code worked. It wasn’t as good as mine (slower, inefficient), yet, as you said, as a starting point, it would have been fine. “This is today, mid 2023. Give it a year or two (maybe less?) and it will be a tool to contend with. People who like to blame everything else rather than their lack of knowledge and experience will not do very well in that world. Why would I pay someone to do <X> when they bring nothing special to the table? Here's the huge paradigm shift (at least for me): I could not care less what someone knows or does not know. I care about the range and breath of their experience and how they approach learning that which you do not know. Someone like that can use any available tool, including AI tools, to deliver value in almost any domain. Someone who blames others (tools, people, the system, whatever), cannot. We might just be entering an era in which experience will be shown to have serious value.”
- ghaff 4y agoThat's my experience with writing text as well. If I know the area well enough that I don't really need to do a lot of fact checking that isn't off the top of my head--and I can correct any areas of nuance it doesn't get quite write--it's easy enough to get a bunch of words down on paper and rework them as needed. I haven't found it to be a big help but it can save me time.
- passion__desire 4y agoProgramming languages, being precise in description and semantics, means it should be possible, for an automated system, to create new programs by unfolding and expanding already created programs to achieve new solutions. These solutions will be then be memoized and become building blocks for complex problems. A system like GPT-4 should be able to do it, sort of like generating your own training data through self-play like AlphaZero does.
- ilaksh 4y agoBut if we find any web page or document that has your knowledge of what makes ARM code fast and feed that into GPT4 (up to say 80kb of assembly with short comments) along with the task then we don't need you to do that anymore. AI in fact can use any tool, including AI tools, to deliver value in almost any domain. So the problem with the comments here in this thread is that the obvious conclusion is that we need some kind of basic income as a floor as well as some fundamental adjustment of the system to handle the new realities of AI. In fact due to the short-sightedness of comments like yours I am tempted to use the AI programming tool I am developing to try to build an ARM assembler expert business and sell it as Robo Martin the Master ARM Assembly Bot. Since GPT4 can write the code for me, it is possible to launch this business within the next two days.
- _fat_santa 4y agoBy this logic, Priya (the subject of the authors post) would not loose her job but rather just become more productive at her job. And I think that's the key to this whole "GPT4 is taking our jobs argument" that people need to understand, GPT is a bit like Autopilot in a Tesla, it can do the work for you but you still need a trained person there to guide it. Folks like Priya will not loose their jobs IMO, the company would still need them to provide that last line of validation on data because you just can't blindly trust GPT (I don't know if we will ever be able to bindly trust it), rather the scenario I see is that their team at work would actually be able to stay on top of their assignments instead of constantly being overworked.
- saulpw 4y agoWith GPT-4, Priya can do the work of 10 data curators in the same amount of time! She won't lose her job, they can afford to pay her more, because they won't need to pay 9 of her colleagues. Unless of course, one of her more experienced, connected, upper caste colleagues gets the 10x job instead.
- deleted 4y ago[deleted]
- surgical_fire 4y agoBulldozers also can do the work of 10 dudes with shovels. Accounting clerks with excel can do the job of 10 accounting clerks with pen and paper.
- nr2x 4y agoHey now, don’t make anybody actually feel responsible for this. We’re just computer nerds /s
- s__s 4y agoToday you need a trained person to guide it. However I’m not convinced that will remain true in the future. The validation of work is interesting. It does seem essential but it also occurs to me that this would be an absolutely horrible and tedious job. I wonder who would actually be interested in, for example, spending 8 hours a day code reviewing a an AI’s output?
- babyshake 4y agoWe're also seeing with the AutoGPT/BabyAGI type apps that while there is an exciting initial demo, GPT-4 is not ready to just autonomously develop any high quality complex app by itself, soup to nuts. This status quo might not last more than a few more years, though.
- criddell 4y agoIt's not bad for something that's pretty new. Now imagine what another decade or two of development will give us. The number of areas in which a typical worker is more knowledgeable is going to get lower and lower.
- tharne 4y agoExactly. Every discussion I see focuses almost exclusively on AI as it exists right now. Everyone is either dismissing the impact due to some minor shortcoming, or wildly overestimating the short-term impact. The one consistent thing among those who dismiss it and those who think it's going to change everything in the next six months is that they're both drastically underestimating the long-term impact. GPT is already behaving in ways that researchers did not anticipate.
- surgical_fire 4y agolol research with LLM is not new. It just became useful for a wider audience recently
- criddell 4y agoI was thinking specifically of all the amazing work that has been done since the Attention is All You Need paper from 2018. There’s been a lot of progress in just the past 5 years.
- ilaksh 4y agoTwo decades? It added 40 IQ points in the last six months. It can already ingest 80kb of information instantly and use it to perform any knowledge task that can be described in that amount of info (with the 32k token model). The number of areas in which the typical worker is better than a GPT3.5 or GPT4 system that has been properly integrated with the correct data and tools is already probably less than 5% for knowledge workers.
- winstonprivacy 4y agoSame here. It's like a well rounded but very junior level coding assistant that produces a probably flawed bit decent first pass in seconds instead of a week. As a very senior programmer, it's the first technology in a long time that has doubled, or maybe even tripled, my productivity. The demand for my skills is only going to accelerate because of how godly this had made me.
- ilaksh 4y agoIn the very short term maybe. But nocode tools like make.com are already very popular for non-programmers and are used by many to create custom software for their own business or department. Existing nocode tools and other new tools such as the one I am building will incorporate LLMs and multimodal models to automatically build software. There is nothing limiting it to only be used by programmers. The systems will also have debugging loops and long term tasks etc. And every 3-6 months the models are getting better or faster. Even if that slows down we should anticipate further dramatic improvement over the course of the next two years.
- pharmakom 4y agoIve been using LLMs like copilot etc and I find them useful, but less useful than: - stack overflow - compile time type checking - functional programming None of those made vast numbers of devs redundant... quite the opposite in fact. interesting times...
- mnky9800n 4y agoThis exactly. I've been using it to write R code because I've never learned R and needed to quickly write a lot of it. This kind of worked because when I get stuck on a bug that I can't figure out that was generated by chatgpt then my colleague helps me. But really this leaves me with questions like, what if it didn't make so many bugs? What is there was an interface that just evaluated the code I asked it to write? What if most code writing jobs become prompt engineering jobs? I agree the more expert I am the better expert chatgpt seems to be. It will never know what question I am thinking. But it does seem to accelerate or replace a lot of human effort.
- ericbarrett 4y agoAI is a ratchet. It doesn't die, or retire, or change careers. Once it's good at something, it stays good. Once it gets better at something, it stays better. (If not, the model is disposed of without pity or remorse.) Right now, it may make you better at your job. But your very use of it is training it to do your job. It seems to me that if AI is currently helpful to you, it is likely to eventually replace you. And me, and perhaps most skilled jobs as they exist today. It won't be instant, but if you are early- or mid-career, take it very seriously.
- visarga 4y agoEvery 10 years the software paradigms change so much that you practically have to start over. Some skills carry over, but many don't. LLMs will do the same thing - they will force us to change our approaches.
- jacquesm 4y agoThat 10 years is not a constant, it used to be 30 years then it was 20 and now it seems to be 10 or it may already be less than 10.
- HealthNeed 4y agoWhat does "take it seriously" mean? For me, taking it seriously means that I am trying to use it as multiplier to be better at my own job. Is there some better course of action you think we should be taking? I personally do not think it is going anywhere, so I figure I had better be the expert.
- s__s 4y ago“Take it seriously” to me means that I’m using AI to be more productive while simultaneously preparing for a future where I can house and feed my family without relying on being employed as a knowledge worker. What exactly that looks like, I’m still not certain. > so I figure I had better be the expert Be an expert at what? Prompting AI to do things? There will be little to no expertise needed in the future as these tools evolve. That’s precisely what makes AI so disruptive. A basic set of instructions can be used to create something that previously would have took an entire team of skilled specialists to produce. Sure, there will always be some experts, but these jobs will be few and far between, and they’ll be much more academic in nature. Having a job like this will be a luxury only a tiny percentage of the earths population will enjoy. The only real question at this point is how fast will AI move to automate all of these jobs away. It could be 5 years it could be 20. The jobs left in the wake of AI won’t require much education, skill, or talent, meaning they’ll be extremely low paying because almost anyone will be able to do them.
- jenadine 4y agoNote that you are using today's version? How much better will it be in 5 years? What if it is coupled with system who understand the programming languages so you don't get code that doesn't compile anymore?
- dirkt 4y agoSo if you have to babysit it anyway (which is actually a correct way of using it, instead of getting fooled into assuming it's an expert: Use ChatGPT for brainstorming, have a human in the loop to validate): Wouldn't it be faster to write the code yourself? And if you have to come up with the code yourself, wouldn't that allow you to come up with a better architecture for your whole project?
- agrittiwari 4y agoBut on the contrary, It can speed up learning a technology as you go, like a senior engineer packed with knowledge available to you. You could be not knowing the frontend!? You can ask it exactly what you don't understood.And it'll never get bore and try to formulate the answer in a frame that it settles you an understanding, not limited to what your senior has settled with. I would say, it's a good tool available to you when senior engineer/mentor isn't.
- jacquesm 4y agoIn that sense it is not different than any other powertool. A handheld circular saw in the hands of a good craftsman is a serious time saver and will help the make more accurate work. In the hands of an beginner it will lead to bloodshed. In the hands of an amateur it will produce faster work but probably not as accurate.