7 ms·
The future of programming: Research at CHI 2023
- Dowwie 4y ago2023 conference program: https://programs.sigchi.org/chi/2023/program https://programs.sigchi.org/chi/2023/program
- Luciatrutth 4y ago[dead]
- dtagames 4y agoThis paper is quite good: Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
- textninja 4y agoI’m not fond of the provocative title because prompting is easy and only getting easier; the advice seems to be predicated on the use of relatively deficient LLMs. I don’t doubt there will still be operator skill involved, but I anticipate the state of the art for LLMs ability to adapt to “bad” prompts will outpace our ability to learn to prompt them effectively. Disclaimer: I watched the video but didn’t read the paper.
- domoritz 4y agoI think there are a lot of instances where writing prompts can be hard just because it’s hard to express your needs in words sometimes. Bad prompts are often ambiguous and there is only so much even a perfect LLM can correct for. That is, until we have direct connections to our brains.
- version_five 4y agoI think you're right about prompts getting "easier" but I don't think it's a good thing. I expect it will evolve like google search. Where initially there are ways to increase specificity, or at least introduce enough randomness to get some different results, it will converge to something that ignores most of what you prompt and gives you what OpenAI wants you to see. That's really the only way adapting to "bad" prompts even could work
- zamfi 4y agoLLMs will definitely get better -- and people will adapt too. But natural language contains inherent ambiguity, and that's not going to change. A large part of the paper talks about how the challenges of programming and developing ML systems don't go away just because you're using natural language and an LLM. For example, users incorrectly extrapolating from a single failed/successful prompt to other contexts: that's not something that will be solved by better LLMs, really. It's solved by getting users to recognize when they do/don't have enough data to believe what they're seeing. Disclaimer: I wrote the paper.
- digdugdirk 4y agoThanks for the official TL/DR. ... And the laughs. Paper is great, too. What's it like trying to write a paper while the field is shifting so rapidly? Is there anything you would have liked to include that had to be cut so you could get it out the door?
- zamfi 4y agoOh, that's not a full TL;DR -- but sounds like you've read the paper. :) Shifting field: yes, we spend a lot of time trying to figure out what is going to change and what isn't, at least in HCI (my field). LLM capabilities are changing by the...day? So anything that's very focused on LLM capabilities won't be relevant for long. Humans don't change quite as fast! Keep in mind this paper was first written in September 2022, reflecting work done prior to then...practically an eternity in which we've since seen ChatGPT take the world by storm, and now GPT-4. But I think the lessons there are still very relevant, and will be for quite some time.
- kleiba 4y agoNote that CHI is not a programming language or software engineering conference, but a conference in human-computer interaction: it's the ACM Conference on Human Factors in Computing Systems.
- gjvc 4y agoin Europe we call it HCI. In America, they put humans second, so they call it CHI.
- Ar-Curunir 4y agoEr what? It's the Conference for HcI = CHI
- asoneth 4y ago> in Europe we call it HCI Well CHI is being held in Europe this year, so apparently you don't! But more seriously, the field is "HCI" everywhere, including in America, and has been for at least thirty years. I have vague memories of hearing the story about why the initial ACM SIGCHI folks didn't go with HCI at the time but I can't recall. Anyway, it wasn't long after CHI was founded that basically everyone was using "HCI" on both sides of the Atlantic.
- gjvc 4y agoso why hasn't it changed, then?
- asoneth 4y agoInertia mostly. It's a question every grad student asks at some point. If anyone is actually curious: The Association for Computing Machinery (ACM) started a Special Interest Group on Computer-Human Interaction (SIGCHI.org). Later, the Association for Information Systems (AIS) started a Special Interest Group on Human-Computer Interaction (SIGHCI.org) which has remained about an order of magnitude smaller in impact and comes from a different originating field (Information Systems). My impression is that nearly all the papers in one journal would also make sense in the other, but CHI sometimes has a little more design-y or consumer product stuff while IS-HCI has some stuff that's more from the perspective of an IT department. The CHI folks can't unilaterally rename to SIGHCI without stomping over the smaller org, which means merging two different organizations, journals, and conferences run by different professional associations. Such a thing is occasionally proposed (e.g. http://www.bridginghci.org/wp-content/uploads/2018/03/chi-2018-bridging-AIS-HCI-and-CHI-tractinsky-cr.pdf http://www.bridginghci.org/wp-content/uploads/2018/03/chi-20...) which would be nice but in reality I think that's unlikely to happen because the cost exceeds the benefits.
- throwaway4good 4y agoI am surprised how quickly the HCI researchers jumped on chatgpt / prompt engineering.
- gjvc 4y agothey are chasing that research funding money
- cflewis 4y agoSadly I think this is a significant part of it. It is so very hard to convince anyone in CS to fund unsexy projects. I think the majority of innovation on the unsexy things happens internally at the large tech companies.
- jasonhong 4y agoOr maybe their chasing it because it's a highly relevant topic that might impact lots of people around the world, you know, a kind of human-computer interaction.
- gwern 4y agoPerhaps they could learn something about HCI from ChatGPT...?
- radarsat1 4y agoWhy? Natural language interaction with computers is like the holy grail of human-computer interaction, of course they jumped on it.
- teragramma 4y agoOh man, wild to see an article about the biggest conference in my field pop up on HN. It's surprising how quickly HCI people managed to pivot to AI stuff - the paper deadline for this conference was Sept. 15, 2022, which was about a month before ChatGPT was even released. So... expect to see even more AI at next year's conference in Honolulu!
- sgt101 4y agoV.interesting Do any of these studies give a map of programming tasks that are enabled by LLM's vs tasks that don't benefit?
- no_butterscotch 4y agoThanks for this. I'm interested in these subjects!