11 ms·
Sorry, but a new prompt for GPT-4 is not a paper
- gumballindie 3y agoAnyone caught doing this should be kicked out of the industry. Period. You're scaming those funding your "research", you are misleading readers, and are producing low quality content wasting everyone's time.
- d4rkp4ttern 3y agoYes these papers are optimizing for social media hype, I.e what is the quickest and easiest path to make noise on social media ?
- u32480932048 3y agoChatGPT droppings have to be at least as relevant and newsworthy as these findings [1] [1] https://www.wbur.org/news/2022/07/27/harvard-shorenstein-research-january-6-insurrection-president
- siva7 3y agoReminds me of what a real programmer is: https://sac.edu/AcademicProgs/Business/ComputerScience/Pages/Hester_James/Real%20Programmer.htm https://sac.edu/AcademicProgs/Business/ComputerScience/Pages...
- deleted 3y ago[deleted]
- henriquez 3y agoReal science is reserved for those with real expertise! As the self-anointed gatekeeper of real science I decree that other peoples’ work fails to meet the minimum standard I have set for real science! Mind you not the work other actors in the scientific community publish and accept among their peers - they are not real scientists and their work is trivial. For shame!
- mmkos 3y agoStrongly disagree. I do think trivial work is not paper-worthy and it would be more beneficial not to publish such work, as it mostly a waste of time for the peer reviewing it and the readers who will gain nothing from reading it. It's no lie that most publish for the sake of publishing and this post just calls it out for what it is.
- henriquez 3y agoTrivial means different things to different people. I’m not really a fan of LLM hype but it seems to me a valid practice of scientific discovery to evaluate the use and optimization of such models.
- brookst 3y agoThe real shame is that even HN has fallen into the trap of missing obvious and funny sarcasm unless it is clearly labeled.
- KyleBerezin 3y agoSarcasm is often spoken with a sarcastic inflection. It doesn't translate well to text, regardless of the community.
- 3cats-in-a-coat 3y agoAttacking the participants in a systemic shift is 100% useless as it doesn't target the culprit. In programming we have a similar phenomenon, that StackOverflow-driven (and I guess now GPT-driven) juniors have overtaken the industry and displaced serious talent. Because sufficient amounts of quantity always beats quality, even if the end result is inferior, this is caused by market dynamics, which operate on much cruder parameters than the sophisticated analysis of an individual noticing everything around them becoming "enshittified". SO-driven juniors are cheap, plentiful, and easily replaceable. And a business that values less expense and less risk therefore prefers them, because it has no way to measure the quality of the final product with simple metrics. The same mechanism is driving AI replacing our jobs currently, the avalanche of garbage papers by academics. This is entropy for you. We see it everywhere in modern society down to the food we eat. Quality goes away, replaced by cheap to produce and long shelf life. If we don't fundamentally alter what the system sees as ACCEPTABLE, and VALUABLE, this process will inevitably continue until our world is completely unrecognizable. And to fundamentally alter the system, we need an impulse that aligns us as a society, startles us into action, all together (or at least significant majority of us). But it seems we're currently in "slowly boiled frog mode".
- ilc 3y agoThis whole thing reminds me a bit of playing Fallout 4. There will be good data, the pre-AI enshitification data. The stuff from before the war. And then... the data after. Tainted by the entropy, and lack of utility of AI. Alas, this means in some senses, human progress will slow and stop in the tech field if we aren't careful and preserve ways to create pre-AI data. But the cost of it is so high in comparison to post... I'm not sold it will be worth it.
- jhbadger 3y agoThis paints a rosy picture of human-generated data now. It's not as if most human data is reliable. Even among peer reviewed scientific literature, most of it is crap and it takes effort to find the good stuff. Also, your analogy kind of misses the point of the Fallout games. The pre-war world was awful, filled with evil corporations like Vault-tech and Nuka Cola that murdered and poisoned their customers, and the point of the games is that people need to move on and not idealize the past.
- karxxm 3y agoIt depends I guess. If you solve a problem that had been around for a while and LLMs offer a new way of approaching it, then it can definitely become a paper. Of cause one has to verify in sophisticated experiments, that this approach is stable.
- H8crilA 3y agoIf you do enough measurements on that new prompt then I don't see why this shouldn't be a paper. People overestimate the value of "grand developments", and underestimate the value of actually knowing - in this case actually knowing how well something works, even if it is as simple as a prompt. Compare with drug trials: Adderall only differs from regular amphetamine in the relative concentration of enantiomers, and the entire value of the drug is in the measurements.
- aleph_minus_one 3y ago> People overestimate the value of "grand developments", and underestimate the value of actually knowing - in this case actually knowing how well something works, even if it is as simple as a prompt. I think this depends a lot on the "culture" of the subject area. For example in mathematics, it is common that only new results that have been thoroughly worked through are typically "publish-worthy".
- brookst 3y agoWouldn’t the “thoroughly worked through” part be analogous to extensive measurements of a prompt?
- aleph_minus_one 3y agoLet me put it this way: you can expect that a typical good math paper means working on the problem for, I would say, half a year (often much longer). I have a feeling that most papers that involve extensive measurements of prompts do not involve 1/2 to 1 year of careful - hypothesis building - experimental design - doing experiments - analyzing the experimental results - doing new experiments - analyzing in which sense the collected data support the hypothesis or not - ... work.
- starbugs 3y ago> Compare with drug trials: Adderall only differs from regular amphetamine in the relative concentration of enantiomers, and the entire value of the drug is in the measurements. Drug trials may be expected to be somewhat reproducible. What I don't get is how it can even be called research if it cannot be expected to be reproducible at all! GPT is a closed source/weights, proprietary product that changes every couple of weeks or so. How can you expect a prompt to do the same for a reasonable length of time for the research to be even rudimentarily reproducible? And if it's not reproducible, what is it actually worth? I don't think much. Could as well have been a fault in the research setup or a fake.
- Kelkonosemmel 3y agoHow to add prompt knowledge into research? By having papers about it. Shouldn't be the tooling around it good enough that a few prompt papers don't overload the system?
- alickz 3y agoStill beats most psychology papers
- margorczynski 3y agoTbh in both they are mostly alchemy with some more or less expert lingo put in to make it sound more scientific. AI/ML resembles more alchemy than e.g. physics, putting stuff into a pot and seeing what comes out. A lot of the math in those papers doesn't provide anything but some truisms, most of it is throwing stuff at the wall and seeing what sticks.
- selfhoster11 3y agoExcuse me? Step by step wasn't paper-worthy? Hard disagree. LLM research is currently in its infancy, because they are no older than a few years old. And a research field in its infancy is bound to have a few noteworthy "no sh*t, Sherlock" papers that would be obvious from hindsight. The fact is, LLMs are a higher-order construct in machine learning, much like a fish is higher-order than a simple cellular colony. Lower-order ML constructs do not demonstrate emergent capabilities like step by step, stream of consciousness thinking, and so on. Academics should be less jaded and approach the field with beginner's eyes. Because we are all beginners here.
- Der_Einzige 3y agoNLP/Computational Lingustics is NOT a new field. The ACL was founded in 1962 LM research is also old, papers using very shitty LMs (I.e Markov chains) and discussing them have existed since the early 2000s and likely before. Check yourself before you try to check others.
- phkahler 3y ago
- zitterbewegung 3y agoBeing dismissive about this tweet or agreeing with the author is one thing. Not realizing that the absolute minimum of a scientific paper can be much lower than a new prompt for GPT-4 is what everyone should be aware of.
- Der_Einzige 3y agoWhile I think the twitter post author is being a bit of an ass, they’re sort of right about the overvaluing we’ve put on simply better prompts. I wrote an opinionated GitHub gist about this exact issue: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c... I do the whole NLP publishing thing and I’ve hesitated to “write a paper” about applying techniques already known and used everywhere in the stable diffusion community to NLP models. That said, the AI community loves to pretend like they discovered something, such as a recent paper purporting to be the first to do “concept slider” Lora’s, despite these existing for months before that work was published on Civit.ai. The authors of course didn’t cite those already existing models. Everyone chasing citations and clout hard right now because these professors and researchers realize that they only have 5-10 years before AI eats their jobs and most other white collar jobs. I don’t blame them. I want my mortgage paid off before I’m automated away!
- SamBam 3y agoTired: Asking participants to sign an ethics pledge at the top of a tax return makes them more honest. Wired: Asking an LLM to write out their steps first makes them more accurate. They seem equally interesting to me, but one is a lot easier to replicate, and the other is easier to lie about.
- jatins 3y agoCan a person just go upload anything on arxiv or is there a review process around these things? What I am really asking is "what makes something a paper and not a blogpost"?
- forgotpwd16 3y agoArXiv is kinda reputation-based. That is to submit something you need to be endorsed, either done automatically, based on institution, or asking established authors. After being endorsed to a subject area, you can submit freely to it, keeping in mind: >Submissions to arXiv are subject to a moderation process that classifies material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv - the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee.
- MzxgckZtNqX5i 3y ago"Articles" on arXiv are not peer-reviewed, they just check whether it looks like it belongs to one of the categories they hosts: "Registered users may submit articles to be announced by arXiv. There are no fees or costs for article submission. Submissions to arXiv are subject to a moderation process that classifies material as topical to the subject area and checks for scholarly value. Material is not peer-reviewed by arXiv - the contents of arXiv submissions are wholly the responsibility of the submitter and are presented “as is” without any warranty or guarantee." [0] They are commonly known as pre-prints, in a similar fashion to IACR ePrint [1] for cryptography. [0]: https://info.arxiv.org/about/index.html https://info.arxiv.org/about/index.html [1]: https://eprint.iacr.org/ https://eprint.iacr.org/
- wongarsu 3y agoI feel this has nothing at all to do with LLMs and more to do with academic incentives in general. Focusing on quality over quantity won't advance your career. Publishing lots of new papers will, as long as they meet the minimum threshold to be accepted into whatever journal or conference you are aiming for. Having one good paper won't increase your h-score, three mediocre papers might. Doubly so when there's a new breakthrough, where one of your low-effort papers might end up being the first saying something obvious that ends up being really important. Because then everyone will end up quoting your paper in perpetuity.
- deleted 3y ago[deleted]
- mensetmanusman 3y agoIt is a _paper_, but it's not science, since GTP-4 is closed source and thus not reproducible in a lab. If OpenAI disappears tomorrow, papers are GTP-4 will likely be of little to no value, which is another tell of a non-scientific exploration. (note: not all explorations are scientific, and that is great! Science is just one of many tools for exploring lived reality.)
- MattRix 3y agoThat’s like saying a biologist studying an endangered species isn’t doing science because the animal could disappear tomorrow. The permanence of a subject has no bearing on whether it is science or not. The idea that science has to happen in a lab is of course absurd as well.
- dncornholio 3y agoI compare a paper on a GPT-4 prompt to a tutorial on how to use Photoshop. It's not science IMO.
- nicce 3y ago> That’s like saying a biologist studying an endangered species isn’t doing science because the animal could disappear tomorrow. The permanence of a subject has no bearing on whether it is science or not. The idea that science has to happen in a lab is of course absurd as well. The main point here is that anyone can likely start studing those endangered species and try to reproduce the results while in GPT4 it is not possible at all. The lab point is related to fact that we are talking about the software here.
- roguas 3y agoWhats not possible? Do they ban people for exploring sota model that they offer?
- nicce 3y agoThat is not reproducible research. To reproduce the research, you need training data, source code and all the parameters. With black-boxed generated models, there is no way to tell how they have been actually generated. It has no value for science to how to improve it further.
- snet0 3y agoIt's hard to draw these lines, because you will certainly filter out a lot of bad (i.e. useless, low contribution to any field) papers, but you might also filter out some really important papers. Research being basic or something anyone could've done doesn't count againt its potential importance, just the expected value of importance I guess. I'd rather we had a few too many bad papers than a few too few great papers.
- darepublic 3y agoA new prompt is not a paper, but you can prompt it for a paper.
- whywhywhywhy 3y agoAcademia needs to get over itself, can't wait to see how amazing this tech is going to get when the next generation who decide never to bother with those stuffy and navel gazing institutions becomes the driving force behind it. Looking forward to "I made this cool thing, here's the code/library you can use" rather than the papers/gatekeeping/ego stroking/"muh PhD". Think if Google had built an AI team around the former rather than the latter, they wouldn't have risked the future of their entire company and squandered their decade head start.
- elif 3y agoNah, this is just an early example of many "this is too easy it doesn't count" defensive human arguments against AI. Parallel to the "you use copilot so your code quality is terrible and you don't really even understand it so it's not maintainable" human coping we are familiar with. If there is any shred of truth to these defenses, it is temporary and will be shown false by future, more powerful AI models. Consider the theoretical prompt that allows one of these models to rapidly improve itself into an AGI. Surely you'd want to read that paper right?
- jdefr89 3y agoId much rather see a PoC…
- daveguy 3y agoNo prompt will cause an LLM to rapidly improve itself, much less into an AGI. Prompts don't cause permanent change in the LLM, only differences in output.
- elif 3y agoYou're talking about how GPT functions in 2023. I am discussing such a point where when LLM outputs become valuable LLM modifications. AI recursing on itself progressing toward an AGI.
- Intralexical 3y ago> AI recursing on itself progressing toward an AGI. Inbreeding LLMs will result in an "AGI"? In this universe, if you try to get something from nothing, you just end up with noise.
- daveguy 3y agoNo matter how many times you feed the output of an LLM back to itself, the underlying model does not change. Online training (of actual model weights not just fine-tuning) would be hugely resource intensive and not guaranteed to do any better than the initial training. Interference will happen whether catastrophic or simply drift. We can fantasize about future architectures all day long, but that doesn't make them capable of AGI or even give us a path forward.
- coldtea 3y agoSorry, but if they can get away with it, they'll release it as a paper. It's not like most papers are much above that anyway...
- JR1427 3y agoThis reminds me of how there was a boom in half-baked studies around COVID, e.g. modelling this or that aspect of the pandemic, or around mask wearing. I imagine that most of these will simply have had little to no impact, and will only serve to bolster the publication list of those who wrote them.
- carbocation 3y agoThe art and science of building these models is not disputed, but I think that the scientific value of prompts is tightly linked to reproducibility. If you’ve developed a new prompt for a model whose weights you can directly access, then this prompt could have scientific value because its utility will not diminish over time or be erased with a new model update. I’m even generally of the view that a closed API endpoint whose expiration date is years into the future could have some value (but much less so). But simply finding a prompt for something like ChatGPT is not useful for science because we don’t even have certainty about which model it’s executing against. Note that some of the best uses of these models and prompting have nothing to do with academics; this is a comment focused on the idea about writing academic papers about prompts.
- gandalfgeek 3y agoIf a new prompt enables a new task or enhances performance on a task then it absolutely should be published. Back in the day would compiler optimizations be not worthy of publishing?
- skilled 3y agoI can maybe understand the frustration from a “scientific” perspective, but for a lot of these “one prompt papers” - you still need someone to sit down and do the analysis and comparisons. Very few papers focus only on GPT/ChatGPT. Additionally, it gives people other ideas to try for themselves. And some of this stuff might be useful to someone in a specific scenario. It’s not glamorous research or even future-proof seeing as how certain prompts can be surgically removed or blocked by the owner of the model, but I don’t think it warrants telling people not to do it.
- jdefr89 3y agoI am sorry but what can ChatGPT do that a couple of minutes of googling couldn’t solved? Write half hearted essays that all contain the same phrase?
- deleted 3y ago[deleted]
- DrawTR 3y agoAnything generative? At its core, Google doesn't 'make' anything when you query it.
- potatoman22 3y agoGenerative LLMs can be turned into classifiers quite easily, search engines cannot.
- empath-nirvana 3y agoWhy don't you spend 15 minutes playing around with it and see what you can get it to do that google can't do?
- etewiah 3y agoBehind all this is a valid question. How does one evaluate prompts and LLMs? As gipeties (custom gpts) become more popular millions of hours will be wasted by ones that have been built badly. Without some sort of automated quality control, gipeties will become a victim of their own success.
- glitchc 3y agoThe current scientific research apparatus is more about being first than about being correct or thorough. A paper that gets out early means more citations, and many of the faculty sit on the editorial boards, and are able to suggest/enforce specific citations during the review process. Academics aren't fully to blame for this, it's just how the incentives are set up in the system. Tenure and promotions are increasingly based on h-index; a measure of impact based largely on the number of citations.
- rollcat 3y agoIt's hard to estimate the impact of an idea in the same way that you can estimate the impact of an investment (stock is a number that goes up or down). You're right that the current incentive system might be to blame, but a simplistic metric will be gamed just as easily - what would you propose?
- glitchc 3y agoHonestly, I don't think any metric can fix it, and don't have any easy solutions to this. The problem is larger than academia, more endemic to society. The root cause is society's values have changed. Previously, prestige mattered for something. Now, people would rather listen to pop stars than learned individuals, and wealth is the only metric that matters. As a result, typical professions that used to confer prestige, and for which prestige was supposed to be just reward, such as a professor, a medical doctor, a judge, are now mainly pursued for pecuniary reasons (money, job security). And because they're not doing it for prestige, they don't necessarily care about being right/correct. Playing the game to maximize the revenue streams is paramount. I happen to know a number of faculty who are quite proud of their multiple revenue streams. This would be unthinkable for an academic 50 years ago.
- yieldcrv 3y agoArxiv is like the MENSA of the tech world The similarity being that it’s ego masquerading as academic. Most things shared from there should have just been a blog post. The last year has showed that AI/ML research and use did not need academic gatekeeping by PhDs and yet many in that scene keep trying self infatuating things with the lowest utility.
- mo_42 3y agoWhy not? A paper is not necessarily scientific nor a breakthrough. In my view, a paper is written and documented communication that's usually approved by peers in the field. Also a blunt observation in nature can be noteworthy. However, we don't see such papers anymore as these fields have matured. Just go back in the history of your field and you will find trivial papers.
- esalman 3y agoIn the medical field, letters and case studies often document observations that may not be groundbreaking. However, scientific journals typically feature content that contributes to existing knowledge, making it somewhat novel. Consequently, presenting a set of POST parameters as an arXiv paper could be perceived as undermining the integrity of the entire preprint service.
- samlhuillier 3y agoTimes are changing. Human researchers will dedicate more and more time towards getting language models to work in desired ways rather than doing the research themselves. Language models will largely be the ones making "research" discoveries. Both should be considered valid research IMO.
- WhitneyLand 3y agoShould this be a paper? https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html https://not-just-memorization.github.io/extracting-training-... There is supporting analysis and measurement, but the essence is a single type of prompt, and DeepMind is a heavyweight lab I think it’s fair to say. Moreover there’s evidence people independently reported this result months beforehand on Reddit based on casual observation.
- sgt101 3y agoYou will notice that they say in the blog post: "If you’re a researcher, consider pausing reading here, and instead please read our full paper for interesting science beyond just this one headline result. In particular, we do a bunch of work on open-source and semi-closed-source models in order to better understand the rate of extractable memorization (see below) across a large set of models." So they are trying to rigorously quantify the behaviour of the model. Is this "look mom no hands"... I don't think so.
- WhitneyLand 3y agoSorry for any confusion, my comment was meant to refer to the linked paper: https://arxiv.org/abs/2311.17035 https://arxiv.org/abs/2311.17035 I think it’s valid work, but the original tweet seems to call a prompt based paper into question. At least enough to clarify where he would stand on an example like this.
- potatoman22 3y agoWhat's the difference between a paper on a new prompt and a paper discussing a new domain-specific model, e.g. heart failure risk? If they analyze the problem and solution equally, they both seem useful. It's not like most other ML papers share their weights or datasets.
- alphazard 3y agoDeveloping prompts for these models isn't a science yet. It does seem to meet most of the criteria for an art though. We recognize some outputs as high quality, and others as low quality, but often can't articulate the exact reason why. It seems that some people are able to reliably produce high quality results, indicating there is some kind of skill involved. More precisely, the quality of an individual artist's last output is positively correlated with the quality of their next output. A kind of imprecise "shop talk" has emerged, self describing as "prompt engineering", which resembles the conversations artists in other mediums have. For people in tech this will seem most similar to graphic designers. They produce much nicer looking interfaces than lay people can. We often can't explain why, but recognize it to be the case. And graphic designers have their own set of jargon, which is useful to them, but is not scientific. "Prompt artist" is a better term than "prompt engineer".
- DonHopkins 3y agoIn the same sense that "Bullshit artist" is a better term than "Bullshit engineer".
- rileymat2 3y agoWhy isn’t it science? Surely people can use the scientific method in investigating?
- alphazard 3y agoFor starters we don't have a way to measure quality objectively, and this is the case for art in general. If you were to develop an objective measure of beauty for example, visual art as a discipline would quickly turn into a science. At some level we know that's possible, we're all just brains in jars. But AFAIK we aren't doing science there yet. The science and engineering parts all have a measure of quality, sometimes that's a human rating, sometimes it's cross-entropy loss. There's nothing stopping someone from using the scientific method to investigate these things, but descriptively I haven't seen anyone, calling themselves a "prompt engineer/scientist", doing that yet. "I used these words, and I got this output which is nice" sounds like, "I tried using these brushes and I made this painting which is nice". I can agree with the painting being nice, but not that science was used to engineer a nice painting.
- lmeyerov 3y agoTo bring some data to a sour grapes fight: https://paperswithcode.com/sota/code-generation-on-humaneval https://paperswithcode.com/sota/code-generation-on-humaneval For code generation, GPT4 is getting beat by the small prompt library LATS wrapped around GPT3.5. Given the recent release of MagicCoder / Instruct-OSS, that means a small prompt library + a small 7B model you can self-host beats the much fancier GPT4. Similar to when simple NNs destroyed a decade of Bayesian modeling theses & research programs, it's frustrating for folks going other paths. But it doesn't make the work 'wrong'.
- harlanlewis 3y ago> GPT4 is getting beat by the small prompt library LATS wrapped around GPT3.5 The link you shared doesn’t quite reflect this. Omitting other models… LATS (gpt-4): 94.4 Reflexion (gpt-4): 91.0 gpt-4: 86.6 … LATS (gpt-3.5): 83.8 … zero-shot (gpt-4): 67.0 zero-shot (gpt-3.5): 48.1 I’m not quite sure how to translate leaderboards like these into actual utility, but it certainly feels like “good enough” is only going to get more accessible and I agree with what I think is your broader point - more sophisticated techniques will make small, affordable, self-hostable models viable in their own right. I’m optimistic we’re on a path where further improvement isn’t totally dependent on just throwing money at more parameters.
- lmeyerov 3y agoAh you're right, LATS GPT3.5 is 84 while standalone GPT4 is 87 Given standalone GPT3.5 is "just" 48.. it's less about beating and more about meeting RE:Good Enough & Feel... very much agreed. I find it very task dependent! For example, GPT4 is 'good enough' that developers are comfortable copy-pasting & trying, even vs stack overflow results. We haven't seen LATS+MagicCoder yet, but as MagicCoder 7b already meets+exceeds GPT3.5 for HumanEval, there's a plausible hope for agent-aided GPT4-grade tools being always-on for all coding tasks, and sooner vs later. We made that bet for Louie.AI's interactive analyst interface, and as each month passes, evidence mounts. We can go surprisingly far with GPT3.5 before wanting to switch to GPT4 for this kind of interaction scenario. Conversely... I've yet to see a true long-running autonomous coding autoGPT where the error rate doesn't kill it. We're experimenting with design partners on directions here -- think autonomous investigations etc -- but there's more on the advanced fringe and with special use cases, guard rails, etc. For most of our users and use cases... we're able to more reliably deliver -- today -- on the interactive scenarios with smaller snippets.
- Racing0461 3y agoDoesn't academia incentivise quantity over quality anyways?
- grepLeigh 3y agoStudying the way LLMs behave to different prompts (or different ways of fine-tuning for a set of prompts) is valuable science. Some of the most interesting papers published this year ("Automatic Multi-Step Reasoning and Tool-Use") compare prompt strategies across a variety of tasks. The results are fascinating, findings are applicable and invite further research in the area of "prompt selection" or "tool selection."