12 ms·

Ask HN: Is prompt engineering just snake oil?

• mirekrusin 4y ago
  It's new and it's different. Don't dismiss it as "snake oil". Approach it as any scientific topic - with curiosity and open mind. Be rational with your skepticism.
• gjvc 4y ago
  Yes
• throwaway019254 4y ago
  Yes
• VoodooJuJu 4y ago
  It's snake oil as soon as someone starts using the term unironically, which I haven't personally observed.
  • wruza 4y ago
    https://news.ycombinator.com/item?id=35642665 https://news.ycombinator.com/item?id=35642665
• meltyness 4y ago
  Yes, we would all prefer "philosopher of corpus-linguistic preference membranes, and domain-specific knowledge manager" because it's more precise, and doesn't trample on my position/salary/pedigree.
• kordlessagain 4y ago
  In general, any product or service that makes exaggerated or false claims of effectiveness without providing concrete evidence to support those claims could be considered a form of "snake oil." Just because we don't know all the workings of something doesn't mean we can't effectively use it, or measure how well that approach of use is valuable or repeatable for similar uses. For example, while we have a good understanding of the basic principles of magnetism, there is still much that we do not fully understand about how magnets work, and ongoing research is focused on unraveling these mysteries and expanding our understanding of this fascinating natural phenomenon. That's not to say we don't know how to use magnets to make sound, however. Prompt engineering, in my opinion, is a process of optimized querying of a frozen model. The approach can be augmented by using "hot" data from vector databases or other types of text storage engines. The approach to picking the right content for prompt building is not "snake oil", but based on well known processes including autocomplete, synonyms and related terms, personalization, and query expansion.
• tkiolp4 4y ago
  Prompt Engineer is the new Scrum Master.
  • Ekaros 4y ago
    Actually, Prompt Masters would be lot better term. And it would work in both common meanings in current usage.
  • ResearchCode 4y ago
    That's finally a real use for large language models. Replace the agile coaches. Needless to say, a team is better off replacing them with nothing.
• iamwil 4y ago
  Think of it more as "prompt-fu", akin to "google-fu"--or how to google. You do have to know how to talk to an LLM, and it will be important. But it's not so much engineering, as we come to think of it in software traditionally. I wouldn't get hung up on the name, but just see it for what it is--a bad name for something we'll need to learn to be effective. It won't go away entirely, but will be mitigated to be less important as LLMs and AI get better.
  • coffeebeqn 4y ago
    LLMs are very quirky. They need the correct context to produce something good and it’s often surprising what that needed context is. OP just play around with midjourney and you’ll soon realize that it is a very real thing
    • ZephyrBlu 4y ago
      100%. People who have not tried to interact with these models don't understand there is actually quite a lot of skill in navigating the latent space.
      • mdp2021 4y ago
        It really should be just an intermediate phase. Proper, solid control system must be created. First you have Stable Diffusion, then you create ControlNet - and this as just a beginning. This is also one of the reasons why the problem of transparency ("but why and how does it work?") is so important in Machine Learning: allowing control.
        • ZephyrBlu 4y ago
          Adding a working control system doesn't take away the fact it requires skill though. It seems like people think that one day they will be able to give a vague prompt and generate exactly what they were looking for, which I don't believe will ever happen. It feels like there will always be some skill in navigating the model.
• screye 4y ago
  Prompt engineering is an 'oil' alright. It's a lubricant to make models work better. The idea of engineering being backed by solid mathematical understanding is a relatively post-calculus idea. For the longest time, we built bridges a certain way, because those are the bridges that stood. We were flying planes long before we understood flight. At the end of the day, engineering is the cycle of "identify a pain point -> launch experiments -> observe the delta -> apply the solution". Prompt engineering meets all of those requirements and therefore, is engineering. It is funny to hear this, because 20 years ago : "is software engineering really engineering" was a rather common phrase around STEM circles.
  • iandanforth 4y ago
    "We were flying planes long before we understood flight." This isn't true. Sir George Cayley elucidated several aerodynamic principles significantly in advance of the Wright Brothers first flights. The Wrights took advantage of this knowledge in designing their propellers and planes. We have since learned a lot more about how flight works, but the guiding principles of lift, drag, and thrust predate the mechanical engineering to realize a heavier than air aircraft.
• cjbprime 4y ago
  It's okay; software engineering isn't real either. (In many countries, the word "engineer" is regulated -- you can't call yourself an engineer without professional qualifications and oversight.) https://en.m.wikipedia.org/wiki/Regulation_and_licensure_in_engineering https://en.m.wikipedia.org/wiki/Regulation_and_licensure_in_...
  • jeffrallen 4y ago
    And in Canada, if you call yourself an engineer, you wear an iron ring to remember your responsibility to society. https://en.m.wikipedia.org/wiki/Iron_Ring https://en.m.wikipedia.org/wiki/Iron_Ring
  • foogazi 4y ago
    When a building collapses you can’t just reboot it But in software you can simulate and test all you want before connecting it to the real world
    • zerr 4y ago
      To be fair, when the building collapses in the CAD simulation, you just "reboot" it with different parameters. Civil engineers don't actually put bricks or weld a metal.
      • Nextgrid 4y ago
        Problem is that CAD is just an approximation of the real world and will never simulate it 100% faithfully. Software on the other hand can technically be simulated 100% accurately, barring hardware or interpreter bugs.
    • kaba0 4y ago
      No amount of testing will prevent all errors, and certain bugs can cost human lives.
  • voidhorse 4y ago
    Yeah, pretty much this. Only a subset of the actual activity that falls under the umbrella of "software engineering" could be called engineering in the traditional sense. The engineers working on aviation software and other low-level, real-time, performance critical systems probably need to use quite a bit of maths and are closest to doing what we traditionally consider engineering. All those teams working on web apps and advertising...yeah not really. This isn't to say what they do isn't complicated or difficult, but it's not really engineering in the classical sense insofar as it doesn't (typically) require the use of continuous mathematics and deep systems theory. If you're primarily writing systems design docs and performing calculations, you're probably doing some kind of engineering. If you're mostly writing code to spec you're just programming. It's sort of akin to a carpenter calling themselves an engineer. But hey, this is all symptomatic of a general trend here in the states of the trivialization and devaluing of expertise; people think it's fair to identify as whatever their heart desires regardless of qualification, knowledge, or experience. A world in which people regularly accept advice from total randos on social media is not one that has any strong sense of value around knowledge or critical capacity.
• eternalban 4y ago
  The “engineering” aspect will rear its head one fine day when company A, that has bet the house on ‘prompt engineering’ their service or product, resorts to panicked uttering of LLM chants, spells, and whatnot, to make a misbehaving model work again. But that it’s a bit further down the line. Today, like previous hype cycles, opportunistic types will cash out on the money train. And tomorrow you will do whiteboard interviews for them (just like the last cycles..) So the relevant question for the thoughtful geek isn’t “Is crafting prompts engineering?”. No. The Q is “should I sit out this hype cycle and then end up doing leet code monkey dance for the opportunistic types who made it big by riding the hype cycle, yet again?”
• amelius 4y ago
  I think alchemy is a better word. Anyway, if you're trying things iteratively it becomes more a science. Engineering is more like designing stuff and building them without trying lots of things first.
  • than3 4y ago
    I believe the words you are looking for would be, it becomes more a science when you attempt to reduce to first principles (which happens iteratively but not necessarily the other way around). Engineering takes a first-principled approach, where upon knowing those principles you can then construct abstractly without physically building first.
• broast 4y ago
  Prompt engineering is just communication skills. I could tell you which of my coworkers would make good or bad prompt engineers just by looking at their transfer notes on tickets.
• karaterobot 4y ago
  Is this just about gatekeeping the word 'engineer', so that its purity isn't tarnished? I know that software engineers were scoffed at by electrical engineers, who I bet were scoffed at by mechanical engineers, who were probably scoffed at by, you know, people who built siege engines or whatever. Is prompt engineering 'real' engineering? It seems easy enough to test whether knowledgeable, self-proclaimed prompt engineers can outperform a random person with only moderate experience requesting information from an AI in a reproducible way. If they can't, then there is probably no engineering involved in prompt engineering, at least at present. If they can, then it seems like they're probably not doing it with magic, so there is some set of reproducible techniques involved. At that point, would it be fair to call it engineering?
  • wg0 4y ago
    IMHO - software engineering really isn't real engineering. Not in the traditional sense. But it certainly is based on knowing the inner workings of all or as many and as much of the abstraction layers. But who's a prompt engineer? Is anyone who understands how Tensors and Transformers work is a prompt engineer? Are prompt engineers model specific? I'm GPT-3.5 prompt engineer and we have a job openings for Alpca or StableML prompt engineers? Lastly, if these LLMs are so intelligent and have so much deeper understanding as their emergent capability - why prompt engineering is needed in the first place? You generally don't need prompt engineering even with average IQ humans unless one is trying to swindle someone.
    • LawTalkingGuy 4y ago
      > software engineering really isn't real engineering Most people who call themselves software engineers have not studied engineering at all, let alone passed an engineering course. I think there are two broad classes of benefit a real SE can provide. The first is an understanding of how software can be proven correct, and the limitations of that vis-a-vis the real world - power failures, etc. They can help understand what portion of the software needs what level of performance, understandability, correctness, etc. Your core trade-pricing engine's needs are different than the webserver showing the status pages - both are required but the standards for both are not the same. Second is an ability (and this is what apprenticing and is for in a real engineering career) to judge and design entire systems. We "devs" often get treated as fungible work units and tasked with various little subsystems. A professional engineer entering such a space would be required to obtain a pretty good understanding of the entirety of the company's systems and very good of what they integrate with, such that they can characterize the cost, risk, etc, of the solution space before even beginning to plan the technical solution. Engineers often tell clients that they've envisioned the wrong solution. They're trained to break down silos and integrate solutions where possible. (Not that they all succeed, but that's in the training.) > if these LLMs are so intelligent and have so much deeper understanding as their emergent capability - why is prompt engineering needed in the first place? Because they're language models and people are trying to solve things that aren't pure language problems with them. So you need to map the problem to something it can represent and where a solution can be formed, even if that solution may not use an LLM call. A simple example is taking a mathematical word problem and solving it. Currently the LLMs do not model math well so you'd use the LLM to separate out the clauses of the problem and turn it into variables and write an equation from it and then you'd evaluate that snippet of code for the real answer. And then, depending on what part of what system this was, you may need to verify the parsing and other parts of the pipeline so you would need to build a chain where the results from one call are handed to other systems, maybe just back to the same LLM, with a bunch of examples, for multiple instances of more detailed checking, and then if those answers aren't the same, to another round where it tries to explain the difference and feeds that explanation into a round where you try to reprompt the initial layer and try again. Token limits often constrain how good the instructions and examples can be so you often have a few levels of prompts, nearly bulletproof ones, and shorter ones which are cheaper and allow other use of the token budget but may have more failure cases. Building with non-deterministic tools isn't impossible, all ropes are different and yet we have rope bridges, but you have to know how they work and how they fail. > But who's a prompt engineer? [If] I'm a GPT-3.5 prompt engineer do we need to have a job openings for Alpca or StableML prompt engineers? No, just like architects learning different materials during their career. But all those LLMs are different and you need to experiment (scientifically) to find their characteristics in your area before building on them.
• atahanacar 4y ago
  Is social engineering just snake oil?
  • wg0 4y ago
    Good question. Is political engineering also engineering? My question is - for social or political engineering, you can gain definite knowledge (spying, observing, researching, information extraction) and then devise something. How to do the prompt engineering exactly based on what knowledge? How to formulize and document that knowledge so that it isn't just about intuition and gut feeling but rather a learnable and transferable skill?
    • ImPostingOnHN 4y ago
      > Is political engineering also engineering? Good question, even if I've never heard of "political engineering" Is social engineering snake oil, too?
      • wg0 4y ago
        Political engineering is pretty common in societies where deep state or establishment has a firm control and wants to engineer results their way. Most Western democracies are not that.
        • ImPostingOnHN 4y ago
          would you mind answering this question, please? it's been ignored twice despite being kind of the key point: > is social engineering snake oil, too?
          • wg0 4y ago
            Already answered above: >For social or political engineering, you can gain definite knowledge (spying, observing, researching, information extraction) and then devise something.
• azubinski 4y ago
  To begin with, this is definitely not related to engineering... And, yes. Of course, yes.
• docandrew 4y ago
  It’s “engineering” the same way we joke about “social engineering.” Both are ways of using language to subtly manipulate another. But the term should be understood as tongue-in-cheek and I’d run from anyone trying to sell you something to do with it.
• skilled 4y ago
  I think the word Engineering means your own grasp on the subject/task at hand. For example, you can't engineer a prompt that would lead to a result outside of the scope of what the model/interface is capable of doing. You can, however, engineer a prompt that will refine the result to get exactly what you want. In other words, you also have to understand the text that you're getting back from your prompts, and then "engineer" the prompt to fish for better results. AI itself can do this to an extent, but over-engineering is also a thing, as is the fact that AI itself does not actually know what you want. That's all there is to it. Refine, revise, engineer - all just buzzwords that lead to the same end result: a game of ping-pong between you and the LLM.
• MattGaiser 4y ago
  The model works in a certain way, it just needs to be discovered and interpreted and you might want to hire someone to help you with that. Not really different than getting a solutions architect to have an API explained to you.
• photochemsyn 4y ago
  Any form of engineering involves a certain degree of reliance on abstractions, e.g. a civil engineer really doesn't need to know everything about the molecular structure of steel and concrete to successfully design and build a structure such as a bridge. What does matter is having stable, predictable abstractions that you can rely on, so in software engineering it would be a mistake to rely on system-dependent undefined behavior when using a language like C++. You also don't necessarily need to know what kind of low-level optimizations the compiler and linker are using to generate your executable binary, though in some cases it could be important. With LLMs, however, it's not really clear if the same prompt always gives the same kind of output, and you can do 'regenerate response' with something like ChatGPT to see this in action. In some sense, 'prompt engineering' can be more like 'interviewing an engineer' in that LLMs seem to provide better outputs if you start with a broad, general question and then narrow down to your specific interest over a series of questions. Helping the LLM out by defining context, asking it to expand on a specific output it generated, pointing out where it may be hallucinating etc. all appears to improve the quality of output but I don't know if this kind of iterative process is really 'prompt engineering', maybe 'prompt optimization' is a better word for it? In other words, writing a series of prompts to an LLM that gets you the answer you need feels quite unlike writing a Python script to automate some task. You can plan the latter out from start to end, but the former is this back-and-forth process.
• gwd 4y ago
  One definition of "engineer" is "to plan or do something in a skillful way:". Example: "The administration engineered a compromise." I'd say even in this example, there's more a sense of searching through a range of different options to achieve specific properties. This is the sense of "prompt engineering": you use skills to create prompts -- in part by searching through various alternatives -- that have specific properties. [1] https://dictionary.cambridge.org/dictionary/english/engineer https://dictionary.cambridge.org/dictionary/english/engineer
• cvg 4y ago
  I like this framing by Michael Hashimoto, on Prompt Engineering: https://mitchellh.com/writing/prompt-engineering-vs-blind-prompting https://mitchellh.com/writing/prompt-engineering-vs-blind-pr... Most of what we see on Twitter or YouTube is Blind Prompting. However, it is possible to apply an engineering mindset to prompting and that is what we should call prompt engineering. Check out the article for a much more detailed framing. Dair AI also has some nice info and resources ( with academic papers) about prompt engineering.
  • lmeyerov 4y ago
    Prompt testing, especially when for q/a pairs where there are multiple right answers, has been bugging me a lot The article is reasonable, but also shows a big gap in tooling, as the techniques there feel closer to linting & typing then testing once you do more interesting prompts. They don't check the interesting parts..
    • than3 4y ago
      > The article seems reasonable but ... closer to linting then testing... they don't check the interesting parts can you elaborate a bit more on what those interesting parts are? It could just be a limitation of computation.
      • lmeyerov 4y ago
        We are helping our users with qa tasks involving code generation, where the answers may be either JSON, executable code, or markdown discussions involving the same. We are tuning for a bunch of tools following that pattern so our users don't have to. It's easy to make a labeled training set for grading our homework (catching regressions, ...) in the case of classifiers, and that's basically what the blog post showed. What about for the above qa tasks? We can ask GPT4 whether a generated A was a good answer for a Q, but that's asking it to grade itself. Likewise, in the code case, we can write unit tests for the answers. (Trick: we use the former to more quickly do the latter.) But I feel like there has to be better ways Another: OpenAI always updates models based on use, so we have to be sure our tests are real holdout sets that never get back to them...
• Veen 4y ago
  It's a silly name for a real phenomenon. It's better categorized as a form of rhetoric; instead of persuading people to do what you want, you're persuading LLMs using particular patterns of language. But no one is going to call them "prompt rhetoricians". "Prompt writer" would be better, but "writing" is less of the spirit of the times than engineering.
• jprd 4y ago
  SEO, Software Engineer, Systems Administrator w/a budget, regex geniuses, obfuscated C challenge, demoscene, etc. - none of it snake oil. $$ and demand don't change the skills? I mean, Anakin _WAS_ good with machines.
• djtango 4y ago
  I know someone who studied ML as black boxes kind of as her PhD (I don't really know the specifics so don't ask me) but I did find it interesting that if we're dealing with a sufficiently opaque and complex black box, we have to resort to our old friends experimentation and empiricism. So maybe it's not engineering but science? :p A little tongue in cheek but it did get the noodle working
• ravenstine 4y ago
  Software engineering is also snake-oil. Yes, software can be engineered, but that's not what the vast majority of so-called software engineers are actually doing at their jobs. The title mostly exists to inflate the importance of a programmer with years of experience under their belt. In reality, most of them couldn't explain to you what engineering itself is, and their job primarily consists of duct-taping and building features expediently. It's like taking a carpenter whose job is to crank out barely adequate sheds for a shed company and calling them an architect. Don't even get me started on "computer science." Prompt engineering is a legitimate area of study, and is obviously a practice demanded by LLMs, but you gotta just ignore the "engineering" part. It's the same skill as being a good communicator. Take a room full of "software engineers", tell them "build me an app that will let me sell gadgets", and they'll do their best to build one, but chances are it won't do what you want unless you communicate with greater specificity. It's hilarious how many people think LLMs suck just because they don't do the right thing given a single shitty sentence.
  • intelVISA 4y ago
    Most SWE is snake oil, take one look around and you'll see what I mean... React..? Golang? GUIs?! We stepped too far from The Truth (vim, ghc) and now we are all Oilers.
    • hnfong 4y ago
      SWE snake oil is like "Extreme Programming", "whiteboard interviews", "design patterns", "microservice architecture"...
      • intelVISA 4y ago
        You just know you're in a Bad Shop when you see "unit tests" come out the bag...
    • yawnxyz 4y ago
      if we are the Oilers, who are the Flames?
  • 4y ago
• barbariangrunge 4y ago
  It’s not a durable skill. A year after learning to give perfect prompts for your favourite ai, the mode will get updated and you’ll have to learn it all again
• barbariangrunge 4y ago
  Unity has “customer support engineers” iirc. The USA misuses the term endlessly
  • than3 4y ago
    > The USA misuses the term endlessly Well most of corporate america certainly does, but they can just as easily be multinationals so its not really the USA other than they may have registered the entity here.
• mikewarot 4y ago
  I think the proper term for the job should be Prompt Alchemist. We've seen this before. Back when Google was a search engine, you'd eventually get a feel for how things worked, and build a vocabulary of "spells" that give the results you want, without using the common words with many overloaded meanings. (Example: Annotation instead of mark up, when trying research the Marking Up of Hyperlinked Texts) Similar things proved to be true of Stable Diffusion and the image generators. It seems quite reasonable to conclude the same will be true of GPT4 and its kin.
• ZephyrBlu 4y ago
  Don't take names so literally. They're just useful labels that help us categorize things and communicate shared meaning. At this point, people know what you mean if you say "prompt engineering". Prompt engineering feels like a quite appropriate name because there is a lot of experimentation and refinement to find good ways to interact with these models (I.e. good prompts!). It doesn't perfectly map to "engineering", but it maps pretty well. What else would you call an iterative process of experimentation and refinement?
• uptownfunk 4y ago
  I think the integration of LLM into software interface to accept natural language input is the real “prompt engineering”
• wodenokoto 4y ago
  I’ve always considered it was engineering in the “social engineering “ sense of the word
• notatoad 4y ago
  wait, people are talking about prompt engineering like an actual thing? it's not snake oil, it's literally just a joke.
  • amonith 4y ago
    already a thing in Poland: https://www.pracuj.pl/praca/prompt-engineer-warszawa-czerniakowska-87a,oferta,1002513338 https://www.pracuj.pl/praca/prompt-engineer-warszawa-czernia...
• taeric 4y ago
  Notable to remember that snake oil actually had use. It was everything getting labeled as such that caused problems. Probably makes it an apt comparison, all told. :)
• arroz 4y ago
  Prompt engineer is like search bar engineer sure you can be good at googling, and it helps, but it isn’t a profession
• cloudking 4y ago
  Has anyone solved a real business problem with LLMs that they couldn't solve as efficiently before?
  • codegeek 4y ago
    Funny. I used to ask the same question about blockchains.
    • deleted 4y ago
      [deleted]
• tourgen 4y ago
  [dead]
• ResearchCode 4y ago
  Yes. It requires much less engineering than a Google Search, and there are no Google Search Engineers.
• netdur 4y ago
  I have seen people doing multi steps prompting SD to get exact image they wants, that includes textual inversion, merging models, creating lora, manipulate prompt weights, in/out painting, this is a serious job like graphic designer but with less imagination and more wizarding, now SD created open source LLM, I expect the same to happen
• pkd 4y ago
  I prefer to call it "prompt alchemy" for exactly the reasons you described. It can't be engineering if you can't understand what you are trying to engineer with. Alchemists were sort of proto-scientists and engineers because they were trying to get use out of a physical world that they did not fully understand.
• m3kw9 4y ago
  Is sort of snake oil. People would have an entire arXiv article explaining "Explain Step by step" as prompt engineerning, but in reality, it's just natural language to say "step by step" if you want it to be step by step.
• _throwawayaway 4y ago
  I just see it as proper thinking, but yeah prompt engineering sounds more catchy.
• nanidin 4y ago
  > Below is a list of some known non-speech sounds, but we are finding more every day. Please let us know if you find patterns that work particularly well on Discord! [laughter], [laughs], [sighs].... This is likely a reflection of the data used to train the model - TTS models are trained on labeled audio, one form of which is subtitled audio streams. In subtitles / closed captioning, bracketed words are frequently used when there is something audible that is not speech. Based on this insight, it should be possible to inspect the training data and extract a set of non-speech sounds the model is likely to generate well - but that doesn’t drive engagement of your users like asking them to experiment themselves does ;) For example, closed captioning adds music notes (♪) when music plays or when people sing. According to the Bark docs, adding ♪ causes the model to output things as music.
• oneplane 4y ago
  In essence it's all about getting results, and specialisation might yield better results. Just because people think it sounds cool to call it 'prompt engineering' doesn't mean it's not a specialisation (or optimisation), even if the terminology is based on nothing. I do think that it's not broad or deep enough (right now) to merit a completely separate area of expertise on its own, but it can be too big to fit within the minds and areas of expertise of people who already have their work cut out for them in their existing workdays. Think about most desktop computer users, they really have no clue how the computer works, and even asking them to change a resolution on a mirrored display might be too far removed from their knowledge base (and that is fine). This also applies to tokenisation prompt inputs; it might take too much for someone to simply tack that on to what they already do and know. This is then were we get job postings and those need to be easily identified, so people can search for them, and this is where we get nonsense terms used to distinguish the desired applicants from the pool of work-seeking people.
• ulrashida 4y ago
  Professional engineers are typically: * Regulated by a profession and associated legislation * Work first to ensure the safety and welfare of the public * Perform only within their area of competence * Act as faithful agents or trustees for their clients * Avoid deception and represent matters in an objective and truthful manner * Invest in continuous professional development * Can be subject to penalties for malpractice: removal of licence, fines, or jail It's hard to see any of these applying to LLM input entry people. It's a even a stretch to consider them non-professional engineers.
  • vimax 4y ago
    Hard to apply those to software engineering as well.
    • runjake 4y ago
      In most jurisdictions, a professional engineer is essentially a specific legal thing. In most jurisdictions I am aware of, software engineers are not professional engineers. I am sure there are many exceptions. That said, I think all of the above are rather easy to apply to SWE aside from, arguably, the first bullet point.
      • vimax 4y ago
        And I can list counter examples for each. Civil engineers and architects don’t “move fast and break things” or “fake it til you make it” after a 3 month bootcamp.
        • marcellus23 4y ago
          There are software engineering domains, like aerospace, where the process is very stringent with rigorous planning and testing. There are others, like consumer app development, where there's simply no need for that kind of process, so it doesn't exist. HNers might complain, but average people don't care if their social media app crashes once in a while and they have to restart it — so why should developers care? Better to spend their time on other stuff. Whereas people care a lot if their airplane crashes. I don't think it suggests anything about the relative value of working in those domains.
• rvz 4y ago
  Yes. > So that's the team themselves not knowing what their model is capable of then how come prompt engineering is any engineering at all? This is why.
• slowhadoken 4y ago
  Calling yourself a Prompt Engineer makes me think you don’t know college math and computer science. Prompt Engineer sounds like a Subway employee calling themselves a sandwich artist.
  • grepfru_it 4y ago
    Welcome to Clown World! Where things that should not be are, and things that should are not. I first noticed CW when someone reached out to me during COVID and their title was "Professional LinkedIn Specialist". In hindsight, it was probably when people were making gobs of funny money from running their GPU 24x7 My advice? Don't fight it, accept it and embrace the change. Don't sell yourself short, you only need to put in like 3 months of work to earn that Principal Prompt Engineer title!
    • slowhadoken 4y ago
      Yeah people are always trying to celebrate laziness and mediocrity.
• xbar 4y ago
  Purely.
• gnicholas 4y ago
  I've wondered if some of the value of being a skilled prompt engineer will be eroded as the LLMs get better, especially as they can reply to ask clarifying questions or ask if you want XYZ tweaks (e.g., in the voice of Author X, or from the perspective of an ant, with backlighting). I'm sure there will always be some people who are better at this than others, but if the interface is sufficiently chatty and 'smart', that could significantly reduce the gap between newbies and seasoned prompters.
• turtleyacht 4y ago
  Prompt engineering is as valid as software engineering (of the non-certified kind). Why is it engineering? Technical expertise. It is technical in nature, or only one level removed from having to look at code or internals. Unlike customer support--strictly speaking--you have access to additional tools or authorization to investigate the matter. Domain expertise. You know that some phrases just don't help. You know about temperature and other jargon. You can look at someone's paragraph of prompt and immediately suggest something. Stakeholder safety. You act in a manner that eliminates or reduces stakeholder harm. You care about your work. If any of these are missing, it's not "engineering." Not to mention the importance of measuring things, data, etc--but I would hazard that being able to stare at graphs all day does not an engineer make. At the end of the day, if your job title is "Prompt Engineer," who will object? Versus professional (certified) engineers: https://news.ycombinator.com/item?id=35669226 https://news.ycombinator.com/item?id=35669226 Engineering as artful action: https://news.ycombinator.com/item?id=35670444 https://news.ycombinator.com/item?id=35670444 Tools of the AI Engineer: https://news.ycombinator.com/item?id=35669249 https://news.ycombinator.com/item?id=35669249
• yarg 4y ago
  Probably closer to SEO. Works for the time being while there's low hanging fruit, but researchers will use those exploits to help build more robust systems. Eventually there'll be no point in doing it anymore.
• deleted 4y ago
  [deleted]
• antegamisou 4y ago
  Isn't it another tech-influencer LinkedIn fad BS job title to make those who are related to it feel important?
• layer8 4y ago
  The “engineering” in “prompt engineering“ is more like in “social engineering”. It’s a secondary, metaphorical meaning. For example, Google defines the second meaning of "engineering" as: 2. the action of working _artfully_ to bring something about. "if not for his shrewd engineering, the election would have been lost" (https://www.google.com/search?q=define%3AEngineering https://www.google.com/search?q=define%3AEngineering) Merriam-Webster has: 3 : calculated manipulation or direction (as of behavior), giving the example of “social engineering” (https://www.merriam-webster.com/dictionary/engineering https://www.merriam-webster.com/dictionary/engineering) Random House has: 3. skillful or artful contrivance; maneuvering (https://www.collinsdictionary.com/dictionary/english/engineering https://www.collinsdictionary.com/dictionary/english/enginee...) Webster's has: The act of maneuvering or managing. (https://www.yourdictionary.com/engineering https://www.yourdictionary.com/engineering) This is a well-established, nontechnical meaning of “engineering”.
• codeptualize 4y ago
  There are some degradations, just like other "engineering" like software engineering. I am pretty sure most people don't know the ins and outs of everything that makes code run, from compiler to CPU, especially not the CPU's of today, I doubt any one person fully understands them. You just know the abstraction layer you manipulate and learn techniques how you can make it do the things you it want. That is quite similar to prompt writing. You can certainly get better at it by learning the basics of LLM models, as well as finding and applying certain methods that improve the results. You can actually see quite a lot of difference between someone trying out some things, or someone who has spent quite a bit of time trying to make it do non standard things. What you call "engineering" is of course up for debate, but it's not snake oil.
• nathias 4y ago
  I'm sure one can develop great skill in their arcane incantations, but how we call it is a matter of taste
• _pdp_ 4y ago
  I don't know if it is fair to call it "engineering" but it is not snake oil. Prompt engineering is programming but with natural language. Even between us humans being able to communicate clearly is essential part of understanding each other. It is also a form of programming. Prompt engineering, or whatever you want to call it, will be an important part of how we communicate with machines in the future.
• wwilim 4y ago
  Prompt wrangling, prompt whispering?
• dbavaria 4y ago
  It seems akin to SEO experts identifying as engineers. Can we just call them experts or wizards if they want to be fantastical.
• aaron695 4y ago
  [dead]
• james-revisoai 4y ago
  Prompt engineering made more sense before RLHF/instruct models. Back then text just continued onwards, and many papers found ridiculous gains on conforming with intended output (and therefore properties of output) by finding a suitable widespread online format. Example: I think all Question/Answer forums, and places like StackOverflow, used a certain format like 'Q:' - that worked much better for question content with high quality answers. One paper, "Ask me anything" showed that formulating tasks as questions-answers resulted in much more effective prompts: https://arxiv.org/abs/2210.02441 https://arxiv.org/abs/2210.02441 - another for reasoning is the "Lets think things through step by step" example. It was a challenge to output JSON data with more than 3 correct keys, unless you finetuned. However, the instruct models and recent 3.5/conversational RLHF are just so good at doing tasks, that such "engineering" - which supported/improved a wide base of prompts across domains and functionalities - broadly entirely stopped. Now prompt engineering sounds so cool and was also used in the way of specific use cases back then, that everybody uses the term for prompts to get certain outputs. This however, is not really the same level of engineering and to some expert, reminiscent of old-school knowledge system finesse, that we saw before 2022.
• Art9681 4y ago
  Communication is a highly valued skill that everyone thinks everyone else is bad at. But some people are clearly better than others. Describing a problem or abstracting a process into a prompt will become a highly valued skill that some people will be better at than others. I also think that it's not as simple as just writing a prompt and calling it engineering. If these jobs become commonplace in a professional capacity, I would assume that developing or being good at the tooling that will be built around LLMs will be part of it. The people that are really good at it will hook in their own systems such as LangChain, Microsoft Semantic Kernel, etc.
• j45 4y ago
  Prompting feels like it might be in part measured how social engineering skills are.
• mathgladiator 4y ago
  It's like learning the limits of the tech. For example, the evening I was messing around and asked it to give the philosophy of Conan and what makes life best. It pulled the right quote from the movie and then went on the deep end telling about how violence is bad and not fit for the real world. For me, prompt engineering was about getting it to stop moralizing and adding trigger warnings.
• sam0x17 4y ago
  I mean what is engineering though but trying to understand the unknown and come up with rules of thumb that work in practice
• kazinator 4y ago
  It is pure, unadulterated snake oil. You cannot create a prompt or sequence of prompts such that you can substitute some variable data into it, to generate responses of predictable with the modified prompts. GPT4 isn't promoted as offering that capability. The output of each chat must be evaluated on a case by case basis for its quality and suitability. If you want repeatable results, you get GPT4 to write you code. Then use that code. If you run into cases it doesn't handle, or have new ones, then you can revise the code, perhaps with the help of the AI once again. The problem is, that the bullshit artists out there want repeatable results in areas for which it can't write you working code. For instance, a problem like taking a textual product description and producing a glib advertisement. They want to be able to write a prompt where you can just substitute different product descriptions and have quality ads come out that nobody has to proofread (so they can be immediately served), just by "turning a crank".
• zqk6 4y ago
  10000000%
• gremlinsinc 4y ago
  I'm building a startup in this space between prompting and crafting agents, but for me using lang chain to create vector embeddings to enhance an LLMs memory and context is a core skill a prompt engineer should have. I've already considered myself a prompt engineer instead of a software engineer, because I've been coding 15 years and now 80 percent of the code is generated for me, so I really don't think I'm ever going to want to go back to engineering without the aid of ai. If you're a software engineer and you use chatGPT to help you code, you've switched careers from software to prompting.
• hackernoteng 4y ago
  Its hacking not engineering. But what helps is understanding a bit how the attention mech works and how your prompt effects the output. This "prompt engineering" is a vastly different programming paradigm. The notion of time is different, as output is being formed (one token at a time), the attention mechanism can focus more or less on different parts of the original prompt (along with the previous output tokens). This means that your "code", (your original prompt), is essentially getting "revisited" at different spots and different times while the output is getting generated and you are essentially "steering" the output in different directions based on the instructions in the prompt. This is different than how traditional code works which is always sequential and top-down while prompt writing, its better to think and speak like a human with human thoughts than to try to narrow things down to logic and code-like syntax. The LLM isnt going to process your prompt in a top-down one-time manner - its going to keep revisiting it (reweighing different parts of it, emphasising different parts of it) based on the output tokens chosen, and those weights will help determine the next token chosen, etc. Its like a whole new world of hacking, and it can be super frustrating but also amazingly effective.
• dbbk 4y ago
  I thought it was a joke?