3 ms·

Prompt engineering isn't as simple as writing prompts in english. It's still engineering data flow, when data is relevant, systems that the AI can access and se

Prompt engineering isn't as simple as writing prompts in english. It's still engineering data flow, when data is relevant, systems that the AI can access and search, tools that the AI can use, etc.

drowsspa 1y ago
Is it, though? Apparently the current best practice is just to allow the LLM untethered access to everything and try to control access by preventing prompt injection...
- mettamage 1y ago
  You still need to find the correct data, and get it to the LLM. IMO, a lot of it is data engineering work with API calls to an LLM as an extra step. I'm currently doing a lot of ETL work with Airflow (and whatever data {warehouses, lakes, bases} are needed) to get the right data to a prompt engineering flow. The prompt engineering flow is literally a for loop of Google Docs in a Google Drive that non-tech people, but domain experts in their field, can access. It's up to the domain experts and me to understand where giving it data will tone down the hallucinative nonsense an LLM puts out, and where we should not give data because we need the problem solving skills of the LLM itself. A similar process is for tool-use, which in our case are pre-selected Python scripts that it is allowed to run.
  - apwell23 1y ago
    can you describe what the usecase is ?
- kgeist 1y ago
  Well it took me 2 full-time weeks to properly implement a RAG-based system so that it found actually relevant data and did not hallucinate. Had to: - write an evaluation pipeline to automate quality testing - add a query rewriting step to explore more options during search - add hybrid BM-25+vector search with proper rank fusion - tune all the hyperparameters for best results (like weight bias for bm25 vs. vector, how many documents to retrieve for analysis, how to chunk documents based on semantics) - parallelize the search pipeline to decrease wait times - add moderation - add a reranker to find best candidates - add background embedding calculation of user documents - lots of failure cases to iron out so that the prompt worked for most cases There's no "just give LLM all the data", it's more complex than that, especially if you want best results and also full control of data (we run all of that using open source models because user data is under NDA)
  - altcognito 1y ago
    whoa, two weeks
    - rynn 1y ago
      @apwell23 while the author didn’t say how s/he measured QA, creating the QA process was literally the first bullet.
  - deleted 1y ago
    [deleted]
  - mbesto 1y ago
    "did not hallucinate" Sorry to nitpick, but this is not technically possible no matter how much RAG you throw at it. I assume you just mean "hallucinates a lot less"
    - kgeist 1y ago
      You're right, bad wording
  - saberience 1y ago
    Sounds like you vibe coded a RAG system in two weeks, which isn't very hard. Any startup can do it. I've debugged single difficult bugs before for two weeks, a whole feature that takes two weeks is an easy feature to build.
    - kgeist 1y ago
      I already had experience with RAG before so I had a head start. You're right that it's not rocket science, but it's not just "press F to implement the feature" either P.S. No vibe coding was used. I only used LLM-as-a-judge to automate quality testing when tuning the parameters, before passing it to human QA
lomase 1y ago
Imagine you are a top of the line engenier... Engineering data flow... sure, we all like to use big words.
- goostavos 1y ago
  The new 10x engineering is writing "please don't write bugs" in a markdown file.
dustbunny 1y ago
Human speech is "engineering data flow" Painting is "engineering data flow" Directing a movie is "engineering data flow" Playing the guitar is "engineering data flow" This statement merely reveals a bias to apply high value to the word "engineering" and to the identity "engineer". Ironic in that silicon valley lifted that identity and it's not even legally recognized as a licensed profession.
nradov 1y ago
Nah. There's no such thing as prompt engineering. It doesn't exist. Engineering involves applying scientific principles to solve real world problems. There are no clear scientific principles to apply here. It's all instinct, hunches, educated guesses, and heuristics with maybe some sort of feedback loop. And that's fine, it can still produce useful results. Just don't call it engineering. Maybe artisanal prompt crafting? Or prompt alchemy?
- exmicrosoldier 1y ago
  Prompt engineering is the new Search Engine Optimization. Not sure if we called it engineering ten years ago.

Prompt engineering isn't as simple as writing prompts in english. It's still engineering data flow, when data is relevant, systems that the AI can access and search, tools that the AI can use, etc.

drowsspa 1y ago

Is it, though? Apparently the current best practice is just to allow the LLM untethered access to everything and try to control access by preventing prompt injection...

mettamage 1y ago
You still need to find the correct data, and get it to the LLM. IMO, a lot of it is data engineering work with API calls to an LLM as an extra step. I'm currently doing a lot of ETL work with Airflow (and whatever data {warehouses, lakes, bases} are needed) to get the right data to a prompt engineering flow. The prompt engineering flow is literally a for loop of Google Docs in a Google Drive that non-tech people, but domain experts in their field, can access. It's up to the domain experts and me to understand where giving it data will tone down the hallucinative nonsense an LLM puts out, and where we should not give data because we need the problem solving skills of the LLM itself. A similar process is for tool-use, which in our case are pre-selected Python scripts that it is allowed to run.
- apwell23 1y ago
  can you describe what the usecase is ?
kgeist 1y ago
Well it took me 2 full-time weeks to properly implement a RAG-based system so that it found actually relevant data and did not hallucinate. Had to: - write an evaluation pipeline to automate quality testing - add a query rewriting step to explore more options during search - add hybrid BM-25+vector search with proper rank fusion - tune all the hyperparameters for best results (like weight bias for bm25 vs. vector, how many documents to retrieve for analysis, how to chunk documents based on semantics) - parallelize the search pipeline to decrease wait times - add moderation - add a reranker to find best candidates - add background embedding calculation of user documents - lots of failure cases to iron out so that the prompt worked for most cases There's no "just give LLM all the data", it's more complex than that, especially if you want best results and also full control of data (we run all of that using open source models because user data is under NDA)
- altcognito 1y ago
  whoa, two weeks
  - rynn 1y ago
    @apwell23 while the author didn’t say how s/he measured QA, creating the QA process was literally the first bullet.
- deleted 1y ago
  [deleted]
- mbesto 1y ago
  "did not hallucinate" Sorry to nitpick, but this is not technically possible no matter how much RAG you throw at it. I assume you just mean "hallucinates a lot less"
  - kgeist 1y ago
    You're right, bad wording
- saberience 1y ago
  Sounds like you vibe coded a RAG system in two weeks, which isn't very hard. Any startup can do it. I've debugged single difficult bugs before for two weeks, a whole feature that takes two weeks is an easy feature to build.
  - kgeist 1y ago
    I already had experience with RAG before so I had a head start. You're right that it's not rocket science, but it's not just "press F to implement the feature" either P.S. No vibe coding was used. I only used LLM-as-a-judge to automate quality testing when tuning the parameters, before passing it to human QA

lomase 1y ago

Imagine you are a top of the line engenier... Engineering data flow... sure, we all like to use big words.

goostavos 1y ago
The new 10x engineering is writing "please don't write bugs" in a markdown file.

dustbunny 1y ago

Human speech is "engineering data flow" Painting is "engineering data flow" Directing a movie is "engineering data flow" Playing the guitar is "engineering data flow" This statement merely reveals a bias to apply high value to the word "engineering" and to the identity "engineer". Ironic in that silicon valley lifted that identity and it's not even legally recognized as a licensed profession.

nradov 1y ago

Nah. There's no such thing as prompt engineering. It doesn't exist. Engineering involves applying scientific principles to solve real world problems. There are no clear scientific principles to apply here. It's all instinct, hunches, educated guesses, and heuristics with maybe some sort of feedback loop. And that's fine, it can still produce useful results. Just don't call it engineering. Maybe artisanal prompt crafting? Or prompt alchemy?

exmicrosoldier 1y ago
Prompt engineering is the new Search Engine Optimization. Not sure if we called it engineering ten years ago.