4 ms·

Fixing this is the promise of enterprise deployments of LLMs, correct? That you can take the list of initial result texts from a shitty elasticsearch implementa

Fixing this is the promise of enterprise deployments of LLMs, correct? That you can take the list of initial result texts from a shitty elasticsearch implementation, pass it into an LLM with a prompt "you are a user looking for <query>, please use common sense to comb through these results and find a few good ones" and get better intramural search quality. The only other enterprise application I am aware of is applying the homework cheating app to customer facing roles that require email/copy generation.

regularjack 3y ago
Implementing search correctly would solve the problem too, without requiring a square kilometer of GPUs.
- Spivak 3y ago
  Right but implementing search correctly often means humans combing through the data and applying common sense to tag the products with their actual facets. So really you're just choosing between a square kilometer of office space for cheap labor or GPUs.
  - cratermoon 3y ago
    > you're just choosing between a square kilometer of office space for cheap labor or GPUs. No, with LLMs you're getting both, because a. LLMs need the human-curated data to ingest and b. after training the model is going to get RLHF. Also, a square kilometer of office space still uses much less electricity than a square kilometer of GPUs.
- pixl97 3y ago
  I'm guessing we're at or very near the point where LLMs could tag the data better and cheaper than humans.
- lukevp 3y ago
  If you have >100k items, “implementing search correctly” is an intractable problem if you don’t have a hundred person team of search relevance optimizers and taggers. Search is terrible until you gather tons of metadata about each item. Search is not nearly as simple as text matching within product titles. An LLM by comparison is peanuts. You could run it overnight and just have it build tags and metadata for all your items and it would probably be 75% accurate and a hell of a lot cheaper. Things can then be fine tuned by people later.
  - lightbendover 3y ago
    At Amazon scale that last x% is a multi-decade problem and still going. It’s intractable full stop.

regularjack 3y ago

Implementing search correctly would solve the problem too, without requiring a square kilometer of GPUs.

Spivak 3y ago
Right but implementing search correctly often means humans combing through the data and applying common sense to tag the products with their actual facets. So really you're just choosing between a square kilometer of office space for cheap labor or GPUs.
- cratermoon 3y ago
  > you're just choosing between a square kilometer of office space for cheap labor or GPUs. No, with LLMs you're getting both, because a. LLMs need the human-curated data to ingest and b. after training the model is going to get RLHF. Also, a square kilometer of office space still uses much less electricity than a square kilometer of GPUs.
pixl97 3y ago
I'm guessing we're at or very near the point where LLMs could tag the data better and cheaper than humans.
lukevp 3y ago
If you have >100k items, “implementing search correctly” is an intractable problem if you don’t have a hundred person team of search relevance optimizers and taggers. Search is terrible until you gather tons of metadata about each item. Search is not nearly as simple as text matching within product titles. An LLM by comparison is peanuts. You could run it overnight and just have it build tags and metadata for all your items and it would probably be 75% accurate and a hell of a lot cheaper. Things can then be fine tuned by people later.
- lightbendover 3y ago
  At Amazon scale that last x% is a multi-decade problem and still going. It’s intractable full stop.