3 ms·

Is this not actually kind of powerful? Having linguists write up a bunch of rules seems a lot more predictable than "rolling a bunch of dice and hoping that som

by rtpg 2y ago

Is this not actually kind of powerful? Having linguists write up a bunch of rules seems a lot more predictable than "rolling a bunch of dice and hoping that some LLM spits out a coherent set of steps". It feels very fractal but on the other hand if Alexa has only a specific gamut of responses it's not exactly a limitless state space right? Very curious about how those rules look like though

factormeta 2y ago
wouldn't that just be some kind of NLP https://en.wikipedia.org/wiki/Natural_language_processing https://en.wikipedia.org/wiki/Natural_language_processing? May just be a long list of if/else and/or switch statements or isomorphism.
ihkasfjdkabnsk 2y ago
Honestly the FSTs themselves were actually really cool, it's very much GOFAI. It automatically creates lots of permutations, i.e. `play taylor swift`, `please play taylor swift`, play taylor swift now`. etc. And once the FST is built it always works deterministically. It's compiled to a graph and an incoming command is pushed through the state machine, if you get to an end state it "matched the fst" and some specific behaviour would be triggered. the rule were really just strings and we had efficient matching against it. I didn't work on that, I would assume some sort of LHS.
- phito 2y ago
  what do all these acronyms mean
  - 39896880 2y ago
    https://en.wikipedia.org/wiki/Finite-state_transducer https://en.wikipedia.org/wiki/Finite-state_transducer https://medium.com/fetch-ai/classical-roots-of-ai-unpacking-gofai-1afce52de53b#:~:text=The%20term%20GOFAI%2C%20introduced%20by,methods%20to%20mimic%20intelligent%20behavior https://medium.com/fetch-ai/classical-roots-of-ai-unpacking-....
IshKebab 2y ago
The problem is it's completely undiscoverable. You can tell Alexa "play some music" because you're pretty sure one of these linguists added a rule for that. But can you tell it "play me a song that lasts longer than 5 minutes"? Doubtful. The only way to know is to try it. The problem is the space of possible commands is waaaaay bigger than the space of commands you can manually handle, which means if you just randomly try stuff 95% of the time it won't work. Users learn that very quickly and end up sticking to the few commands they know work. The one exception is "search" queries - "how tall is Everest" and so on, but that only really works well on Google's platform because they've done all the work for that already. Contrast that with LLMs which basically at least understand everything you're asking of them. If you give them a simple API to carry out actions they can do really complex commands like "send a WhatsApp to my wife telling her how when I'll get home if I start cycling in 10 minutes". That's impossible without LLMs but pretty trivial with them. Obviously the downside is they are prone to bullshitting and might do completely the wrong thing.
- 39896880 2y ago
  It’s worse than that. These systems can be adapted by looking at failed user commands, but people don’t really sit around trying out fun things and watch it fall on its face for longer than the first day or so. After that, the novelty wears off, so you’ve trained your users to accept the device’s limitations. Then, even when you do improve the functionality, your users won’t know! They won’t try it, and those commands will never get traction in the system or get more testing beyond the initial launch criteria. It’s a death spiral. The same thing happens with the tone of voice people use.
  - mycall 2y ago
    > people don’t really sit around trying out fun things and watch it fall on its face for longer than the first day or so. Isn't this what thumbs-up/down RL is for? To improve the quality of the results.
    - 39896880 2y ago
      That’s the intention but very few users enjoy being unpaid QA for trillion dollar corps.

by rtpg 2y ago

factormeta 2y ago

wouldn't that just be some kind of NLP https://en.wikipedia.org/wiki/Natural_language_processing https://en.wikipedia.org/wiki/Natural_language_processing? May just be a long list of if/else and/or switch statements or isomorphism.

ihkasfjdkabnsk 2y ago

Honestly the FSTs themselves were actually really cool, it's very much GOFAI. It automatically creates lots of permutations, i.e. `play taylor swift`, `please play taylor swift`, play taylor swift now`. etc. And once the FST is built it always works deterministically. It's compiled to a graph and an incoming command is pushed through the state machine, if you get to an end state it "matched the fst" and some specific behaviour would be triggered. the rule were really just strings and we had efficient matching against it. I didn't work on that, I would assume some sort of LHS.

phito 2y ago
what do all these acronyms mean
- 39896880 2y ago
  https://en.wikipedia.org/wiki/Finite-state_transducer https://en.wikipedia.org/wiki/Finite-state_transducer https://medium.com/fetch-ai/classical-roots-of-ai-unpacking-gofai-1afce52de53b#:~:text=The%20term%20GOFAI%2C%20introduced%20by,methods%20to%20mimic%20intelligent%20behavior https://medium.com/fetch-ai/classical-roots-of-ai-unpacking-....

IshKebab 2y ago

The problem is it's completely undiscoverable. You can tell Alexa "play some music" because you're pretty sure one of these linguists added a rule for that. But can you tell it "play me a song that lasts longer than 5 minutes"? Doubtful. The only way to know is to try it. The problem is the space of possible commands is waaaaay bigger than the space of commands you can manually handle, which means if you just randomly try stuff 95% of the time it won't work. Users learn that very quickly and end up sticking to the few commands they know work. The one exception is "search" queries - "how tall is Everest" and so on, but that only really works well on Google's platform because they've done all the work for that already. Contrast that with LLMs which basically at least understand everything you're asking of them. If you give them a simple API to carry out actions they can do really complex commands like "send a WhatsApp to my wife telling her how when I'll get home if I start cycling in 10 minutes". That's impossible without LLMs but pretty trivial with them. Obviously the downside is they are prone to bullshitting and might do completely the wrong thing.

39896880 2y ago
It’s worse than that. These systems can be adapted by looking at failed user commands, but people don’t really sit around trying out fun things and watch it fall on its face for longer than the first day or so. After that, the novelty wears off, so you’ve trained your users to accept the device’s limitations. Then, even when you do improve the functionality, your users won’t know! They won’t try it, and those commands will never get traction in the system or get more testing beyond the initial launch criteria. It’s a death spiral. The same thing happens with the tone of voice people use.
- mycall 2y ago
  > people don’t really sit around trying out fun things and watch it fall on its face for longer than the first day or so. Isn't this what thumbs-up/down RL is for? To improve the quality of the results.
  - 39896880 2y ago
    That’s the intention but very few users enjoy being unpaid QA for trillion dollar corps.