5 ms·

> We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on t

by momojo 2y ago

> We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs. This part interests me the most. I want to know how small yet functional we can get these models. I don't want an AI that can solve calculus, I just want a dumb AI that pretty consistently recognizes "lights off" and "lights on".

MobiusHorizons 2y ago
why would you use an LLM for that? Seems like there are much better options available.
- londons_explore 2y ago
  It's actually pretty hard to design a non-llm system that can detect all the possible variations: Lights on. Brighter please. Turn on the light. Is there light in here? Turn the light on. Table lamp: on. Does the desk lamp work? It's a bit dim here, anything you can do? More light please. Put the lights on for the next 5 mins. Turn the light on when I come home. Turn all the lights off together. Switch the lights off whenever its daytime or quiet at home unless I say otherwise. etc. If you don't support every possible way of saying a command, then users will get frustrated because they effectively have to go and learn the magic incantation of words for every possible action, which is very user-unfriendly.
  - anon373839 2y ago
    I suspect ModernBERT can also be very helpful with these sorts of tasks, if you decompose them into an intent classification step and a named entity recognition step.
    - simcop2387 2y ago
      that entity extraction is where it actually gets really really difficult, even for LLMs since people will use 10 different names for the same thing and you'll have to know them ahead of time to handle them all properly. For either BERT based or llm based there's a bit of a need for the system to try to correct and learn those new names unless you require users to put them all in ahead of time. That said i've seen LLMs handle this a lot better with a list of aliases in the prompt for each room and then type of device when playing with home assistant + llm.
      - ted_dunning 2y ago
        It isn't unreasonable to imagine one recognizable intent to be teaching new terminology. That would allow dialogs where the machine doesn't understand the object of the command and the human then says something like "when I say x, I mean the y" and the computer updates the training set for the named entity recognizer and does a quick fine-tuning pass.
  - phkahler 2y ago
    Your examples include complex instructions and questions, but for simple ON/OFF commands you can go far by pulling key words and ignoring sentence structure. For example, pick out "on" "off" and "light" will work for "turn the light on", "turn off the light", "light on", "I want the light on", etc... Adding modifiers like "kitchen" or "all" can help specify which lights (your "Table lamp: on" example), regardless of how they're used. I'm not saying this a great solution, but it covers pretty much all the basic variations for simple commands and can run on anything.
    - astrange 2y ago
      Original Siri used a primitive sentence embedding and then nearest-neighbor of the vector to supported commands.
  - MobiusHorizons 2y ago
    That is certainly a hard problem, but how do you know such a system is better than simpler command based ones? Systems like you describe have a much higher chance of taking the wrong action. Even humans do, when given ambiguous instructions. I know everyone loves to hate on Siri because it doesn’t do a good job of understanding anything complex, but I have always found it to be very reliable when you find the right command. As a result I use it much more often than google assistant (was team android/ pixel until 2020). I especially use it for timers, reminders, and navigation, and if my hands are not free, texts. Taking the wrong action breeds distrust, which I also think is not user friendly.

by momojo 2y ago

MobiusHorizons 2y ago

why would you use an LLM for that? Seems like there are much better options available.

londons_explore 2y ago
It's actually pretty hard to design a non-llm system that can detect all the possible variations: Lights on. Brighter please. Turn on the light. Is there light in here? Turn the light on. Table lamp: on. Does the desk lamp work? It's a bit dim here, anything you can do? More light please. Put the lights on for the next 5 mins. Turn the light on when I come home. Turn all the lights off together. Switch the lights off whenever its daytime or quiet at home unless I say otherwise. etc. If you don't support every possible way of saying a command, then users will get frustrated because they effectively have to go and learn the magic incantation of words for every possible action, which is very user-unfriendly.
- anon373839 2y ago
  I suspect ModernBERT can also be very helpful with these sorts of tasks, if you decompose them into an intent classification step and a named entity recognition step.
  - simcop2387 2y ago
    that entity extraction is where it actually gets really really difficult, even for LLMs since people will use 10 different names for the same thing and you'll have to know them ahead of time to handle them all properly. For either BERT based or llm based there's a bit of a need for the system to try to correct and learn those new names unless you require users to put them all in ahead of time. That said i've seen LLMs handle this a lot better with a list of aliases in the prompt for each room and then type of device when playing with home assistant + llm.
    - ted_dunning 2y ago
      It isn't unreasonable to imagine one recognizable intent to be teaching new terminology. That would allow dialogs where the machine doesn't understand the object of the command and the human then says something like "when I say x, I mean the y" and the computer updates the training set for the named entity recognizer and does a quick fine-tuning pass.
- phkahler 2y ago
  Your examples include complex instructions and questions, but for simple ON/OFF commands you can go far by pulling key words and ignoring sentence structure. For example, pick out "on" "off" and "light" will work for "turn the light on", "turn off the light", "light on", "I want the light on", etc... Adding modifiers like "kitchen" or "all" can help specify which lights (your "Table lamp: on" example), regardless of how they're used. I'm not saying this a great solution, but it covers pretty much all the basic variations for simple commands and can run on anything.
  - astrange 2y ago
    Original Siri used a primitive sentence embedding and then nearest-neighbor of the vector to supported commands.
- MobiusHorizons 2y ago
  That is certainly a hard problem, but how do you know such a system is better than simpler command based ones? Systems like you describe have a much higher chance of taking the wrong action. Even humans do, when given ambiguous instructions. I know everyone loves to hate on Siri because it doesn’t do a good job of understanding anything complex, but I have always found it to be very reliable when you find the right command. As a result I use it much more often than google assistant (was team android/ pixel until 2020). I especially use it for timers, reminders, and navigation, and if my hands are not free, texts. Taking the wrong action breeds distrust, which I also think is not user friendly.