3 ms·

We're doing a successor to this, working hard and going public in month or so, hopefully. But HN gets a preview of course: https://huggingface.co/datasets/lenna

by fi-le 2y ago

We're doing a successor to this, working hard and going public in month or so, hopefully. But HN gets a preview of course: https://huggingface.co/datasets/lennart-finke/SimpleStories https://huggingface.co/datasets/lennart-finke/SimpleStories And here's a more interactive explorer: https://fi-le.net/simplestories https://fi-le.net/simplestories

jmward01 2y ago
This looks like a great dataset! Thanks for posting. I'm looking for projects just like this to try my training modifications against. Do you have initial results shown? It is a small model/dataset so training the gpt2 model in the repo probably wouldn't be too hard but it would be good to have reference runs to make sure things are set up right when I run it.
- fi-le 2y ago
  So glad you like it! If I understand your question correctly, yes, we are also putting together a small library for training small language models. It's not mature at all yet, but can keep up with our progress here: https://github.com/danbraunai/simple_stories_train https://github.com/danbraunai/simple_stories_train
  - jmward01 2y ago
    Yeah. I looked at the dataset and there are a lot of possible tasks you could train against here since it has some great annotations. So, having a simple reference baseline, like a pretrain gpt2 run (which I think your repo is set up to do), helps give a starting point for other work. It looks like the dataset is small enough and the gpt2 ref code in your repo is lightweight enough to do a quick run and plot some curves on. Thanks!
coder543 2y ago
Does template_plural actually work well / offer any benefits?
- fi-le 2y ago
  It does, we use it as a default. Some possible benefits are that 1) it saves input tokens 2) in theory allows for different variations on a theme, whereas with two seperate prompts you run the risk of repeating one topic.

3 ms·

We're doing a successor to this, working hard and going public in month or so, hopefully. But HN gets a preview of course: https://huggingface.co/datasets/lenna

by fi-le 2y ago

jmward01 2y ago
This looks like a great dataset! Thanks for posting. I'm looking for projects just like this to try my training modifications against. Do you have initial results shown? It is a small model/dataset so training the gpt2 model in the repo probably wouldn't be too hard but it would be good to have reference runs to make sure things are set up right when I run it.
- fi-le 2y ago
  So glad you like it! If I understand your question correctly, yes, we are also putting together a small library for training small language models. It's not mature at all yet, but can keep up with our progress here: https://github.com/danbraunai/simple_stories_train https://github.com/danbraunai/simple_stories_train
  - jmward01 2y ago
    Yeah. I looked at the dataset and there are a lot of possible tasks you could train against here since it has some great annotations. So, having a simple reference baseline, like a pretrain gpt2 run (which I think your repo is set up to do), helps give a starting point for other work. It looks like the dataset is small enough and the gpt2 ref code in your repo is lightweight enough to do a quick run and plot some curves on. Thanks!
coder543 2y ago
Does template_plural actually work well / offer any benefits?
- fi-le 2y ago
  It does, we use it as a default. Some possible benefits are that 1) it saves input tokens 2) in theory allows for different variations on a theme, whereas with two seperate prompts you run the risk of repeating one topic.