3 ms·

Sorry, looking more, it doesn't seem like you are doing what you are saying. This is just poorly breaking text into bad chunks with no regard for semantics and

• stephantul 1y ago
  It doesn’t break text into chunks at all. These models can handle sequences of arbitrary length.
  • jasonjmcghee 1y ago
    I believe parent is referring to: https://github.com/MinishLab/model2vec-rs/blob/480ec988d7f4a41b333df92fbdea0e570e723844/src/main.rs#L36 https://github.com/MinishLab/model2vec-rs/blob/480ec988d7f4a... https://github.com/MinishLab/model2vec-rs/blob/480ec988d7f4a41b333df92fbdea0e570e723844/src/model.rs#L158 https://github.com/MinishLab/model2vec-rs/blob/480ec988d7f4a...
• Tananon 1y ago
  I think you are referring to for "batch in sentences.chunks(batch_size)"? This is not actually chunking sentences, chunks() is simply an iterator over a slice (in this case, a slice of all our input sentences of length batch_size). We don't have an actual constraint on input length. We truncate to 512 tokens by default, but you can easily set that to any amount by directly calling encode_with_args. There's an example in our quickstart: https://github.com/MinishLab/model2vec-rs/tree/main?tab=readme-ov-file#quick-start https://github.com/MinishLab/model2vec-rs/tree/main?tab=read....