4 ms·

Thanks for the reference to haystack. I didn't know it existed! I was looking into huggingface that seems to allow to build your own language model and train (s

by mrg3_2013 5y ago

Thanks for the reference to haystack. I didn't know it existed! I was looking into huggingface that seems to allow to build your own language model and train (still learning - but thats what I've learnt so far). I don't know how expensive these get (for example, if you have 100K lines?). Any thoughts on how this compares to HuggingFace and any anecdotes on time it would take to custom train ?

m3at 5y ago
Not the author but I can answer that! Haystack does not really compare to huggingface as they are directly using it as a library, it's more of a layer on top. Serving a search system for 100K lines (short document?) is trivial on a single CPU machine, so the costs will be low. For training if you stick to fine tuning and your dataset is relatively small (<10M) you could use collab with a free gpu. Training time really depends on your use case, the simplest classification may be handled in a couple of hours.
- mrg3_2013 5y ago
  Thank you @m3at!