4 ms·
The two most interesting things to me in a "minimal" framework would be eliminating dependency on HF transformers and helping customize chunking. Not a knock a
by andy99 3y ago
The two most interesting things to me in a "minimal" framework would be eliminating dependency on HF transformers and helping customize chunking.
Not a knock against this project, I see where it can be helpful.
- dmezzetti 3y agoMy assumption is that this gives you the ability to locally encode vectors. This is useful for those not using API services to build their vectors.
- andy99 3y agoTransformer inference is ~60 lines of numpy[0] (closer to 500 once you add tokenization etc). It would be nice to just have this and not all of pytorch and transformers. [0] https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/
- dmezzetti 3y agoWhat about models besides GPT? Most of the popular vector encoding models aren't using this architecture. If you really didn't want PyTorch/Transformers, you could consider exporting your models to ONNX (https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime).
- cherryteastain 3y agoIt's 60 lines for CPU only inference, which'll be slow. If you want GPU acceleration it'll be a lot more than 60 lines.
- jsimian 3y agoYeah, I just tried this out (props to the devs, super easy to set up) and my main gripe is the chunking algorithms aren't great - could be alot more useful with a context option that gives surrounding results to search results. The sliding window chunking method always cuts off the start of sentences.
- andy99 3y agoI've found it works better to chunk by some logical sections in the document, e.g header h2 h3 h4 etc or 1.1 1.1.1 ... plus to be able to ignore some stuff (header and footer) plus other customizations. At least for use cases where there are clusters of many similarly formatted documents, it would be cool to have a way of easily customizing chunking.