4 ms·
I've been playing a bit with stacking transformer adapters to add knowledge to models and so far it has met my needs. It doesn't have the same illusion of intel
by batch12 3y ago
I've been playing a bit with stacking transformer adapters to add knowledge to models and so far it has met my needs. It doesn't have the same illusion of intelligence, but so far it's just as good as a multitasking intern, so I am still having fun with it. I wonder if this is basically doing the same thing.
- leobg 3y agoInteresting. Do you know if this can be done with Sentence Transformers, too? Picking a good performing one from HF. Then training an adapter for the domain (unsupervised). Then adding another one using actual training triplets (base, similar, non-similar)?
- batch12 3y agoI haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers. Check out https://github.com/huggingface/peft https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA (https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf) That should get you started.
- leobg 3y agoThank you. Peft and adapters seem to be two different things though, no? AFAIK there are other libraries for adapters (forgot the name). Is peft what you were talking about when you said adapters in your original comment?
- batch12 3y agoI was of the understanding that Lora was one flavor of adapters, but I am still learning so I may be wrong. I yet gotten too deep into other transformer adapters yet (still reading).