4 ms·

Ok so I’ve evaluated spacy a few years ago, but nowadays we’re using huggingface’s transformers / tokenizers / etc to train our own language models + fine tuned

Ok so I’ve evaluated spacy a few years ago, but nowadays we’re using huggingface’s transformers / tokenizers / etc to train our own language models + fine tuned models. I see there’s now transformer based pipeline support, how do the two relate? Phrased differently, how does spacy fit in with today’s world of transformers? Would it still be interesting for me?

binarymax 6y ago
I have lots of experience with both, and I use both together for different use cases. SpaCy fills the need of predictable/explainable pattern matching and NER - and is very fast and reasonably accurate on a CPU. Huggingface fills the need for task based prediction when you have a GPU.
- danieldk 6y ago
  Huggingface fills the need for task based prediction when you have a GPU. With model distillation, you can make models that annotate hundreds of sentences per second on a single CPU with a library like Huggingface Transformers. For instance, one of my distilled Dutch multi-task syntax models (UD POS, language-specific POS, lemmatization, morphology, dependency parsing) annotates 316 sentences per second with 4 threads on a Ryzen 3700X. This distilled model has virtually no loss in accuracy compared to the finetuned XLM-RoBERTa base model. I don't use Huggingface Transformers, but ported some of their implementations to Rust [1], but that should not make a big difference since all the heavy lifting happens in C++ in libtorch anyway. tl;dr: it is not true that tranformers are only useful for GPU prediction. You can get high CPU prediction speeds with some tricks (distillation, length-based bucketing in batches, using MKL, etc.). [1] https://github.com/tensordot/syntaxdot/tree/main/syntaxdot-transformers/src/models https://github.com/tensordot/syntaxdot/tree/main/syntaxdot-t...
  - binarymax 6y ago
    Interesting. Did you start from a Distilled base model (like DistilRoBerta), or did you distill your fine-tuned model?
    - danieldk 6y ago
      Sorry for the late reply. I distilled from my own finetuned XLM-RoBERTa model.
  - aabhay 6y ago
    Is there a standard template for creating a distilled model? I didn’t see a public hugging face implementation, just the models
syllogism 6y ago
The improved transformers support is definitely one of the main features of the release. I'm also really pleased with how the project system and config files work. If you're always working with exactly one task model, I think working directly in transformers isn't that different from using spaCy. But if you're orchestrating multiple models, spaCy's pipeline components and Doc object will probably be helpful. A feature in v3 that I think will be particularly useful is the ability to share a transformer model between multiple components, for instance you can have an entity recogniser, text classifier and tagger all using the same transformer, and all backpropagating to it. You also might find the projects system useful if you're training a lot of models. For instance, take a look at the project repo here: https://github.com/explosion/projects/tree/v3/benchmarks/ner_conll03 https://github.com/explosion/projects/tree/v3/benchmarks/ner.... Most of the readme there is actually generated from the project.yml file, which fully specifies the preprocessing steps you need to build the project from the source assets. The project system can also push and pull intermediate or final artifacts to a remote cache, such as an S3 bucket, with the addressing of the artifacts calculated based on hashes of the inputs and the file itself. The config file is comprehensive and extensible. The blocks refer to typed functions that you can specify yourself, so you can substitute any of your own layer (or other) functions in, to change some part of the system's behaviour. You don't _have_ to specify your models from the config files like this --- you can instead put it together in code. But the config system means there's a way of fully specifying a pipeline and all of the training settings, which means you can really standardise your training machinery. Overall the theme of what we're doing is helping you to line up the workflows you use during development with something you can actually ship. We think one of the problems for ML engineers is that there's quite a gap between how people are iterating in their local dev environment (notebooks, scrappy directories etc) and getting the project into a state that you can get other people working on, try out in automation, and then pilot in some sort of soft production (e.g. directing a small amount of traffic to the model). The problem with iterating in the local state is that you're running the model against benchmarks that are not real, and you hit diminishing returns quite quickly this way. It also introduces a lot of rework. All that said, there will definitely be usage contexts where it's not worth introducing another technology. For instance, if your main goal is to develop a model, run an experiment and publish a paper, you might find spaCy doesn't do much that makes your life easier.

binarymax 6y ago

I have lots of experience with both, and I use both together for different use cases. SpaCy fills the need of predictable/explainable pattern matching and NER - and is very fast and reasonably accurate on a CPU. Huggingface fills the need for task based prediction when you have a GPU.

danieldk 6y ago
Huggingface fills the need for task based prediction when you have a GPU. With model distillation, you can make models that annotate hundreds of sentences per second on a single CPU with a library like Huggingface Transformers. For instance, one of my distilled Dutch multi-task syntax models (UD POS, language-specific POS, lemmatization, morphology, dependency parsing) annotates 316 sentences per second with 4 threads on a Ryzen 3700X. This distilled model has virtually no loss in accuracy compared to the finetuned XLM-RoBERTa base model. I don't use Huggingface Transformers, but ported some of their implementations to Rust [1], but that should not make a big difference since all the heavy lifting happens in C++ in libtorch anyway. tl;dr: it is not true that tranformers are only useful for GPU prediction. You can get high CPU prediction speeds with some tricks (distillation, length-based bucketing in batches, using MKL, etc.). [1] https://github.com/tensordot/syntaxdot/tree/main/syntaxdot-transformers/src/models https://github.com/tensordot/syntaxdot/tree/main/syntaxdot-t...
- binarymax 6y ago
  Interesting. Did you start from a Distilled base model (like DistilRoBerta), or did you distill your fine-tuned model?
  - danieldk 6y ago
    Sorry for the late reply. I distilled from my own finetuned XLM-RoBERTa model.
- aabhay 6y ago
  Is there a standard template for creating a distilled model? I didn’t see a public hugging face implementation, just the models

syllogism 6y ago

The improved transformers support is definitely one of the main features of the release. I'm also really pleased with how the project system and config files work. If you're always working with exactly one task model, I think working directly in transformers isn't that different from using spaCy. But if you're orchestrating multiple models, spaCy's pipeline components and Doc object will probably be helpful. A feature in v3 that I think will be particularly useful is the ability to share a transformer model between multiple components, for instance you can have an entity recogniser, text classifier and tagger all using the same transformer, and all backpropagating to it. You also might find the projects system useful if you're training a lot of models. For instance, take a look at the project repo here: https://github.com/explosion/projects/tree/v3/benchmarks/ner_conll03 https://github.com/explosion/projects/tree/v3/benchmarks/ner.... Most of the readme there is actually generated from the project.yml file, which fully specifies the preprocessing steps you need to build the project from the source assets. The project system can also push and pull intermediate or final artifacts to a remote cache, such as an S3 bucket, with the addressing of the artifacts calculated based on hashes of the inputs and the file itself. The config file is comprehensive and extensible. The blocks refer to typed functions that you can specify yourself, so you can substitute any of your own layer (or other) functions in, to change some part of the system's behaviour. You don't _have_ to specify your models from the config files like this --- you can instead put it together in code. But the config system means there's a way of fully specifying a pipeline and all of the training settings, which means you can really standardise your training machinery. Overall the theme of what we're doing is helping you to line up the workflows you use during development with something you can actually ship. We think one of the problems for ML engineers is that there's quite a gap between how people are iterating in their local dev environment (notebooks, scrappy directories etc) and getting the project into a state that you can get other people working on, try out in automation, and then pilot in some sort of soft production (e.g. directing a small amount of traffic to the model). The problem with iterating in the local state is that you're running the model against benchmarks that are not real, and you hit diminishing returns quite quickly this way. It also introduces a lot of rework. All that said, there will definitely be usage contexts where it's not worth introducing another technology. For instance, if your main goal is to develop a model, run an experiment and publish a paper, you might find spaCy doesn't do much that makes your life easier.