4 ms·
Glad to finally share this release! I'll paste in something I shared on Twitter, about one of the features I think some will find surprising (at least, in the s
by syllogism 6y ago
Glad to finally share this release! I'll paste in something I shared on Twitter, about one of the features I think some will find surprising (at least, in the sense I might have been skeptical about it if it had been pitched to me a year ago). The twitter thread is here: https://twitter.com/honnibal/status/1316792607205470209 https://twitter.com/honnibal/status/1316792607205470209
This release was SO much work! Glad to finally have it out.
:beer:
The big impact for users will definitely be the transformer models and config system, but I want to talk about a feature I wasn't expecting to build until a few months ago: the new workflow system, spaCy Projects. :thread:
spaCy Projects was inspired by @DVCorg
, and has an easy integration for DVC users. But it's also standalone: you can write a single YML file and spaCy will get your data, trigger your processing steps, and use remote cache. It even generates readmes: https://t.co/uRcfJWZlsQ?amp=1 https://t.co/uRcfJWZlsQ?amp=1
The requirements that spaCy Projects handles are more general than spaCy. So why put it in the library? The case against it -- simple minimalism -- is pretty clear. The case for it is a bit more subtle.
We built spaCy Projects for the design-space it opens up for the rest of the library. If we can assume a workflow system (by giving it to you!), it's much easier to let things happen in separate steps. We can embrace approaches that would otherwise be awkward.
A good example is the way we've stream-lined the training utilities. spaCy used to have this awkward class, GoldParse. That's gone now: the training annotations are represented with the Doc class, and the training data is read in from DocBin format -- which is about 100x smaller.
Why didn't we ditch GoldParse sooner? Well, there's definitely a downside to reading data from a binary format. It adds an extra conversion step. But with the projects system, that's no longer such a big deal. We now have a conventional way to describe such multi-step workflows.
I think the impact will be especially big for the ecosystem. It's extremely standard in NLP to have a bunch of steps to run to build counts, lists, models etc. Previously every spaCy Universe project had to build and describe their own conventions. Now we have a standard.