3 ms·
Neat how do you compare yourself on the annotation capabilities with Datasaur.ai which launched in the last YC batch? In terms of training the models for deplo
by Grimm1 6y ago
Neat how do you compare yourself on the annotation capabilities with Datasaur.ai which launched in the last YC batch?
In terms of training the models for deployment -- do we own the artifact? Can I move that into my own model repository?
Also how do you feel this compares to using fine tuning on a publicly available BERT family model which is already fairly fast and easy not requiring a huge corpus, speaking from experience of recently having done so?
Are the benefits more from the tight feedback loop and already standing infrastructure?
- jordn 6y agoAll great questions! Datasaur are great. I hope Ivan would think it's fair that I'd describe their current product as as a modern, cloud-hosted Brat (https://brat.nlplab.org/ https://brat.nlplab.org/ – this remains very popular!) with the features to make that work with teams. As you point out we're focusing on the tight integration of annotation and training enabling you to move faster and iterate on NLP ideas... essentially trying for move a waterfall ML lifecycle to a an agile one. Fine tuning on BERT is the way to go. It's what we do, and that already reduces the data annotation requirements by an order of magnitude. Doing that offline in a notebook is still wanted by some (you can use our tool just as the annotation platform, and download the data and you'll still get the efficiency benefit through active learning) but integrating or deploying that model is still a time-suck. Having the model deployed in the cloud immediately has a load of supplementary benefits (easy to update, can always use the latest models etc) too, we hope. (edit: typos)
- julvo 6y agoFirstly, congrats on the launch! Active learning is a super interesting space. You say it's possible to download the data and use Humanloop for annotation only while still benefitting from active learning. I'm curious about your experience with how much active learning depends on the model. Are the examples that the online model selects for labelling generally also the most useful ones for a different model trained offline?
- jordn 6y agoCheers. It's a good thing to be wary of. Poor use of active learning will end up biasing the data according to the model it's trained on – so that data won't be the best X samples to train on a different model. Most of this issue comes from bad active learning selection methods. If you have well calibrated uncertainty estimates and sample for diversity and representiveness too, it's far less of a concern.
- Grimm1 6y agoGreat answers, thank you very much!
- flyx 6y agoCongrats to the Humanloop team for the launch! Ivan chiming in from Datasaur here. As jordn pointed out, Datasaur does view itself as a full labeling platform, which encompasses an optimized labeling interface, a workforce management tool in addition to intelligence and active learning. Unlike Humanloop, we are focused solely on the labeling step of the process and do not offer a trained model at the end of the process. Our users have separate pipelines for this. Thanks for the question!