2 ms·
Looks promising and definitely a needed tool. I signed up for the beta and I used the demo version and have a couple of thoughts. 1. This seems closer to a rei
by AndrewKemendo 9y ago
Looks promising and definitely a needed tool. I signed up for the beta and I used the demo version and have a couple of thoughts.
1. This seems closer to a reinforcement learning system than a pure annotation system. That seems to be by design, however based on the demo, I am not able to change or add to the annotations as I go, which is a big limitation. It's just yes, no (no feedback), ignore and undo. This is in contrast to something like the VGG annotations system: http://www.robots.ox.ac.uk/~vgg/software/via/via.html http://www.robots.ox.ac.uk/~vgg/software/via/via.html
2. I don't see an actual annotations capability for images in the demo. Not sure if that is just a pretotype page, but IMO image classification/segmentation is the place where this tool would really benefit the community.
3. It's unclear to me how or if I retrieve my trained model or even just the annotated structure (.csv?, .json?) from this system. Do I get a .pb somehow that I can import into TF or am I locked into an API with my new model served from Prodigy? My guess would be the latter.
I think what this wants to be is a human validation system for training, which also improves the Prodigy nets through crowd sourcing. Definitely a win-win in the short term, but it has the limitations of the initial model and the ability for the user/client to tweak the system and output the results.
Matroid is doing something similar here, but I have been unimpressed with their offering so far.
- syllogism 9y agoThanks for the engaging questions! Reading between the lines, I think there's an important point that hasn't come across. Prodigy isn't SaaS --- it's a library you download and run. You can extend and customise every aspect of it, and there's definitely no lock-in. The model (and annotations) never have to leave your servers. For the specific questions: 1. The built-in web views all have binary annotation interfaces. This is more of a design choice than a fundamental limitation, and the front-end is extensible --- you can add your own web views if you need to. The binary interface is sort of a position statement. We think this is The Way, so we want you to try it. We'll have more input components in future, but at the start we want to guide people towards the intended workflow. 2. The beta focuses on NLP support, but there's a front-end for image classification, and a workflow page: https://prodi.gy/docs/workflow-image-classification https://prodi.gy/docs/workflow-image-classification . 3. You can usually get some accuracy improvement by retraining once all the annotations are available. I've not found a streaming SGD algorithm that works as well as the simple iterate-and-shuffle batch process. Batch training also lets you tune the hyper-paramers. You can read more about this here: https://prodi.gy/docs/workflow-named-entity-recognition#training https://prodi.gy/docs/workflow-named-entity-recognition#trai... I would suggest writing a Prodigy recipe to do the batch training. That way you can pass in the dataset ID, instead of exporting the annotations. There's no problem with exporting the annotations and running a script, though. Again --- it's all on your computer. You can run it however you like.
- AndrewKemendo 9y agoI think there's an important point that hasn't come across. Prodigy isn't SaaS --- it's a library you download and run. You're right, I totally missed that. Re-reading that comes through but it is definitely different than what I would have expected. Thanks for the response, I'll dig in further.