3 ms·

Hi Mike, let me know if I'm getting too into specifics for casual conversation. You mentioned a smaller dataset for training and I'm curious as to how much smal

Hi Mike, let me know if I'm getting too into specifics for casual conversation. You mentioned a smaller dataset for training and I'm curious as to how much smaller. Like, for static image recognition, what type of dataset trade off are we looking at. Also, randomly, is there a compute cost trade off or is it just a smarter process? I read the site, and no one likes cleaning data (no one), and answering questions from a "toddler machine" (for lack of a better term) doesn't sound as bad, but I was curious what potential trade offs there might be.

peadarohaodha 6y ago
We've found you can get an order of magnitude improvement in the amount of labelled data you need - but there is some variance based on the difficulty of the problem. Because you are retraining the model in tandem with the data labelling process, there is additional compute associated to an active learning powered labelling process versus just selecting the data at random to label next. But this additional compute consideration is almost always outweighed by the saving of human time spent labelling.
- talolard 6y ago
  I have a question on the compute aspect regarding your business model, hope I’m not being to nosy.. I tried HL, the experience was stellar (well done!) and it made me think... To get AL working with a great user experience you need quite a bit of compute. How are you thinking about your margins, e.g the cost to produce what you’re offering versus what customers will pay for it?
  - peadarohaodha 6y ago
    Thanks for the feedback! It's a good question re compute. There are some fun engineering and ML research challenges that we are constantly iterating on that are related to this. A few examples - how to most efficiently share compute resources in a JIT manner (e.g. GPU memory) during model serving for both training and inference (where the use case and privacy requirements permit) - how to construct model training algorithms that operate in a more online manner effectively (so you don't have to retrain on the whole dataset when you see new examples) - how to significantly reduce the model footprint (in terms of memory and flops) of modern deep transformer models given they are highly over-parameterised and can contain a lot of redundancy. this stuff helps us a lot on the margins point!
talolard 6y ago
I wrote a blog post a few years ago about possible downsides to active learning. https://www.lighttag.io/blog/active-learning-optimization-is-not-imporvement/ https://www.lighttag.io/blog/active-learning-optimization-is... To be fair to the Humanloop folks, the criticisms probably don’t apply to the kinds of models their using (modern transformers).

peadarohaodha 6y ago

We've found you can get an order of magnitude improvement in the amount of labelled data you need - but there is some variance based on the difficulty of the problem. Because you are retraining the model in tandem with the data labelling process, there is additional compute associated to an active learning powered labelling process versus just selecting the data at random to label next. But this additional compute consideration is almost always outweighed by the saving of human time spent labelling.

talolard 6y ago
I have a question on the compute aspect regarding your business model, hope I’m not being to nosy.. I tried HL, the experience was stellar (well done!) and it made me think... To get AL working with a great user experience you need quite a bit of compute. How are you thinking about your margins, e.g the cost to produce what you’re offering versus what customers will pay for it?
- peadarohaodha 6y ago
  Thanks for the feedback! It's a good question re compute. There are some fun engineering and ML research challenges that we are constantly iterating on that are related to this. A few examples - how to most efficiently share compute resources in a JIT manner (e.g. GPU memory) during model serving for both training and inference (where the use case and privacy requirements permit) - how to construct model training algorithms that operate in a more online manner effectively (so you don't have to retrain on the whole dataset when you see new examples) - how to significantly reduce the model footprint (in terms of memory and flops) of modern deep transformer models given they are highly over-parameterised and can contain a lot of redundancy. this stuff helps us a lot on the margins point!

talolard 6y ago

I wrote a blog post a few years ago about possible downsides to active learning. https://www.lighttag.io/blog/active-learning-optimization-is-not-imporvement/ https://www.lighttag.io/blog/active-learning-optimization-is... To be fair to the Humanloop folks, the criticisms probably don’t apply to the kinds of models their using (modern transformers).