8 ms·

I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition. Paying per us

by mlthoughts2018 7y ago

I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition. Paying per use certainly doesn’t make sense, because it has to be qualified by the accuracy you get per use. And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place. But once you need to hire them anyway, you might as well utilize them to build this type of thing in-house in ways that are much more tailored and optimized around your in-house data models and data integration tools. To put in perspective, I’ve worked in several companies (from small start-ups to large ecommerce sites) that have a variety of search needs spanning plug and play Lucene all the way to highly customized joint embedding neural network based nearest neighbor search, and tons in between. The distribution of text in e.g. the support center search use case was totally different than the product search use case or the document store use case, where highly unique word distribution, special words, frequency of required updates to the search index, asymmetric costs of surfacing bad or deleted content items, etc., was the norm. Every search use case was different and needed care to develop unique annotated result sets to measure mean reciprocal rank, NDCG, etc., as well as simple stakeholder subjective opinion of quality. Short of basically hiring Algolia to be a gigantic consultant on all these things, I don’t see how it could actually be valuable. I suspect it’s just an easy sell to CTO types that don’t really understand. They want “search” to be one problem with one little component to drop in to solve it, but it’s just not real.

apalmer 7y ago
You are absolutely factually correct in your analysis, but you are completely missing how business works. Fundamentally, there is value to most businesses in being able to just buy a decent solution to a non core competency. That’s where Algolia and AWS and basically all service companies come in... a medium scale clothing manufacturer with a booming e-commerce site may well know they have no clue how to do search, and no clue how to assess and hire individuals who could implement it, and no clue how to find and hire a cio who could put together a team from scratch who could do this on a reasonable timeline.
- mlthoughts2018 7y ago
  I’m saying in my experience there is no such thing as one singular “decent solution” for search. It varies enormously from use case to use case, customer cohort to customer cohort, etc. To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house. I think the fundamental myth, just like with Rekognition, is that if you ship off your data and the third party trains some model (most likely fine-tuning a base model), then you’re done, problem solved. Even for businesses where search is not a core part of their direct value proposition to customers this is flagrantly untrue.
  - hnaccy 7y ago
    >To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house. You just have to pay Algolia, wire up the APIs, and then see if whatever stakeholder that was complaining about search stops complaining. If they do then it's good enough.
    - wtvanhest 7y ago
      This is correct. Lets say you have an ecom site with product search. How do you know if the search is good? You search for stuff and compare to what you expect. Even if it sucks, as long as their is revenue lift, you will keep it until the next solution comes along.
      - briandear 7y ago
        And Algolia has great analytics, so you can actually mesure business value from actual search queries: you can tie a query to a purchase and then run further analysis on that. It’s powerful and doesn’t require any exceptional engineering talent to use.
    - mlthoughts2018 7y ago
      Totally false. This is like massively overfitting a high-order polynomial regression to your data. The fit looks good enough, then the next data point comes in and breaks in a way the existing model cannot be hacked to account for. The search results you believed were implicitly tuned to some feedback mechanism slowly experience creep as the customer cohort changes and data distribution changes until before you knew it your management of the search solution is a ceaseless game of whack-a-mole siphoning off engineering resources at a rapidly increasing rate. It’s the same false promise of just having some engineers stand up Elastic Search.
deleted 7y ago
[deleted]
majormajor 7y ago
> And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place. You may simply not be able to do this at all. You might not know how to tell good ML/Stats people from bad. You might not be able to pay them competitively. You might simply want something that's better than nothing, with "nothing" being your realistic alternative. "Expert in-house ML team" is not an alternative many companies can get, and even for the ones that could, it'll take a while. What are you going to do in the meantime?
- Sholmesy 7y ago
  At our company we have an "Expert in house ML team", but I don't want them wasting there time with search since it isn't our differentiating factor.
Mister_Snuggles 7y ago
> I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition. I signed up for AWS specifically to use Rekognition. I use it to screen alerts from my security cameras. In short, Blue Iris detects motion, a Node-RED flow grabs an image and uses Rekognition to see what's in it, if there's a person detected the Node-RED flow notifies me via PushOver. This significantly reduces the false-positives that inevitably happen on windy days - I've already done a lot of work in Blue Iris on this, but passing alerts through Rekognition makes it almost perfect. Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative. Based on my usage I expect my costs to be ~$5/mo once I'm no longer in the free tier. This is cheaper than the person detection service that Blue Iris natively integrates with and is significantly less effort to get up and running compared to, for example, TensorFlow. I also assume that Amazon will periodically update their detection models to make it better, which is one less thing for me to worry about. For me, all of these benefits are worth the ~$5/mo.
- mlthoughts2018 7y ago
  > “Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative.” But you’re just proving my point. It wouldn’t make sense to use Rekognition unless you had someone with skills to assess the classifier accuracy in the context of your specific problem. For example, it seems like your loss function places an asymmetrically higher cost on false negatives. (Incidentally, it’s interesting you claim it hasn’t produced a false negative ... did you watch every frame of video and make sure?) If you replace your simple one man operation with a simple loss function on an amount of data you can manually evaluate with instead a complex computer vision workflow, say where face or person detection has legal consequences for a company that sells or licenses stock photography, or an image or video search tool trying to avoid surfacing porn or pirated content, etc. then Rekognition becomes no longer useful, because you’ll need not just one person doing cursory evaluation of false negatives, but a team of people building out a benchmark-like battery of automated evaluations with probably IoU metrics in addition to classifier metrics and will need to figure out how many errors they can tolerate in some cost budget combined with the normal cost budget of usage to Rekognition. Basically, for some tiny hobbyist use case, I guess it’s fine (though really you could literally just load some Keras model pre-trained on imagenet or some off the shelf version of yolo and save yourself $5/mo) but the value proposition falls apart as soon as the cost function becomes a complicated business one.
jackdh 7y ago
Another big difference here is that Algolia does not use machine learning in its algorithms. This according to an old friend who worked there allowed them to really drill down to why which search results are shown and hence the pay per use does actually make sense.
- mlthoughts2018 7y ago
  This sounds like comedy to me. Either people say machine learning is magic and solves everything or people say not to use machine learning at all and this lets them drill down and understand? Many search tasks really do need machine learning, especially variations on collaborative filter and matrix factorization. Mixed modality search often truly does need deep learning and wasn’t even really possible at a level of fidelity suitable for real use cases until maybe 10 years ago. If Algolia was categorically omitting a whole class of possible solutions, that would be a big red flag, certainly not a reason to think they can drill down to understand search results better. I worked once on a large ecommerce search engine that had been built with Solr, and the sort order involved crazy hand-tuned boosting scores applied to ngrams of different sizes. None of it was reproducible, nobody knew where the magic boost weights came from, and as the quality of results started to plummet, there was no way to fix it. Everyone was too afraid to modify the magic constants because even slight perturbations created stark visual errors. And this was just for a super simple non-normalized term frequency matrix with boosts. “Not using machine learning” is not at all a signal that your solution won’t end up as a black box with no interpretability.
ArtWomb 7y ago
If you are building a core product, such as Spotify's Discover Weekly. Then no question, you need a dedicated team, and substantial commitment. Maybe even a PhD or two. But as an unabashed Angolia devotee, I think the value prop of InstantSearch is a no brainer. It's worthwhile looking at the product itself, as an almost textbook example of how to package services to enterprise customers. https://www.algolia.com/products/instantsearch/ https://www.algolia.com/products/instantsearch/ https://www.algolia.com/enterprise/customers/ https://www.algolia.com/enterprise/customers/

apalmer 7y ago

You are absolutely factually correct in your analysis, but you are completely missing how business works. Fundamentally, there is value to most businesses in being able to just buy a decent solution to a non core competency. That’s where Algolia and AWS and basically all service companies come in... a medium scale clothing manufacturer with a booming e-commerce site may well know they have no clue how to do search, and no clue how to assess and hire individuals who could implement it, and no clue how to find and hire a cio who could put together a team from scratch who could do this on a reasonable timeline.

mlthoughts2018 7y ago
I’m saying in my experience there is no such thing as one singular “decent solution” for search. It varies enormously from use case to use case, customer cohort to customer cohort, etc. To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house. I think the fundamental myth, just like with Rekognition, is that if you ship off your data and the third party trains some model (most likely fine-tuning a base model), then you’re done, problem solved. Even for businesses where search is not a core part of their direct value proposition to customers this is flagrantly untrue.
- hnaccy 7y ago
  >To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house. You just have to pay Algolia, wire up the APIs, and then see if whatever stakeholder that was complaining about search stops complaining. If they do then it's good enough.
  - wtvanhest 7y ago
    This is correct. Lets say you have an ecom site with product search. How do you know if the search is good? You search for stuff and compare to what you expect. Even if it sucks, as long as their is revenue lift, you will keep it until the next solution comes along.
    - briandear 7y ago
      And Algolia has great analytics, so you can actually mesure business value from actual search queries: you can tie a query to a purchase and then run further analysis on that. It’s powerful and doesn’t require any exceptional engineering talent to use.
  - mlthoughts2018 7y ago
    Totally false. This is like massively overfitting a high-order polynomial regression to your data. The fit looks good enough, then the next data point comes in and breaks in a way the existing model cannot be hacked to account for. The search results you believed were implicitly tuned to some feedback mechanism slowly experience creep as the customer cohort changes and data distribution changes until before you knew it your management of the search solution is a ceaseless game of whack-a-mole siphoning off engineering resources at a rapidly increasing rate. It’s the same false promise of just having some engineers stand up Elastic Search.

deleted 7y ago

[deleted]

majormajor 7y ago

> And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place. You may simply not be able to do this at all. You might not know how to tell good ML/Stats people from bad. You might not be able to pay them competitively. You might simply want something that's better than nothing, with "nothing" being your realistic alternative. "Expert in-house ML team" is not an alternative many companies can get, and even for the ones that could, it'll take a while. What are you going to do in the meantime?

Sholmesy 7y ago
At our company we have an "Expert in house ML team", but I don't want them wasting there time with search since it isn't our differentiating factor.

Mister_Snuggles 7y ago

> I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition. I signed up for AWS specifically to use Rekognition. I use it to screen alerts from my security cameras. In short, Blue Iris detects motion, a Node-RED flow grabs an image and uses Rekognition to see what's in it, if there's a person detected the Node-RED flow notifies me via PushOver. This significantly reduces the false-positives that inevitably happen on windy days - I've already done a lot of work in Blue Iris on this, but passing alerts through Rekognition makes it almost perfect. Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative. Based on my usage I expect my costs to be ~$5/mo once I'm no longer in the free tier. This is cheaper than the person detection service that Blue Iris natively integrates with and is significantly less effort to get up and running compared to, for example, TensorFlow. I also assume that Amazon will periodically update their detection models to make it better, which is one less thing for me to worry about. For me, all of these benefits are worth the ~$5/mo.

mlthoughts2018 7y ago
> “Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative.” But you’re just proving my point. It wouldn’t make sense to use Rekognition unless you had someone with skills to assess the classifier accuracy in the context of your specific problem. For example, it seems like your loss function places an asymmetrically higher cost on false negatives. (Incidentally, it’s interesting you claim it hasn’t produced a false negative ... did you watch every frame of video and make sure?) If you replace your simple one man operation with a simple loss function on an amount of data you can manually evaluate with instead a complex computer vision workflow, say where face or person detection has legal consequences for a company that sells or licenses stock photography, or an image or video search tool trying to avoid surfacing porn or pirated content, etc. then Rekognition becomes no longer useful, because you’ll need not just one person doing cursory evaluation of false negatives, but a team of people building out a benchmark-like battery of automated evaluations with probably IoU metrics in addition to classifier metrics and will need to figure out how many errors they can tolerate in some cost budget combined with the normal cost budget of usage to Rekognition. Basically, for some tiny hobbyist use case, I guess it’s fine (though really you could literally just load some Keras model pre-trained on imagenet or some off the shelf version of yolo and save yourself $5/mo) but the value proposition falls apart as soon as the cost function becomes a complicated business one.

jackdh 7y ago

Another big difference here is that Algolia does not use machine learning in its algorithms. This according to an old friend who worked there allowed them to really drill down to why which search results are shown and hence the pay per use does actually make sense.

mlthoughts2018 7y ago
This sounds like comedy to me. Either people say machine learning is magic and solves everything or people say not to use machine learning at all and this lets them drill down and understand? Many search tasks really do need machine learning, especially variations on collaborative filter and matrix factorization. Mixed modality search often truly does need deep learning and wasn’t even really possible at a level of fidelity suitable for real use cases until maybe 10 years ago. If Algolia was categorically omitting a whole class of possible solutions, that would be a big red flag, certainly not a reason to think they can drill down to understand search results better. I worked once on a large ecommerce search engine that had been built with Solr, and the sort order involved crazy hand-tuned boosting scores applied to ngrams of different sizes. None of it was reproducible, nobody knew where the magic boost weights came from, and as the quality of results started to plummet, there was no way to fix it. Everyone was too afraid to modify the magic constants because even slight perturbations created stark visual errors. And this was just for a super simple non-normalized term frequency matrix with boosts. “Not using machine learning” is not at all a signal that your solution won’t end up as a black box with no interpretability.

ArtWomb 7y ago

If you are building a core product, such as Spotify's Discover Weekly. Then no question, you need a dedicated team, and substantial commitment. Maybe even a PhD or two. But as an unabashed Angolia devotee, I think the value prop of InstantSearch is a no brainer. It's worthwhile looking at the product itself, as an almost textbook example of how to package services to enterprise customers. https://www.algolia.com/products/instantsearch/ https://www.algolia.com/products/instantsearch/ https://www.algolia.com/enterprise/customers/ https://www.algolia.com/enterprise/customers/