5 ms·

I find this article a bit funny, and see it mostly as hype. I worked very revently on a large internal face detection service for a legal and compliance applic

by mlthoughts2018 8y ago

I find this article a bit funny, and see it mostly as hype. I worked very revently on a large internal face detection service for a legal and compliance application at a large US company. Being pragmatic engineers, the first thing we did was to pilot test Rekognition against an in-house prototype built by modifying some open source deep learning approaches. Rekognition performed so poorly compared to our prototype (which was not state of the art or anything, just good enough for our company’s bespoke conpliance problem). It was just so bad. Overall intersection-over-union scores on our training & acceptance data was really poor, and perhaps worse, the latency of the requests to Rekognition was abysmal and completely unusable for us, especially for images that had a large number of identifiable faces (Rekognition service calls isually timed out at 5 seconds and returned whatever amount of faces it could process). Our very simple prototype could fully process upwards of 30 faces with 1 second of latency, and so if using a similar 5 second timeout limit, we could process around an order of magnitude more faces (which happens more often than you think, e.g. crowds, people in public places). I just could not believe that the web service, backed by a team surely comprised of at least hundreds of top Amazon engineers, was so slow and inaccurate. We didn’t even bother trying to price out how expensive it would be for our expected amount of throughput, because we knew from a simple accuracy and per-request latency point of view, Rekognition just couldn’t cut it. As I’ve moved into a more lead role with machine learning teams, this experience really opened my eyes to how seductive cloud solutions can appear to non-technical managers. They will act like severe bean counters when they see a high salary request from a really experienced machine learning engineer, but they won’t even bother with due diligence or estimating cost-per-request or cost-per-unit-accuracy for dropping money on a cloud service. Instead of empowering a small team to do something much more cost-effective for the company, they’ll ball at their internal engineers while setting a pile of money ablaze.

eatbitseveryday 8y ago
> Rekognition performed so poorly compared to our prototype (which was not state of the art or anything, just good enough for our company’s bespoke conpliance problem). It was just so bad. Overall intersection-over-union scores on our training & acceptance data was really poor, and perhaps worse, the latency of the requests to Rekognition was abysmal and completely unusable for us, especially for images that had a large number of identifiable faces (Rekognition service calls isually timed out at 5 seconds and returned whatever amount of faces it could process). Criticizing the technology's performance is tangential to the emergence of the technology on public streets, and its adoption by local governments for surveillance. It does not matter what performance it has now; I argue and predict it will improve, and then your statement becomes moot. The issue is not how well it performs. The issue is allowing it to happen, and without trust from the people. My fear is: government wants surveillance, and businesses pursue revenue, so there is no incentive among those two to question their actions.
- mlthoughts2018 8y ago
  I agree with you generally. But I differ in the sense that my default belief is that these technologies are over hyped and cannot provide the scale of performance or detail of prediction that could make them effective as oppressive surveillance tools, at least not for a long time. Mostly they are “surveillance theater” akin to useless (but costly) TSA surveillance. The way it interacts with society is by being vague and scary, not by actually enabling truly scary scales of surveillance. Another way to look at it is that the government buys these contracts and services and rarely holds the provider accountable, perhaps especially in fields like machine learning where people can more easily make political arguments out of the debate over whether it’s working or not. Since agencies are happy to drop big money even when the technology does not work then providers like Rekognition actually do not have incentives to invest in making the scale of accurate surveillance larger. Why would they when they’ll make more money yanking an engineer off of a material project like that and slotting them to work on some braindead TED talk sort of demoware that gets them the next big contract? I worry about the spread of these surveillance contracts more as a taxpayer seeing government money wasted that could help people through much simpler means than I do as a concerned citizen who sees the danger in Orwellian surveillance.
  - cryoshon 8y ago
    >my default belief is that these technologies are over hyped and cannot provide the scale of performance or detail of prediction that could make them effective as oppressive surveillance tools, at least not for a long time. you've never heard of the chinese police state's facial surveillance which they're using to throw countless uighurs in jail for stuff like jaywalking, i suppose? >as a concerned citizen who sees the danger in Orwellian surveillance. you don't see the danger because your paycheck relies on you not seeing it. the rest of us are not so willfully blind.
    - mlthoughts2018 8y ago
      But I do see danger. Both in the misapplication of this unproven tech, and in the opportunity cost of wasting public funds on it.
cascom 8y ago
I wonder how many innocent people are going to be harmed due to poor facial recognition matches...
- deepakg 8y ago
  Indeed! I recently tried Rekognition's "celebrity detection" and was surprised at the no. of false positives - "doppelgänger detection" would've been more like it. Some examples: https://medium.com/@deepak.gulati/trying-amazon-rekognition-on-indian-album-art-c2b2a343f081 https://medium.com/@deepak.gulati/trying-amazon-rekognition-... That said, it could flag faces for manual screening. Or cross reference other data (License plates, Geo data from phone) to boost confidence. Also, once they get more data as part of some of these deals they'll get better.
amzn-throw 8y ago
> backed by a team surely comprised of at least hundreds of top Amazon engineers, was so slow and inaccurate. You'd be surprised. Many many AWS services are launched by a single 2-pizza team (8-10 developers). It's in Amazon's DNA to act like a startup, ship something quickly, collect feedback, then iterate, and scale. Rekognition will get much better in time.
- seanp2k2 8y ago
  Also, from what I’ve been able to deduce, the AWS services are all just built on top of EC2 [un]modified open-source software until they reach a level of complexity where they roll their own, which leads to lots of issues that can take years to fix. Some things are telling, like banning two consecutive hyphens because something somewhere in that pipeline is shelling out.
- fixermark 8y ago
  Sidebar: I've never heard of teams measured in terms of "pizza count," and now I can't un-think it. Good metric. :)
  - exclusiv 8y ago
    It's been reported that Bezos uses a two pizza rule for meetings.
- sadamznintern 8y ago
  Also, for the record, AWS is hardly top of the industry in terms of talent. It's not Google or Facebook. I've never heard of an ML researcher joining Amazon when they could have joined Google/FB/Apple.
  - p0rkbelly 8y ago
    I've known plenty of people that have left Google/FB/Apple for Amazon. And I know plenty of people who have left Amazon for Google/FB/Apple. The biggest tech companies are quite incestuous. Some people do thrive in one environment, some in others. They all pay top dollar.
jakelarkin 8y ago
interesting data point on its performance but the reason decision makers might be cloud happy is amazon has the resources to scale this to thousands of RPS and develop the backend codebase up to hundreds of engineer contributions for many years, even as key talent comes and goes. The same might not be true of your prototype especially if the core competency of your business is sales or legal compliance not distributed computing platforms.
- mlthoughts2018 8y ago
  Our team was just a general machine learning team with no expertise in compliance issues, and only one or two of us had experience in face detection. We only cared about providing a web service that gave good face detection performance (the downstream consumer was the team that had to then take the face detection output and do business logic with it for compliance). > amazon has the resources to scale this to thousands of RPS Actually, this part is mostly straight forward. Not easy, but straight forward, and Amazon already has this part solved for most things. Our ML team did have expertise in making very large-scale services (our company operated a large ecommerce web store, so our traffic was actually very high, top 500 Alexa rank), but our concern over Amazon had nothing to do with whether it could handle enough requests per second. But this also misses the point. The latency for a single request was too poor for a variety of natural images (crowd scenes, public streets, etc., -- exactly the types of images relevant for many surveillance applications). It was not a throughput issue, I guess unless you expect consumer teams to do something like chopping images up into sub-images, replacing one Rekognition request with a bunch of sub-image requests, and then post-processing to stitch the results together and account for error introduced by the choice of where to split the original image. Yikes. It's not an issue of "scaling it up" -- it's an issue that the fundamental machine learning algorithm underpinning it suffered too high latency, likely because it was a deep neural network combined with a bunch of post-processing gadgets that are conceptually not good enough for bespoke use cases -- things which engineers on a centralized platform have little incentive to care about. The real question is whether Amazon has incentives to make this part better, and if so, how? Will a general one-size-fits-all face detector work for customers? Or will they need application-by-application customer-specific tailored models (and if so, why aren't customers savvy enough to realize that the all-in costs to build and operate it in-house are so demonstrably cheaper than Amazon, and might be the only way to meet specific performance goals)? I hear this so often, that Amazon can "just scale it up" -- but I think if you really work in face detection and have studied the problem deeply, you know this is not right. There's a whole different bucket of problems and issues and cost-effectiveness trade-offs going on.
gscott 8y ago
Your post makes me think of this article which pretty comprehensively reviews it https://cloudacademy.com/blog/google-vision-vs-amazon-rekognition/ https://cloudacademy.com/blog/google-vision-vs-amazon-rekogn...
- mlthoughts2018 8y ago
  The scenario-based costs table at the bottom is particularly interesting. If you need to run the detectors on all incoming images of a site ingesting, say, ~10 million images per week, you're looking at nearly $15,000/week, or close to $800K/year with AWS, and nearly $1.2MM/year with Google, and either way you'll still probably need to employ some type of backend engineering team to maintain the wrappers for calling it and manipulating the response data. We were able to solve this in-house using only about 15 8-core servers (we used a few general GPU servers for training, but found we only needed CPU machines for runtime, and did not have the poor latency of the AWS calls) with quite a lot of redundancy for traffic spikes and a pretty easy deployment system if we needed to add or remove nodes, and it was only one of several dozen different machine learning projects going on, so whatever portion of the total cost of operation could be allocated to paying the salary of a machine learning team, it would further be amortized by work on many projects. For example, amortizing the cost of GPU machines for model training across a variety of projects.
tgsovlerkhgsel 8y ago
As it get customers, it will likely get more and more resources thrown at it and get better and better. Even if the tech may not be useful right now, it probably will be in 5 years, and unless something is done now, another large slice of the little bit of privacy that remains will be gone.
- mlthoughts2018 8y ago
  I think this is unlikely because of the nature of the problem. It's kind of like when Paul Graham said consulting is fundamentally not scalable. Throwing more people at a problem like this one only works if you have dedicated consultants for every bespoke business case, because there's no one-size-fits-all model, and the way that performance limitations interacts with which kind of face detection models you can use is very complicated. Companies would end up paying huge premia to Amazon essentially to have Amazon build in-house teams that would be equivalent, only much more expensive, than the companies own in-house team just building an in-house solution. With some forms of consulting this can make sense, because the consulting services are temporary and should set the company in a position such that for the long run, the company can maintain the solution cost-effectively. But for something like face detection, assuming it's a pivotal service you'll need on an on-going basis, this would become the worst sort of vendor lock-in imaginable, combined with the fact that if you don't appear to be creating growth for Amazon, you're liable to be deprioritized at any point, and at the mercy of Amazon's choices regarding when and how to address your bespoke need. For these specialized machine learning services it doesn't fit the same kind of commodity model that AWS infrastructure uses, though that's currently how Amazon is pursuing it. Don't get me wrong, Amazon is very good at making money and convincing nervous management to buy their brand. I suspect they will find ways to sell these services despite the services not being cost-effective or customizeable for bespoke customer situations. But when I point out that Amazon's machine learning services don't offer cost-effective performance, it's very different from a "just throw more devs at it" kind of problem, fundamentally.

eatbitseveryday 8y ago

> Rekognition performed so poorly compared to our prototype (which was not state of the art or anything, just good enough for our company’s bespoke conpliance problem). It was just so bad. Overall intersection-over-union scores on our training & acceptance data was really poor, and perhaps worse, the latency of the requests to Rekognition was abysmal and completely unusable for us, especially for images that had a large number of identifiable faces (Rekognition service calls isually timed out at 5 seconds and returned whatever amount of faces it could process). Criticizing the technology's performance is tangential to the emergence of the technology on public streets, and its adoption by local governments for surveillance. It does not matter what performance it has now; I argue and predict it will improve, and then your statement becomes moot. The issue is not how well it performs. The issue is allowing it to happen, and without trust from the people. My fear is: government wants surveillance, and businesses pursue revenue, so there is no incentive among those two to question their actions.

mlthoughts2018 8y ago
I agree with you generally. But I differ in the sense that my default belief is that these technologies are over hyped and cannot provide the scale of performance or detail of prediction that could make them effective as oppressive surveillance tools, at least not for a long time. Mostly they are “surveillance theater” akin to useless (but costly) TSA surveillance. The way it interacts with society is by being vague and scary, not by actually enabling truly scary scales of surveillance. Another way to look at it is that the government buys these contracts and services and rarely holds the provider accountable, perhaps especially in fields like machine learning where people can more easily make political arguments out of the debate over whether it’s working or not. Since agencies are happy to drop big money even when the technology does not work then providers like Rekognition actually do not have incentives to invest in making the scale of accurate surveillance larger. Why would they when they’ll make more money yanking an engineer off of a material project like that and slotting them to work on some braindead TED talk sort of demoware that gets them the next big contract? I worry about the spread of these surveillance contracts more as a taxpayer seeing government money wasted that could help people through much simpler means than I do as a concerned citizen who sees the danger in Orwellian surveillance.
- cryoshon 8y ago
  >my default belief is that these technologies are over hyped and cannot provide the scale of performance or detail of prediction that could make them effective as oppressive surveillance tools, at least not for a long time. you've never heard of the chinese police state's facial surveillance which they're using to throw countless uighurs in jail for stuff like jaywalking, i suppose? >as a concerned citizen who sees the danger in Orwellian surveillance. you don't see the danger because your paycheck relies on you not seeing it. the rest of us are not so willfully blind.
  - mlthoughts2018 8y ago
    But I do see danger. Both in the misapplication of this unproven tech, and in the opportunity cost of wasting public funds on it.

cascom 8y ago

I wonder how many innocent people are going to be harmed due to poor facial recognition matches...

deepakg 8y ago
Indeed! I recently tried Rekognition's "celebrity detection" and was surprised at the no. of false positives - "doppelgänger detection" would've been more like it. Some examples: https://medium.com/@deepak.gulati/trying-amazon-rekognition-on-indian-album-art-c2b2a343f081 https://medium.com/@deepak.gulati/trying-amazon-rekognition-... That said, it could flag faces for manual screening. Or cross reference other data (License plates, Geo data from phone) to boost confidence. Also, once they get more data as part of some of these deals they'll get better.

amzn-throw 8y ago

> backed by a team surely comprised of at least hundreds of top Amazon engineers, was so slow and inaccurate. You'd be surprised. Many many AWS services are launched by a single 2-pizza team (8-10 developers). It's in Amazon's DNA to act like a startup, ship something quickly, collect feedback, then iterate, and scale. Rekognition will get much better in time.

seanp2k2 8y ago
Also, from what I’ve been able to deduce, the AWS services are all just built on top of EC2 [un]modified open-source software until they reach a level of complexity where they roll their own, which leads to lots of issues that can take years to fix. Some things are telling, like banning two consecutive hyphens because something somewhere in that pipeline is shelling out.
fixermark 8y ago
Sidebar: I've never heard of teams measured in terms of "pizza count," and now I can't un-think it. Good metric. :)
- exclusiv 8y ago
  It's been reported that Bezos uses a two pizza rule for meetings.
sadamznintern 8y ago
Also, for the record, AWS is hardly top of the industry in terms of talent. It's not Google or Facebook. I've never heard of an ML researcher joining Amazon when they could have joined Google/FB/Apple.
- p0rkbelly 8y ago
  I've known plenty of people that have left Google/FB/Apple for Amazon. And I know plenty of people who have left Amazon for Google/FB/Apple. The biggest tech companies are quite incestuous. Some people do thrive in one environment, some in others. They all pay top dollar.

jakelarkin 8y ago

interesting data point on its performance but the reason decision makers might be cloud happy is amazon has the resources to scale this to thousands of RPS and develop the backend codebase up to hundreds of engineer contributions for many years, even as key talent comes and goes. The same might not be true of your prototype especially if the core competency of your business is sales or legal compliance not distributed computing platforms.

mlthoughts2018 8y ago
Our team was just a general machine learning team with no expertise in compliance issues, and only one or two of us had experience in face detection. We only cared about providing a web service that gave good face detection performance (the downstream consumer was the team that had to then take the face detection output and do business logic with it for compliance). > amazon has the resources to scale this to thousands of RPS Actually, this part is mostly straight forward. Not easy, but straight forward, and Amazon already has this part solved for most things. Our ML team did have expertise in making very large-scale services (our company operated a large ecommerce web store, so our traffic was actually very high, top 500 Alexa rank), but our concern over Amazon had nothing to do with whether it could handle enough requests per second. But this also misses the point. The latency for a single request was too poor for a variety of natural images (crowd scenes, public streets, etc., -- exactly the types of images relevant for many surveillance applications). It was not a throughput issue, I guess unless you expect consumer teams to do something like chopping images up into sub-images, replacing one Rekognition request with a bunch of sub-image requests, and then post-processing to stitch the results together and account for error introduced by the choice of where to split the original image. Yikes. It's not an issue of "scaling it up" -- it's an issue that the fundamental machine learning algorithm underpinning it suffered too high latency, likely because it was a deep neural network combined with a bunch of post-processing gadgets that are conceptually not good enough for bespoke use cases -- things which engineers on a centralized platform have little incentive to care about. The real question is whether Amazon has incentives to make this part better, and if so, how? Will a general one-size-fits-all face detector work for customers? Or will they need application-by-application customer-specific tailored models (and if so, why aren't customers savvy enough to realize that the all-in costs to build and operate it in-house are so demonstrably cheaper than Amazon, and might be the only way to meet specific performance goals)? I hear this so often, that Amazon can "just scale it up" -- but I think if you really work in face detection and have studied the problem deeply, you know this is not right. There's a whole different bucket of problems and issues and cost-effectiveness trade-offs going on.

gscott 8y ago

Your post makes me think of this article which pretty comprehensively reviews it https://cloudacademy.com/blog/google-vision-vs-amazon-rekognition/ https://cloudacademy.com/blog/google-vision-vs-amazon-rekogn...

mlthoughts2018 8y ago
The scenario-based costs table at the bottom is particularly interesting. If you need to run the detectors on all incoming images of a site ingesting, say, ~10 million images per week, you're looking at nearly $15,000/week, or close to $800K/year with AWS, and nearly $1.2MM/year with Google, and either way you'll still probably need to employ some type of backend engineering team to maintain the wrappers for calling it and manipulating the response data. We were able to solve this in-house using only about 15 8-core servers (we used a few general GPU servers for training, but found we only needed CPU machines for runtime, and did not have the poor latency of the AWS calls) with quite a lot of redundancy for traffic spikes and a pretty easy deployment system if we needed to add or remove nodes, and it was only one of several dozen different machine learning projects going on, so whatever portion of the total cost of operation could be allocated to paying the salary of a machine learning team, it would further be amortized by work on many projects. For example, amortizing the cost of GPU machines for model training across a variety of projects.

tgsovlerkhgsel 8y ago

As it get customers, it will likely get more and more resources thrown at it and get better and better. Even if the tech may not be useful right now, it probably will be in 5 years, and unless something is done now, another large slice of the little bit of privacy that remains will be gone.

mlthoughts2018 8y ago
I think this is unlikely because of the nature of the problem. It's kind of like when Paul Graham said consulting is fundamentally not scalable. Throwing more people at a problem like this one only works if you have dedicated consultants for every bespoke business case, because there's no one-size-fits-all model, and the way that performance limitations interacts with which kind of face detection models you can use is very complicated. Companies would end up paying huge premia to Amazon essentially to have Amazon build in-house teams that would be equivalent, only much more expensive, than the companies own in-house team just building an in-house solution. With some forms of consulting this can make sense, because the consulting services are temporary and should set the company in a position such that for the long run, the company can maintain the solution cost-effectively. But for something like face detection, assuming it's a pivotal service you'll need on an on-going basis, this would become the worst sort of vendor lock-in imaginable, combined with the fact that if you don't appear to be creating growth for Amazon, you're liable to be deprioritized at any point, and at the mercy of Amazon's choices regarding when and how to address your bespoke need. For these specialized machine learning services it doesn't fit the same kind of commodity model that AWS infrastructure uses, though that's currently how Amazon is pursuing it. Don't get me wrong, Amazon is very good at making money and convincing nervous management to buy their brand. I suspect they will find ways to sell these services despite the services not being cost-effective or customizeable for bespoke customer situations. But when I point out that Amazon's machine learning services don't offer cost-effective performance, it's very different from a "just throw more devs at it" kind of problem, fundamentally.