6 ms·

On-Device Supermarket Product Recognition

83 points by theafh 6y ago

cs702 6y ago
This is only the beginning. At the moment, mobile devices must run sub-SOTA models, such as a MobileNetV3[a] for visual recognition, to provide an acceptable user experience. As mobile devices continue to get more powerful and start incorporating dedicated AI hardware for multiply-add-accumulate ops, I expect to see devices that can run today's SOTA models locally within a couple of years. Exciting stuff! [a] https://ai.googleblog.com/2019/11/introducing-next-generation-on-device.html https://ai.googleblog.com/2019/11/introducing-next-generatio...
dillonmckay 6y ago
License this to Instacart so I can get the proper size package I ordered.
ganzuul 6y ago
This is one of those obvious killer-apps that with a moment's thought was obviously computationally prohibitive or much too complex for a sole developer. If this is in the pipelines then we can soon also expect a bunch of semantic apps.
armitron 6y ago
Privacy nightmare not to mention solution looking for a problem.
- olyjohn 6y ago
  You know... Every time some new product comes out, I can't help but think "how are they going to use this to track my habits." And how can you blame anybody for that? Think about everything you own that is already tracking you. Your TV, your Phone, your smart speaker... all the apps you use... Can you really blame somebody for being pessimistic? You know this is a useful tool for visually impaired people, but I can't help but think that nobody should have to give up their privacy in order to have some level of equality. That is total horseshit IMO.
  - anoncareer0212 6y ago
    this is a spot-on caricature of a lot of comments over the past 5 years, great stuff
- gundmc 6y ago
  What specific aspects of the design do you consider to a "privacy nightmare"? Typically on-device computing is lauded from a privacy perspective when compared to cloud equivalents.
  - ffpip 6y ago
    Doing it on-device, and storing it on-device are 2 different things. Shopping data is the most valuable. It is the end result of advertising. What ad caused you to but that specific product? Can Google show a competitors ad to you?
awinter-py 6y ago
interesting that one of the uses of NN here is to shrink the amount of storage required per product in the product recognition DB -- compression is a neat application of NN that generally follows from recognition / synthesis tricks, and will matter a lot on mobile. I'm wondering if 'on device' is an exaggeration. The post is saying 64 bytes per product, and a DB of millions of products, but the video is showing product details, prices, and a box image. I wonder if once it identifies the product it then loads metadata from ze cloud?
- anoncareer0212 6y ago
  'it had to load an image from the cloud' sounds much more like a nitpick than 'it recognized the product locally' sounds like an exaggeration
  - awinter-py 6y ago
    it's a spectrum, right? 'on-device' is a big claim with different implications if you're thinking about this as an NN problem or a privacy problem. Loading an image from the cloud = sending telemetry on your supermarket shopping to a giant ads company.
swimfar 6y ago
I would love something like this paired with augmented reality to show/hide/highlight only the products that meet my search criteria. Try going to a big supermarket to look for whole-milk yogurt, for example. It's a pain trying to filter them out of the rest of the 100 varieties. Another thing I would like to do is compare products by price/quanity. Maybe I'm trying to get the most protein/$, or oz/$, etc. You could also hook it into some database to only show you ethically caught fish, organic projects, fair trade products, local products, etc. I've thought about this kind of technology a lot. Cool to see that it's actually becoming possible.
- azalemeth 6y ago
  > It's a pain trying to filter them out of the rest of the 100 varieties. Arguably, shops specifically design themselves to do things like this. When I was growing up, it used to drive my dad mad that the local supermarket would move almost literally everything around seemingly at random "to prevent shoplifting". He was a pilot, and usually had little time -- he liked to go into supermarket, grab x, pay, and leave. Suddenly, that stoped working, as finding x became increasingly difficult. It really annoyed him. For a long time the floor manager said that they moved their stock around to prevent shoplifting – until one day the store manager, when asked the same question, said it was "to promote footfall". Suddenly the real reason became clear. >Another thing I would like to do is compare products by price/quanity. Maybe I'm trying to get the most protein/$, or oz/$, etc. You could also hook it into some database to only show you ethically caught fish, organic projects, fair trade products, local products, etc. In the UK at least, all of this information is required to be printed on the label: product per unit money on the shop label and misery per unit product on the product's label. Are you asking for a filter / search engine for that on top? Either way, I don't think your average supermarket would let that stay un-advertised for too long. Products at eyeball level pay a premium to be there.
  - squeaky-clean 6y ago
    I don't know if it's legally required, but that's also the standard here in the US. One problem though is the units may not always be equal. I was buying canned coconut milk this weekend and among all the different choices were $/pound, $/gram, and $/liter. It's easier to do the math from scratch yourself at that point.
greesil 6y ago
Interesting how it isn't just one giant neural net mashed together, and seems to leverage "hand-crafted" features.
arkanciscan 6y ago
People who can't recognize products on a shelf often can't use a touchscreen either. I live with a Boomer and he can't even use a Chromecast because he can't see his screen, let alone use screen readers or voice commands.
- pkaye 6y ago
  Its another option for the visually impaired.
lifeisstillgood 6y ago
An RFID scanner will do a lot of this surely ? The issues are presumably access to the TFID number database, but for me the big open question is that as we progress to an RFID chip in everything world, that access becomes a public good - a complete supply chain record will be the next "food labelling" fight
- ladberg 6y ago
  That basically already exists with barcodes.
  - lifeisstillgood 6y ago
    1. Barcode UPN lists are closely guarded secrets. Knowing that 123456 is a lot of hummus is valuable to a lot of retailers. 2. The supply chain for who made the hummus, who grew the chickpeas, were they sprayed with what. That basically does not exist, let alone in public.
deleted 6y ago
[deleted]
polskibus 6y ago
Is this app open source?
mjs33 6y ago
Does it take into account flavors? Yoghurt packages and Oreos come to mind. Lots of different flavors and the packaging is subtly different
apapli 6y ago
This is interesting - I couldn’t determine how they train them system to recognise all the products though. Flavour variants, special promotional packaging etc make training hard due to the sheer size of images that need to be in a training dataset to yield accurate results. How is this achieved?
- joshvm 6y ago
  This is likely done using OCR: > OCR is executed on the ROI for each camera frame in order to extract additional information, such as packet size, product flavor variant, etc. You could probably also do some very simple statistics like a colour histogram on the box, once you had high certainty of the product line.
riyadparvez 6y ago
I am interested into why they haven't released this feature in their main Google Lens app. My hunch is they released a separate app for experimental and beta testing. Once these features get robust and reliable, they will probably move these features to the main Google Lens app.
tolidano 6y ago
Didn’t Amazon do this with the Fire Phone and FireFly in 2015? What I need is to comparison shop this item at 5 other grocery stores and FreshDirect and Amazon Fresh and have it tell me if I’m getting a good deal.
nmstoker 6y ago
Would be handy if they applied this to an internal shop level Google Maps Would cut the need to speak to shop staff, who in many places in larger cities don't have the language skills to know what you're looking for.

83 points by theafh 6y ago

This is only the beginning. At the moment, mobile devices must run sub-SOTA models, such as a MobileNetV3[a] for visual recognition, to provide an acceptable user experience. As mobile devices continue to get more powerful and start incorporating dedicated AI hardware for multiply-add-accumulate ops, I expect to see devices that can run today's SOTA models locally within a couple of years. Exciting stuff! [a] https://ai.googleblog.com/2019/11/introducing-next-generation-on-device.html https://ai.googleblog.com/2019/11/introducing-next-generatio...

dillonmckay 6y ago

License this to Instacart so I can get the proper size package I ordered.

ganzuul 6y ago

This is one of those obvious killer-apps that with a moment's thought was obviously computationally prohibitive or much too complex for a sole developer. If this is in the pipelines then we can soon also expect a bunch of semantic apps.

armitron 6y ago

Privacy nightmare not to mention solution looking for a problem.

olyjohn 6y ago
You know... Every time some new product comes out, I can't help but think "how are they going to use this to track my habits." And how can you blame anybody for that? Think about everything you own that is already tracking you. Your TV, your Phone, your smart speaker... all the apps you use... Can you really blame somebody for being pessimistic? You know this is a useful tool for visually impaired people, but I can't help but think that nobody should have to give up their privacy in order to have some level of equality. That is total horseshit IMO.
- anoncareer0212 6y ago
  this is a spot-on caricature of a lot of comments over the past 5 years, great stuff
gundmc 6y ago
What specific aspects of the design do you consider to a "privacy nightmare"? Typically on-device computing is lauded from a privacy perspective when compared to cloud equivalents.
- ffpip 6y ago
  Doing it on-device, and storing it on-device are 2 different things. Shopping data is the most valuable. It is the end result of advertising. What ad caused you to but that specific product? Can Google show a competitors ad to you?

awinter-py 6y ago

interesting that one of the uses of NN here is to shrink the amount of storage required per product in the product recognition DB -- compression is a neat application of NN that generally follows from recognition / synthesis tricks, and will matter a lot on mobile. I'm wondering if 'on device' is an exaggeration. The post is saying 64 bytes per product, and a DB of millions of products, but the video is showing product details, prices, and a box image. I wonder if once it identifies the product it then loads metadata from ze cloud?

anoncareer0212 6y ago
'it had to load an image from the cloud' sounds much more like a nitpick than 'it recognized the product locally' sounds like an exaggeration
- awinter-py 6y ago
  it's a spectrum, right? 'on-device' is a big claim with different implications if you're thinking about this as an NN problem or a privacy problem. Loading an image from the cloud = sending telemetry on your supermarket shopping to a giant ads company.

swimfar 6y ago

I would love something like this paired with augmented reality to show/hide/highlight only the products that meet my search criteria. Try going to a big supermarket to look for whole-milk yogurt, for example. It's a pain trying to filter them out of the rest of the 100 varieties. Another thing I would like to do is compare products by price/quanity. Maybe I'm trying to get the most protein/$, or oz/$, etc. You could also hook it into some database to only show you ethically caught fish, organic projects, fair trade products, local products, etc. I've thought about this kind of technology a lot. Cool to see that it's actually becoming possible.

azalemeth 6y ago
> It's a pain trying to filter them out of the rest of the 100 varieties. Arguably, shops specifically design themselves to do things like this. When I was growing up, it used to drive my dad mad that the local supermarket would move almost literally everything around seemingly at random "to prevent shoplifting". He was a pilot, and usually had little time -- he liked to go into supermarket, grab x, pay, and leave. Suddenly, that stoped working, as finding x became increasingly difficult. It really annoyed him. For a long time the floor manager said that they moved their stock around to prevent shoplifting – until one day the store manager, when asked the same question, said it was "to promote footfall". Suddenly the real reason became clear. >Another thing I would like to do is compare products by price/quanity. Maybe I'm trying to get the most protein/$, or oz/$, etc. You could also hook it into some database to only show you ethically caught fish, organic projects, fair trade products, local products, etc. In the UK at least, all of this information is required to be printed on the label: product per unit money on the shop label and misery per unit product on the product's label. Are you asking for a filter / search engine for that on top? Either way, I don't think your average supermarket would let that stay un-advertised for too long. Products at eyeball level pay a premium to be there.
- squeaky-clean 6y ago
  I don't know if it's legally required, but that's also the standard here in the US. One problem though is the units may not always be equal. I was buying canned coconut milk this weekend and among all the different choices were $/pound, $/gram, and $/liter. It's easier to do the math from scratch yourself at that point.

greesil 6y ago

Interesting how it isn't just one giant neural net mashed together, and seems to leverage "hand-crafted" features.

arkanciscan 6y ago

People who can't recognize products on a shelf often can't use a touchscreen either. I live with a Boomer and he can't even use a Chromecast because he can't see his screen, let alone use screen readers or voice commands.

pkaye 6y ago
Its another option for the visually impaired.

lifeisstillgood 6y ago

An RFID scanner will do a lot of this surely ? The issues are presumably access to the TFID number database, but for me the big open question is that as we progress to an RFID chip in everything world, that access becomes a public good - a complete supply chain record will be the next "food labelling" fight

ladberg 6y ago
That basically already exists with barcodes.
- lifeisstillgood 6y ago
  1. Barcode UPN lists are closely guarded secrets. Knowing that 123456 is a lot of hummus is valuable to a lot of retailers. 2. The supply chain for who made the hummus, who grew the chickpeas, were they sprayed with what. That basically does not exist, let alone in public.

deleted 6y ago

[deleted]

polskibus 6y ago

Is this app open source?

mjs33 6y ago

Does it take into account flavors? Yoghurt packages and Oreos come to mind. Lots of different flavors and the packaging is subtly different

apapli 6y ago

This is interesting - I couldn’t determine how they train them system to recognise all the products though. Flavour variants, special promotional packaging etc make training hard due to the sheer size of images that need to be in a training dataset to yield accurate results. How is this achieved?

joshvm 6y ago
This is likely done using OCR: > OCR is executed on the ROI for each camera frame in order to extract additional information, such as packet size, product flavor variant, etc. You could probably also do some very simple statistics like a colour histogram on the box, once you had high certainty of the product line.

riyadparvez 6y ago

I am interested into why they haven't released this feature in their main Google Lens app. My hunch is they released a separate app for experimental and beta testing. Once these features get robust and reliable, they will probably move these features to the main Google Lens app.

tolidano 6y ago

Didn’t Amazon do this with the Fire Phone and FireFly in 2015? What I need is to comparison shop this item at 5 other grocery stores and FreshDirect and Amazon Fresh and have it tell me if I’m getting a good deal.

nmstoker 6y ago

Would be handy if they applied this to an internal shop level Google Maps Would cut the need to speak to shop staff, who in many places in larger cities don't have the language skills to know what you're looking for.