10 ms·
Show HN: I built an AI that recognizes food
- subcosmos 10y agoBuilt using those newfangled convolutional neural network things. Trained caffe on a set of about a million food related images. Can't believe how accurate the classifier is... UI needs work. Would love feedback on how to make food logging more intuitive.
- teLeopardthy 10y agoAny info on the network itself (number of layers, input resolution, pooling, dropout etc)?
- subcosmos 10y agoStock Cafenet, trained from scratch.
- sandGorgon 10y agowhy caffe ? why not something like h2o or tensorflow ? just curious.
- AJRF 10y agoYou could probably outsource enough of the UI work to Fiverr to make a really decent looking app. Things like Main screen, App Icon and colour palette.
- swah 10y agoYour app kinda sucks now, but the detection is great. You could also do food identification as a service... I'm trying to use fatsecret/myfitnesspal for counting calories. Something like this could make it much easier! To me, a better workflow would be: take pictures of everything you eat and classify later (at night, maybe on the PC). Of course, this software would simplify the classification, the user would only fix any errors and adjust sizes. This workflow would be well suited to your app, which operates on a server? (my understanding). Congrats, I understand the relevance of AI for simple apps now ;)
- subcosmos 10y agoIve been thinking of that 'classify later' workflow. I think of it kind of like facebooks photo tagging, and it might even be better as web app that you get email reminders for to complete entries. It would also work nicely for background geolocation services, since if the photograph was taken near a restaurant the service could extrapolate those things. Anyways, I might make this project opensource and build a nice community around it. Depends on whether or not that model would best support my research goals.
- swah 10y agoGreat! And think about using the new data from the users to improve the system (that should probably be in your terms). How well does it work on plates with multiple foods "kinda mixed"?
- subcosmos 10y agoThats another huge design challenge Ive faced, specifically because it really complicates the data model and UI. Implicitly, since the app presents a list of possible identified foods, and the user can easily check multiple boxes, thats straightforward, but then having the app pull each of those items AND ask the user to estimate portions for each .... gets a little messy. So I put that complexity off for v2. That use case in particular would also be best off for a web-interface, I think.
- Keats 10y agoI would love a blog post about that!
- imacomputer2 10y agoLogin via Facebook? Come on! See you next app.
- imacomputer2 10y agoLogin via Facebook? Come on! See you next app.
- imacomputer2 10y agoLogin via Facebook? Come on! See you next app.
- deleted 10y ago[deleted]
- krashidov 10y agoHow much experience with machine learning did you have before you started this project?
- subcosmos 10y agoCut my teeth on the netflix prize several years back. Took ML in my graduate coursework, and have dabbled with it on and off since. I love the whole theory of matrix factorization and its myriad uses.
- krashidov 10y agoVery cool. What do you think helped you most? Would you recommend grad school?
- zopf 10y agoWhere'd you get your images? Food-101 plus something else? We're working on a similar component for our commercial behavioral-economics-driven app suite, targeted toward patients with chronic diseases. Do you use location as a way to filter the set of possible foods, as in Google's im2calories project/paper last year? They do some pretty awesome depth calculation stuff too: https://www.google.com/?ion=1&espv=2#q=im2calories+type:pdf https://www.google.com/?ion=1&espv=2#q=im2calories+type:pdf Edit: oops, should have read further. I see that you pulled appropriate terms from WordNet and used ImageNet to gather training images from Flickr. Cool!
- amelius 10y agoHow does it measure the glucose content of that sauce?
- AJRF 10y agoSomething like this would be a fantastic addition to MyFitnessPal. I find myself sometimes not logging food on that app because I couldn't be bothered to search for it. On the other hand if the product has a barcode I always scan it in because that is convenient.
- subcosmos 10y agoSame - its one of the reasons I wanted to make my own. That and data export functionality, which is requisite for research.
- swah 10y agoI feel like its easy to make a better app if you don't try to please all use cases - just try to make the food logging in less taps. Logging in those apps takes me around 5-10 taps plus typing the food name. Ify ou could do it in 2 or 3, it would be great.
- jcoffland 10y agoIf you're eatting a lot of food with barcodes on it then your doing it wrong.
- lj3 10y agoI wonder if somebody would buy a "highly opinionated calorie tracker" that only has non-barcode foods stored in it and mocks you when you try to add bad food to its database. If you miss your macros for the day, it would then text everyone in your phone that you're a fat weenie with no willpower.
- AJRF 10y agojcoffland would. He would throw money at it. A tracker that doesn't even allow for barcodes to be scanned? Perfect!
- SolaceQuantum 10y agoI don't understand this sentiment. My grocery store allows us to print barcodes that describe the weight and price of any produce or bulk order we purchase. This ranges from quinoa, oats, nuts, seeds, tea, along with the usual fruits and veggies. Also, don't bulk packages of vegetables tend to have barcodes on them inherently?
- aembleton 10y agoThis is not compatible with my device which is an LeTV X500, running Android 5.0.2. Is there any reason that it might not work? I'm in the UK, if that might be the reason.
- subcosmos 10y agoIm new to cordova (the toolkit I used to build the app), and I did notice that the number of supported devices is kind of low for my build. I may have included a plugin that narrows down the pool. I need to dig into this. Sorry about this!
- rymate1234 10y agoThis happening for me as well with a Moto X Play running Android 6.0.2 Looking forward to trying the app once I can install it though, looks neat
- subcosmos 10y agoFound my bug! Try again. Way more droids are supported now.
- lj3 10y agoThis is happening for me too. Samsung Galaxy S3 on US ATT running Android 4.4.4 Cyanogenmod 11. :(
- devstack 10y agoThat's very interesting indeed. Are you using any API?
- hardworker7882 10y agoMy Contacts are : Email: hack702j002d77s@gmx.us Phone number: +14437633327 ( Text me First if you want me to pick your call )
- subcosmos 10y agoFatsecret API for nutritional information and Foursquare for checkins, but installed and trained the neural network on my AWS instance for the prediction stuff.
- aantix 10y agoI remember the latest Android camera app being able to extrapolate distances in the app (thus simulating photos with large apertures/shallow depth of field). I wonder if you could use this visual distance algorithm to somehow judge portion sizes/plate sizes?
- subcosmos 10y agoHaven't thought of that! What I have considered is letting users put something like a quarter or ID card in the field of view for that purpose. Aside from that though, I figured Id do portion estimation down the road should this preliminary version take off.
- zaroth 10y agoPlease do blog about the technical details, if you could post a walkthrough of setting up and training the net it would be great!
- subcosmos 10y agoSomeday! Right now I need to put the fire out in my server room. Didn't quite prepare for HN traffic ...
- misiti3780 10y agowhere did you get the annotated training data ?
- subcosmos 10y agowordnet: https://wordnet.princeton.edu/wordnet/ https://wordnet.princeton.edu/wordnet/
- franciscop 10y agoIsn't it strange that in [1] the beginning of the "Normal" BMI has more risk of death than the beginning of the "Obese"? I'd even venture to say that the "Overweight" range has, in average, lower Mortality Risk than the "Normal". PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height. [1] https://www.infino.me/join/ https://www.infino.me/join/
- deleted 10y ago[deleted]
- subcosmos 10y agoIts been contested in a few papers now that overweight people can outlive normal weight, according to these BMI subcategories. But I am not surprised that things can be fuzzy, since ultimately its a continuum.
- bjterry 10y agoThe finding that people with an "Overweight" BMI have a lower rate of all-cause mortality is consistent and has been called the "obesity paradox." A very interesting study came out this month that addressed it.[0] Basically, they did a prospective study of 55,000 (mostly elderly women) where they measured their body fat percentage directly using DEXA scans. It turns out that when you control for body fat percentage, it's actually better for all cause mortality to have the highest BMI category and the lowest body fat percentage. Of course, the measure "BMI after controlling for body fat percentage" is kind of unintuitive, since BMI is supposed to be a crappy proxy for body fat percentage. But basically, as long as your body fat percentage is low, being bigger is healthier than being smaller. 0: http://annals.org/article.aspx?articleid=2499472 http://annals.org/article.aspx?articleid=2499472 (this paper used to be available, because I read the whole thing, but I guess it's paywalled now)
- subcosmos 10y agoThanks for posting this! I wonder if part of this paradox is also explainable because people with high muscle mass will tip towards the high end of the scale. Very cool!
- sandGorgon 10y agoWhere did you get your training data set? I'm trying to build a network to recognize faces from ID cards and match them to faces/selfies. Onenof the hard things has been to get a training data set.
- subcosmos 10y agoCheck out wordnet! I think they have face categories.
- zodPod 10y agoThis sounds like it could be used for the wrong purpose for sure.. Are you working for some nosy employer?
- sandGorgon 10y agoEmployer? No - we are automating paper applications for financial institutions. This includes a photo ID and a photo. That is checked manually today.
- zodPod 10y agoHaha nice! I was assuming someone was going to do some web screening before hiring people.
- virtualritz 10y agoFirst thing I tried: logging /two/ Granny Smith apples manually. I had do the entire process twice because there is no (obvious) way to change the serving size to 2.
- subcosmos 10y agoIf you click the 'no barcode' button to pull down a fatsecret nutrition record, there is a servings pulldown that is displayed. Its kinda hard to see. Working on the CSS
- npalli 10y agoThe classification looks good, congrats!. Where did you get the million labeled images to train your classifier?
- subcosmos 10y agoWordnet! Got a list of URLs from there and then hit flikr with some python.
- micro_cam 10y agoNice work! Another big user group of stuff like this is diabetics who need carb estimates to adjust insulin doses.
- subcosmos 10y agoI earned my PhD in diabetes research at the Mayo clinic. My whole goal for this project is to cure that damned illness. Aim high ...
- natch 10y agoIs there a writeup about how you approached this? All I see are a couple of app store links.
- subcosmos 10y agoIve been meaning to make a blog for some time now, since my nonprofit also has some preliminary research results with the genetic basis of obesity. Just havent had the time really. It will happen though. These neural networks are amazing and worth sharing.
- vinidy 10y agoGreat work! I'd love to chat with you to help feed your data set outside of wordnet. I've been collecting my own data set where I breakdown the macros in images for anyone who sends me their meals.
- mrfusion 10y agoI think something like this could get really good at estimating calories too. I'm saying within +\- 50 calories.
- subcosmos 10y agoI've heard that Google is working on that problem. They have way more resources than my one-man part time nonprofit operation, so I look forward to how awesome their thing will be when they finish it. Will probably opensource this, since I want to grow up a nice research community around quantified self fanatics.
- mark_l_watson 10y agoThat is very cool subcosmos! I am also very interested in both cooking+nutrition and AI (see for example my web site cookingspace.com). My contact information is in my HN profile - shoot me an email if you want to talk about AI, food technology and business, etc.
- gchristen 10y agoHey Subcomos, great work! We've been working on something pretty similar, an app called Logameal (only available in the UK). We have run into (are running into) a lot of those UI issues too and would be glad to help. Let me know if you'd like to chat.
- subcosmos 10y agoYour UI looks great!
- koolba 10y agoSomeone should combine this AI with something like Google glass (so you don't have to pull out your phone), a basic calorie/nutrition filter, and a shock bracelet[1]. That way when I look at a Krispy Kreme donut I get immediate feedback! [1]: http://pavlok.com/hello.php http://pavlok.com/hello.php
- subcosmos 10y agoI absolutely LOVE this idea .... :)
- misiti3780 10y agogoogle glass is dead though right ?
- internaut 10y agoI'm dying here. Sign me up for your startup. Let's call it: "American Dystopia"
- deleted 10y ago[deleted]
- Zolomon 10y agoThis should be in use at those self-registry machines at food markets. I imagine that you place the food item on the scanner and it will automatically register it or ask for human input when something unrecognized is found. Would decrease cheating perhaps?
- solaxun 10y agoI've been thinking of this exact thing for a few years now, good to see others are interested too. I think there is HUGE potential for an app that can accurately (within a reasonable error range) count macro-nutrients through pictures of food, but I think that is WAY more difficult than you would imagine. How can you train a classifier to tell how much butter is in a dish? How can you account for food that is literally underneath other food? Maybe a combination of spectrometer, weight, and photos would be a future solution. I certainly don't have an idea of how it would be implemented, but I think using data from all 3 of those sources could be a good step.
- subcosmos 10y agoAnother angle is post-consumption. Ive always wondered if precisely monitoring body temperature might tell you something about consumption, since there is this 'thermic effect of food'. Would be hard to get right though ... In the lab, we can use doubly-labelled water to precisely measure calorie intake and burn, but its so damned expensive. If only it were commercially viable.
- nickpsecurity 10y agoYou have a business opportunity here if you train that on fresh vegetables. Many people have trouble with these issues: 1. Knowing what the hell a (example) rhubarb is or looks like. 2. Knowing if it looks fresh or about to go bad. Being able to aim a camera slowly across an isle with it getting highlighted would solve first problem. Spinning vegetable around in front of the camera would could solve second. So, try that and make it a paid app.
- subcosmos 10y agoYou know, its funny. being a former fat guy, Im not entirely familiar with all of my vegetables. I took this app to a farmers market recently and it was actually helping me identify a few things ... Had never seen a beet before un-cut. Ashamed of my ignorance ;)
- nickpsecurity 10y agoI live in a semi-rural area. Gives me a bit more experience. I still don't know what all these things are in the supermarket, though. So, I can't judge. ;) That said, you should try training it on different kinds of produce that look very similar. One example I recall is Zucchini vs Cucumber as the stem at one side is the giveaway. Another is Nappa Cabbage vs Romaine Lettuce. Only two examples I could think of off top of head that would confuse people.
- nathan_f77 10y agoI would pay something like $5 per day for this if it was a real person who looks at my photos, categorizes them accurately, and enters them into MyFitnessPal. I won't trust an AI until it has a massive database and a million users. Until then, it's just not going to work well enough. You also need to convince users why they should use your app, instead of MyFitnessPal (which already has a massive database, including barcodes.)
- subcosmos 10y agoI have considered something like this as a business model, although I was thinking of nutrition coaching instead of mere logging itself, but thats actually a really great idea.
- jqueryin 10y agoThere's nothing stopping you from having a human aspect do the training and verification behind the scenes. It's how companies such as Expensify were able to handle their receipt OCR gracefully for such a wide variety of invoices and receipts. I'd err on the side of saying this is a more palpable sales approach to selling it as AI. Letting your customers know that while AI is running the show, things are being monitored closely and fixed by real people when misclassifications arise.
- subcosmos 10y agoIve been running the underlying nonprofit (infino.me) for about 3 years with this ambition to study gene-exercise-nutrition interactions, with very little interest in making any money at it. But this thought of having a paid service for people who don't want to go through the asspain of logging .... this does have me intrigued. Everyone tells me they want an effortless meal logging app ... and photo snaps might be the closest we can get short of having something like google glass persistently monitor intake.
- kbenson 10y agoI wouldn't pay $5 a day. I would pay $30 a month. Mechanical turk, with a note that the picture needs to contain some sort of scale (and a request to manually scale the item if you didn't include one and it's not obvious), would probably take you pretty far.
- KaoruAoiShiho 10y agoHow much hardware and how much time was used in training?
- subcosmos 10y agoTook a single AWS instance about a month. Not too bad.
- zeveb 10y agoPretty nifty, but please consider allowing creation of an account with a simple email address, or even anonymously. I really don't want to have to connect my Facebook or other social-networking information to your app in order to use it. Thanks for allowing one to try it out without logging in via another service; it'd be great to be able to actually have an account.
- subcosmos 10y agoEmail accounts are definitely on the long todo list.
- doppenhe 10y agoThis is great. Where are you hosting your caffe model? Just AWS with API Gateway?
- subcosmos 10y agoIts on the instance right now, but I could certainly post it to github at some point. I think I need to retrain it though and swap a few categories out. Theres a few animals in there ;) Also, been needing to train on non-food items to weed out abuse.
- holografix 10y agoThis has HUGE potential. Some of the comments here disregard the fact that AI can do a better job than humans. An expert chef looking at dishes will make an educated guess on the ingredients and quantities that went into a dish and will have a better idea of the caloric content. The AI could learn a similar skill, identifying whether a dish contains pastry, buttery sauces, fatty meat etc etc.
- johnPullman 10y agoAnother app that is similar to this is called tuingle http://tuingle.com http://tuingle.com
- halonewstza 10y agohalonewstza.blogspot.com
- manyama66 10y agoVisit halonewstza.blogspot.com