6 ms·
Ha didn't expect this to end up here. If anyone is interested, I'm working on a blogpost explaining how we built the app in detail… It uses embedded TensorFlow
by timanglade 9y ago
Ha didn't expect this to end up here. If anyone is interested, I'm working on a blogpost explaining how we built the app in detail… It uses embedded TensorFlow on device (better availability, better speed, better privacy, $0 cost), with a custom neural net inspired by last month's MobileNets paper, built & trained with Keras. It was loads of fun to build :)
- skraelingjar 9y agoI would love to look at the source code for this!
- DonHopkins 9y agoAnd the training set! Mmmmmm!
- asadlionpk 9y agoDamn! I thought it was using GCloud ML API but it felt so fast. The one in the show felt so slow in comparison! Kudos Jian Yang!
- Swizec 9y agoHow does one get this job? This sounds ridiculously fun.
- timanglade 9y agoIt was random, I was already working on the show as what Hollywood calls a (technical) “consultant”: advising on storylines, dialogue, background assets, etc. When this idea popped up, someone suggested we build the app for real. We took a try and ended up building the entire thing in-house with the crew, as opposed to hiring an external agency to do it for us.
- scrollaway 9y agoSounds like you're handling yourself even better than the characters in the show. :) Edit: I guess it's official: The time of XKCD 1425 has passed. https://xkcd.com/1425/ https://xkcd.com/1425/ September 2014; Randall undershot that by a bit :)
- timanglade 9y agoHa object recognition is still far from foolproof if you ask me, but we're getting there!
- jrimbault 9y agoIf I understood correctly by reading the comments, this was built for a show ? What show ? EDIT: ah, it's in your profile ;)
- eriknstr 9y agoFor future reference, the show is Silicon Valley.
- deleted 9y ago[deleted]
- gm-conspiracy 9y agoWhat was your path to becoming a Hollywood technical "consultant"? Would you write a blog post about that journey?
- laktek 9y agoHow long did it take you to build the app? How many devs worked on it and who got to feed the training data set?
- msdog 9y agoYou can fetch a dataset of 1273 labeled hot-dog images and 123287 non-hot-dog images from mscoco training dataset: search for "hot dog" or click on the hot dog icon: http://mscoco.org/explore/ http://mscoco.org/explore/
- elldoubleyew 9y agoWere the images gathered by Stanford undergrads?
- timanglade 9y agoI should say a lot of people made the app possible, including the show's awesome producers, writers, designers, and a lot of kind folks at HBO. To answer your question, I was the only dev on the project, and I've been working on it since last Summer, on a very part-time basis (some nights and weekends). A lot of time was spent learning Deep Learning to be honest. The last revision of the neural net was designed & trained in less than a month of nights/weekend work but obviously couldn't have been achieved without the preceding months of work — but if I was starting today knowing what I know now yeah it'd probably be about a month of work. The React Native shell around the neural net was just a few weekends worth of work — mostly it was about finding the right extensions, tuning a few things for rendering/performance, and like a whole weekend dealing with the UX around iOS permissions to access the camera & photos (lol it's seriously so complicated).
- DonHopkins 9y agoWere you able to expense the hot dogs? In the early 90's, I had the honor of corresponding with "Uncle Frank" Webster [1], the curator of the Hot Dog Hall of Fame [2]! It boasts of more than 2,500 frankfurter items, including the Lamborweenie, a dog on wheels that "Uncle Frank" hopes some day to race against the Oscar Meyer Weinermobile [3]. Unfortunately the hot dog museum is currently closed and in (hopefully refrigerated) storage [4], otherwise the museum, gallery and gift shop would be a great place to train your app. He asked for permission to publish in his newsletter a gif [5] of a photo I'd taken and put on my web site of the Doggie Diner head [6] on Sloat Boulevard in San Francisco [7]. (This was years before John Law acquired all the Doggie Diner heads he could find for his restoration project [8], so there weren't so many photos on them on the internet at the time.) Of course I gave him permission because he asked so politely, and although at first he seemed a little creepy, I could tell he was authentic and sincere since he signed his correspondence: "With Relish, Uncle Frank." [9] And he even delivered on his promise to send me copies of his newsletter! He enthusiastically informed me that the highest quality hot dogs he's ever found are from Top Dog in Berkeley [10]. He admitted that he went through their dumpster to find out where they sourced them from because they wouldn't tell him, and he vouches that they are made from the finest possible ingredients. I agree with Uncle Frank that Top Dog has really excellent hot dogs (ask for them cooked butterfly style), and I take him at his word that they come from a reputable source. You can now actually order them online [11] if you would prefer not to go through their dumpster. [1] http://www.smithsonianmag.com/arts-culture/hot-dogs-are-us-68009224/ http://www.smithsonianmag.com/arts-culture/hot-dogs-are-us-6... [2] http://thehotdoghalloffame.blogspot.nl/ http://thehotdoghalloffame.blogspot.nl/ [3] https://www.questia.com/magazine/1G1-19007874/the-almost-all-american-all-time-best-hall-of-fame https://www.questia.com/magazine/1G1-19007874/the-almost-all... [4] http://www.roadsideamerica.com/tip/20747 http://www.roadsideamerica.com/tip/20747 [5] http://donhopkins.com/home/catalog/images/DoggieDiner.gif http://donhopkins.com/home/catalog/images/DoggieDiner.gif [6] https://en.wikipedia.org/wiki/Doggie_Diner https://en.wikipedia.org/wiki/Doggie_Diner [7] http://www.roadsideamerica.com/story/14441 http://www.roadsideamerica.com/story/14441 [8] https://www.kickstarter.com/projects/2118888480/doggie-diner-makeover https://www.kickstarter.com/projects/2118888480/doggie-diner... [9] http://www.downtownmakeover.com/comments.asp?CZID=17 http://www.downtownmakeover.com/comments.asp?CZID=17 [10] https://www.yelp.com/biz/top-dog-berkeley https://www.yelp.com/biz/top-dog-berkeley [11] http://www.topdoghotdogs.com/mailorder.html http://www.topdoghotdogs.com/mailorder.html
- deleted 9y ago[deleted]
- minimaxir 9y agoHow large is the training set / how did you get a sufficient amount of hot dog pics for a custom model, especially since you did not have a class of naive Stanford CS students at your disposal?
- timanglade 9y agoAbout 150k total images, 3k of which were hotdogs. The results are far from perfect (there's ton of subtle — or hilarious — ways to trick the app) but it was better than using a pre-trained model or doing transfer learning, accuracy-wise (honestly it was even better than using Cloud APIs). As for the difficulty in preparing the training set, I'll just say I definitely empathize with Dinesh and Jian Yang’s feelings in episode 4 :D
- austenallred 9y ago> there's ton of subtle — or hilarious — ways to trick the app ಠ_ಠ
- alexcnwy 9y agoAWESOME! Can you please elaborate a bit more on the model architecture and what you tried with respect to transfer learning? Did you use an imagenet architecture e.g. VGG and retrain from scratch or a custom architecture? Did you try chop off the last 1/2/3 layers of a prerrained mode and fine-tune? Bonus points: 1. How much better were your results trained from scratch vs fine-tuned? 2. How long did it take to train your model and on what hardware? :)
- timanglade 9y agoHey so I actually tried Vgg, Inception and SqueezeNet, out of the box, chopped and trained from scratch (SqueezeNet only for the latter due to resource constraints). We ended up with a custom architecture trained from scratch due to runtime constraints more so than accuracy reasons (the inference runs on phones, so we have to be efficient with CPU + memory), but that model also ended up being the most accurate model we could build in the time we had. (With more time/resources I have no doubt I could have achieved better accuracy with a heavier model!) Training the final model took about 80 hours on a single Nvidia GTX 980 Ti (the best thing I could hook to my MacBook Pro at the time). That's for 240 epochs (150k images in an epoch) ran in 3 rate annealing phases, each phase being a handful of CLR (cyclical learning rate) phases. I'll answer in more detail in the full blogpost, it's a bit complicated to explain in a comment. I'll have charts & figures for y'all :)
- bitmapbrother 9y agoIs there a reason you didn't release on Android? The app was even demoed on a Pixel.
- pkulak 9y agoIt was written in Objective C?
- timanglade 9y agoIt's actually written in React Native with a fair bit of C++ (TensorFlow), and some Objective-C++ to glue the two. One cool thing we added on top of React Native was a hack to let us inject new versions of our deep learning model on the fly without going through an App Store review. If you thought injecting JavaScript to change the behavior of your app was cool, you need to try injecting neural nets, it's quite a feeling :D
- codesternews 9y agocan you please open source it. It would be great if you open source it.
- kgc 9y agoWhat was the hack?
- asadlionpk 9y agoWhen using React Native (and also storing your network in JS) you can choose to push updates and fixes to JS part of your code without going through store again. Check: http://microsoft.github.io/code-push/ http://microsoft.github.io/code-push/
- timanglade 9y agoYup that's basically it. The hack was just in getting Tensorflow to accept/load its neural network definition from the JS bundle (what CodePush distributes for you) rather than from the main Cocoa bundle.
- rcamp 9y agoI instantly noticed the speed and thought, "did they put TensorFlow on the device?!". props
- alexcnwy 9y agoyes please - would love to read that blogpost!
- deleted 9y ago[deleted]
- gm-conspiracy 9y agoGreat work, man. The effort you put into the show is much appreciated.
- vernie 9y agoHow did you not expect it to end up here?
- timanglade 9y agoI honestly thought the app itself would come across as too limited — and I wasn't quite sure how HN felt about the show it's attached to. I was preparing that technical blogpost specifically for HN because I thought that would be a more hacker-centric way of looking at the same thing.
- imsofuture 9y agoYou can tell a lot about SV-types based on how they feel about the show SV...
- a_c 9y agoHad not heard MobileNets before. Mind sharing which paper are you referring to?
- ip_addr 9y agoI believe they are referring to this paper titled "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications" https://arxiv.org/pdf/1704.04861.pdf https://arxiv.org/pdf/1704.04861.pdf
- timanglade 9y agoYup that's the one! It was probably the most instrumental paper in making this app, along with the CLR paper: https://arxiv.org/abs/1506.01186 https://arxiv.org/abs/1506.01186
- 659087 9y ago> It uses embedded TensorFlow on device (better availability, better speed, better privacy, $0 cost) It's sad that this isn't more common outside of hot dog detectors.
- timanglade 9y agoWell, in fairness, it's relatively difficult to do today: first of all, most neural architectures don't fit on mobile, because most of the research is either more academic than production-focused, or because people are chasing the best possible accuracy as opposed to accuracy under constraints. There's still great applicable research, and in particular the SqueezeNet, QuickNet and MobileNets papers are great in that respect. Then there are some issues with just the tooling. TensorFlow is great, and works great on mobile, but it was a bit hard to get it working. The whole team there is doing an awesome job documenting and improving everything though! It's just early and not enough users have built apps with it to get the kind of production-readiness you get out of non-Machine Learning components.
- redxblood 9y agoDefinitely! Link please?
- dnautics 9y agothe funny thing is that there are already "seefood"s out there. Among others: https://www.infino.me/hungrybot https://www.infino.me/hungrybot
- subcosmos 9y agoDaww! I'm the author, thanks! I was REALLY depressed after watching these last two episodes. HungryBot was easily the coolest thing I've done in my unfunded nonprofit. But it was too buggy to fly. Kicked it out of the nest too early, but to the best of my knowledge it was the first deep learning food recognition app. Looking for philanthropists if anyone wants to help me study diabetes with apps like this.
- dnautics 9y agopsst this is isaac
- craigm26 9y agoI would love to read this.