8 ms·

How to build a robot that “sees” with $100 and TensorFlow

365 points by nogaleviner 10y ago

urvader 10y ago
I would like to know how long it "thinks"- it is clear the camera is paused for a while while the robot parses the image..
- lukas 10y ago
  It thinks for around 3 seconds. It doesn't use the PI's GPU - I bet that if it did it could get a lot faster. I bet you could monkey around with the compiler flags and speed it up. If anyone working on TensorFlow has some ideas, I'd love to hear them.
visarga 10y ago
Great project. Locomotion and vision are pretty advanced compared to grasping and complex handling of objects. If we could have a workable arm, it would be much more interesting in applications.
- Animats 10y ago
  There are some affordable robot arms. uArm has a hobbyist-level arm. I have one, and I made a 6DOF force sensor out of a 3Dconnexion SpaceNavigator on the end and got it to talk to ROS. I want to hook a classifier system up to it so it can use tools like end wrenches and get them onto a bolt by feel. Robot manipulation in unstructured situations still sucks, though. Willow Garage was making progress, but didn't last.
icemelt8 10y ago
This was amazing, I am amazed at your command of both hardware and software technology. Even as a Software Engineer, I have a hard time trying to make TensorFlow do something for me.
- dharma1 10y ago
  To do the task in the article (classify images, pretrained model) - it's pretty easy - just follow the tutorial here: https://www.tensorflow.org/versions/r0.10/tutorials/image_recognition/index.html https://www.tensorflow.org/versions/r0.10/tutorials/image_re...
  - sidarape 10y ago
    I tried that (https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html https://www.tensorflow.org/versions/r0.9/how_tos/image_retra...) which seems to work well.
    - dharma1 10y ago
      yep, that's good if you want to retrain a pre-trained model for specific categories on new image data (that are pretty close to imagenet type images)
      - sidarape 10y ago
        What do you mean by "pretty close"? What would be "not close"?
        dharma1 10y ago
        Something that looks very different to the ImageNet dataset. Microscope, ultrasound, satellite, x-ray etc images
        sidarape 10y ago
        Ok, I see. Thank you.
  - Omnipresent 10y ago
    Is this available in keras as well?
rbanffy 10y ago
> recognizing arbitrary objects within a larger image has been the Holy Grail of artificial intelligence The Holy Grail is general AI. Recognizing objects is a side quest, perhaps a required step, but, by no means, the end goal.
- ryandamm 10y ago
  I think we're going to get better vision before we get hard AI. Just my two cents. (Then again, I'm biased.)
- taneq 10y ago
  Like so many other things in the field of AI, general object recognition was the "holy grail" because it was assumed that it required AGI. Now we've figured out a way to do general object recognition without AGI.
  - Swizec 10y ago
    Whenever we figure out how to do something, we stop calling it AI or AGI. If the trend continues, will we eventually have a general AI, but won't consider it anything special? Will it have been just a small incremental step by then?
    - kowdermeister 10y ago
      I think it will need some kind of breakthrough. Current advancements are probably incremental as you stated, but having an AGI might need some new theory we don't have currently.
      - xiphias 10y ago
        Deep learning is the opposite of incremental. For a long time it was not clear whether/how we can learn multi layer networks efficiently. ImageNet changed everything.
        10y ago
dharma1 10y ago
Dis the author publish a repo for this? It's easy getting tensorflow going for basic image classification but the hard part is actually making the robot move in a way that makes sense - using the camera and the sonar data to make decisions and then drive the motors. Or is this not autonomous?
- OilDerek 10y ago
  "I then built a simple Python webserver to spin the wheels of the robot based on keyboard commands that made for a nifty remote control car." So, not autonomous it would seem. With that and an arm, though, you could eventually get it to play fetch...
  - dharma1 10y ago
    Ah, right. You could do this project with a cheap RC car and a phone running TF stuck on it
    - jd20 10y ago
      The mention of sonar sensors, used to prevent the car from running into obstacles, made me think the author was planning to make the robots autonomous too, but maybe just hasn't gotten that part worked out yet. I was also curious how the robot would know when to stop and try classifying something, without the operator explicitly telling it so.
    - ultrasounder 10y ago
      Since when did TF started to run on stock phones?
      - dharma1 10y ago
        A few months ago. https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/ios_examples https://github.com/tensorflow/tensorflow/tree/master/tensorf... https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android https://github.com/tensorflow/tensorflow/tree/master/tensorf...
bernardopires 10y ago
Just a nit, but the author keeps talking about object recognition while what he was actually doing is image classification. Object recognition actually consists of two tasks, one is classifying the object (this is a beer bottle) and the other is also says where in the image the object is. Additionally it can/should detect multiple objects in the image. This is a more complex than classification, which only associates one category with the image.
- dharma1 10y ago
  http://pjreddie.com/darknet/yolo/ http://pjreddie.com/darknet/yolo/, https://github.com/daijifeng001/MNC https://github.com/daijifeng001/MNC, https://bitbucket.org/aquariusjay/deeplab-public-ver2 https://bitbucket.org/aquariusjay/deeplab-public-ver2 or similar should do the job. Choose depending on how fast it needs to be, and how accurate the segmentation boundaries need to be
  - oxymoron 10y ago
    Here's another one: https://github.com/facebookresearch/deepmask https://github.com/facebookresearch/deepmask
- bonoboTP 10y ago
  I don't think there's a consistent terminology. In my computer vision class we called it "object recognition" when it was about recognizing one specific object (this particular car) and "object classification" when deciding the category of the object in the image (in general, like 'car', 'bottle'). One may also call the localization of the object as object detection and subsequent classification. But I don't think it's too important how we call it as long as we understand what the task is.
- AndrewKemendo 10y ago
  Actually the tensorflow implementation he uses does both segmentation and classification and returns a probabilistic graph of objects. For his application, it's only returning the top result, so it looks more basic than it is.
salex89 10y ago
My biggest current question is which keyboard is this, on the image in the article?! https://d3ansictanv2wj.cloudfront.net/Figure_2-985cd20ea0c0bcd68a7d896f11312edf.jpg https://d3ansictanv2wj.cloudfront.net/Figure_2-985cd20ea0c0b...
- selckin 10y ago
  http://gadgetsin.com/flexible-bluetooth-mni-wireless-keyboard.htm http://gadgetsin.com/flexible-bluetooth-mni-wireless-keyboar...
- cezor 10y ago
  Looks like Karnotech Foldable Silicone Keyboard
  - salex89 10y ago
    Looks like you are right. Based on the image I thought it was something mechanical. Looks nice, but I'm not fond of foldable keyboards :( .
- lukas 10y ago
  This was the keyboard - https://www.amazon.com/gp/product/B0179N39KS/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 https://www.amazon.com/gp/product/B0179N39KS/ref=oh_aui_sear... - it's incredibly light and cheap but one month later I'm not sure I'd recommend it. A bunch of the keys were randomly mapped incorrectly and it's started to flake out on me a little bit.
criddell 10y ago
This reminds me of a low res vision system I read about 20 years ago: http://www.seattlerobotics.org/encoder/jan97/lowresv.html I've always been kind of intrigued by what is possible with very simple hardware.
- gugagore 10y ago
  Do you know what ever happened to the Encoder? I used to be so excited as a kid when a new issue came out.
  - criddell 10y ago
    No I don't. It was a fantastic resource and I bet it was a lot of work to put one together. I sure appreciated the people that took the time to make it. I always wanted to find some way to get to one of their meetings but it never worked out.
forgotAgain 10y ago
Sorry for the off topic but is anyone else getting very high cpu usage from O'Reilly websites? Any known resolution or work around? With Chrome developer tools I see one error: "Uncaught SecurityError: Failed to read the 'localStorage' property from 'Window': Access is denied for this document."
nojvek 10y ago
Oh my god. You are trying to build the exact thing I am trying to build. Albeit you've made much more progress. I'm still soldering wires into the motors. You should take off the paper from acrylic. The transparent effect makes it look awesome. My goal is to make a raspberry pi bot that plays indoor fetch. I would love to have a chat with you.

365 points by nogaleviner 10y ago

I would like to know how long it "thinks"- it is clear the camera is paused for a while while the robot parses the image..

lukas 10y ago
It thinks for around 3 seconds. It doesn't use the PI's GPU - I bet that if it did it could get a lot faster. I bet you could monkey around with the compiler flags and speed it up. If anyone working on TensorFlow has some ideas, I'd love to hear them.

visarga 10y ago

Great project. Locomotion and vision are pretty advanced compared to grasping and complex handling of objects. If we could have a workable arm, it would be much more interesting in applications.

Animats 10y ago
There are some affordable robot arms. uArm has a hobbyist-level arm. I have one, and I made a 6DOF force sensor out of a 3Dconnexion SpaceNavigator on the end and got it to talk to ROS. I want to hook a classifier system up to it so it can use tools like end wrenches and get them onto a bolt by feel. Robot manipulation in unstructured situations still sucks, though. Willow Garage was making progress, but didn't last.

icemelt8 10y ago

This was amazing, I am amazed at your command of both hardware and software technology. Even as a Software Engineer, I have a hard time trying to make TensorFlow do something for me.

dharma1 10y ago
To do the task in the article (classify images, pretrained model) - it's pretty easy - just follow the tutorial here: https://www.tensorflow.org/versions/r0.10/tutorials/image_recognition/index.html https://www.tensorflow.org/versions/r0.10/tutorials/image_re...
- sidarape 10y ago
  I tried that (https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html https://www.tensorflow.org/versions/r0.9/how_tos/image_retra...) which seems to work well.
  - dharma1 10y ago
    yep, that's good if you want to retrain a pre-trained model for specific categories on new image data (that are pretty close to imagenet type images)
    - sidarape 10y ago
      What do you mean by "pretty close"? What would be "not close"?
      - dharma1 10y ago
        Something that looks very different to the ImageNet dataset. Microscope, ultrasound, satellite, x-ray etc images
        sidarape 10y ago
        Ok, I see. Thank you.
- Omnipresent 10y ago
  Is this available in keras as well?

rbanffy 10y ago

> recognizing arbitrary objects within a larger image has been the Holy Grail of artificial intelligence The Holy Grail is general AI. Recognizing objects is a side quest, perhaps a required step, but, by no means, the end goal.

ryandamm 10y ago
I think we're going to get better vision before we get hard AI. Just my two cents. (Then again, I'm biased.)
taneq 10y ago
Like so many other things in the field of AI, general object recognition was the "holy grail" because it was assumed that it required AGI. Now we've figured out a way to do general object recognition without AGI.
- Swizec 10y ago
  Whenever we figure out how to do something, we stop calling it AI or AGI. If the trend continues, will we eventually have a general AI, but won't consider it anything special? Will it have been just a small incremental step by then?
  - kowdermeister 10y ago
    I think it will need some kind of breakthrough. Current advancements are probably incremental as you stated, but having an AGI might need some new theory we don't have currently.
    - xiphias 10y ago
      Deep learning is the opposite of incremental. For a long time it was not clear whether/how we can learn multi layer networks efficiently. ImageNet changed everything.
      - 10y ago

dharma1 10y ago

Dis the author publish a repo for this? It's easy getting tensorflow going for basic image classification but the hard part is actually making the robot move in a way that makes sense - using the camera and the sonar data to make decisions and then drive the motors. Or is this not autonomous?

OilDerek 10y ago
"I then built a simple Python webserver to spin the wheels of the robot based on keyboard commands that made for a nifty remote control car." So, not autonomous it would seem. With that and an arm, though, you could eventually get it to play fetch...
- dharma1 10y ago
  Ah, right. You could do this project with a cheap RC car and a phone running TF stuck on it
  - jd20 10y ago
    The mention of sonar sensors, used to prevent the car from running into obstacles, made me think the author was planning to make the robots autonomous too, but maybe just hasn't gotten that part worked out yet. I was also curious how the robot would know when to stop and try classifying something, without the operator explicitly telling it so.
  - ultrasounder 10y ago
    Since when did TF started to run on stock phones?
    - dharma1 10y ago
      A few months ago. https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/ios_examples https://github.com/tensorflow/tensorflow/tree/master/tensorf... https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android https://github.com/tensorflow/tensorflow/tree/master/tensorf...

bernardopires 10y ago

Just a nit, but the author keeps talking about object recognition while what he was actually doing is image classification. Object recognition actually consists of two tasks, one is classifying the object (this is a beer bottle) and the other is also says where in the image the object is. Additionally it can/should detect multiple objects in the image. This is a more complex than classification, which only associates one category with the image.

dharma1 10y ago
http://pjreddie.com/darknet/yolo/ http://pjreddie.com/darknet/yolo/, https://github.com/daijifeng001/MNC https://github.com/daijifeng001/MNC, https://bitbucket.org/aquariusjay/deeplab-public-ver2 https://bitbucket.org/aquariusjay/deeplab-public-ver2 or similar should do the job. Choose depending on how fast it needs to be, and how accurate the segmentation boundaries need to be
- oxymoron 10y ago
  Here's another one: https://github.com/facebookresearch/deepmask https://github.com/facebookresearch/deepmask
bonoboTP 10y ago
I don't think there's a consistent terminology. In my computer vision class we called it "object recognition" when it was about recognizing one specific object (this particular car) and "object classification" when deciding the category of the object in the image (in general, like 'car', 'bottle'). One may also call the localization of the object as object detection and subsequent classification. But I don't think it's too important how we call it as long as we understand what the task is.
AndrewKemendo 10y ago
Actually the tensorflow implementation he uses does both segmentation and classification and returns a probabilistic graph of objects. For his application, it's only returning the top result, so it looks more basic than it is.

salex89 10y ago

My biggest current question is which keyboard is this, on the image in the article?! https://d3ansictanv2wj.cloudfront.net/Figure_2-985cd20ea0c0bcd68a7d896f11312edf.jpg https://d3ansictanv2wj.cloudfront.net/Figure_2-985cd20ea0c0b...

selckin 10y ago
http://gadgetsin.com/flexible-bluetooth-mni-wireless-keyboard.htm http://gadgetsin.com/flexible-bluetooth-mni-wireless-keyboar...
cezor 10y ago
Looks like Karnotech Foldable Silicone Keyboard
- salex89 10y ago
  Looks like you are right. Based on the image I thought it was something mechanical. Looks nice, but I'm not fond of foldable keyboards :( .
lukas 10y ago
This was the keyboard - https://www.amazon.com/gp/product/B0179N39KS/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 https://www.amazon.com/gp/product/B0179N39KS/ref=oh_aui_sear... - it's incredibly light and cheap but one month later I'm not sure I'd recommend it. A bunch of the keys were randomly mapped incorrectly and it's started to flake out on me a little bit.

criddell 10y ago

This reminds me of a low res vision system I read about 20 years ago: http://www.seattlerobotics.org/encoder/jan97/lowresv.html I've always been kind of intrigued by what is possible with very simple hardware.

gugagore 10y ago
Do you know what ever happened to the Encoder? I used to be so excited as a kid when a new issue came out.
- criddell 10y ago
  No I don't. It was a fantastic resource and I bet it was a lot of work to put one together. I sure appreciated the people that took the time to make it. I always wanted to find some way to get to one of their meetings but it never worked out.

forgotAgain 10y ago

Sorry for the off topic but is anyone else getting very high cpu usage from O'Reilly websites? Any known resolution or work around? With Chrome developer tools I see one error: "Uncaught SecurityError: Failed to read the 'localStorage' property from 'Window': Access is denied for this document."

nojvek 10y ago

Oh my god. You are trying to build the exact thing I am trying to build. Albeit you've made much more progress. I'm still soldering wires into the motors. You should take off the paper from acrylic. The transparent effect makes it look awesome. My goal is to make a raspberry pi bot that plays indoor fetch. I would love to have a chat with you.