5 ms·

Queries like this already work well in my experiments. For a large fraction of useful queries it's mostly a matter of engineering. My current model (which is ri

Queries like this already work well in my experiments. For a large fraction of useful queries it's mostly a matter of engineering. My current model (which is right now at state of the art performance) doesn't yet take into account the spatial relationships of things, but it would be good at retrieving pictures that have you, backpack and wood logs somewhere in it. Reasoning about relations is harder because in the image space of x,y coordinates spatial relationships don't make too much sense, and we haven't yet figured out good ways to do spatial, 3D understanding from images (this is to a large part a lack of data problem). All we can do is "see" that there is a backpack detection nearby a person detection, and that it seems to be of relatively compatible size in the image plane. All that being said, a large fraction of queries like this will work _today_, with what we have and the challenge is simply implementing it. However, telling the difference between whether the wood logs are inside your backpack, on your head, or in your hands might take a few more years :)

drcomputer 12y ago
Is it because of what the items are, specifically? Can you add an attribute to every potential item detection that includes a proportionality measure to something standardized? Wouldn't this be a "simple" (that is, definable) calculation if every item that can be detected (meaning the item is already cataloged) has a standardized dimensional measurement in relation to the camera?
- karpathy 12y ago
  I'm not sure if I fully understand, but consider that "a person with backpack" tells you extremely little about the actual positions of those two items in the 2-D coordinate system of the image: It could be an extreme closeup where only the torso/head of the person is visible and a bit of the backpack, or it could be a tiny person anywhere in the image with a small blob sticking out from their back. The backpack could also be large, small, very occluded, etc. So there is less information in those locations than you might think.
  - drcomputer 12y ago
    > I'm not sure if I fully understand, but consider that "a person with backpack" That's not the only information that you have to work with though. If I understand correctly, object recognition is trained from a data set. That means that every classification is a map from a cluster of images to a single label. This cluster of images can be quantified in terms of proportionality, in relation to other objects identified in the scene. We are beginning with the assumption we can compute the identification of individual objects when they are grouped. The next thing to train on is groups of objects as a unit structure, where we infer and learn a proportion relation. Consider a macro shot of an ant with a baseball in the background and perhaps someone's finger, and a shot of all three with only one dimension altered to the camera. In the first, all three objects appear to be the same size. In the other, object one and object two express a ratio. Then we can normalize vectors based on the 3 computed ratios (object a : object b). Distance is not solved, but relative distance is. Then, if you can build a catalog of standardized measurements (common objects with a defined distance from a camera, like a face 3 feet away from a camera), then you can start training for actual distances. I'm thinking about this like how I would reason about the objects in a raycaster: their definition in the machine versus what gets projected onto a 2d plane, and how moving the objects and camera affects the final projected plane. I'm certain I'm glossing over a lot of details that would actually be difficult in implementation, not to mention computational complexity. > So there is less information in those locations than you might think. I agree, but the only way I understand machine learning at all, as a computer scientist, is "gradual accumulation, definition, and relational organization of humanly defined atomic units". I don't think there are any magic tricks, just a lot of carefully constructed computation, even if it spans over generations of computers, groups, people.
ilyaeck 12y ago
Andrej, I totally get it, but I think what you have some far may already be pretty good to form the foundation of a product that has a clear/big need in the market. If you'd like to dig in a bit more and bounce ideas, ping me at ilya@robinlabs.com.
ilyaeck 12y ago
Oh, and congrats - this is super-exciting!

drcomputer 12y ago

Is it because of what the items are, specifically? Can you add an attribute to every potential item detection that includes a proportionality measure to something standardized? Wouldn't this be a "simple" (that is, definable) calculation if every item that can be detected (meaning the item is already cataloged) has a standardized dimensional measurement in relation to the camera?

karpathy 12y ago
I'm not sure if I fully understand, but consider that "a person with backpack" tells you extremely little about the actual positions of those two items in the 2-D coordinate system of the image: It could be an extreme closeup where only the torso/head of the person is visible and a bit of the backpack, or it could be a tiny person anywhere in the image with a small blob sticking out from their back. The backpack could also be large, small, very occluded, etc. So there is less information in those locations than you might think.
- drcomputer 12y ago
  > I'm not sure if I fully understand, but consider that "a person with backpack" That's not the only information that you have to work with though. If I understand correctly, object recognition is trained from a data set. That means that every classification is a map from a cluster of images to a single label. This cluster of images can be quantified in terms of proportionality, in relation to other objects identified in the scene. We are beginning with the assumption we can compute the identification of individual objects when they are grouped. The next thing to train on is groups of objects as a unit structure, where we infer and learn a proportion relation. Consider a macro shot of an ant with a baseball in the background and perhaps someone's finger, and a shot of all three with only one dimension altered to the camera. In the first, all three objects appear to be the same size. In the other, object one and object two express a ratio. Then we can normalize vectors based on the 3 computed ratios (object a : object b). Distance is not solved, but relative distance is. Then, if you can build a catalog of standardized measurements (common objects with a defined distance from a camera, like a face 3 feet away from a camera), then you can start training for actual distances. I'm thinking about this like how I would reason about the objects in a raycaster: their definition in the machine versus what gets projected onto a 2d plane, and how moving the objects and camera affects the final projected plane. I'm certain I'm glossing over a lot of details that would actually be difficult in implementation, not to mention computational complexity. > So there is less information in those locations than you might think. I agree, but the only way I understand machine learning at all, as a computer scientist, is "gradual accumulation, definition, and relational organization of humanly defined atomic units". I don't think there are any magic tricks, just a lot of carefully constructed computation, even if it spans over generations of computers, groups, people.

ilyaeck 12y ago

Andrej, I totally get it, but I think what you have some far may already be pretty good to form the foundation of a product that has a clear/big need in the market. If you'd like to dig in a bit more and bounce ideas, ping me at ilya@robinlabs.com.

ilyaeck 12y ago

Oh, and congrats - this is super-exciting!