13 ms·

Editing Photographs in Three Dimensions

74 points by scapbi 12y ago

Alupis 12y ago
Very interesting project! I'm just curious about her degrees. It seems she achieved both a B.S. and M.S. at the same time from the same university. How is that possible/allowed? The schools around me don't even accept people into their masters programs until after you have completed a B.S. and have some relevant work experience as well.
- dragonwriter 12y ago
  Some schools have combined B.S./M.S. programs.
  - Alupis 12y ago
    Oh man, I'm at the wrong University then! No but really, I think it's more beneficial for a student to take a year or two off after completing their BS and work in their field. The experience you pick up in the workplace actually doing the thing you went to school for is invaluable. That is unless you worked in your field prior-to or while you were attending university.
    - zo1 12y ago
      Some universities, such as the one I went to, have degrees that combine some normal B.Sc degrees with honors. Such that after completing it, you can start with your masters.
    - emddudley 12y ago
      RIT requires a full year of work experience (co-op) for a BS in Computer Engineering.
- emddudley 12y ago
  Rochester Institute of Technology (RIT), the school in question, has a dual degree program where you can use some graduate classes to satisfy both the BS and MS credit requirements. She still had to write a research thesis.
therobot24 12y ago
Cool research but something bothers me about students/professors asking for money on sites like indiegogo/kickstarter/etc.
- deleted 12y ago
  [deleted]
- StavrosK 12y ago
  Yeah, and it doesn't look like you get anything for your money, either? It wouldn't be very nice if I paid $20 and helped them make a product only to have to buy it later, or if nothing but a research paper came out. This looks more like a donation than crowdfunding.
  - mistercow 12y ago
    >This looks more like a donation than crowdfunding. Those aren't mutually exclusive concepts.
- mistercow 12y ago
  Looks like that's the whole point of this site. Why does crowdfunding science bother you?
  - gioele 12y ago
    > Why does crowdfunding science bother you? There is already a fair crowdfunding scheme for science called "taxes". I feel the increase of popularity of crowdfunding schemes for science or health care is detrimental to the cause. They will never be able to approach the same level of funding that a state can give away, yet they will be a good excuse for the politics to reduce the amount of money spent on these issues: "The crowdfunding schemes are already taking care of it, aren't they? Why should we waste taxpayers' money? If prof. X wants its experiment to be funded why doesn't it ask for money on indiegogo?" For those who may thing "nobody is going to ever suggest that", well, the current government of UK as been elected exactly on this promise: http://en.wikipedia.org/wiki/Big_Society http://en.wikipedia.org/wiki/Big_Society
etiam 12y ago
Cool application. I don't see much justification for double precision arithmetic in this case though, so why would they need the Teslas? Going with Titan Z cards instead I believe they'd come out with comparable single precision performance in the ballpark of half the price? That might be closer to what could be reasonably expected in the target workstation too...
- taion 12y ago
  It's also a bit weird because NVIDIA are usually pretty generous in giving out free K40s to academics. Wonder why they couldn't get some from NVIDIA.
zo1 12y ago
9 hours to figure out the 3 dimensional rotation of the chair?? Wow, I don't think that solution is the path to go down. Why not allow the user to put down some sort of control points to narrow down the search space? You know, it's okay to ask the user for help sometimes. *Edit: The reason it took 9 hours is because they rendered the model in every single possible camera position and orientation until they found one that matched.
- Apofis 12y ago
  Another problem,is that they need to have a 3D model ready of whatever it is you're trying to manipulate. Have a slightly different chair or different car make? Tough cookies.
marknadal 12y ago
This is amazing technology, I'm excited for it to actually come to photo editing/manipulation. Sometimes I wish the programming world and academic world intersected more to solve problems.
- Joeboy 12y ago
  You can already do it. Use content aware fill to cover your object, then find a 3D model of your object and composite it over the top. Afaict this is intended to streamline that process a bit, but it's basically something you can already do, eg using Gimp + Resynthesizer and Blender.
orbifold 12y ago
Hm, I would imagine a university like Carnegie Mellon had plenty of money, or that they could use some centralised computing resource to do the work.
Joeboy 12y ago
Unless I'm missing something, for this to be useful, at least to normal people, there would seem to be a huge step missing whereby every object in the world has to be modelled and put in a database. Also you'd have to be able to locate your 2d objects in the 3d database, but that is at least theoretically tractable I guess. Edit: It sounds cool, but maybe a bit misleadingly advertised.
- rayalez 12y ago
  I think that 3D scanners would be a perfect solution to this. It becomes easier and easier to digitize real world objects, even based on a single-camera video. http://youtu.be/gu5Ywwb4RaU http://youtu.be/gu5Ywwb4RaU
  - Joeboy 12y ago
    That stuff looks cool, and I want to play with photogrammetry stuff when I have some spare time. I think it's a bit optimistic to say it's a "solution" to this though. It's another tool that can help with the modelling + positioning + texturing + lighting + rendering + compositing workflow, but I don't think we're anywhere near a "3D photo editing tool" that can be used by normal people.
imaginenore 12y ago
I don't believe 90% of the examples in that video. They perfectly fill the gaps left by the object every single time, including the person's fingerprint, the carpet pattern, the patterns on the road. I call BS.
- Joeboy 12y ago
  The paper says: """ We compute a mask for the object pixels, and use this mask to inpaint the background using the PatchMatch algorithm [Barnes et al. 2009]. For complex backgrounds, the user may touch up the background image after inpainting. """ So they're not really claiming perfect object removal is part of their thing. What they are actually doing is quite interesting, but their pitch is making some rather extravagant claims IMO.
  - imaginenore 12y ago
    Go ahead and try it. No algorithm will fill in the details like that.
    - Joeboy 12y ago
      I am not disagreeing with you.

74 points by scapbi 12y ago

Very interesting project! I'm just curious about her degrees. It seems she achieved both a B.S. and M.S. at the same time from the same university. How is that possible/allowed? The schools around me don't even accept people into their masters programs until after you have completed a B.S. and have some relevant work experience as well.

dragonwriter 12y ago
Some schools have combined B.S./M.S. programs.
- Alupis 12y ago
  Oh man, I'm at the wrong University then! No but really, I think it's more beneficial for a student to take a year or two off after completing their BS and work in their field. The experience you pick up in the workplace actually doing the thing you went to school for is invaluable. That is unless you worked in your field prior-to or while you were attending university.
  - zo1 12y ago
    Some universities, such as the one I went to, have degrees that combine some normal B.Sc degrees with honors. Such that after completing it, you can start with your masters.
  - emddudley 12y ago
    RIT requires a full year of work experience (co-op) for a BS in Computer Engineering.
emddudley 12y ago
Rochester Institute of Technology (RIT), the school in question, has a dual degree program where you can use some graduate classes to satisfy both the BS and MS credit requirements. She still had to write a research thesis.

therobot24 12y ago

Cool research but something bothers me about students/professors asking for money on sites like indiegogo/kickstarter/etc.

deleted 12y ago
[deleted]
StavrosK 12y ago
Yeah, and it doesn't look like you get anything for your money, either? It wouldn't be very nice if I paid $20 and helped them make a product only to have to buy it later, or if nothing but a research paper came out. This looks more like a donation than crowdfunding.
- mistercow 12y ago
  >This looks more like a donation than crowdfunding. Those aren't mutually exclusive concepts.
mistercow 12y ago
Looks like that's the whole point of this site. Why does crowdfunding science bother you?
- gioele 12y ago
  > Why does crowdfunding science bother you? There is already a fair crowdfunding scheme for science called "taxes". I feel the increase of popularity of crowdfunding schemes for science or health care is detrimental to the cause. They will never be able to approach the same level of funding that a state can give away, yet they will be a good excuse for the politics to reduce the amount of money spent on these issues: "The crowdfunding schemes are already taking care of it, aren't they? Why should we waste taxpayers' money? If prof. X wants its experiment to be funded why doesn't it ask for money on indiegogo?" For those who may thing "nobody is going to ever suggest that", well, the current government of UK as been elected exactly on this promise: http://en.wikipedia.org/wiki/Big_Society http://en.wikipedia.org/wiki/Big_Society

etiam 12y ago

Cool application. I don't see much justification for double precision arithmetic in this case though, so why would they need the Teslas? Going with Titan Z cards instead I believe they'd come out with comparable single precision performance in the ballpark of half the price? That might be closer to what could be reasonably expected in the target workstation too...

taion 12y ago
It's also a bit weird because NVIDIA are usually pretty generous in giving out free K40s to academics. Wonder why they couldn't get some from NVIDIA.

zo1 12y ago

9 hours to figure out the 3 dimensional rotation of the chair?? Wow, I don't think that solution is the path to go down. Why not allow the user to put down some sort of control points to narrow down the search space? You know, it's okay to ask the user for help sometimes. *Edit: The reason it took 9 hours is because they rendered the model in every single possible camera position and orientation until they found one that matched.

Apofis 12y ago
Another problem,is that they need to have a 3D model ready of whatever it is you're trying to manipulate. Have a slightly different chair or different car make? Tough cookies.

marknadal 12y ago

This is amazing technology, I'm excited for it to actually come to photo editing/manipulation. Sometimes I wish the programming world and academic world intersected more to solve problems.

Joeboy 12y ago
You can already do it. Use content aware fill to cover your object, then find a 3D model of your object and composite it over the top. Afaict this is intended to streamline that process a bit, but it's basically something you can already do, eg using Gimp + Resynthesizer and Blender.

orbifold 12y ago

Hm, I would imagine a university like Carnegie Mellon had plenty of money, or that they could use some centralised computing resource to do the work.

Joeboy 12y ago

Unless I'm missing something, for this to be useful, at least to normal people, there would seem to be a huge step missing whereby every object in the world has to be modelled and put in a database. Also you'd have to be able to locate your 2d objects in the 3d database, but that is at least theoretically tractable I guess. Edit: It sounds cool, but maybe a bit misleadingly advertised.

rayalez 12y ago
I think that 3D scanners would be a perfect solution to this. It becomes easier and easier to digitize real world objects, even based on a single-camera video. http://youtu.be/gu5Ywwb4RaU http://youtu.be/gu5Ywwb4RaU
- Joeboy 12y ago
  That stuff looks cool, and I want to play with photogrammetry stuff when I have some spare time. I think it's a bit optimistic to say it's a "solution" to this though. It's another tool that can help with the modelling + positioning + texturing + lighting + rendering + compositing workflow, but I don't think we're anywhere near a "3D photo editing tool" that can be used by normal people.

imaginenore 12y ago

I don't believe 90% of the examples in that video. They perfectly fill the gaps left by the object every single time, including the person's fingerprint, the carpet pattern, the patterns on the road. I call BS.

Joeboy 12y ago
The paper says: """ We compute a mask for the object pixels, and use this mask to inpaint the background using the PatchMatch algorithm [Barnes et al. 2009]. For complex backgrounds, the user may touch up the background image after inpainting. """ So they're not really claiming perfect object removal is part of their thing. What they are actually doing is quite interesting, but their pitch is making some rather extravagant claims IMO.
- imaginenore 12y ago
  Go ahead and try it. No algorithm will fill in the details like that.
  - Joeboy 12y ago
    I am not disagreeing with you.