4 ms·

I wonder what the details of the computer vision tracking are. I am thinking its simple color tracking which has challenges under lighting and contrast with oth

I wonder what the details of the computer vision tracking are. I am thinking its simple color tracking which has challenges under lighting and contrast with other colors. Object detection using deep learning could make the tracking more robust.

curuinor 8y ago
Can't do deep learning easily and without a lot of specialized effort at the latencies that they need to work with, iirc.
jonnydubowsky 8y ago
This summer at NYU ITP Camp, a fellow itp'r setup https://paperprograms.org https://paperprograms.org which "recreates just a tiny part of what makes Dynamicland so interesting" (quote from JP, one of the creators) http://janpaulposma.nl http://janpaulposma.nl) The outline answers some of your questions. It's really easy to setup, all you need is a web camera and a projector, and the colored dots and a printer. We were up and running playing pong with paper controllers within twenty minutes... Lots of fun. How does Paper Programs work? Programs are stored on a server (using Node.jsand PostgreSQL), hosted on paperprograms.org. Each program has a number, and the dots on the paper encode that number. Currently each corner is uniquely identified with 5 dots of 5 possible colours, which means you can have about 600 unique papers currently (this is a significant limitation). A camera detects the dots and retrieves the program associated with each paper. This is done in a browser, using OpenCV compiled to WebAssembly, and some custom Javascript code. Calibration happens manually, using a UI built in React. Program code and configuration are stored in the browser’s local storage. Projection and execution of programs happens in a separate browser window. Each program runs asynchronously in a Web Worker, and can request access to a canvas, coordinates of other programs, and so on. Then there is an editor page, which anyone in the space with a laptop or tablet can use to edit programs, using Monaco. When having created a new program, you can click a print button to print out a new paper that runs that program. It has the program text printed on the paper itself. Any edited program can be reverted to its original state.
- jonnydubowsky 8y ago
  The calibration is pretty good at dealing with the background noise of your table top area, and we set this up in a pretty average open workspace environment and didn't encounter any issues with the image detection. Dynamicland has even lore advanced image detection from what I've gathered from various posts, where robots and action figures, and other 3d objects can be placed within a tabletop and integrated into a variety of different interactive programs. If you'd like to follow this down another path, nyc's Recurse Center has been supporting development of a Roomdb framework that also uses concepts from the Dynamicland universe. This goes more into the database side of things in the vein of Smalltalk and Linda, and is really interesting stuff. https://www.recurse.com/blog/132-living-room-making-rc-programmable https://www.recurse.com/blog/132-living-room-making-rc-progr... https://livingroomresearch.tumblr.com https://livingroomresearch.tumblr.com

curuinor 8y ago

Can't do deep learning easily and without a lot of specialized effort at the latencies that they need to work with, iirc.

jonnydubowsky 8y ago

This summer at NYU ITP Camp, a fellow itp'r setup https://paperprograms.org https://paperprograms.org which "recreates just a tiny part of what makes Dynamicland so interesting" (quote from JP, one of the creators) http://janpaulposma.nl http://janpaulposma.nl) The outline answers some of your questions. It's really easy to setup, all you need is a web camera and a projector, and the colored dots and a printer. We were up and running playing pong with paper controllers within twenty minutes... Lots of fun. How does Paper Programs work? Programs are stored on a server (using Node.jsand PostgreSQL), hosted on paperprograms.org. Each program has a number, and the dots on the paper encode that number. Currently each corner is uniquely identified with 5 dots of 5 possible colours, which means you can have about 600 unique papers currently (this is a significant limitation). A camera detects the dots and retrieves the program associated with each paper. This is done in a browser, using OpenCV compiled to WebAssembly, and some custom Javascript code. Calibration happens manually, using a UI built in React. Program code and configuration are stored in the browser’s local storage. Projection and execution of programs happens in a separate browser window. Each program runs asynchronously in a Web Worker, and can request access to a canvas, coordinates of other programs, and so on. Then there is an editor page, which anyone in the space with a laptop or tablet can use to edit programs, using Monaco. When having created a new program, you can click a print button to print out a new paper that runs that program. It has the program text printed on the paper itself. Any edited program can be reverted to its original state.

jonnydubowsky 8y ago
The calibration is pretty good at dealing with the background noise of your table top area, and we set this up in a pretty average open workspace environment and didn't encounter any issues with the image detection. Dynamicland has even lore advanced image detection from what I've gathered from various posts, where robots and action figures, and other 3d objects can be placed within a tabletop and integrated into a variety of different interactive programs. If you'd like to follow this down another path, nyc's Recurse Center has been supporting development of a Roomdb framework that also uses concepts from the Dynamicland universe. This goes more into the database side of things in the vein of Smalltalk and Linda, and is really interesting stuff. https://www.recurse.com/blog/132-living-room-making-rc-programmable https://www.recurse.com/blog/132-living-room-making-rc-progr... https://livingroomresearch.tumblr.com https://livingroomresearch.tumblr.com