4 ms·

This was a very important missing piece for mixed/augmented and virtual reality. The semantic understanding of the objects around you (without upload to a serv

This was a very important missing piece for mixed/augmented and virtual reality. The semantic understanding of the objects around you (without upload to a server). A detailed set of actual meshes of the objects in your living and working spaces. So much can be built on this key building block, to actually help people with their daily lives, low vision people are only one group that will see massive benefits. I also think this technique could and should be extended generally into computer vision applications, understanding semantic meaning, volumes and masses via the often highly structured environments around us. Starting from a principled position of meaning, volumes and meshes and then training to assign gaussian splats to those meshes seems like something clever people could solve a lot of problems with.

nox101 3y ago
A huge problem is privacy > without upload to a server Doesn't matter. If the app has to render them then it can still send them to the server. If the app does not render directly it can still infer them by doing collision testing against them. If it can't even check for collisions then there isn't much point in having the data in the first place. It gets worse. The more you want AR/MR to understand the scene the more fine grained data you need. The camera sees things like: the underwear you left on the floor, the medicines/pills on your table, the posters on your wall, the bank account browser tab still on your computer screen, the adult toys you forgot to put away, your undressed body, your dressed body, your political baseball cap or t-shirt, your naked roommate/spouse/kids who just walked into the room. I don't see how this can be solved in a trust-less way. In other words, you'll have to trust each and every individual AR/MR app you use, that they aren't spying on you. The platform/device/OS itself can not prevent this and still allow the majority of the AR/MR uses that people imagine.
- littlestymaar 3y ago
  Companies don't spy on you because they don't like you, they spy on you because they make money out of it. Outlaw making money out of personal data (by outlawing targeted ads, data purchase for insurance, and data brokers) and the incentives to spy on you disappear. You can't be probusiness and pro-privacy at the same time though, you need to limit businesses in order to protect the people.
  - deleted 3y ago
    [deleted]
- thoughtlede 3y ago
  Yes, the camera sees everything. But we could avoid teaching the model certain things. For instance, an undressed body and a dressed body could both be taught as a body. Likewise, medicine pills as well as regular mints could just be taught as mints. While this approach doesn’t address privacy completely, it avoids certain elements of our life that are considered really private. I think half of the point of using the simulated dataset to train the model was to safeguard the above mentioned kind of privacy (with the other half being lack of real-world dataset).
- jonplackett 3y ago
  I think there must be a way of doing this so you just have to trust the OS. The camera feed itself could be essentially private and only pass on locations and types of objects. In this way you could whitelist / blocklist items or categories. But you would defo have to trust the OS, and in this case Meta…
vergessenmir 3y ago
Isn't this more of A semantic understanding of how language maps onto geometry rather than understanding of actual space and geometry? I sometimes feel like this may not work without language helping it along
- WhatIsDukkha 3y ago
  With a multimodal llm watching a stream of these objects, should be able answer the question - "where are my keys?" It should be able to respond - "Here on the dining room table" with an astar path drawn to them in the headset. It's a stream of tokens that you can almost just reverse search on to find the last instance of "keys" in it (thats a simplification but...).
- pyinstallwoes 3y ago
  So, potentially how humans turn language into a rendered version of reality? :)

nox101 3y ago

A huge problem is privacy > without upload to a server Doesn't matter. If the app has to render them then it can still send them to the server. If the app does not render directly it can still infer them by doing collision testing against them. If it can't even check for collisions then there isn't much point in having the data in the first place. It gets worse. The more you want AR/MR to understand the scene the more fine grained data you need. The camera sees things like: the underwear you left on the floor, the medicines/pills on your table, the posters on your wall, the bank account browser tab still on your computer screen, the adult toys you forgot to put away, your undressed body, your dressed body, your political baseball cap or t-shirt, your naked roommate/spouse/kids who just walked into the room. I don't see how this can be solved in a trust-less way. In other words, you'll have to trust each and every individual AR/MR app you use, that they aren't spying on you. The platform/device/OS itself can not prevent this and still allow the majority of the AR/MR uses that people imagine.

littlestymaar 3y ago
Companies don't spy on you because they don't like you, they spy on you because they make money out of it. Outlaw making money out of personal data (by outlawing targeted ads, data purchase for insurance, and data brokers) and the incentives to spy on you disappear. You can't be probusiness and pro-privacy at the same time though, you need to limit businesses in order to protect the people.
- deleted 3y ago
  [deleted]
thoughtlede 3y ago
Yes, the camera sees everything. But we could avoid teaching the model certain things. For instance, an undressed body and a dressed body could both be taught as a body. Likewise, medicine pills as well as regular mints could just be taught as mints. While this approach doesn’t address privacy completely, it avoids certain elements of our life that are considered really private. I think half of the point of using the simulated dataset to train the model was to safeguard the above mentioned kind of privacy (with the other half being lack of real-world dataset).
jonplackett 3y ago
I think there must be a way of doing this so you just have to trust the OS. The camera feed itself could be essentially private and only pass on locations and types of objects. In this way you could whitelist / blocklist items or categories. But you would defo have to trust the OS, and in this case Meta…

vergessenmir 3y ago

Isn't this more of A semantic understanding of how language maps onto geometry rather than understanding of actual space and geometry? I sometimes feel like this may not work without language helping it along

WhatIsDukkha 3y ago
With a multimodal llm watching a stream of these objects, should be able answer the question - "where are my keys?" It should be able to respond - "Here on the dining room table" with an astar path drawn to them in the headset. It's a stream of tokens that you can almost just reverse search on to find the last instance of "keys" in it (thats a simplification but...).
pyinstallwoes 3y ago
So, potentially how humans turn language into a rendered version of reality? :)