5 ms·

Holy sh*t, can you imagine a year from now if they start using something like this for concerts or basketball games? Like imagine rewatching a basketball game b

Holy sh*t, can you imagine a year from now if they start using something like this for concerts or basketball games? Like imagine rewatching a basketball game but being able to move the camera on the court???? Might not be possible yet but this shows the techs possible. Let alone someday being able to scale it to realtime someday maybe lol

soylentgraham 3y ago
We did this (as a side effect) for the premier league ~2009-2012 (liquidated JUST before VR appeared, where the content worked fantastically, and then ~2014 with the moverio glasses, even better in AR) We did live player tracking (~33 cameras) on-site at every game, and for fun rendered players fifa-style free-camera. We even did some renders (capture of realtime engine) for canal+ highlights as an experiment. edit: my own gpgpu-only,(frag shaders :), sub-100ms, uncalibrated-cameras (footage directly from sky/match of the day) r&d a few years later, also works really well on a LookingGlass https://twitter.com/HoloSports/status/1327375694884646913 https://twitter.com/HoloSports/status/1327375694884646913 (I took this to sky sports but they said it was a bit too in-the-future)
MBCook 3y ago
You want a killer app for VR/AR goggle style things? You’re right this would be amazing. Apple demoed some kind of volumetric video to the press with the Vision Pro. There was a short clip of a concert and an NBA game (Nuggets?) among other things. I heard a number of people said it was like being there. This is a step past that. Apple’s was recorded from some kind of special camera rig (I assume), but I seriously doubt it was full volumetric video from a large number of angles. It sounded more like volumetric video if you were stuck in a (very good) seat in the venue. I’d be curious to know just how much horsepower it takes to play these back.
- keras2 3y ago
  From the paper, it seems a 3060 is enough for 60FPS for the DNA-Rendering dataset. On the full-screen datasets it manages 25 fps. A 4090 might be needed to stay above 60 fps. Still pretty heavy I’d say but it certainly came a long way and shows us real volumetric video is doable.
  - 16bitvoid 3y ago
    That's real-time rendering though, right? Is there anything preventing it from being pre-rendered in non-real-time first? Or does it have to be rendered in real-time? I'm not familiar with any of this at all, so I'm genuinely curious.
    - keras2 3y ago
      [dead]
    - l33tman 3y ago
      If the context was to use in a VR/AR headset it has to be real-time. And I guess that use-case, and the related that you interactively want to walk around a scene are two of the main use-cases
    - detourdog 3y ago
      I think the best way to consider it.. is as a 3-d cloud of individually addressable pixels. The size of the cloud is dimensions of real-time rendering.
    - jayd16 3y ago
      The interesting bit is that it's feasible to render in realtime. We have plenty of ways to render a 3D scene if render time is not an issue.
- TaylorAlexander 3y ago
  I thought they recorded their special videos using the Vision Pro itself, which has enough sensors to build depth maps of the scene and provide novel views within a small range from the original position. But I am half speculating and I don't really remember. That's just the impression I remember having.
  - MBCook 3y ago
    That’s a feature of the headset, but I think some of the demo videos were recorded in some other manor. I seem to remember hearing a discussion on a podcast (The Talk Show? Dithering?) where it was mentioned you could see a camera rig somehow. Perhaps that was something you could see on TV when they accidentally showed the rig that was at the same event? Of course it’s possible they were only using a camera rig so that no one would see the device before they were ready to invade it. Which would be very Apple like. So I’m speculating a bit. But even if they were going to professionally record events I would think they would do better than just have someone sit there with a headset on.
    - kridsdale1 3y ago
      I saw the image you’re referencing yesterday. The unit appeared to be a plane, briefcase sized. It had two fisheye lenses about 10cm diameter each and separated by what I expect is an average human IPD. So I think that unit was simply capturing 180 degree stereo video. Not enough to compute volume without most of it being some ML inference.
      - MBCook 3y ago
        It’s not full volumetric video. Probably can simulate a few degrees off axis but nothing like what this article shows.
nomel 3y ago
I started working on something like that a couple decades ago. I figured, with all of the camera feeds at a football game, there would be plenty of views generate 3d models, even with relatively naive approaches. Then the NFL did it shortly after (2008) [1], and it didn't catch on. It's definitely the future! [1] 2008, https://www.cnet.com/tech/services-and-software/nfl-demos-live-3d-broadcasts/ https://www.cnet.com/tech/services-and-software/nfl-demos-li... [2] Some renewed interest, in 2016: https://bleacherreport.com/articles/2659861-future-of-the-nfl-the-virtual-augmented-3d-360-degree-football-experience https://bleacherreport.com/articles/2659861-future-of-the-nf...
- Cthulhu_ 3y ago
  This is what they use in the latest FIFA / FC games, they call it Volumetric Data Capture, basically using video footage from real sports events to capture model and animation data for the players, allowing their unique mannerisms and movements to translate to the game. In previous iterations they would have football players in motion capture suits, but they wouldn't get all the players, plus if they did it would be in stifled, studio conditions, not their natural environment. Anyway, not quite the same as turning a match into 3D, but definitely related. https://gamermatters.com/ea-sports-fc-24-makes-use-of-volumetric-data-capture-and-ai-for-next-gen-realism/ https://gamermatters.com/ea-sports-fc-24-makes-use-of-volume...
ggurface 3y ago
They don't even stream the NBA games in 4k because TV networks only support 1080p. I doubt they'd buy into such an expensive technology for such a niche audience.
- jcims 3y ago
  If they could sell it for $100 ppv they might consider it.
js98 3y ago
Actually a company has been working on this for a few years now, and I believe they are currently in production. Their focus is football/soccer I believe. I was going to do a research internship at them before I dropped it for a different one. Here it is: https://www.beyondsports.nl/ https://www.beyondsports.nl/ Looking at it, they heavily focus on tracking the movements of players now to replay in AR
chedabob 3y ago
They were demoing this last year https://www.theverge.com/2022/3/16/22982243/nets-mavericks-espn-nba-volumetric-video-canon https://www.theverge.com/2022/3/16/22982243/nets-mavericks-e...
o-o- 3y ago
A thought experiment I like to employ when imagining the impact of a new piece of tech: reverse the timeline. If the tech were the status quo, how would the current status quo be marketed? If moving the camera were the norm, the current status quo would probably be marketed along the lines of "No more micro-managing camera views, arguing over playback speed or fiddling with the timeline – this next leap in technology introduces pre-edited video, where each game is preprocessed by a team of highly-skilled, professional producers, selecting the best viewing angle and playback speed so you and your friends can just sit back, relax and enjoy the game." If the fictional press release sounds good enough, the tech probably won't hit. I remember 30 years ago during switching to digital TV broadcasting that proponents of the tech tried to sell a future where viewers of sports events would be able to select which camera to watch. Then again, imagine watching a game with 10 friends and trying to agree over cameras...
- pinko 3y ago
  I think for routine game-watching you're 100% right. But for replaying amazing/interesting/controversial plays, this tech would an enormous improvement over being captive to the broadcast team. Everyone would love to be able to grab control and fly around and zoom in on that one power dunk, critical fumble, bad foul call, etc. on demand.
- oasisbob 3y ago
  Great point. Are you familiar with McLuhan's media tetrad? The hypothetical status quo is a wonderful example of retrieval. Every new technology should be expected to have the effect of reemphasizing something previously obsolete.
- MPSimmons 3y ago
  >10 friends Wait, you have 10 friends?
barbecue_sauce 3y ago
Will probably be just as boring as watching a basketball game on TV.
pedalpete 3y ago
This is what we were building at my previous start-up, though we had a focus on outdoor sports. We had built a 3D virtual world and used GPS tracks to follow athletes (ultra-marathons, paragliders), etc. We theorized that 2 go-pro cameras on the athlete would let us completely re-create the entire scene from all angles, and inform an AI of how to re-paint our virtual world with real-world weather environments etc. Unfortunately, 5 years ago, everyone said I was crazy to think any of this was possible. There is a video capture of our 3D scenes from 2017 on our old website (we were a full 3d world, not video) https://ayvri.com https://ayvri.com - the tech was acquired just over a year ago.

soylentgraham 3y ago

We did this (as a side effect) for the premier league ~2009-2012 (liquidated JUST before VR appeared, where the content worked fantastically, and then ~2014 with the moverio glasses, even better in AR) We did live player tracking (~33 cameras) on-site at every game, and for fun rendered players fifa-style free-camera. We even did some renders (capture of realtime engine) for canal+ highlights as an experiment. edit: my own gpgpu-only,(frag shaders :), sub-100ms, uncalibrated-cameras (footage directly from sky/match of the day) r&d a few years later, also works really well on a LookingGlass https://twitter.com/HoloSports/status/1327375694884646913 https://twitter.com/HoloSports/status/1327375694884646913 (I took this to sky sports but they said it was a bit too in-the-future)

MBCook 3y ago

You want a killer app for VR/AR goggle style things? You’re right this would be amazing. Apple demoed some kind of volumetric video to the press with the Vision Pro. There was a short clip of a concert and an NBA game (Nuggets?) among other things. I heard a number of people said it was like being there. This is a step past that. Apple’s was recorded from some kind of special camera rig (I assume), but I seriously doubt it was full volumetric video from a large number of angles. It sounded more like volumetric video if you were stuck in a (very good) seat in the venue. I’d be curious to know just how much horsepower it takes to play these back.

keras2 3y ago
From the paper, it seems a 3060 is enough for 60FPS for the DNA-Rendering dataset. On the full-screen datasets it manages 25 fps. A 4090 might be needed to stay above 60 fps. Still pretty heavy I’d say but it certainly came a long way and shows us real volumetric video is doable.
- 16bitvoid 3y ago
  That's real-time rendering though, right? Is there anything preventing it from being pre-rendered in non-real-time first? Or does it have to be rendered in real-time? I'm not familiar with any of this at all, so I'm genuinely curious.
  - keras2 3y ago
    [dead]
  - l33tman 3y ago
    If the context was to use in a VR/AR headset it has to be real-time. And I guess that use-case, and the related that you interactively want to walk around a scene are two of the main use-cases
  - detourdog 3y ago
    I think the best way to consider it.. is as a 3-d cloud of individually addressable pixels. The size of the cloud is dimensions of real-time rendering.
  - jayd16 3y ago
    The interesting bit is that it's feasible to render in realtime. We have plenty of ways to render a 3D scene if render time is not an issue.
TaylorAlexander 3y ago
I thought they recorded their special videos using the Vision Pro itself, which has enough sensors to build depth maps of the scene and provide novel views within a small range from the original position. But I am half speculating and I don't really remember. That's just the impression I remember having.
- MBCook 3y ago
  That’s a feature of the headset, but I think some of the demo videos were recorded in some other manor. I seem to remember hearing a discussion on a podcast (The Talk Show? Dithering?) where it was mentioned you could see a camera rig somehow. Perhaps that was something you could see on TV when they accidentally showed the rig that was at the same event? Of course it’s possible they were only using a camera rig so that no one would see the device before they were ready to invade it. Which would be very Apple like. So I’m speculating a bit. But even if they were going to professionally record events I would think they would do better than just have someone sit there with a headset on.
  - kridsdale1 3y ago
    I saw the image you’re referencing yesterday. The unit appeared to be a plane, briefcase sized. It had two fisheye lenses about 10cm diameter each and separated by what I expect is an average human IPD. So I think that unit was simply capturing 180 degree stereo video. Not enough to compute volume without most of it being some ML inference.
    - MBCook 3y ago
      It’s not full volumetric video. Probably can simulate a few degrees off axis but nothing like what this article shows.

nomel 3y ago

I started working on something like that a couple decades ago. I figured, with all of the camera feeds at a football game, there would be plenty of views generate 3d models, even with relatively naive approaches. Then the NFL did it shortly after (2008) [1], and it didn't catch on. It's definitely the future! [1] 2008, https://www.cnet.com/tech/services-and-software/nfl-demos-live-3d-broadcasts/ https://www.cnet.com/tech/services-and-software/nfl-demos-li... [2] Some renewed interest, in 2016: https://bleacherreport.com/articles/2659861-future-of-the-nfl-the-virtual-augmented-3d-360-degree-football-experience https://bleacherreport.com/articles/2659861-future-of-the-nf...

Cthulhu_ 3y ago
This is what they use in the latest FIFA / FC games, they call it Volumetric Data Capture, basically using video footage from real sports events to capture model and animation data for the players, allowing their unique mannerisms and movements to translate to the game. In previous iterations they would have football players in motion capture suits, but they wouldn't get all the players, plus if they did it would be in stifled, studio conditions, not their natural environment. Anyway, not quite the same as turning a match into 3D, but definitely related. https://gamermatters.com/ea-sports-fc-24-makes-use-of-volumetric-data-capture-and-ai-for-next-gen-realism/ https://gamermatters.com/ea-sports-fc-24-makes-use-of-volume...

ggurface 3y ago

They don't even stream the NBA games in 4k because TV networks only support 1080p. I doubt they'd buy into such an expensive technology for such a niche audience.

jcims 3y ago
If they could sell it for $100 ppv they might consider it.

js98 3y ago

Actually a company has been working on this for a few years now, and I believe they are currently in production. Their focus is football/soccer I believe. I was going to do a research internship at them before I dropped it for a different one. Here it is: https://www.beyondsports.nl/ https://www.beyondsports.nl/ Looking at it, they heavily focus on tracking the movements of players now to replay in AR

chedabob 3y ago

They were demoing this last year https://www.theverge.com/2022/3/16/22982243/nets-mavericks-espn-nba-volumetric-video-canon https://www.theverge.com/2022/3/16/22982243/nets-mavericks-e...

o-o- 3y ago

A thought experiment I like to employ when imagining the impact of a new piece of tech: reverse the timeline. If the tech were the status quo, how would the current status quo be marketed? If moving the camera were the norm, the current status quo would probably be marketed along the lines of "No more micro-managing camera views, arguing over playback speed or fiddling with the timeline – this next leap in technology introduces pre-edited video, where each game is preprocessed by a team of highly-skilled, professional producers, selecting the best viewing angle and playback speed so you and your friends can just sit back, relax and enjoy the game." If the fictional press release sounds good enough, the tech probably won't hit. I remember 30 years ago during switching to digital TV broadcasting that proponents of the tech tried to sell a future where viewers of sports events would be able to select which camera to watch. Then again, imagine watching a game with 10 friends and trying to agree over cameras...

pinko 3y ago
I think for routine game-watching you're 100% right. But for replaying amazing/interesting/controversial plays, this tech would an enormous improvement over being captive to the broadcast team. Everyone would love to be able to grab control and fly around and zoom in on that one power dunk, critical fumble, bad foul call, etc. on demand.
oasisbob 3y ago
Great point. Are you familiar with McLuhan's media tetrad? The hypothetical status quo is a wonderful example of retrieval. Every new technology should be expected to have the effect of reemphasizing something previously obsolete.
MPSimmons 3y ago
>10 friends Wait, you have 10 friends?

barbecue_sauce 3y ago

Will probably be just as boring as watching a basketball game on TV.

pedalpete 3y ago

This is what we were building at my previous start-up, though we had a focus on outdoor sports. We had built a 3D virtual world and used GPS tracks to follow athletes (ultra-marathons, paragliders), etc. We theorized that 2 go-pro cameras on the athlete would let us completely re-create the entire scene from all angles, and inform an AI of how to re-paint our virtual world with real-world weather environments etc. Unfortunately, 5 years ago, everyone said I was crazy to think any of this was possible. There is a video capture of our 3D scenes from 2017 on our old website (we were a full 3d world, not video) https://ayvri.com https://ayvri.com - the tech was acquired just over a year ago.