22 ms·
Our VR surgical training startup has been working for the last few months towards a big medical conference this week where we're showing multiple training proce
by mattnewport 9y ago
Our VR surgical training startup has been working for the last few months towards a big medical conference this week where we're showing multiple training procedures for multiple customers on Oculus Rift, as well as having our own booth. The headsets all stopped working the morning of the conference.
Fortunately one of our engineers figured out we could get our demo rigs working by setting the clock back a few days. This could have been a huge disaster for our company if we hadn't found that workaround though. Pretty annoyed with Oculus about this
- fijal 9y agoImagine how terrible it would be for your customers once that happened in the production....
- konschubert 9y agoHe said it's a surgical training startup so I think it would have been fairly okay.
- Piskvorrr 9y agoThose things aren't cheap for simulators, either - not to mention knock-on costs. "What do you mean - I got the doctors in, which alone took a month of herding cats, and now it won't work, just because?" How low has the SW development bar gone, if "it's okay" now means "at least it's not directly killing people"?
- babuskov 9y ago> How low has the SW development bar gone, if "it's okay" now means "at least it's not directly killing people"? I though that was the way ever since OS/2 failed. Getting stuff out to customers has priority over quality control.
- johnchristopher 9y agoDisrupt, innovate, first-mover advantage, growth marketing, yadaa-yadaa.
- FLUX-YOU 9y ago>How low has the SW development bar gone, if "it's okay" now means "at least it's not directly killing people"? It gets that low every time a hospital underfunds IT staff and makes horrible project management decisions and product buying decisions. I've seen that first hand. There's a bunch of corpses at the IT entrance of people who've tried to turn that around.
- Piskvorrr 9y agoHospital...corpses...I have a hard time distinguishing the literal and figurative context here.
- jdietrich 9y agoThere has always been a tradeoff between reliability and development time. There wouldn't be a games industry if every video game had the same level of software assurance as a mars lander, because Tetris would cost $200m to develop. A medical simulator lies somewhere between a mars lander and a video game - it needs to provide accurate simulation, but the odd crash isn't a complete dealbreaker. The bug is now patched, so the downtime appears to be less than 24 hours from discovery to fix. The original error is clearly a major blunder, but Oculus have responded properly.
- konschubert 9y agoThe GP was suggesting that this could kill people. I simply implied that it wouldn't, and compared to killing people, I would say a lost day is "okay".
- Piskvorrr 9y agoI'll try that for my next programming blunder: "Sure, I've set back hundreds of people one day, but hey, didn't kill them! No big deal, they should even be grateful!" In other words, comparing to the worst possible outcome is, by definition, not a very high bar.
- ams6110 9y agoYes, but it's an interesting question to ponder. The last decade or two government and Military procurement has been leaning more heavily towards COTS (commercial off-the-shelf ) hardware and software. The thought was it's cheaper and possibly more reliable than the bespoke solutions that vendors were delivering in the past. Now we see that it isn't a guarantee of anything. Though still probably cheaper, at least initially. Something like this probably will happen with computer assisted surgery or medical procedures, or an aircraft in flight. Just a matter of time.
- tudorw 9y agoGive the engineer the day off, that's classic side think :)
- moccachino 9y agoThe engineer went on to figuring out that if he set management's clocks back a few days he could take them off, since management clearly remembered him being on premises for those days.
- Agnosco 9y agoSimple and elegant.
- oooomm 9y agoWell every comment thread on the Internet related to the Rift issue mentioned this as a solution so it probably wasn't his idea.
- rublev 9y agoSo what? He got them out of a bind, that's what counts.
- heroprotagonist 9y agoYeah, but if he's the guy who Googled it, he should get the day off anyway if it wasn't really in his realm of responsibility. It seems ridiculous in this modern age, but there are a huge number of people who will never bother to look into their problems on their own before asking someone else. Then this other person does a simple Google search and becomes the hero expert. This all too often results in further dependence, with no real reward for the guy who took this basic step except more requests in the future. If this one guy can get a day off in this instance, it'll be a victory for every person who has ever said "Oh, if you google that, you'll see one of the first results with instructions to do x, y, z." to a time-draining coworker.
- 9y ago
- DanAndersen 9y agoWow! I'm glad you were able to get it figured out. Sounds like a nightmare scenario. By the way, do you have any links to your surgical training startup? I'm doing some research into VR/AR for surgical telementoring and training and would be interested in seeing how it's in use.
- mattnewport 9y agoOsso VR - http://ossovr.com/ http://ossovr.com/ We've had a fair amount of press coverage in specialist press recently so a search for Osso VR should turn up some recent articles too.
- dmix 9y ago"I was a former game developer turned orthopaedic surgeon." [1] Wow. That guy sounds interesting... [1] https://www.youtube.com/watch?v=bqra7wslwCM https://www.youtube.com/watch?v=bqra7wslwCM
- corndoge 9y agoHardly original, kids have been using this to extend the trial periods of shareware since forever. Most Rift users have been using RunAsDate which hooks the kernel's time APIs.
- pc86 9y agoYup, hardly original to think of a non-obvious security work around the morning of a conference that could potentially make or break the company's future success, while dealing with what must have been insane pressure from the everyone there to figure it out.
- corndoge 9y agoIt's extremely obvious to anyone who's ever used shareware and it was also posted on reddit within like 5 minutes. Kudos for not using google I guess.
- barkingcat 9y agoThis does not bode well for real VR surgery. Imagine if this were surgery day for someone, and because of an expiring certificate the rift shuts down ...
- teawrecks 9y agoIt's not like a cert is necessary for it to function. A VR headset is basically a monitor you wear on your face. This is their own poor design choice that just ensures they're going to lose business of anyone who needs reliability in their headset.
- agar 9y agoThe "monitor you wear on your face" trope is simply inaccurate, and essentially a misunderstanding of the state of VR today. I say this not to either criticize you or excuse the mistake by Oculus (they really needed to countersign their cert with a timestamp server), but to educate. These are non-obvious issues to people that don't follow the VR sector. Monitors work without low-level drivers because their maturity (and lack of innovation) allows the hard stuff to be embedded in the operating system. VR is not at that state; it is emergent, and the capability stacks require additional integration into the OS. Vendors frequently add unique features, and will continue to do so for some time, making standardization difficult. Even at its simplest level, a VR headset with 6 degrees of freedom is two monitors that must remain in absolute synchronization while also returning positional information to the CPU. This alone is enough to go beyond "standard monitor driver" functionality. But there's much more. Here is a paste of a comment I made elsewhere: Oculus (and Steam, via SteamVR) engineers a plethora of low-level code to reduce latency and add features. It's not just a monitor, but a whole set of SDKs, APIs, devices, and drivers. For the Rift, the hand controllers are wireless input devices refreshing at 1,000Hz; the sensors (to know where you are in the room) are USB devices with tight 60 fps synchronization to LEDs on the headset; there is a custom audio stack with spacialized audio and ambisonic sound; video needs specialized warping to correct lens distortion, interpolate frames, and maintain a 90 fps image, etc. Not to mention, the system creates a virtual monitor so you can see your 2D desktop while in VR. You can reach out and "grab" a window, rip it from the desktop and position it in your VR world. Pin it, and when you enter a game that window remains and is fully interactive with the Touch controllers emulating a mouse. Maybe you want to play Spotify in the background of a table tennis game, or be able to check a Discord screen while working through a puzzler, or watch YouTube videos while flying your ship in Elite:Dangerous. One guy set up a video feed from his baby monitor so he could watch his kid napping while in VR. This is obviously not a standard feature of the Windows compositor. All this needs to work across AMD and Nvidia, in Unity, Unreal, or any custom game engine. It's not off-the-shelf driver stuff. Not to mention, the premise that monitors don't have drivers is also mistaken. They may not be necessary, but they are available[1]. And, the decision to sign kernel drivers is not a poor choice by Oculus, but a mandate from Microsoft for Windows 10 build 1607 and above.[2] A cert is, indeed, necessary to function. Hope that was informative. [1] http://www.aocmonitorap.com/my/download_driver.php http://www.aocmonitorap.com/my/download_driver.php [2] "Starting with new installations of Windows 10, version 1607, Windows will not load any new kernel mode drivers which are not signed by the Dev Portal." - https://docs.microsoft.com/en-us/windows-hardware/drivers/install/kernel-mode-code-signing-policy--windows-vista-and-later- https://docs.microsoft.com/en-us/windows-hardware/drivers/in...
- seba_dos1 9y agoWhy are you basing medical appliance on such a walled-garden technology you aren't in control of, while there are more accessible alternatives? Oculus was already known for locking up fiascos, this really shouldn't be a surprise for you.
- mattnewport 9y agoOur product is a training aid for medical professionals not a medical appliance. We're not fundamentally tied to any particular VR device but Oculus has been our primary platform due to better ergonomics and an easier setup experience for a portable demo rig than the Vive.
- Kelbit 9y agoThis is fairly common practice in the medical device field. Volumes for specialized equipment are far too low to justify the NRE on custom solutions, so many low-volume medical devices integrate COTS solutions wherever possible. It doesn't help that vendors are generally nervous about liability in medical equipment (this fear is often unfounded, but persistent). As a result, vendors of commercial and industrial equipment generally don't want to engage medical device OEMs with engineering and customization support. If there had been that sort of support in this case, Oculus might have made a custom build without the cert check, just as a de-risking measure. This vendor reluctance is especially present at the FDA Class III (high risk device) level - most vendors outright prohibit use of their devices in these products. It's an open secret that this still happens anyway in a wink-wink nudge-nudge fashion, just without vendor support - which is arguably worse, but it keeps the lawyers happy.
- SkyPuncher 9y agoI think you underestimate how many companies build technologies off of walled gardens - especially in Healthcare.
- CaptSpify 9y agoAlthough you are correct, I don't see how that really matters. Just because a lot of people do it doesn't make it a good idea.
- AboutTheWhisles 9y agoWelcome to the future.
- borispavlovic 9y agoThat engineer earned himself a nice bonus. Hope he'll get it
- retox 9y ago>one of our engineers figured out we could get our demo rigs working by setting the clock back a few days The real MVP of this story. Sometimes a dirty hack is good enough.