3 ms·

When dealing with inputs of diverse latencies, another approach is a replayable event stream with app state that's checkpointed or otherwise retractable. My lap

When dealing with inputs of diverse latencies, another approach is a replayable event stream with app state that's checkpointed or otherwise retractable. My laptop had cameras tracking hands on keyboard, creating fused input. For key events with finger id and position on keycap, keycap touching as modifiers, and keycap touch events without pressing. But while you want ui response to keypress to be very fast, tracking info wouldn't become available until the next video frame finally showed up and was processed. Integrating voice faces similar issues - speech-to-text "launch the missiles!" might shortly later become "lunch is mussels, in butter!", with launch needing retraction. Which required adapting app state and its modification.

julik 2y ago
Replayable event streams (but careful there as, say, a complete drag interaction of multiple minutes may produce tens of thousands of events, so deduping/vector addition is needed). Also - "interlocked" interactions (if one interaction is in progress, no other interaction may start), orchestrated global shortcut installation...
yjftsjthsd-h 2y ago
> My laptop had cameras tracking hands on keyboard, creating fused input. What laptop is that?
- mncharity 2y ago
  ThinkPad P53 "portable workstation" with DIY cameras on sticks mounted on back of screen. Here mostly a keyboard-cam that flopped up-and-over to look down on the keyboard, and a convex mirror bar along the base of screen for crappy touch detection. But also fisheyes hinged up to corners. Python/MediaPipe/libev under an Electron/browser stack.
  - yjftsjthsd-h 2y ago
    Ah, so the laptop is normal and you've tacked on a DIY rig so it can see your keyboard. Is there a writeup/blogpost/repo somewhere? That's an interesting idea
    - mncharity 2y ago
      Nod. Sigh, regrettably not. A photo[1] got snagged for a twitter conversation, but little else. A different part of the effort, diy shutter glasses, got a brief reddit post. Failure mode: Big scope, "these kludges only matter if they end up day-to-day usable", "just for me - user community potential is zero". Motivated with an local in-person tech community ended by covid - I didn't even do a "ok, this looks dead, so let's scavenge a wrap-up". Ah well. I've been wondering what to do differently with a new potential project. Maybe try an offline live-blogging-flavored research journal, with a story of "when things comes up in conversation, I'll scavenging together a live-blogging-effort-level post in answer"? Thanks for the interest. Happy to answer questions. Hand/head/gaze/stylus tracking, electron-based tall stack (low-level-input to full-screen, X11 apps via screen capture - eg emacs presenting adjacent slightly different buffers for stereo), shallow-3D ui, integration with shutter glasses / nreal / diy-passthrough-ar-in-vr/drone-headsets, laptop keyboard as multitouch surface. And none of it usable. Lots of extra "well, that didn't work out" and "interesting, but no" spikes. Lots of "this only becomes usable if/when this other thing really works" dependencies. Blech. Might be able to pull out something bite-sized for a student thesis project or such. :/ [1] https://imgur.com/a/Z1VipaL https://imgur.com/a/Z1VipaL

julik 2y ago

Replayable event streams (but careful there as, say, a complete drag interaction of multiple minutes may produce tens of thousands of events, so deduping/vector addition is needed). Also - "interlocked" interactions (if one interaction is in progress, no other interaction may start), orchestrated global shortcut installation...

yjftsjthsd-h 2y ago

> My laptop had cameras tracking hands on keyboard, creating fused input. What laptop is that?

mncharity 2y ago
ThinkPad P53 "portable workstation" with DIY cameras on sticks mounted on back of screen. Here mostly a keyboard-cam that flopped up-and-over to look down on the keyboard, and a convex mirror bar along the base of screen for crappy touch detection. But also fisheyes hinged up to corners. Python/MediaPipe/libev under an Electron/browser stack.
- yjftsjthsd-h 2y ago
  Ah, so the laptop is normal and you've tacked on a DIY rig so it can see your keyboard. Is there a writeup/blogpost/repo somewhere? That's an interesting idea
  - mncharity 2y ago
    Nod. Sigh, regrettably not. A photo[1] got snagged for a twitter conversation, but little else. A different part of the effort, diy shutter glasses, got a brief reddit post. Failure mode: Big scope, "these kludges only matter if they end up day-to-day usable", "just for me - user community potential is zero". Motivated with an local in-person tech community ended by covid - I didn't even do a "ok, this looks dead, so let's scavenge a wrap-up". Ah well. I've been wondering what to do differently with a new potential project. Maybe try an offline live-blogging-flavored research journal, with a story of "when things comes up in conversation, I'll scavenging together a live-blogging-effort-level post in answer"? Thanks for the interest. Happy to answer questions. Hand/head/gaze/stylus tracking, electron-based tall stack (low-level-input to full-screen, X11 apps via screen capture - eg emacs presenting adjacent slightly different buffers for stereo), shallow-3D ui, integration with shutter glasses / nreal / diy-passthrough-ar-in-vr/drone-headsets, laptop keyboard as multitouch surface. And none of it usable. Lots of extra "well, that didn't work out" and "interesting, but no" spikes. Lots of "this only becomes usable if/when this other thing really works" dependencies. Blech. Might be able to pull out something bite-sized for a student thesis project or such. :/ [1] https://imgur.com/a/Z1VipaL https://imgur.com/a/Z1VipaL