5 ms·

Hi HN, I created this program out of my personal need to be able to do some handwriting on my Linux system. I know, I could just have bought a graphic tablet bu

by HMH 6y ago

Hi HN, I created this program out of my personal need to be able to do some handwriting on my Linux system. I know, I could just have bought a graphic tablet but as I already have a tablet with the necessary hardware for note taking around I was unwilling to do so. And after quite some fruitless search for a solution to this problem Weylus (= Web + Stylus) was born. It is written mostly in Rust, some C and a tiny bit Typescript and works by serving a webpage that uses the browser API to capture pen, touch and mouse events from the tablet. Additionally the desktop or a selected window is captured and encoded to a video stream via ffmpeg. This stream is played in the browser using the Media Source Extensions API. This should also run on macOS and Windows but with very limited features.

felixr 6y ago
I have to say it is quite cool how easy it is got it set up. However, I for me the screen updates on my tablet were too laggy. I wonder how much it would improve if the stream had a lower resolution and limited colors.
- HMH 6y ago
  Thanks for giving it a try! If you are on Linux you could try to select a window and check if making it smaller still results in lags. Unfortunately the ability to playback videos via MSE API with low latency seems to vary quite a bit from device to device. You could also try an older release (v5.*). Those work by repeatedly sending PNGs and updating an image tag, actually the gif in the Readme is from that version. If this works better for you I will probably add it back as fallback in case video playback is to slow.
  - netsharc 6y ago
    I'm guessing it's slow because capturing the display and encoding it takes a while? I wonder if you could integrate this HTML5 VNC client: https://github.com/novnc/noVNC https://github.com/novnc/noVNC , since the complicated stuff above will be taken care of by the VNC server. I feel like the boss in Office Space saying "if you could just go ahead and...".
    - HMH 6y ago
      > I'm guessing it's slow because capturing the display and encoding it takes a while? This is a possibility but encoding is already pretty optimized (ffmpeg and libx264 with ultrafast as preset), which is faster than my previous approach of just streaming PNGs. And streaming PNGs was not that slow either, even on a rather old laptop. That's why I suspect there's something wrong on the tablet. Regarding noVNC: What latencies do they achieve? I am not a webdev so I guess they might have some more tricks up their sleeves, I will probably have a look. As far as I can tell noVNC does not support things like multitouch and pressure sensitivity so that rules out just using it (at least for my usecase).
  - lultimouomo 6y ago
    Have you thought about having a companion mobile app instead of using the browser? If so, what made you go for a web based solution?
    - HMH 6y ago
      Yes I have. I have decided to go with using a browser for now as I simply do not have the time to support multiple mobile apps and I am not too inclined to pay Apple 99$ for a Developer account. Personally I need this to work on my iPad, so you could consider using the browser as my way to break free of Apples walled garden :) If someone wants to build a mobile app I am more than happy to provide other means of connection to Weylus than websockets. This also might bring the latency down as udp streaming and more control over video decoding and playing is possible then.
bzzzt 6y ago
If you're on macOS and iPad this is exactly what the 'sidecar' feature does. Needs recent hardware though since it uses the video encoder built in recent Intel CPUs to make it less laggy.
- bdcravens 6y ago
  I've used Sidecar, and I thought it was more of extending your screen to the iPad, with some basic input controls on the device as well. It looks like this is more of a mirroring setup where you can use the tablet for touch-enabled input.
  - pkage 6y ago
    Sidecar works great with touch-enabled input as well—I personally use it for Photoshop/Illustrator projects, and it has completely replaced a traditional tablet (one less thing to carry). Regardless, this project looks incredibly cool. The cross-OS support is a killer feature.
  - arthurcolle 6y ago
    You need to sign in with the same Apple ID, which is such a joke. I have an iPad that I created a unique Apple ID for and I can't use it with my maxed out MacBook Pro without losing the purchases. Truly infuriating
    - joeberon 6y ago
      Why did you do that?
      - arthurcolle 6y ago
        Because my personal devices are personal devices, and I got the iPad in order to do research for a business I'm building on the side. I didn't think I wouldn't be able to use both seamlessly just because of an arbitrary decision on Apple corporate's side
- IggleSniggle 6y ago
marci 6y ago
Did you know about GfxTablet* (limited to Linux)? Any thoughts? * https://github.com/rfc2822/GfxTablet https://github.com/rfc2822/GfxTablet
- HMH 6y ago
  Yes I do! That is where I got the idea to use uinput from. But as this is no longer maintained and does not mirror the screen I decided to create Weylus. Something that is different in GfxTablet is that inputs are not directly mapped to screen coordinates but it works like a regular touch pad.
Novukus 6y ago
This is a pretty damn cool idea. Do you only use this at home? I'm thinking about how I would use this when I'm not home, I guess the wifi hotspot on my tablet would suffice?
- HMH 6y ago
  I already used it the other way around with a hotspot created on my laptop. As long as you can connect your browser with decent speed it should work just fine.
masnaox 6y ago
VNC should be the ideal solution to this on a fast enough network ...and if vnc client can send a pen input to the host and host can see it as a proper HID device.
- HMH 6y ago
  > ...and if vnc client can send a pen input to the host and host can see it as a proper HID device. As far as I can tell this is not possible: https://tools.ietf.org/html/rfc6143#section-7.5.5 https://tools.ietf.org/html/rfc6143#section-7.5.5 Also selecting a specific window, which is possible in Weylus, does not seem to work with vnc.
rafobr 6y ago
Hi, -H-M-H- . Totally newbie here. I tried installing the deb package. I set up the uinput, but I still can't make it work. I'm using xubuntu along with an IPAD. Do I need to change the "bind address" on weylus? What could I be missing here? Thx

by HMH 6y ago

felixr 6y ago

I have to say it is quite cool how easy it is got it set up. However, I for me the screen updates on my tablet were too laggy. I wonder how much it would improve if the stream had a lower resolution and limited colors.

HMH 6y ago
Thanks for giving it a try! If you are on Linux you could try to select a window and check if making it smaller still results in lags. Unfortunately the ability to playback videos via MSE API with low latency seems to vary quite a bit from device to device. You could also try an older release (v5.*). Those work by repeatedly sending PNGs and updating an image tag, actually the gif in the Readme is from that version. If this works better for you I will probably add it back as fallback in case video playback is to slow.
- netsharc 6y ago
  I'm guessing it's slow because capturing the display and encoding it takes a while? I wonder if you could integrate this HTML5 VNC client: https://github.com/novnc/noVNC https://github.com/novnc/noVNC , since the complicated stuff above will be taken care of by the VNC server. I feel like the boss in Office Space saying "if you could just go ahead and...".
  - HMH 6y ago
    > I'm guessing it's slow because capturing the display and encoding it takes a while? This is a possibility but encoding is already pretty optimized (ffmpeg and libx264 with ultrafast as preset), which is faster than my previous approach of just streaming PNGs. And streaming PNGs was not that slow either, even on a rather old laptop. That's why I suspect there's something wrong on the tablet. Regarding noVNC: What latencies do they achieve? I am not a webdev so I guess they might have some more tricks up their sleeves, I will probably have a look. As far as I can tell noVNC does not support things like multitouch and pressure sensitivity so that rules out just using it (at least for my usecase).
- lultimouomo 6y ago
  Have you thought about having a companion mobile app instead of using the browser? If so, what made you go for a web based solution?
  - HMH 6y ago
    Yes I have. I have decided to go with using a browser for now as I simply do not have the time to support multiple mobile apps and I am not too inclined to pay Apple 99$ for a Developer account. Personally I need this to work on my iPad, so you could consider using the browser as my way to break free of Apples walled garden :) If someone wants to build a mobile app I am more than happy to provide other means of connection to Weylus than websockets. This also might bring the latency down as udp streaming and more control over video decoding and playing is possible then.

bzzzt 6y ago

If you're on macOS and iPad this is exactly what the 'sidecar' feature does. Needs recent hardware though since it uses the video encoder built in recent Intel CPUs to make it less laggy.

bdcravens 6y ago
I've used Sidecar, and I thought it was more of extending your screen to the iPad, with some basic input controls on the device as well. It looks like this is more of a mirroring setup where you can use the tablet for touch-enabled input.
- pkage 6y ago
  Sidecar works great with touch-enabled input as well—I personally use it for Photoshop/Illustrator projects, and it has completely replaced a traditional tablet (one less thing to carry). Regardless, this project looks incredibly cool. The cross-OS support is a killer feature.
- arthurcolle 6y ago
  You need to sign in with the same Apple ID, which is such a joke. I have an iPad that I created a unique Apple ID for and I can't use it with my maxed out MacBook Pro without losing the purchases. Truly infuriating
  - joeberon 6y ago
    Why did you do that?
    - arthurcolle 6y ago
      Because my personal devices are personal devices, and I got the iPad in order to do research for a business I'm building on the side. I didn't think I wouldn't be able to use both seamlessly just because of an arbitrary decision on Apple corporate's side
IggleSniggle 6y ago

marci 6y ago

Did you know about GfxTablet* (limited to Linux)? Any thoughts? * https://github.com/rfc2822/GfxTablet https://github.com/rfc2822/GfxTablet

HMH 6y ago
Yes I do! That is where I got the idea to use uinput from. But as this is no longer maintained and does not mirror the screen I decided to create Weylus. Something that is different in GfxTablet is that inputs are not directly mapped to screen coordinates but it works like a regular touch pad.

Novukus 6y ago

This is a pretty damn cool idea. Do you only use this at home? I'm thinking about how I would use this when I'm not home, I guess the wifi hotspot on my tablet would suffice?

HMH 6y ago
I already used it the other way around with a hotspot created on my laptop. As long as you can connect your browser with decent speed it should work just fine.

masnaox 6y ago

VNC should be the ideal solution to this on a fast enough network ...and if vnc client can send a pen input to the host and host can see it as a proper HID device.

HMH 6y ago
> ...and if vnc client can send a pen input to the host and host can see it as a proper HID device. As far as I can tell this is not possible: https://tools.ietf.org/html/rfc6143#section-7.5.5 https://tools.ietf.org/html/rfc6143#section-7.5.5 Also selecting a specific window, which is possible in Weylus, does not seem to work with vnc.

rafobr 6y ago

Hi, -H-M-H- . Totally newbie here. I tried installing the deb package. I set up the uinput, but I still can't make it work. I'm using xubuntu along with an IPAD. Do I need to change the "bind address" on weylus? What could I be missing here? Thx