6 ms·
Ghostwriter – use the reMarkable2 as an interface to vision-LLMs
- newman314 2y agoI wonder if this can be abstracted to accept interaction from a Daylight too.
- rpicard 2y agoThis is so cool. I’m going to try it this weekend. I’ve been playing with the idea of auto creating tasks when I write todos by emailing the PDF and sending it to an LLM. This just opened up a whole realm of better ways to accomplish that goal in realtime.
- r2_pilot 2y agoThis works pretty well when I did a proof of concept with Claude and rMPP a couple of months ago. It even handles scheduling fuzzy times ("I want to do this sometime but I don't have any real time I want to do it, pick a time that doesn't conflict with my actually scheduled tasks"). All with minimal prompting. I just didn't have a decent workflow and did exactly what you considered, emailed the pdf. I should probably revisit this but I haven't had the inclination since I just ignored the tasks anyway lol
- rpicard 2y agoHa, automating the doing of the task is the next step.
- awwaiid 2y agoLet me know if you need any help, I think only one other person has tried to get this working. I'm over on the reMarkable discord server, https://discord.gg/u3P9sDW https://discord.gg/u3P9sDW (linked from https://github.com/reHackable/awesome-reMarkable https://github.com/reHackable/awesome-reMarkable) Rust binary so should be easy to install. In theory :)
- rpicard 2y agoWill do! My wife and I love Harry Potter so I’m motivated to show her my investment in the tablet actually got me Tom Riddle’s diary. I don’t use discord much but I’ll find you somewhere around here!
- awwaiid 2y agoI'm on at awwaiid@gmail.com and probably other places :) "proof" to partner of tablet investment value based on interactive fiction conversation == excellent strategy and nothing could go wrong
- owulveryck 2y agoAwesome. I wanted to try to implement this for months. You did a really good job.
- owulveryck 2y agoAt some point I wanted to turn goMarkableStream into a MCP server (model context protocol). I could get the screen, but without “hack” I couldn’t write the response back.
- awwaiid 2y agoThe trick here is to inject events as if they came from the user. The virtual-keyboard works really reliably, you can see it over at https://github.com/awwaiid/ghostwriter/blob/main/src/keyboard.rs https://github.com/awwaiid/ghostwriter/blob/main/src/keyboar... . It is the equivalent of plugging in the reMarkable type-folio. Main limitation is that the reMarkable drawing app is very very minimal, it doesn't let you place text in arbitrary screen locations and is instead sort of a weird overlay text area spanning the entire screen.
- awwaiid 2y agoThank you! Still a WIP, but a very fun learning / inspiration project. Got a bit of Rust jammed in there, bit of device constraint dancing, bit of multiple LLM api normalization, bit of spacial vision LLM education, etc.
- tony_francis 2y agoHarry potter half-blood prince vibes. Interesting just how much the medium changes the feeling of interacting with a chat model
- s2l 2y agoNow only if llm response font is some handwritten style.
- memorydial 2y agoActually if you figure that out please post it here!! I'd love to see that!
- satvikpendem 2y agoThat's definitely pretty easy to achieve, just change the font settings to use a particular handwritten style font [0]. [0] https://fonts.google.com/?categoryFilters=Calligraphy:%2FScript%2FHandwritten https://fonts.google.com/?categoryFilters=Calligraphy:%2FScr...
- memorydial 2y agoThat would be next-level immersion! You could probably achieve this by rendering the LLM’s response using a handwritten font—maybe even train a model on your own handwriting to make it feel truly personal.
- dharma1 2y agoScript fonts don’t really look like handwriting - too regular. But one of the early deep learning papers from Alex Graves does this really well with LSTMs - https://arxiv.org/abs/1308.0850 https://arxiv.org/abs/1308.0850 Implementation - https://www.calligrapher.ai/ https://www.calligrapher.ai/
- awwaiid 2y agoooo -- thanks for the link!
- cancelself 2y ago@apple.com add to iPadOS Notes?
- ghfhghg 2y ago[flagged]
- deleted 2y ago[deleted]
- complex1314 2y agoReally cool. Would this run on the remarkable paper pro too?
- awwaiid 2y agoBuy me one and I'll find out! hahahaha But also -- the main thing that might be different is the screenshot algorithm. I'm over on the reMarkable discord; if you want to take up a bit of Rust and give it a go then I'd be happy to (slowly/async) help!
- complex1314 2y ago:) Thanks! Been looking into learning rust recently, so will keep that in mind if I get it off the ground.
- awwaiid 2y agoInitially most of the Rust was written by copilot or Sourcegraph's Cody; then I learn more and more rust as I disagree with the code-helper's taste and organization. Though I have a solid foundation in other programming languages which accelerates the process ... it's still a weird way to learn a language that I'm getting used to and kinda like. That said, I based the memory capture on https://github.com/cloudsftp/reSnap/tree/latest https://github.com/cloudsftp/reSnap/tree/latest which is a shell script that slurps out of process space device files. If you can find something like that which works on the rPP then I can blindly slap it in there and we can see what happens!
- 0xferruccio 2y agoThis is so cool! I love to see people hacking together apps for the reMarkable tablet I made a little app for reMarkable too and I shared it here some time back: https://digest.ferrucc.io/ https://digest.ferrucc.io/
- memorydial 2y agoThat’s awesome! Love seeing the reMarkable get more functionality through creative hacks. Just checked out your app—what was the biggest challenge you faced while developing for the reMarkable?
- 0xferruccio 2y agoI think the thing I really didn't like was the lack of an OAuth like flow with fine-grained permissions Basically authentication with devices is "all-access" or "no-access". I would've liked it if a "write-only" or "add-only" api permission scope existed
- pieterhg 2y agoBlocked for AI reply @dang
- defrost 2y agoGood catch, the last few pages of comment history are inhumanly insincere. https://news.ycombinator.com/threads?id=memorydial https://news.ycombinator.com/threads?id=memorydial " @dang " isn't a thing, he doesn't watch for it - take credit and email him direct.
- kordlessagain 2y agoDo you have proof this is true?
- awwaiid 2y agoI might be biased because memorydial was complimentary to me ... but they SEEM like a human! Also I'm not all that opposed to robot participation in the scheme of things. Especially if they are nice to me or give good ideas :)
- t0bia_s 2y agoHow about this on android driven Onyx Boox ereaders? Would it be possible?
- awwaiid 2y agoThe limitations for the reMarkable made it so that I took a screenshot and then injected input events to interact with the proprietary drawing app. Cross-app screenshots with the right permission are probably possible on Android, I'm not sure about injecting the drawing events. The other way to go would be to make a specific app. I just picked up an Apple Pencil and am thinking of porting the concepts to a web app which so far works surprisingly well ... but for a real solution it'd be better for this Agent to interact with existing apps.
- memorydial 2y agoThis is a brilliant use case—handwriting input combined with LLMs makes for a much more natural workflow. I wonder how well it handles messy handwriting and if fine-tuning on personal notes would improve recognition over time.
- r2_pilot 2y agoI did this a few months ago with the Remarkable Paper Pro and Claude. It worked quite well, my handwriting is pretty terrible, and I even had a clunky workflow where I could just write down stuff I wanted to do, and roughly(or specifically) when I wanted to do it, and it was able to generate an ical I could load into my calendar.
- awwaiid 2y agoGenerally if I can read my handwriting then it can! It has no issues with that. Really the problem is more in spacial awareness -- it can't reliably draw an X in a box, let alone play tic-tac-toe or dots-and-boxes.
- account-5 2y ago[flagged]
- deleted 2y ago[deleted]
- chrismorgan 2y ago> Things that worked at least once: I like it.
- awwaiid 2y agoTop quality modern AI Eval!!!
- 8bithero 2y agoNot to distract from the project but if anyone is interested in eink tablets with LLMs, the ViWoods tablet might be of interest to you.
- deleted 2y ago[deleted]
- Ensign35 2y agoIs this a Remarkable rebrand? Even the UI looks the same! edit: https://viwoods.com/ https://viwoods.com/ (based in Hong Kong) edit 2: It's a blatant copy of the Remarkable 2 for sure :/ LLM integration is interesting --> Remarkable are you listening?
- seethedeaduu 2y agoKinda unrelated but should I go for kobo or the remarkable? I mostly want to read papers and maybe take notes. How do tthey compare in terms of hackability and freedom?
- vessenes 2y agoLove this! There are some vector diffusion models out there; why not use tool calling to outsource to one of those if the model decides to draw something? Then it could specify coordinate range and the prompt.
- awwaiid 2y agoTwo reasons. One, because I haven't gotten to it yet. Two... er no just the one reason! Do you have a particular one, ideally with a hosted API, that you recommend?
- vessenes 2y agoI recall flux.ai had a couple models — a quick google search turned up these guys: https://github.com/ximinng/SVGDreamer https://github.com/ximinng/SVGDreamer I’ve been working on a different angle - in place updating of PDFs on the Remarkable, so it’s cool to see what you’re working on. Thanks for sharing it.
- 3abiton 2y agoI own a boox tablet (full fledge Android tablet with eink screen), and this sort of things would be perfect for it. I wonder if in 5 years the mobile hw would support something like that locally!
- xtiansimon 2y agoFor PDF paper readers, is the Remarkable’s 11” size sufficient? I have the Sony DPT 2nd version at 13”, and it’s perfect viewing experience. But projects like this keep drawing me to the Remarkable product.
- pilotneko 2y agoI have used the Remarkable 2 for papers, but it is slightly too small to read text comfortably. I’m also an active reader, so I miss the color highlighting. Annotations are excellent. For now, I’m sticking to reviewing papers in the Zotero application on my iPad.
- kordlessagain 2y agoIt’s barely usable for PDFs
- freedomben 2y agoDepends mostly on the font size in the PDF. For dense PDFs I agree, it's barely usable. For most PDFs though I'd call it "acceptable." If you have control over the font size (such as when you're converting some source material to PDF) you can make it an excellent reading experience IMHO.
- xtiansimon 2y agoSo close. The advertised diagonal screen for the reMarkable Pro is 11.8". The DPT-RP1 is advertised as 13.3" (my unit measures 13.125"). Hopefully in the future reMarkable will make a full-size unit. As mobile phones, tablets, laptops and monitor sales indicate, larger sized screen is important buying factor.
- abawany 2y agoI got the reMarkable Pro tablet recently and as a result was able to move on from my Sony DPT-S1 and reMarkable 2. The latter was nice for its hackability but the screen size of the Pro, its color functionality, and size have made it a great replacement.
- awwaiid 2y agoProject author here -- happy to elaborate on anything; a continuous WIP project. The biggest insight has been limitations of vision models in spacial awareness -- see https://github.com/awwaiid/ghostwriter/blob/main/evaluation_results/2024-12-29_21-05-47/results.md https://github.com/awwaiid/ghostwriter/blob/main/evaluation_... for some sketchy examples of my rudimentary eval. Next top things: * Continue to build/extract into a yaml+shellscript agentic framework/tool * Continue exploring pre-segmenting or other methods of spacial awareness * Write a reSvg backend that sends actual pen-strokes instead of lots of dots
- rybosome 2y agoThis is a really cool effect. How do you envision this being used? Thinking about it as a product, I’d want a way to easily slip in and out of “LLM please respond” so it wasn’t constantly trying to write back the moment I stopped the stylus - maybe I’d want awhile to sketch and think, then restart a conversation. Or maybe for certain pages to be LLM-enabled, and others not. Does it require any sort of jailbreak to get SSH access to the device?
- awwaiid 2y agoThe reMarkable comes with root-ssh out of the box, so installation here is scp'ing a rust-built binary over, and then ssh'ing and running it. I haven't wrapped it in a startup-on-boot service yet. It is triggered right now by finger-tapping in the upper-right corner, so you can ask it to respond to the current contents of the screen on-demand. I think it would be cool to have another out-of-band communication, like voice, but this device has no microphone. Also right now it is one-shot, but on my long long TODO list is a second trigger that would _continue_ a back and forth multi-screenshot (like multi-page even) conversation.
- rybosome 2y agoAh great, I will definitely give this a try later then, thanks! I’m curious if this is becoming something that you are using in your own day-to-day, or if your focus right now is on building it? The context for my question is just a general interest in the transition to AI-enabled workflows. I know that I could be much more productive if I figured out how to integrate AI assistance into my workflows better.
- vendiddy 2y agoI wish the remarkable tablets weren't so locked down. It's one of my favorite pieces of hardware and wish there were more apps for it.
- thrtythreeforty 2y agoLocked down? You can get a shell by ssh'ing to it. Call me when an iPad lets you do that...
- freedomben 2y agoI agree I definitely wouldn't call them "locked down." I do however think they could do a lot more to make it usable/hackable. This slightly undermines their cloud service ambitinos, but I think the hackability is what makes the Remarkable so ... well .. remarkable. Certainly that's why I bought one!