105 ms·

Very cool demo OP. Not sure why it's only for Apple Silicone, is it because of it's superior ML support compared to windows? Side oservarion, Olama is not avail

Very cool demo OP. Not sure why it's only for Apple Silicone, is it because of it's superior ML support compared to windows? Side oservarion, Olama is not available for Windows. Sadly I won't be able to test this out since I don't own a Apple Silicon notebook, I only have a Apple Intel and beefy Windows. I don't know if I am a basic programmer or lack the idea on how do folks go build something like this from scratch with no Swift programming language. If I was OP, I would first do bunch of Swift tutorials. This will be wishful thinking but it is a legitimate side project to make a clone of this to work on Linx oe WIndows on programming language I am most comfortable in, Java and C#. I have zero background in building anything in ML and not at all familiar with Direct X api or Linux Desktop API or Direct X. The point I am trying to make is there is crap ton of API and tools to be familiar with before even taking the step to code. How did OP crack this with no exp in Swift to build this? Is it simplar to build project on Apple Silicon and I should get one? Mind you, I have 4 YOE and code in Java and C# doing vanilla web API and bit of WinFoem/DevExpress work.

allenu 3y ago
I suspect it's Apple Silicon only because there are simple APIs that Apple has provided to take screenshots of the desktop and OCR text from images. I don't think OP necessarily built any low-level code from scratch here (not that their provided code isn't useful). Someone else has previously looked into how Rewind.ai may be doing its thing under the hood and there are more details about it here: https://kevinchen.co/blog/rewind-ai-app-teardown/ https://kevinchen.co/blog/rewind-ai-app-teardown/ OP may have used some of the info there.
- Sakos 3y ago
  I think it's Apple only because OP made this on an Apple M1 and that's all he intended to support. As for OCR on Windows, there seems to be that functionality available on an OS level: https://medium.com/dataseries/using-windows-10-built-in-ocr-with-c-b5ca8665a14e https://medium.com/dataseries/using-windows-10-built-in-ocr-... There's a Power Toy that uses it, which is where I found out about it. I have no idea how it compares to Live Text or how good it is.
jasonjmcghee 3y ago
I have been in software for a while now. I just love learning and doing cool stuff. As for being able to hack in Swift specifically... I'm comfortable in 5-6 languages, and play with many more. The language itself feels like a mix of C# and Kotlin to me. But I had no familiarity with Mac OS APIs or SwiftUI / MVVM etc. but there's lots of docs, despite them having effectively no examples. The repo has a lot of room to grow in terms of quality, which is perfectly ok in my book. Re: Apple Silicon - I should have just said "I built it on my laptop which is an M1 Air and it's managing to keep up with the screenshot -> OCR -> ffmpeg rendering pipeline and not completely drain the battery and have a strong suspicion it will require a lot more work to get it to perform the same on Intel computers" As for clone it and build it in Linux / Windows - do it. And there are other comments here suggesting others want to do the same. I personally want this thing, if I can impact speed of it happening positively, awesome. I want to record everything and have this nice big dataset which is what I've experienced (on my laptop) the last X amount of time, and be able to do stuff with it - whether it's chat with a local llm or have a really good way to search back in time. I _constantly_ need things from the past.
davidy123 3y ago
You could set up screen sharing for any system that supports it, and on another system receive the stream and use an LLM(s) to run object ("cat, piano") and text detection on images and sound. Mac silicon can do that, but it's not as efficient or cost effective as a dedicated GPU. This approach would be system neutral since models work on (increasingly) any GPU including Apple's. On your workstation, you'd just have the overhead (and concerns) of the video stream.

allenu 3y ago

I suspect it's Apple Silicon only because there are simple APIs that Apple has provided to take screenshots of the desktop and OCR text from images. I don't think OP necessarily built any low-level code from scratch here (not that their provided code isn't useful). Someone else has previously looked into how Rewind.ai may be doing its thing under the hood and there are more details about it here: https://kevinchen.co/blog/rewind-ai-app-teardown/ https://kevinchen.co/blog/rewind-ai-app-teardown/ OP may have used some of the info there.

Sakos 3y ago
I think it's Apple only because OP made this on an Apple M1 and that's all he intended to support. As for OCR on Windows, there seems to be that functionality available on an OS level: https://medium.com/dataseries/using-windows-10-built-in-ocr-with-c-b5ca8665a14e https://medium.com/dataseries/using-windows-10-built-in-ocr-... There's a Power Toy that uses it, which is where I found out about it. I have no idea how it compares to Live Text or how good it is.

jasonjmcghee 3y ago

I have been in software for a while now. I just love learning and doing cool stuff. As for being able to hack in Swift specifically... I'm comfortable in 5-6 languages, and play with many more. The language itself feels like a mix of C# and Kotlin to me. But I had no familiarity with Mac OS APIs or SwiftUI / MVVM etc. but there's lots of docs, despite them having effectively no examples. The repo has a lot of room to grow in terms of quality, which is perfectly ok in my book. Re: Apple Silicon - I should have just said "I built it on my laptop which is an M1 Air and it's managing to keep up with the screenshot -> OCR -> ffmpeg rendering pipeline and not completely drain the battery and have a strong suspicion it will require a lot more work to get it to perform the same on Intel computers" As for clone it and build it in Linux / Windows - do it. And there are other comments here suggesting others want to do the same. I personally want this thing, if I can impact speed of it happening positively, awesome. I want to record everything and have this nice big dataset which is what I've experienced (on my laptop) the last X amount of time, and be able to do stuff with it - whether it's chat with a local llm or have a really good way to search back in time. I _constantly_ need things from the past.

davidy123 3y ago

You could set up screen sharing for any system that supports it, and on another system receive the stream and use an LLM(s) to run object ("cat, piano") and text detection on images and sound. Mac silicon can do that, but it's not as efficient or cost effective as a dedicated GPU. This approach would be system neutral since models work on (increasingly) any GPU including Apple's. On your workstation, you'd just have the overhead (and concerns) of the video stream.