11 ms·
Screenpipe: 24/7 local AI screen and mic recording
- luna4151 2y ago[dead]
- jumploops 2y agoLove the focus on just recording your system, and then using that as a building block for more advanced apps. When I prototyped doing this for a sub-problem (terminal agent), it was nice to have a tight feedback loop between read/write. Curious how difficult it would be to add “actions” on top of this, or if it’s mostly geared towards a read-only/replay mindset.
- vouaobrasil 2y agoWith such technology, we are becoming less and less like human beings and more like technological beings augmented with a biological base. I think it's a bad thing because at least the average human being in modern society is not brought up with wisdom, but only the drive to advance technology and operate in a highly capitalistic world. The augmentation of human beings with tech like this is a proto-type for a dismal world where wisdom is lacking and the pure pursuit of knowledge is becoming a more and more seductive path to destruction.
- esafak 2y agoIf it's all done locally, this is just a more efficient way of taking notes.
- vouaobrasil 2y agoIt's an efficient way, but I vehemently disagree with "just". No technology is "just" anything. All of these little "improvements" constitute a very advanced modification of human beings to become more mechanical and less empathetic towards life.
- deleted 2y ago[deleted]
- ImPostingOnHN 2y agoCyber augmentation (hearing aids, cochlear implants, etc.) are often a means to become more empathetic, because it allows deeper connections. As someone who suffers from ADD, I simply won't be able to recall forever everything someone and I said, so I use technological augmentation in the form of writing down birthdays, for example. When I'm at meetups, when a conversation huddle ends, I'll write down notes, or more likely, send a custom linkedin connection request mentioning what we talked about. The result is that we have the same, empathetic, human conversation as before, and also next time we talk, I can ask them about their startup or hobby, and also I wish them a happy birthday every year, which I think is a net positive without any downsides.
- vouaobrasil 2y agoA common rebuttal, but I don't think the tradeoff (on average) is worth it when the technology becomes sufficiently advanced. (Of course, it's worth it for some people, but the resulting technology makes society worse on average.) And you are forgetting all the destructive technology required to get to the "benign" ones.
- ImPostingOnHN 2y agoWhat do you think the negative trade-offs (the less empathetic you speak of) of the aforementioned examples would be? From my experience, society is better on average as a result of my using notes and calendar entries in the ways I described.
- vouaobrasil 2y agoNegative trade-offs are not directly related to individual products, but to the technology they depend on and the technology that can follow from them, plus our tendency in capitalistic society to invent whatever can be invented for incremental advantages. For example, AI note taking (benign) requires AI (overall bad) and can imply future technologies (greater surveillance). The bad parts cannot be separated from the good in modern global capitalism because we have no oversight mechanism to do so.
- iterateoften 2y agoIts hard for me to see how you could think it is “just” taking notes. Objectively the notes are being constructed and filtered by an external 3rd party (even if it is run on device locally, someone external is still training and choosing the agent). It is the homogenization of thought and note taking of everyone using AI to record their lives that is the potential problem.
- bondarchuk 2y ago>the pure pursuit of knowledge is becoming a more and more seductive path to destruction. How so?
- vouaobrasil 2y agoAs knowledge becomes more powerful in the sense of enabling us to do more things, it becomes more tempting to use it to gain short-term advantages that typically have long-term detrimental consequences. Such as AI for example, which is too quick at disrupting employment or cheap energy to generate bitcoin but is problematic for local energy grids. The more powerful the knowledge, the easier it is for people to ignore the downsides at the expense of fellow human beings. That is especially true because we have an economic system that rewards short-term improvements in the efficiency of the system, regardless of the long-term costs. Fossil fuel use, cutting down local forests (has relatively litle short-term impact, but adds up). And, as we pursue knowledge and technology more vigorously, we slowly lose other forms of gaining knowledge such as a relationship with nature. Human society is advanced with regard to its knowledge capability, but exceptionally primitive with regard to basic wisdom about community, love, nature, and friendship. We continually donwgrade these things to make way for new technology, and the prisoner's dilemma (tech gives some people advantages, so everyone is pressured to use it), makes it hard to make decisions for the long-run like the Amish do.
- XorNot 2y agoWhat a bizzare attitude to take on a site fundamentally based on the advancement and refinement of information handling technology Like what do you think everyone who posts here does?
- vouaobrasil 2y agoI understand exactly what everyone else does here. And nothing intrinsically wrong with that -- technology is unquestionably fun and interesting. I like programming myself. BUT, and this is a huge BUT, I think we as people who are well versed in technology should take a little more responsibility for what we create. So, not bizarre at all.
- dylan604 2y agoi guarantee you not everyone here has benevolent intent. for example, we know people from FB,TikTok,Snap,Twit...er,X are here. what do you think everyone does?
- sirsinsalot 2y agoDo you want an echo chamber? Perhaps just being fed with opinions you agree with? We should celebrate opinions that go against local conventions
- 7952 2y agoArguably this kind of viewpoint can be particuarly interesting to this group. People are well placed to see some of the more worrying aspects of technology.
- deleted 2y ago[deleted]
- CaptainFever 2y ago> we are becoming less and less like human beings and more like technological beings augmented with a biological base I would actually love to be a technological being. This is transhumanism, isn't it?
- vouaobrasil 2y agoYes and I think it's a horrible thing.
- cparish 2y agoI resonate with your concerns, but I believe the best way to secure a human-centric future is fully diving in to technology and our constant surveillance realities. My goal is to empower people to collect their own data to help them objectively understand the impact technology has on themselves: https://hindsight.life/ https://hindsight.life/
- vouaobrasil 2y agoI am absolutely against a human-centric future. I advocate for a biological future where animals have equal rights to exist.
- darknavi 2y agoHow is the data stored? Isn't this the feature Windows got royally roasted for because it stored the data in something like a SQLite db? How are you protecting the data locally? Sorry if it's in the README, I didn't see it when skimming.
- bertman 2y agoThey store everything in an SQLite db :) https://github.com/mediar-ai/screenpipe/blob/4f2391c6dfd8775919a5f59f72313ea0d5d353eb/screenpipe-server/src/db.rs#L242 https://github.com/mediar-ai/screenpipe/blob/4f2391c6dfd8775...
- botanical76 2y agoWhat is the recommended approach for this? I feel as though the specific database should be irrelevant. All OSes are equipped with permissions models that can restrict a SQLite file to use by a single application.
- g_p 2y agoThe issue so far seems to be that most OSs don't really have an effective way to restrict that file to a single application. User-oriented filesystem permissions don't work, as all software runs "as" the user. If you assume there's a way to restrict permissions by application (a bit like TCC on Mac for certain folders), you need to then go down a rabbit-hole of what matcher you use to decide what is a "single application" - Mac OS can use developer Team ID (i.e. app signature identity), or similar. You wouldn't want to rely on path or binary name, as those could be spoofed or modified by a rogue app. So in short, in a multi-user OS, generally the filesystem (asides from Mac OS, under certain circumstances) is fairly widely readable by other software running as the current user. At least in my experience, Mac OS is the desktop OS that is closest to having some level of effective protections against apps accessing "everything" owned by the user (but belonging to other apps).
- chankstein38 2y agoAdmittedly I have little knowledge on the Windows feature's functionality but my problem with that is that I want to choose whether or not something like this is happening on my computer and have control over it. I barely trust Microsoft and Windows anymore as it is but it's a somewhat-necessary evil in my case. I don't trust them to record my data and actually keep it local to me and I want to actively find software to do it not have them auto install something and have full control over the data.
- vid 2y agoI tried it a week or so ago, it didn't really go well. Transcription was terrible, even though using the same model (whisper-large) on its own works well. The UI messages were trying to make it sound like a straightforward app, they should have been much more helpful as many things went wrong with a "try again later" type message. I also wish they'd bundled more functionality in the service, so the app could be a simple web front end that uses its API. At a greater level, I've always wanted something like this, but we shouldn't be able to collect info on other people without their permission. The dangers of breaches or abuse are too great. Since corporations are holding back the greatest benefits, we should be able to remember commercial works we've accessed, and collect and organize info on corporations, and some info on governments, but by their nature corporations will be better at getting the upper hand and the resulting attacks on government might not be for the better for society at large. Yes, some parts of some governments conduct abuses which should be called out, but that is specific departments, other departments work to other principles, sometimes pushing back against the abuse. Otherwise comparing governments to business or computers is a downward spiral. This[1] is an interesting series on this topic. 1. https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_of_Loving_Grace_(TV_series) https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_o...
- louis030195 2y agohey sorry about this, we improved voice activity detection since then, also you can use deepgram if you want higher quality
- vid 2y agoGood to hear you improved it. I just need it to be as good as standalone whisper-large. I certainly don't want to send all my conversations to a third party.
- layoric 2y agoWhat’s the power consumption like for running this 24/7? Is it event based to reduce the need to process data when pc is idle? It’s an interesting idea for sure, but seems like a lot for an alternative of providing specific context.
- inhumantsar 2y agoscreen recording isn't generally that hard on a system, depending on display config of course. video compression would be piped thru the GPU's media encoders, which are extremely efficient these days.
- layoric 2y agoYeah but tokenising all of it regardless of if it is ever used would be a lot more intensive yes?
- elif 2y agoHmm this might unintentionally be the glue that brings us actually smart Star Trek computer assistants. Think about it, you can access every single kind of automation through a desktop app or android emulator. Screenpipe becomes a working memory that the LLM develops to have context of its previous actions AND their results. "Computer, lights" can not only send out a signal like a TV remote, it can check a camera to make sure that the light for the room the user is looking at actually turned on at an appropriate light level for the situation.
- ericd 2y agoI’ve dabbled with building something like this for myself, I’m guessing it’s not totally unintentional, it’s the first step. After getting it to interpret what it sees on your computer, give it the ability to use mouse/keyboard, maybe in a sandbox vm, and trying to do some basic tasks, working up from there. No way I’d use something like this that wasn’t local-only, though.
- cloudking 2y agohttps://www.rewind.ai/ https://www.rewind.ai/ tries to do this also. Ultimately I found over the course of a work week, there was way too much noise vs signal for it be useful.
- replwoacause 2y agoI heard some not so great things about the CEO of that company on Reddit. Now sure how true it was but I recall it putting me off the product at the time, considering the sensitivity of the data they would have on me and my company. Reputation matters in this space and his seemed questionable.
- nerdjon 2y agoI can no doubt see the value in this, even being a big skeptic of AI the ability for a computer to just pull up details from throughout the day or interject is genuinely a powerful idea (if it works in practice is another question). However, I find it very frustrating that all of the tools that just record your mic or sell a pin or whatever never think about privacy. They seem to take the Google approach that the person who owns the device is allows to consent for everyone else around them (or any data they share with certain people) when that is not ok. That is before talking about the major concern of people doing this at work without their company knowing, bringing it into a meeting and potentially leaking sensitive information. I realize that unfortunately there is nothing that can be done at this point to stop this technology from being used by people, but I just wish the people that were making these tools thought about the fact that you are not asking for the consent of people around you.
- amelius 2y agoFrom the readme: "Open. Secure. You own your data."
- Cheer2171 2y agoThat's not what they were saying. What comes across your screen and can be picked up by your device's mic includes other people's data. This is a nightmare for anyone in the EU or two-party consent state. If we are on a Zoom call, this is recording both sides of the conversation without notice or consent. If we are meeting in person, and I leave my device on the table with this installed, same problem.
- piltdownman 2y agoWhy would it be a 'nightmare' for anyone in the EU? Only single party consent is required for recording conversations whether on a phone or in person - Ireland, Italy, Czechia, Latvia, Poland, The Netherlands... in fact the only prominent country that comes to mind re: two party consent is Germany.
- nerdjon 2y ago
- Cheer2171 2y agoThis is a nightmare and illegal for anyone in the EU or two-party consent state OR anyone who interacts with others in the EU or any two-party consent state. What comes across your screen and what can be picked up by your device's mic includes other people's data. If we are on a Zoom call, this is recording both sides of the conversation without notice or consent. If we are meeting in person, and I leave my device on the table with this installed, same problem. I can't find anything in the docs about how to pause or temporarily stop data collection. People don't like others recording every interaction they have, which is what killed Google Glass.
- xd1936 2y agoAre you sure Google Glass looking absurd wasn't what killed Google Glass?
- brokencomb 2y agoNot all EU countries are two-party consent.
- KaoruAoiShiho 2y agoNo it's not what killed google glass lol.
- ls612 2y agoOn the other hand it's my system and I don't care what the Euros or anyone else thinks about the software I run on it. Maybe if you are a big business with compliance departments and whatnot this would be an issue but this product seems more geared towards tech savvy individuals.
- lagniappe 2y agoI miss BonziBuddy :/
- neilv 2y agoBefore things like this become widespread, can we get some etiquette (and legal awareness)? On one recent videoconf with a startup founder, it turned out that they were using some random AI product with access to video and audio on the call, without my knowledge or consent. (Thanks a lot; now some AI gold rush company has video and audio recording of me, as well as ingested the non-public information on the call.) Their response to me asking about that made me sure I wanted nothing to do with that startup. But some AI company probably still has that private data, and if so, presumably will go on to leak it various ways. And it seems the only way to get an AI goldrush company to actually remove all trace of data it's obtained sketchily/illegally, might be to go legal scorched-earth on the company and its executives, as well as to any affiliates to whom they've leaked. We shouldn't need random people undertaking filing lawsuits and criminal complaints, just because some random other person was oblivious/indifferent about "consenting" on their behalf. Which "consent" I don't think is legal consent (especially not in "two-party" states), but AI goldrush companies don't care, yet.
- goldfeld 2y agoIf really it is to become widespread it means we as society really got completely lost in the fold. It would be an act of defeat of the human element. But the magnates gonna love it.
- grendelt 2y agoA number of states have laws where you must disclose if the call is being recorded. I pointed this out to a former supervisor that was using otter.ai to record and transcribe calls he would make with potential leads and conference contacts. He said that law didn't apply in the state of our home-office. I told him it applies on calls made to subjects in other states. He disagreed. Not a lawyer, but I don't think that's how it works. And something like this... yikes. Sure, boss, you're not doing eyeball tracking, but all they would have to do is install this on a work computer and just ask AI for a percentage of time not directly spent hammering out code and pay you piecemeal per keystroke or something truly awful. (and the webcam module is coming later, I'm sure.) The future is dystopian.
- 2y ago
- BrouteMinou 2y agoMicrosoft is doing Recall, encrypted and what not with the TPM => Microsoft BAD! A random dude is doing the same thing, but in rust (super important to mention right), storing the data in sqlite => Magnificent project, we want more! You guys are fucking weird.
- ekianjo 2y agoYou understand that the data stays on the computer right? With Microsoft you never do what they do down the road even if they start local
- mldbk 2y agoRandom dude making the same thing is not the same thing what MS does. 1. You have a choice. With MS you don't. You can't opt-out, at least for now. 2. And as prev buddy said, you never know what MS will do with your data. 3. Recall will be heavily targeted and from day-1 some malware will target it. Random dude's pet project doesn't (even though it is a security through obscurity).
- wkat4242 2y ago> Microsoft is doing Recall, encrypted and what not with the TPM => Microsoft BAD! 1) It was not in fact encrypted and any user could mess with it. AND the data was stored in SQLite too. Microsoft only started fixing this after their totally negligent security was brought to light. 2) Recall is maintained and auto updated by Microsoft which can change the rules at any point (e.g. add datamining at a later point). At least with Screenpipe the end user decides. This solution is open-source so you know what's happening.
- beefnugs 2y agoThere is such a big difference between this being shoved down the throat of every single windows user in the world, vs someone choosing to download and run it themselves
- elintknower 2y agoAn intelligence agency's dream :)
- wkat4242 2y agoTrue, though those will probably install loggers on their targets' computers anyway.
- wkat4242 2y agoSo, basically this is Windows Recall for Mac. I'm still very iffy about this. It opens a huge can of worms in terms of privacy. However at least in this case it's not managed by a big company but installed and maintained by the users themselves.
- deleted 2y ago[deleted]
- benhurmarcel 2y ago> So, basically this is Windows Recall for Mac. This concept on Mac has been an existing product for years: https://www.rewind.ai/ https://www.rewind.ai/
- hu3 2y agoworks on windows too
- qntmfred 2y agoI've been playing around with this concept this year as well. Not surprised to see the negativity in the comments. But there is value in this concept, and some of us are going to explore it and find out how to make it work without the dystopian weirdness some of y'all immediately jump to. I've recorded roughly ~1000 hours of livestream of myself this year. 90%+ of it is boring nothingness, but it is not intended for public consumption anyways. I have and will continue to chop up and post on youtube/twitter the interesting moments I do end up capturing, but it's mostly for myself anyways. I haven't quite gotten to the point where I've been able to articulate the benefits of this practice in a way that might sufficiently pique the interest or at least acceptance of others in the tech community or otherwise, but that's ok. I did make one video where I started to touch on this idea https://www.youtube.com/watch?v=2zqXkNhaJx0 https://www.youtube.com/watch?v=2zqXkNhaJx0 and I will keep working on it and hopefully demonstrate the personal value (and perhaps societal at scale) as time goes on. The future is going to get pretty weird. Keep an open mind.
- varispeed 2y agoHopefully one day we will reach the place where such AI contraption after watching us for hours, days and months will be able to just do stuff we do and we could switch off the computers. Check email, respond to some, see some news, pull up tasks, do some code, ship some code, read some more news, watch a tutorial, do some more code, check mails, respond to some etc etc. At the end of the day send you a summary, some highlights, occasionally call you for manual intervention.
- Scea91 2y agoHow do you as a user validate the security of tools like this? I am avoiding many potentially useful tools (e.g. Some Obsidian or Chrome plugins) out of paranoia. Is there a better way than going through the code or running Wireshark myself? Even these are not bulletproof… For now, I am resorting to building the simple things for my use myself. The benefit is that its often quite fun.
- buffer1337 2y agoThis could turn into a market changing app. When they finish the desktop interaction API, you chain it together with babyagi et al, and you get an ai agent that operates on a variety of simple tasks, in many different software applications.
- matthewddy 2y ago[dead]
- liendolucas 2y agoI can't understand why we have to inject AI in every piece of technology be hardware or software. Few days ago another thread about HP printers having an AI feature (yes, WTF?). It feels like tomorrow we will be taking a s*t on AI featured toilettes. Madness.
- zild3d 2y agodoesn't sound unlike the IoT craze, where companies were liberally "enabling" various household items with internet connectivity. We're in the "lets see what sticks" phase here https://staceyoniot.com/connected-toilets-have-a-lesson-for-the-iot/ https://staceyoniot.com/connected-toilets-have-a-lesson-for-...
- signatoremo 2y ago> about HP printers having an AI feature (yes, WTF?) Sounds like you are talking about a new feature that allows content to fit pages better —- saving paper, saving ink. Not sure if the same can be achieved without AI, but it’s a useful feature, not gimmick. https://arstechnica.com/gadgets/2024/09/in-rare-move-from-printing-industry-hp-actually-has-a-decent-idea/ https://arstechnica.com/gadgets/2024/09/in-rare-move-from-pr...
- louis030195 2y agoThanks for sharing screenpipe :) (author here)
- tomComb 2y agoI was liking what I was seeing, and I appreciate you making your business model very clear. But then I saw that users can get it free by posting "about screenpipe 10 times on social media". If you want ads, pay for proper ads! Don’t pay people to turn user content into sneaky ads. I understand why people hate regular ads, but IMO affiliate promotion (when done without disclosure) and stuff like what you are doing is much worse.
- spullara 2y agoI tried using this during a meeting and my CPU was hitting 700% and my fans started blasting on a MacBook M3 Max... ctrl-c'd and moved on.
- maxglute 2y agoUndetermined legalities aside, is there any doubt this is how personal digital assistance will trend going forward? It's just like messaging read reciepts, I want to be privy to others info, but not vice versa. Maybe that's the eventual legal compromise, these are going to be networked database, if you want to retain others info, you need to opt in yourself. If one opts out, recordings will turn into placeholder image and written transcripts. Also, if you transcribe a conversation, but do not record the audio, is that still relevant to recording consent laws? Even if it conversation exists temperarily in some hardware buffer?
- tofof 2y agoAs another user points out, Screenpipe currently allows users to obtain the app for free[1][2] by posting 10 times on social media. This story is not organic; it is Screenpipe scummily leveraging their client base into sneaking advertising into social media including HN. If their clients are already just leverage at the outset, I have no doubt their clients' data will just be leverage later. 1: https://screenpi.pe/onboarding https://screenpi.pe/onboarding 2: https://i.imgur.com/UvjXc1I.png https://i.imgur.com/UvjXc1I.png
- musicale 2y agoIs that you Satya Nadella?
- systemerror06 2y agoit's a helpful tool
- hacka 2y agoThey're building something with a lot of potential if they can get good vertical specific apps developed on top of it. I'm sure they'll build more mindful privacy features over time (or they'll have to because of various pressures). I wonder if they could build a LinkedIn prospecting tool that doesn't send spammy, fully automated messages? LinkedIn seems to flag and block accounts using other AI automation tools.
- aantix 2y agoI love this idea. But why does it say "70 users run screenpipe 24/7!" What's being reported from the app, that they know this number?