11 ms·
Ok, crazy tangent; Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated,
by MyFirstSass 3y ago
Ok, crazy tangent;
Where agents will potentially become extremely useful/dystopian is when they just silently watch your entire screen at all times. Isolated, encrypted and local preferably.
Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. "I noticed you code this way, may i recommend this pattern" or "i noticed you have signs of this diagnosis from the way you move your mouse and consume content, may i recommend this lifestyle change".
I wonder how long before something like that is feasible, ie a model you install that is constantly updated, but also constantly merged with world data so it becomes more intelligent on two fronts, and can follow as hardware and software advances over the years.
Such a model would be dangerously valuable to corporations / bad actors as it would mirror your psyche and remember so much about you - so it would have to be running with a degree of safety i can't even imagine, or you'd be cloneable or loose all privacy.
- huytersd 3y agoIf that much processing power is that cheap, this phase you’re describing is going to be fleeting because at that point I feel like it could just come up with ideas and code it itself.
- MetalGuru 3y agoIsn’t this what rewind does?
- az226 3y agoRewind.ai
- evaneykelen 3y agoI have tried Rewind and found it very disappointing. Transcripts were of very poor quality and the screen capture timeline proved useless to me.
- Falimonda 3y agoIf it wasn't for the poor transcript quality would you consider Rewind.ai to be valuable enough to use day-to-day? Could you elaborate on what was useless about the screen capture timeline?
- evaneykelen 3y agoI would probably not consider using it, and it's likely due to these factors: 1. I use a limited set of tools (Slack, GitHub, Linear, email), each providing good search capabilities. 2. I can remember things people said, and I said, in a fairly detailed way, and accessing my memory is faster than using a UI. Other minor factors include: I take screenshots judiciously (around 2500-3000 per year) and bookmark URLs (13K URLs on Pinboard). Rewind did not convince me that it was doing all of this twice as well.
- deleted 3y ago[deleted]
- wingerlang 3y agoIf I may do some advertising, I specifically disliked the timeline in Rewind.ai so much so that I built my own application https://screenmemory.app https://screenmemory.app. In fact the timeline is what I work on the most and have the most plans for.
- CamperBob2 3y agoI liked this idea better in THX-1138.
- MyFirstSass 3y agoOne of the movies i've had on my watch list for far too long, thanks for reminding me. But yeah, dystopia is right down the same road we're all going right now.
- mdanger007 3y agoReading The Four by Scott Galloway, Apple, Facebook, Google, and Amazon were dominating the market 7 years ago generating 2.3 trillion in wealth. They're worth double that now. The Four, especially with its AI, is going to control the market in ways that will have a deep impact on government and society.
- MyFirstSass 3y agoYeah, that's one of the developments i'm unable to spin positively. As technological society advances the threshold to enter the market with anything not completely laughable becomes exponentially harder, only consolidating old money or the already established right? What i found so amazing about the early internet, or even just the internet 2.0 was the possibility to create a platform/marketplace/magazine or whatever, and actually have it take off and get a little of the shared growth. But now it seems all growth has become centralised to a few apps and marketplaces and the barrier to entry is getting harder by the hour. Ie. being an entrepreneur is harder now because of tech and market consolidation. But potentially mirrored in previous eras like the industrialisation - i'm just not sure we'll get another "reset" like that to allow new players. Please someone explain how this is wrong and there's still hope for the tech entrepreneurs / sideprojects!
- jjjjj55555 3y agoSeems like the big tech cos are going to build the underlying infrastructure but you'll still be able to identify those small market opportunities and develop and sell solutions to fit them.
- system2 3y agoIf 7 second video consumed 1k token, I'd assume the budget must be insane to process such prompt.
- yazaddaruvala 3y agoUnlikely to be a prompt. It would need to be some form of fine tuning like LORA.
- MyFirstSass 3y agoYeah not feasible with todays methods and rag / lora shenanigans, but the way the field is moving i wouldn't be surprised if new decoder paradigms made it possible. Saw this yesterday, 1M context window but haven't had any time to look into it, just an example new developments happening every week: https://www.reddit.com/r/LocalLLaMA/comments/1as36v9/anyone_tried_the_new_1m_context_window_7b_large/ https://www.reddit.com/r/LocalLLaMA/comments/1as36v9/anyone_...
- Invictus0 3y agoThat's a 7 second video from an HD camera. When recording a screen, you only really need to consider whats changing on the screen.
- nostrebored 3y agoThat’s not true. What content is important context on the screen might change dependent on the new changes.
- MetalGuru 3y agoThe point is you can do massive compression. It’s more like a sequence of sparse images than video.
- zoogeny 3y agoWhy watch your screen when you could feed in video from a wearable pair of glasses like those Instagram Ray Bans. And why stop at video when you could have it record and learn from a mic that is always on. And you might as well throw in a feed of your GPS location and biometrics from your smart watch. When you consider it, we aren't very far away from that at all.
- Animats 3y ago> Imagine it just watching you coding for months, planning stuff, researching things, it could potentially give you personal and professional advice from deep knowledge about you. And then announcing "I can do your job now. You're fired."
- ghxst 3y agoThat's why we would want it to run locally! Think about a fully personalized model that can work out some simple tasks / code while you're going out for groceries, or potentially more complex tasks while you're sleeping.
- underdeserver 3y agoIt's local to your employer's computer.
- albumen 3y agoHave it running on your personal comp, monitoring a screen-share from your work comp. (But that would probably breach your employment contract re saving work on personal machines.)
- eru 3y agoYou could point your local computer's webcam at the work computer. It probably breaks the spirit of the employment contract just as hard, but it's essentially undetectable for the work computer.
- pbhjpbhj 3y agoIs there an app that recreates documents this way? Presumably a ML model that works on images and text could take several overlapping images of a document and piece then together as a reproduction of that document? Kinda like making a 3D CAD model from a few images at different angles, but for documents?
- frizlab 3y agoI would hate that so much.
- FirmwareBurner 3y agoIKR, Who wouldn't want another Clippy constantly nagging you, but this time with a higher IQ and more intimate knowledge of you? /s
- chamomeal 3y agoNot crazy! I listened to a software engineering daily episode about pieces.app. Right now it’s some dev productivity tool or something, but in the interview the guy laid out a crazy vision that sounds like what you’re talking about. He was talking about eventually having an agent that watches your screen and remembers what you do across all apps, and can store it and share it with you team. So you could say “how does my teammate run staging builds?” or “what happened to the documentation on feature x that we never finished building”, and it’ll just know. Obviously that’s far away, and it was just the ramblings of excited founder, but it’s fun to think about. Not sure if I hate it or love it lol
- jerbear4328 3y agoBeing able to ask about stuff other people do seems like it could be ripe with privacy issues, honestly. Even if the model was limited to only recording work stuff, I don't think I would want that. Imagine "how often does my coworker browse to HN during work" or "list examples of dumb mistakes my coworkers have made" for some not-so-bad examples.
- bonoboTP 3y agoEven later it will be ingesting camera feeds from your AR glasses and listening in on your conversations, so you can remember what you agreed on. Just like automated meeting notes with Zoom which already exists, but it will be for real life 24/7. Speech-to-text works. OCR works. LLMs are quite good at getting the semantics of the extracted text. Image understanding is pretty good too already. Just with the things that already exist right now, you can go most of the way. And the CCTV cameras will also all be processed through something like it.
- slg 3y ago>Isolated, encrypted and local of course. And what is the likelihood of that "of course" portion actually happening? What is the business model that makes that route more profitable compared to the current model all the leaders in this tech are using in which they control everything?
- fragmede 3y agoGiven that http://rewind.ai http://rewind.ai is doing just that, the odds are pretty good!
- slg 3y agoNo they aren't. Rewind uses ChatGPT so data is sent off your local device[1]. I understand the actual screen recordings don't leave your machine, but that just creates a catch-22 of what does. Either the text based summaries of those recordings are thorough enough to still be worthy of privacy or the actual answers you get won't actually include many details from those recordings. [1] - https://help.rewind.ai/en/articles/7791703-ask-rewind-s-privacy-policy https://help.rewind.ai/en/articles/7791703-ask-rewind-s-priv...
- fragmede 3y agoah yeah fair point. it's the screen recordings I'm worried about leaving my computer
- worldsayshi 3y agoMaybe it doesn't have to be more profitable. Even if open source models would always be one step behind the closed ones that doesn't mean they won't be good enough.
- shostack 3y agoThis. I want an AI assistant like in the movie Her. But when I think about the realities of data access that requires, and my limited trust in companies that are playing in this space to do so in a way that respects my privacy, I realize I won't get it until it is economically viable to have an open source option run on my own hardware.
- foolfoolz 3y agoyou could design a similar product to do the opposite and anonymize your work automatically
- mixmastamyk 3y ago"It looks like you're writing a suicide note... care for any help?" https://www.reddit.com/r/memes/comments/bb1jq9/clippy_is_quite_perceptive/ https://www.reddit.com/r/memes/comments/bb1jq9/clippy_is_qui...
- pier25 3y ago> encrypted and local of course Only for people who'd pay for that. Free users would become the product.
- fillskills 3y agoUnless its open sourced :)
- troupo 3y agoIn modern world open code often doesn't mean much. E.g. Chrome is opensourced. And yet no one really contributes to it or has any say over the direction its going: https://twitter.com/RickByers/status/1715568535731155100 https://twitter.com/RickByers/status/1715568535731155100
- DariusKocar 3y agoOne needs to follow the money to find the true direction. I think the ideal setup is that such a product is owned by a public figure/org who has no vested interest in making money or using it in a way.
- stavros 3y agoOpen source isn't meant to give everyone control over a specific project. It's meant to make it so, if you don't like the project, you can fork it and chart your own direction for it.
- freedomben 3y agoexactly. open source doesn't mean you can tell other people what to do with their time and/or money. it does mean that you can use your own time and/or money to make it what you want it to be. The fact that there are active forks of Chromium is a pretty good indicator that it is working
- userbinator 3y agoIt's meant to make it so, if you don't like the project, you can fork it and chart your own direction for it. ...accompanied by the wrath of countless others discouraging you from trying to fork if you even so much as give slight indications of wanting to do so, and then when you do, they continue to spread FUD about how your fork is inferior. I've seen plenty of discussions here and elsewhere where the one who suggests forking got a virtual beating for it.
- philips 3y agoI have a friend building something like that at https://perfectmemory.ai https://perfectmemory.ai
- DariusKocar 3y agoI'm working on this! https://www.perfectmemory.ai/ https://www.perfectmemory.ai/ It's encrypted (on top of Bitlocker) and local. There's all this competition who makes the best, most articulate LLM. But the truth is that off-the-shelf 7B models can put sentences together with no problem. It's the context they're missing.
- crooked-v 3y agoI feel like the storage requirements are really going to be these issue for these apps/services that run on "take screenshots and OCR them" functionality with LLMs. If you're using something like this a huge part of the value proposition is in the long term, but until something has a more efficient way to function, even a 1-year history is impractical for a lot of people. For example, consider the classic situation of accidentally giving someone the same Christmas that you did a few years back. A sufficiently powerful personal LLM that 'remembers everything' could absolutely help with that (maybe even give you a nice table of the gifts you've purchased online, who they were for, and what categories of items would complement a previous gift), but only if it can practically store that memory for a multi-year time period.
- DariusKocar 3y agoIt's not that bad. With Perfect Memory AI I see ~9GB a month. That's 108 GB/year. HDD/SSDs are getting bigger than that every year. The storage also varies by what you do, your workflow and display resolution. Here's an article I wrote on my finding of storage requirements. https://www.perfectmemory.ai/support/storage-resources/storage-requirements https://www.perfectmemory.ai/support/storage-resources/stora... And if you want to use the data for LLM only, then you don't need to store the screenshots at all. Then it's ~ 15MB a month
- jascination 3y ago> That's 108 GB/year. HDD/SSDs are getting bigger than that every year. Cries in MacBook Pro
- behat 3y agoHeh. Built a macOS app that does something like this a while ago - https://github.com/bharathpbhat/EssentialApp https://github.com/bharathpbhat/EssentialApp Back then, I used on device OCR and then sent the text to gpt. I’ve been wanting to re-do this with local LLMs
- chancemehmu 3y agoThat's impel - https://tryimpel.com https://tryimpel.com
- dweekly 3y agoThere's limited information on the site - are you using them or affiliated with them? What's your take? Does it work well?
- chancemehmu 3y agoI have been using their beta for the past two weeks and it's pretty good. Like I am watching youtube videos and it just pops up automatically. I don't know if it's public yet, but they sent me this video with the invite: https://youtu.be/dXvhGwj4yGo https://youtu.be/dXvhGwj4yGo
- isaac-sway 3y agoI'd be very keen to beta test as well. If you or anyone else has an invite code, please do get in touch.
- crooked-v 3y agoThe "smart tasks" functionality looks like the most compelling part of that to me, but it would have to be REALLY reliable for me to use it. 50% reliability in capturing tasks is about the same as 0% reliability when it comes to actually being a useful part of anything professional.
- dmix 3y agoThe hard part of any smart automation system, and probably 95% of the UX is timing and managing the prompts/notifications you get. It can do as much as it wants in the background turning that into timely and non-intrusive actionable behaviours is extremely challenging. I spent a long time thinking about a global notification consumption system that would parse all desktop, mobile, email, slack, web app, etc notifications into a single stream and then intelligently organizes it with adaptive timing and focus streams. The cross platform nature made it infeasible but it was a fun thought experiment because we often get repeated notifications on every different device/interface and most of the time we just zone it out cuz it’s overload. Adding a new nanny to your desktop is just going to pile it on even more so you have to be careful.
- oconnor663 3y agoA version of this that seems both easier and less weird would be an AI that listens to you all the time when you're learning a foreign language. Imagine how much faster you could learn, and how much more native you could ultimately get, if you had something that could buzz your watch whenever you said something wrong. And of course you'd calibrate it to understand what level you're at and not spam you constantly. I would love to have something like that, assuming it was voluntary...
- lawlessone 3y ago>assuming it was voluntary... Imagine if it was wrong about something. But every time you tried to submit the bug report it disables your arms via Nueralink.
- lucubratory 3y agoI think even aside from the more outlandish ideas like that one, just having a fluent native speaker to talk to as much as you want would be incredibly valuable. Even more valuable if they are smart/educated enough to act as a language teacher. High-quality LLMs with a conversational interface capable of seamless language switching are an absolute killer app for language learning. A use that seems scientifically possible but technically difficult would be to have an LLM help you engage in essentially immersion learning. Set up something like a pihole, but instead of cutting out ads it intercepts all the content you're consuming (webpages, text, video, images) and translates it to the language you're learning. The idea would be that you don't have to go out and find whole new sources of language to set yourself with a different language's information ecosystem, you can just press a button and convert your current information ecosystem to the language you want to learn. If something like that could be implemented it would be incredibly valuable.
- RogerL 3y agoDon't we have that? My browser offers to translate pages that aren't in English, youtube creates auto generated closed captions, which you can then have it translate to English (or whatever), we have text to speech models for the major languages if you want to hear it verbally (I have no idea if the youtube CC are accessible via an api, but it is certainly something google could do if they wanted to). I'll probably get pushback on the quality of things like auto-generated subtitles, but I did the above to watch and understand a long interview I was interested in but don't possess skill in the language they were using. That was to turn the content into something I already know, but I could do the reverse and turn English content into French or whatever I'm trying to learn.
- cush 3y agohttps://www.rewind.ai/ https://www.rewind.ai/ seems to be exactly this
- Buttons840 3y agoPerhaps even more valuable is if AI can learn to take raw information and display it nicely. Maybe would could finally move beyond decades of crusty GUI toolkits and browser engines.
- busymom0 3y agoAnd then imagine when employers stop asking for resume, cover letters, project portfolios, github etc and instead ask you to upload your entire locally trained LLM.
- bagful 3y agoAmplified Intelligence - I am keenly interested in the future of small-data machine learning as a potential multiplier for the creative mind
- abrichr 3y agoWe are building this at https://openadapt.ai https://openadapt.ai, except the user specifies when to record.
- EGreg 3y agoImagine if it starts suggesting the ideal dating partner as both of you browse profiles. Actually, dating sites can do that now.
- searchableguy 3y agoI pre-ordered the rewind pendant. It will listen 24/7 and help you figure out what happened. I bet meta is thinking of doing this with quest once the battery life improves. https://rewind.ai/pendant https://rewind.ai/pendant
- ramenbytes 3y agoBlack Mirror strikes again.
- 1shooner 3y agoThis service says it's local and privacy-first, but it sends to OpenAI? >Our service, Ask Rewind, integrates OpenAI’s ChatGPT, allowing for the extraction of key information from your device’s audio and video files to produce relevant and personalized outputs in response to your inputs and questions.
- vineyardmike 3y agoI'm not related to the project, but I think they mean that it stores the audio locally, and can transcribe locally. They (plan to) use GPT for summarization. They said you should be able to access the recording locally too. The rest of the company has info on their other free/paid offerings and the split is pretty closely "what do we need to pay for an API to do vs do locally". Again, I'm not associated with them, but that was my expectation after looking at it.
- dustingetz 3y agothoughtcrime
- bonoboTP 3y agoIt doesn't even have to coach you at your job, simply a LLM-powered fuzzy retrieval would be great. Where did I put that file three weeks ago? What was that trick that I had to do to fix that annoying OS config issue? I recall seeing a tweet about a paper that did xyz about half a year ago, what was it called again? Of course taking notes and bookmarking things is possible, but you can't include everything and it takes a lot of discipline to keep things neatly organized. So we take it for granted that every once in a while we forget things, and can't find them again with web searching. But with the new LLMs and multimodal models, in principle this can be solved. Just describe the thing you want to recall in vague natural language and the model will find it. And this kind of retrieval is just one thing. But if it works well, we may also grow to rely on it a lot. Just as many who use GPS in the car never really learn the mental map of the city layout and can't drive around without it. Yeah, I know that some ancient philosopher derided the invention of books the same way (will make our memory lazy). But it can make us less capable by ourselves, but much more capable when augmented with this kind of near-perfect memory.
- Nition 3y agoEventually someone will realise that it'd also be great for telling you where you left your keys, if it'd film everything you see instead of just your screen.
- bonoboTP 3y agoTrue but that's still a bit further away. The screen contents (when mostly working with text) is a much better constrained and cleaner environment compared to camera feeds from real life. And most of the fleeting info we tend to forget appears on screens anyway.
- goatlover 3y agoI simply am not going to have my entire life filmed by an form of technology, I don't care what the advantages are. There's a limit to the level of dystopian dependent uses of these technologies I'm going to put up with. I sincerely hope the majority of the human race feels the same way.
- nebula8804 3y agoIt would be dangerously valuable to bad actors but what if it is available to everyone? Then it may become less dangerous and more of a tool to help people improve their lives. The bad actor can use the tool to arbitrage but just remove that opportunity to arbitrage and there you go!
- parentheses 3y agoAside. Is this your first Sass or Saas?
- spaceman_2020 3y agoThe dystopian angle would be when companies install agents like these on your work computer. The agent learns how you code and work. Soon enough, an agent that imitates you completely can code and work instead of you. At that point, why pay you at all?
- bushbaba 3y agoBasically Google’s current search model, just expanded to ChatGPT style. Great….
- psychoslave 3y agoPerfect, finally I can delegate that lengthy hours spent reading HN fantasies about AI and the laborious art of crafting sarcastic comments.
- delegate 3y agoCan also add the photos you take and all the chats you have with people (eg. whatsapp, fb, etc), the sensor information from your phone (eg. location, health data, etc). This is already possible to implement today, so it's very likely that we'll all have our own personal AIs that know us better than we do.
- te_chris 3y agoI could've used this before where I accidentally booked a non-transferrable flight on a day where I'd also booked tickets to a sold out concert I want(ed) to attend.