8 ms·
Show HN: Hacky Meta Glasses GPT4 Vision Integration
Super hacky implementation due to the lack of an SDK. Fun project though.
In the foodlog demonstration I just made a fake fb account (sorry zucc) called "Mye Food-Log".
- amne 3y agosomewhat off-topic but this is the first time I see bun actually used for something other than "let's compare it to node"
- ge96 3y agoSomeday, witty remark about anaconda and bun
- devon_c 3y agoI generally just follow the "It's a drop in replacement for node" and hope the APIs I need are there. Also makes me feel like a 2023 bleeding edge blazingly fast developer which is 90% of the use case right?
- pzmarzly 3y agoThe project seems to have 0 external runtime dependencies, it should work without "bun install". That is really cool. In the code, it is using typescript, fetch API (so no need for node-fetch) and Bun.serve (so no need for express). From my experience, a lot of users fail to get Node.js projects running - the distributions (Debian, Ubuntu) still ship ancient 10.x or 12.x versions, npm may or may not be installed, "npm install -g" will try writing to /usr by default and fail with EACCESS, etc. Bun gets the UX right - I think that author could even try "bun build --compile" to get ELF binary with bun interpreter and JS payload.
- soderfoo 3y agoAwesome demo, short simple and to the point. I did not know Meta released smart glasses. Has anybody found them useful or is it more of a novelty?
- devon_c 3y agoThey're actually good and have nearly replaced my airpods. I don't need glasses and don't live in SF so I do feel a bit pretentious though. The potential is definitely there and I'd be very surprised if the Meta Reality Labs team haven't already added heaps of features internally and are just waiting or staggering releases.
- TheHumanist 3y agoOk, so how long have you had these? I also was unaware they had released these but they look pretty fun AND the price point is surprisingly approachable. Are you wearing these out in public pretty regularly? Are people noticing the lenses? My main concern with buying these is that I will end up not wearing them in public eventually, which kind of makes them not super useful. I worry about that because I can see some people seeing those lenses and going off on the whole bit about am I filming them? am I filming their kids? And we all now the rabbit hole of uncomfortable public social iteractions this can take one down. So, I worry I would run into too many of these people downtown or something and end up being anxious about even wearing them out anywhere. What is your experience and how long have you been wearing them around in public (if you are)?
- devon_c 3y agoWore them everyday for 2 weeks and I actually only bought them as I thought I could make something like this (through any hacky means). No one has noticed they're any different, especially outside as I got transition lenses so they just go into normal rayban sunglasses mode. You could have a similar effect if you wore those old "new" 3D glasses out and about. Even colleagues had to squint. If I do ever have a bad interaction with strangers I'll just tell them about the lengths meta have gone through to prevent bad actors. (i.e: knowing you're trying to cover up the flash).
- TheHumanist 3y agoWhat has Meta done to prevent bad actors?
- digitcatphd 3y agoReally cool and actually a real use case.
- laputan_machine 3y agoVery cool! Hacker ethos (make a fake account to leverage sending photos), short and simple demo. Nice job
- devon_c 3y agoThanks! I first attempted a more standard approach via a webhook hosted on Heroku linked to an fb page, which theoretically would of worked. But, Meta has limited you to only sending messages/pictures to "real people". I believe they have to be on your friends list for a certain amount of time too but haven't properly tested all the edge cases.
- TheHumanist 3y agoOh is this your project? It's really cool! Ok, so how long have you had these? I was unaware they had released these but they look pretty fun AND the price point is surprisingly approachable. Are you wearing these out in public pretty regularly? Are people noticing the lenses? My main concern with buying these is that I will end up not wearing them in public eventually, which kind of makes them not super useful. I worry about that because I can see some people seeing those lenses and going off on the whole bit about am I filming them? am I filming their kids? And we all now the rabbit hole of uncomfortable public social iteractions this can take one down. So, I worry I would run into too many of these people downtown or something and end up being anxious about even wearing them out anywhere. What is your experience and how long have you been wearing them around in public (if you are)? Also, legit question... how did you succesfully make an alt facebook account? lol Anytime I try with a different name so I can have an alt for buying stuff on marketplace it blocks that new account out because I can't verify it is me (because it's a fake name).
- anonzzzies 3y agoHow about for the quest 2/3? Anyone did that? What similar quality vision models are there outside gpt4 ? It’s annoying it has so maybe (albeit mostly hack-prompt-able) restrictions.
- Philpax 3y agoNo camera access on those, unfortunately.
- anonzzzies 3y agoAh didn’t know that, thanks.
- DevX101 3y agoI love this demo! 35 seconds, terrible video quality, but it shows you can unlock so much more. Great job.
- devon_c 3y agoThanks! Recorded on OBS, uploaded to linkedin, screen recorded on iOS then uploaded to Youtube. Is that not a standard workflow haha (initially was just to send to 1 mate who didn't have linkedin lol)
- webappguy 3y agoSorry did I miss something in the demo, I must have as it appeared you just took a photo and saved it, a function native to the glasses. Where did the vision come in? I'll watch again. Or was it that you saved it to a feed and not the native app
- BowBun 3y agoWhere did the details about his food item come from in the list? That was GPT4 turning his vision into data.
- devon_c 3y agoJust via prompt engineering, was something like: `Using this image, estimate the nutritional information of it and output it as a JSON using this data structure: { "calories":int; "protein":int; etc... } `
- _1 3y agoHe's created another Facebook account. When he takes a photo, he tells the glasses to share it with his other account. He has a service that checking for messages sent to that account. When a new message arrives, theres GPT4-v script that analyses the image and logs the results in his food tracker.
- jcims 3y agoThis is awesome! I got my glasses a couple weeks ago. I’m in Vegas and actually wore them for the first time yesterday. They’re actually really nice and i was thinking how cool it world be to hook it into gpt4v.
- _giorgio_ 3y agoWhat "feedback" do you get from the glasses. Imagine taking a picture or sending a message...
- _1 3y agoAudio only.
- TheHumanist 3y agoOk so do you just wear these around in public? Do you think people will see the pretty obvious lenses and be confrontational? That's the only thing stopping me... I love tech like this but I'm not trying to get bitched at in public when people think I'm filming them or their kids or something like that, you know?
- KaiserPro 3y agoIts really hard to notice that they are different from normal raybans, unless you take a picture. The lens are pretty subtle, and they look like normal wayfarers. When the light comes on, as you make a recording or take a picture, then they notice, but not all of the time.
- TheHumanist 3y agoOk, so if I wanted to walk through downtown on a pretty day and livestream that, it will have a bright light to let people know I am recording? I can't just do that subtly?
- KaiserPro 3y ago
- AlecSchueler 3y agoWhen Google Glass was around there was a big social pushback. Have we accepted the use of these devices around ourselves since then?
- ConfusedDog 3y agoI was not gonna spend $1500 for Google glasses, but $300 for a Rayban isn't bad.
- AlecSchueler 3y agoMy understanding was that the pushback was based on people's discomfort around other people wearing these things, hence the "glasshole" name being used.
- soylentcola 3y agoThese don't even have the (more interesting, IMO) part though - the HUD. Glass was obviously an early experimental design with loads of shortcomings, but at least in concept, a personal HUD was fairly interesting. I didn't care about the camera and found it disappointing that it was the only part people seemed to focus on. Isn't this just the camera/mic bit without the cool part, tied to a Facebook/Meta account instead of a Google one? I agree $1500 was a bit steep for something barely above prototype level, but this is basically an earbud/mic that allows me to take hands-free photos - something I do maybe a few times per year.
- Janicc 3y agoI think the pushback will be pretty insignificant simply based on the fact that compared to Google Glass, they really don't stand out as much.
- AlecSchueler 3y agoDoesn't that make the privacy concerns that people previously had even more prominent?
- m00dy 3y agoObviously this is the future just need a bit more powerful tech.
- criddell 3y agoWhen I see demos like this I think of William Gibson who said "The future is already here – it's just not evenly distributed."
- alonsonic 3y agoThis is a fantastic demo. I've tried to build these type of prototypes for all Meta Headsets in the past but their very limited API/SDKs block you from doing anything meaningful with computer vision. They are too scared of devs getting access to the camera. Hope that Apple Vision Pro gives a more robust api to developers and that forces Meta to open up. I will use this approach for a poc I have in mind. Great job and thank you for open sourcing!
- _giorgio_ 3y agoNo jailbreak yet.
- devon_c 3y agoI did recently try to reverse engineer the connection Instagram are doing with the glasses in order to livestream through the glasses. However, not too familiar with trawling through decompiled APKs (also presume there's some sort of internal secret they're using too) No worries!
- fullspectrumdev 3y agoThat’s been on my endless todo list since I got the first version of the View glasses from FB/Meta ages back.
- yunohn 3y ago> Hope that Apple Vision Pro gives a more robust api to developers I was under the impression that Apple is keeping camera access completely locked down?
- procgen 3y agoThird-party apps can access a single composite "front camera", but only if a "spatial persona" is found on the device. https://developer.apple.com/videos/play/wwdc2023/10094/?time=588 https://developer.apple.com/videos/play/wwdc2023/10094/?time...
- codeulike 3y agoHey Meta, send a photo to Eez Discheaperon Amazon
- devon_c 3y agoLmao Defo possible using a mixture of tavily and the Amazon api!
- WhitneyLand 3y agoNice work Devon! It may be “super hacky” as you say, but these projects are important. We may have had a theoretical understanding that this was possible, but seeing your demo opens up our experiential understanding. Lights up totally different parts of the brain.
- johnwheeler 3y agoI was thinking the same thing. Obviously this is not meant to be a polished product, but it lights a spark that will eventually grow into a fire. We’ve always wanted this capability, and this guy has shown us it practically possible now. Doubtless there will be others, but that’s the point.
- fgblanch 3y agoVery nice hack! I did a very similar project integrating ChatGPT bot but using WhatsApp business account instead of fake facebook contact. I got my account blocked when Meta discovered I'm not a business unfortunately. I'll retake the project with the FB account, it seems much easier. Great job!
- stuckkeys 3y agoI will wait for some kind of version that does not require a facebook account. Cool demo tho.
- charcircuit 3y agoIt does not require a facebook account. It uses a separate meta account.
- gourneau 3y agoMeta folks if you are reading this , please give us a real SDK. Great hack though until we get that!
- b8 3y agoMatter of time until someone hacks Meta's glasses to cheat on tests.
- devon_c 3y agoI started to make an implementation of this for leet code by having a bot transcribe an Instagram live and then could send you tips. But the latency was about 30 seconds so during an interview it wouldn't be that useful haha
- Moldoteck 3y agoWhy? Latency too big, must be connected to smartphone. A smartwatch imo is much more versatile for cheating, esp those that look like analog watches with physical needles that indicate the hour.
- padjo 3y agoBookmarklets are still a thing? I just sorta assumed browsers shut them down years ago
- devon_c 3y agoSurprisingly, really good for automating/improving the UX for various niche internal HR tools too!
- paul7986 3y agoWhat have others experience been using Hey Meta with their Meta glasses? Myself not great as it feels like a beta feature.. doesn't run reliably. Tho good effort by them using current technology. What the glasses do now reliably are taking photos, short videos and transferring them into the Meta View app. Bluetooth audio transferring from my car stereo into the glasses is annoying and I couldn't find a setting to change / stop that. As for Meta Vision there's so much innovation to happen here, nice!! gives me tons of ideas
- nojvek 3y agoI wish meta glasses were an open platform and we could directly integrate with vision/llm apis.