30 ms·
If you're interested in eye-tracking, I'm interested in funding you
- mhb 3y agoIn responses, there seem to be dozens of experts and companies already doing this. Where does he think they fall short of meeting his friend's needs?
- soligern 3y agoIt feels like Apple has solved this problem with the Vision Pro? So just wait for a couple more months?
- woodruffw 3y agoThe context here is ALS; someone with only the ability to move their eyes may not have a couple of months free to wait.
- soligern 3y agoI think that makes sense but a startup is going to have a hard time delivering something better between now and January if they start from scratch. But maybe I’m wrong, it might be a purely software based problem that can be solved quicker.
- woodruffw 3y ago> I think that makes sense but a startup is going to have a hard time delivering something better between now and January if they start from scratch. I agree entirely!
- KaiserPro 3y agonot really. There are a bunch of hardware out there to get gaze vectors, the problem is that they rely on a generalised model of the eye. With fine tuning you can go from >5 degree of error, to <1-2
- foooorsyth 3y agoAlso donning the AVP headset might not be possible for someone with advanced ALS. A fixed outside-in apparatus (probably attached to the patient’s wheelchair) would make more sense.
- post_break 3y agoExcept Apple has stated they will encrypt that information and not supply it to apps to avoid targeting or fingerprinting. And second how would that help someone with ALS?
- hparadiz 3y agoPeople use stuff like this in VRChat VR Eye-Tracking Add-on Droolon F1 for Cosmos（Basic Version） https://a.co/d/asAAZwT https://a.co/d/asAAZwT You can use it with any headset cause the data it provides is independent of the headset. Not sure how accurate that one is in particular. Eye tracking data is not particularly complex nor super sensitive in nature. Once the sensor is calibrated it just sends over two vectors.
- gnicholas 3y ago> And second how would that help someone with ALS? They won't share information on where you look, but they will share info on where you 'click', which is used to navigate apps. IIRC you are supposed to use your fingers to do this, and other actions, but I imagine that Apple's accessibility team will have an alternate mode for people with motor limitations. It could be a long-blink, or rapid blinking, for example.
- LegitShady 3y agoEven the large headset is a no go for als sufferers.
- epi3 3y ago[dead]
- fastball 3y agoIs the lack of mentioning Apple deliberate? It seems like they've already poured a lot of R&D into this for the Vision Pro, which might be exactly the kind of thing the friend needs.
- pg 3y agoIt's not available yet. And in any case if this is the future there should be multiple companies doing it, not just Apple.
- Difwif 3y agoGlad to see you still lurk around here. How do you think about going up against a company like Apple when it comes to patents in this space? I think that would be my major hesitation but I don't have a lot of experience evaluating patents. Apples Eye Tracking Patent: https://patents.google.com/patent/US20180113508A1/en https://patents.google.com/patent/US20180113508A1/en
- ladberg 3y agoI used to work on it and have spent tons of time in the headset. The eye tracking is next-level and it's really the only platform that exists with eye tracking as a primary input method. I'm pretty confident it will greatly improve your friend's quality of life. Because of that, I'm also sure that eye tracking will go mainstream in other areas once the Vision Pro is released once everyone else catches on to it as a great input method.
- dgrin91 3y agoDoesn't it rely on external cameras to see users hands and use that as the "click" inputs? Seems like that negates usage for ALS cases. Also, I'm not an ALS expert, but if the only muscular control is in the eyes, then lack of control in the head/neck probably breaks some assumptions about how the vision headset works (just a guess though).
- extra88 3y ago
- aaron695 3y agohttps://nitter.net/paulg/status/1695596853864321055 https://nitter.net/paulg/status/1695596853864321055 to see more of the thread if not logged in. It'd be good to know what rate we need to beat and some other metrics.
- aaron695 3y agoIs anyone speedrunning this? I can't find anything with a quick look. I'd consider an approach like the human powered helicopter parable. I'd create a model for the eyes / face. So a screen like a phone is put in front of a camera with a model face and controllable eyes that you use software to control. Maybe skip the screen in front of a camera and link straight into the video feed. It knows the limits of the eyes (different models for different people and situations) can measure fatigue etc. You could run billions of simulations....
- deleted 3y ago[deleted]
- gwurldz 3y agoThis seems like it would fit https://thinksmartbox.com/products/eye-gaze/ https://thinksmartbox.com/products/eye-gaze/ I once interviewed at this company. Unfortunately didn't get the job but very impressed nonetheless.
- ankaAr 3y agoI work for them.., they have their ins and outs. The stack is very impressive but there is a loooot to improve yet
- Verath 3y agoI was part of developing the Lumin-i variant of this (working for Smart Eye)! Working with the Smartbox team was a pleasure :). The solution actually works pretty well, especially when calibrated to a single individual.
- PBnFlash 3y agoI suspect a foveated system is going to be a big thing in machine vision as well.
- steve_adams_86 3y agoThat's an interesting idea. How do you see it being beneficial to machine learning models, other than (I assume) it could work more efficiently within less foveal regions? Perhaps cases where you want the vision to emulate human vision?
- PBnFlash 3y agoRunning a fast network on sparse data then calling one optimized for a task on a subset seems like a good optimization and dealing with video we are probably going to need them. After you parse what an object is, tracking it doesn't take anywhere near the effort of original segmentation. No need to re-evaluate until something changes. Maybe even use activations to turn on and off networks. "Oh text better load ocr into memory" And it does inform a lot of our built world It's strange to think that when watching a movie only 10% is in focus. Eye movement does provide a lot of information to other people and I think the physical movement produces feedback for velocities and things too. Mimicking biology is often a good bet.
- hardwaregeek 3y agoAw, that's nice of pg to want something better for his friend. As cynical as we are about technology, new developments can be so fantastic for accessibility and better quality of life.
- forgetfreeman 3y agoAnd if that were where the story ended we'd have an honest feel-good going. New developments -could- be fantastic for accessibility and QoL, but without exception they just end up getting sucked into a marketing surveillance suite.
- frenchman99 3y agoUnfortunately we use all that technology on devices that are built with metals mined by kids in Congo and similar states. Talk about quality of life.
- lotsofpulp 3y agoThose are unrelated topics. Not developing the technology is not going to help those kids’ quality of life.
- frenchman99 3y agoBut building it as we currently are is directly putting them at risk of poisoning, dying buried alive when their cobalt mining tunnels collapse, etc.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- CyberDildonics 3y agoThis has nothing to do with anything and it isn't even true. You are talking about cobalt and that is only used in lithium ion batteries. You can avoid cobalt by using lithium iron phosphate batteries. There are active efforts to develop new lithium ion chemistries to avoid cobalt and there are even commercial sodium ion batteries now. You bring this in to a completely irrelevant conversation, what have you done to help solve it?
- turnsout 3y agoSomeone should do this, but for the love of god, DO NOT take any venture capital to do it. No matter how well-intentioned the VCs are at the start, eventually your eye tracking startup will 100% be used for advertising as investors in your Series D need to make an exit or take you public.
- methou 3y agoeyetracking is already used by the advertisers in research for many many years, e.g: https://www.tobii.com/ https://www.tobii.com/
- ta123456789 3y ago[dead]
- turnsout 3y agoYes, in research. But once they can see what real-world users are looking at in context, in realtime, it's a different ballgame.
- ta123456789 3y agoThis already happening in many places. And does not seem to be he intended application in the OP
- deleted 3y ago[deleted]
- shakna 3y agoNetflix already pause their ads if their window is not in focus. Not a huge leap to pausing ads if you're not paying attention.
- 0xcde4c3db 3y agoThe overarching dystopian concept is at least a solid decade old by now. [1] [2] [1] https://patents.google.com/patent/US8246454B2/en https://patents.google.com/patent/US8246454B2/en [2] https://i.imgur.com/dgGvgKF.png https://i.imgur.com/dgGvgKF.png
- FlamingMoe 3y agoNot really a huge PG fan, but this is what billionaires should be doing: see where a need is exists and put some of your insane wealth towards making an improvement. This why I respect Elon even though I don’t really like him; he puts his money to use, in a very public manner.
- borg16 3y ago> Not really a huge PG fan can you elaborate?
- steve_adams_86 3y agoI'm not arguing that Elon doesn't put his money to use publicly, but I'm interested how you think he does. Do you mean he uses his wealth privately in a way that benefits the public, and/or he does it in public view?
- qup 3y agoMusk has a track record of working on what he considers (among) the most important problems, and he supposedly set that course in college.
- frenchman99 3y agoI fail to see how Tesla fixes an "important problem". They're big electric SUV with huge batteries that require tons of mining and destroy the environment. They're largely non repairable. The full self driving feature seems to be vapor ware. The space stuff looked interesting until he started sending all kinds of things into space to become an ISP, polluting space the same way Tesla pollutes the earth. The real issues are still here: poverty, lack of access to clean water, etc. How does Musk fix that?
- xeromal 3y agoThis seems like those conservative talking points "I saw a homeless child on my drive to work today. Why are we sending money to Ukraine". We can work on many problems at once. Tesla also makes a car.
- fartjetpack 3y agoAnother route might also be sub-vocalization[1], like TTS for your thoughts. I recently picked up some cheap toys to get started trying to emulate the results[2]. 1. https://www.nasa.gov/centers/ames/news/releases/2004/subvocal/subvocal.html https://www.nasa.gov/centers/ames/news/releases/2004/subvoca... 2. https://github.com/kitschpatrol/Brain https://github.com/kitschpatrol/Brain
- qup 3y agoHow is it going? It's not obvious to me at a glance.
- fartjetpack 3y agoI should clarify that is not my repo. I just received the Mind Trainers and am in the process of finding suitable EEG pads.
- qup 3y agoI guess I'm just interested in more information. Do people who aren't NASA have this working? I am just learning this sci-fi feature from Ender's universe is possibly a reality, and if I can check it out, I want to. Got any jumping off points?
- fartjetpack 3y agoTBH, other than the NASA resources, I'm building it from scratch. Goal would be to record the signal while prompting a user to say/think a subset of words. Afterwards the HOPE is that I will be able to train a simple- functional- model based on those inputs. EDIT: User `tbenst` linked their thesis above[1]. 1. https://neuroscience.stanford.edu/research/funded-research/silent-speech-decoding-using-flexible-electronics-and-artificial-0 https://neuroscience.stanford.edu/research/funded-research/s...
- tbenst 3y agoI agree! My PhD thesis is on this topic [1]. We’ve also done a very limited pilot test on a patient with ALS, with above random chance. Actual results may vary heavily on individual disease progression—the more motor recruitment that’s intact, the better. [1] https://neuroscience.stanford.edu/research/funded-research/silent-speech-decoding-using-flexible-electronics-and-artificial-0 https://neuroscience.stanford.edu/research/funded-research/s...
- splatcollision 3y agoThis was designed for a graffiti artist with ALS: https://en.wikipedia.org/wiki/EyeWriter https://en.wikipedia.org/wiki/EyeWriter https://github.com/eyewriter/eyewriter https://github.com/eyewriter/eyewriter https://www.instructables.com/The-EyeWriter-20/ https://www.instructables.com/The-EyeWriter-20/ https://www.moma.org/collection/works/145518 https://www.moma.org/collection/works/145518
- dewarrn1 3y agoEEG recording is an alternative that would outlast the potential disease-related degradation of eye movements. Manny Donchin gave a brown bag at UIUC about the possibilities of using this approach to support communication by ALS patients many years ago. It's clever: they use the P300 marker to index attention/intention. I do not recall whether he and his colleagues ever commercialized the tech. I believe that this publication is representative: https://doi.org/10.1016/j.clinph.2005.06.027 https://doi.org/10.1016/j.clinph.2005.06.027
- incongruity 3y agoThis! Eye tracking is slow and not good - but does that just mean we need to “faster horse” it or is there another option for bridging the communication gap for people with ALS and similar diseases? I have to believe there are better answers with other tech - likely EEG (+ AI). My family is one of the unlucky ones that has genes for ALS so I’ve watched enough family members struggle. (I’m lucky, selfishly, because I dodged the gene but I still care deeply about this).
- BoorishBears 3y agoI had this thought, but then I thought about if my friend was struggling with a problem had had practical but imperfect solutions, would I better serve them by funding highly feasible solutions that they're already familiar with, or experimental moonshots that are more likely to fail, will take longer to implement, and my friend may not even care for at all...
- mikpanko 3y agoI did a PhD in brain-computer interfaces, including EEG and implanted electrodes. BCI research to a big extent focuses on helping paralyzed individuals regain communication. Unfortunately, EEG (including P300) doesn’t provide sufficient signal-to-noise ratio to support good communication speeds outside of the lab with Faraday cages and days/weeks of de-noising including removing eye-movement artifacts in the recordings. This is a physical limit due to attenuation of brain’s electrical fields outside of the skull, which is hard to overcome. For example, all commercial “mind-reading” toys are actually working based off head and eye muscle signals. Implanted electrodes provide better signal but are many iterations away from becoming viable commercially. Signal degrades over months as the brain builds scar tissue around electrodes and the brain surgery is obviously pretty dangerous. Iteration cycles are very slow because of the need for government approval for testing in humans (for a good reason). If I wanted to help a paralyzed friend, who could only move his/her eyes, I would definitely focus on the eye-tracking tech. It hands-down beat all BCIs I’ve heard of.
- AndrewKemendo 3y agoI responded to the thread but Senseye has been working on this for a while now. Originally they were working with the US Air Force to help with improving pilot training - fatigue etc.. inference with retinal reading https://senseye.co/ https://senseye.co/ They have generally struggled to find funding for their eye tracking focused work, and have recently had to pivot away from the really exciting but hard to fund stuff into PTSD screening (which is important too). I can connect you with the founder if desired via the email in my bio
- morkalork 3y agoPlease look directly at the screen to continue Whilst it plays an unskippable and unblockable ad (thanks weiapi!)
- deleted 3y ago[deleted]
- jacquesm 3y agoThat's straight out of Black Mirror ("Fifteen million merits").
- blackguardx 3y agoI worked on eye tracking hardware for Microsoft HoloLens. Several AR headsets offer decent eye tracking, including Hololens 2 and Magic Leap's ML2. I think Tobii's eye tracking glasses are probably better as a stand-alone solution though: https://www.tobii.com/products/eye-trackers/wearables/tobii-pro-glasses-3 https://www.tobii.com/products/eye-trackers/wearables/tobii-...
- tootie 3y agoHave used Tobii as well and they are very accurate with a bit of calibration.
- zh3 3y agoAgreed, the eye tracking itself is really a mostly-solved problem (Tobii are indeed leaders in the area). It's how it's used that matters - and as mentioned above, it's likely that it's the usability/interface that needs work.
- ErneX 3y agoPS VR2 uses Tobii's tech to do eye tracking, it is mostly being used for foveated rendering but some games also use it for gameplay, one for example allows you to shoot at enemies with your gaze.
- lostmsu 3y agoHow much do Tobii glasses cost?
- nocoiner 3y agoI don’t know about the Tobii glasses, but I have a Tobii head/eye tracker that attaches to my monitor, and it works incredibly well and was only $200 or so. I’d be surprised if this isn’t an essentially solved problem at this point.
- lostmsu 3y agoI have the tracker too, but it is insufficient for my setup (multiple larger monitor).
- readyplayernull 3y ago> A friend of mine has ALS and can only move his eyes. He has an eye-controlled keyboard, but it's not very good. Can you make him a better one? When I worked for one of the big game engines I got contacted by the makers of the tech that Stephen Hawking used to communicate, which includes an eye tracker: https://www.businessinsider.com/an-eye-tracking-interface-helps-als-patients-use-computers-2015-9 https://www.businessinsider.com/an-eye-tracking-interface-he...
- frakkingcylons 3y agoI hope this effort bears fruit. My uncle passed from ALS eight years ago.
- cratermoon 3y ago[flagged]
- user3939382 3y agoHow about a library that starts loading a link when you look at it with intent. Or maybe with BCI integration that detects the moment you decide you want to access it. Or how about a UI that automatically adapts to your eye movement and access patterns to minimize the amount of eye movement required to complete your most common tasks by rearranging the UI elements.
- foreverobama 3y ago[dead]
- kken 3y agoIn case anyone is interested: There are plenty of companies around. Both apple and Facebook acquired eye tracking companies to kickstart their own development. Here are some Top-lists https://imotions.com/blog/insights/trend/top-eye-tracking-hardware-companies/ https://imotions.com/blog/insights/trend/top-eye-tracking-ha... https://valentinazezelj.medium.com/top-10-eye-tracking-companies-on-the-market-today-3b96ef131ab5 https://valentinazezelj.medium.com/top-10-eye-tracking-compa... Its also an active research field, this is one of the bigger conferences: https://etra.acm.org/2023/ https://etra.acm.org/2023/
- TootsMagoon 3y agoRe: There are plenty of companies around. I believe Paul Graham can Google or use AI and already knows about the companies and links you posted. His post was a call to action to connect with people working on yet to be discovered innovations and inspire those and others quietly working to come forward and connect with him.
- kken 3y agoWell I thought people may be interested in learning what is already out there.
- adamnemecek 3y agoI have a new approach of doing ML, where autodiff is replaced with something better. Magically a lot of things fall into place. This approach should make problems like this relatively straight forward. Interested in hearing more?
- p1esk 3y agoInterested in hearing more? If you can show your new approach works, sure. Usually this is done via papers in ML conferences, but if you have reproducible results on Github I'll take a look.
- adamnemecek 3y agoWorking on it.
- deleted 3y ago[deleted]
- sailplease 3y agoAdhawk, adhawk.io, has the only all day ultralight eye tracking wearable I'm aware of, all MEMS based with ultra high scan rates, 500Hz+ and research grade accuracy. For ALS u likely need something light and frictionless, wearing a hot and heavy headset all day probably doesn't work.
- musesum 3y agoI've been working on the menuing side [1] based on crossing Fitt's Law with Huffman trees. But, don't know the constraints for ALS. Hopefully, whomever takes this on doesn't take the standard Accessibility approach, which is adding an extra layer of complexity on an existing UI. A good friend, Gordon Fuller, found out he was going blind. So, he co-founded one of the first VR startups in the 90's. Why? For wayfinding. What we came up with is a concept of Universal design. Start over from first principles. Seeing Gordon use an Accessible UI is painful to watch, it takes three times as many steps to navigate and confirm. So, what is the factor? 0.3 X? Imagine if we could refactor all apps with a LLM, and then couple it with an auto compete menu. Within that menu is personal history of all your past transversals. What would be the result? A 10X? Would my sister in a wheelchair be able to use it? Would love to find out! [1] https://github.com/musesum/DeepMenu https://github.com/musesum/DeepMenu
- quickthrower2 3y agoWhat about brain waves to control the keyboard?
- jjbcb 3y agoI’m sorry this happened to your friend. I hope we can do something to help them.
- ZeroCool2u 3y agoI literally just bought this last night. Works with just a webcam and is shockingly accurate. https://beam.eyeware.tech/ https://beam.eyeware.tech/
- sprocket 3y agoI used this software when my mom was battling ALS: https://www.optikey.org/ which ran on a < $1k computer At the time, the other options were much more expensive (> $10-15k) which were sadly out of out budget.
- jacquesm 3y agoWhatever happened to GazeHawk and their crew?
- atleastoptimal 3y agoI bet Apple Vision pro eye tracking is already SOTA and will remain that way for years due to their crazy R&D spending
- deleted 3y ago[deleted]
- joos3 3y ago[flagged]
- justinlloyd 3y agoI do hardware. I do software. I do computer vision. I built some software that ran on a cellphone used by LEO (law enforcement officers) to determine if the person they are quizzing is inebriated or impaired through controlled substances by examining the person's eyes and having them focus on images displayed on the phone screen. I've done eye tracking using fully custom solutions and also through a few of the off-the-shelf SDKs such as GazeSense from eyeware and a few other SDKs. The problem is not the eye-tracking, it is reasonably easy to build robust systems that can do that easily enough, even with custom hardware under all sorts of lighting conditions. The hard part is the UX if you are trying to build something that isn't hampered by current UI paradigms. Rapid typing and menus of custom actions with just eye movement, though fatiguing, shouldn't be hard to solve, and then render the output however you want; text, text to speech, commands issued to an machine, etc. Making a usable user interface to do anything else, that's where the rubber hits the road. @pg, which software is your friend using? If it is anything like I've looked in to in the past, it's over-priced accessibility crap with a UI straight out of the 1990s.
- modeless 3y agoYes, UX is the key. The iPhone succeeded because they didn't just take macOS's mouse/keyboard UI and slap it under a touchscreen. They took the limitations and strengths of capacitive touch and designed a bespoke UX with new concepts everywhere. Input modalities define platforms. Eye tracking is a new input modality and will define a new platform. It needs a whole new UX designed around its limitations and strengths. It needs a keyboard, it needs a browser, it needs copy and paste, it needs an app switcher, it needs a whole vocabulary of standard interactions and best practices. Apple has a good start in Vision Pro but they're not going to be the only ones doing UX for eye tracking. There's definitely room for other players with fresh ideas.
- skeeter2020 3y agoAnother good example is the early "handwriting recognition" of Palm Pilot. They didn't try to increase the accuracy of recognizing all letters, they tried to hit the sweet spot between what they coudl reliably interpret and what users would learn/put up with.
- quietthrow 3y ago@paulg look in to this: https://spectrum.ieee.org/brain-implant-speech https://spectrum.ieee.org/brain-implant-speech
- KaiserPro 3y agono. Thats super intrusive, and given the the risks of major surgery with ALS, its not a good option.
- modeless 3y agoI agree, eye tracking is going to have really broad applications. I've been interested in eye tracking for over a decade, and in fact built my own eye tracker, joined a startup, and got acquired by Google[1]. But there's way more to do. We've barely scratched the surface of what's possible with eye tracking and I'd love to take a second crack at it. [1] https://techcrunch.com/2016/10/24/google-buys-eyefluence-eye-tracking-startup/ https://techcrunch.com/2016/10/24/google-buys-eyefluence-eye...
- krsrhe 3y agoWhat happened to the tech after acquisition?
- modeless 3y agoThe "Daydream View" VR headset developed before we were acquired totally bombed. Then a bunch of VR tech, including both long running internal projects and recent acquisitions, got shelved.
- mlajtos 3y agoEyefluence had an extremely impressive demo at AWE2016: https://youtu.be/iCZLll1l92g?t=480 https://youtu.be/iCZLll1l92g?t=480 > But there's way more to do. We've barely scratched the surface of what's possible with eye tracking and I'd love to take a second crack at it. What do you have in mind? What would you like to see?
- Componica 3y agoMy three partners and I have be developing and selling multi-camera arrays specifically for eye tracking as well as measuring other physiological features for several years now. Our main customers are a couple university research groups, a human factors group in Lockheed, and just recently the US Air Force. In fact we just returned from a trip to Wright-Patterson installing an array in a hypobaric chamber to perform gaze tracker and pupil response for pilots under hypoxic conditions. Phase two will be a custom gaze tracker for their centrifuge. Our main features are accurate eye and face tracking up to a meter from the array, minimal calibration per subject (about 10 seconds staring at a dot), pupil response for measuring fatigue and other things, plus we can adapt the array for the client ranging from a cockpit to a large flat screen TV. We've looked into medical usage such as ALS, but we're bootstrapped based in Iowa and found the military niche as a more direct way to generate cash flow. It's ashame we can't apply this work towards people with medical needs, but we don't have the funds nor the clients to make such a pivot.
- t0mas88 3y agoHave you thought about settinng up a subsidiary that licenses your base tech for a reasonable royalty fee and raise capital for the subsidiary to develop a medical product from it? The risk and part of the returns there are for the investors. While it will generate additional revenue (and diversification) for your bootstrapped company allowing you to keep building and mitigate some of the risk of having a narrow (military) client base. And if it becomes a major success (sounds like pg thinks that's possible) you'll co-own it.
- Componica 3y agoOne of our partners is a renowned neuro-ophthalmologist at our local university, and he's always suggested medical applications for our array. However, the combination of low volume, slim margins, and regulatory challenges (acquiring patient training sets and navigating FDA approval) makes this direction potentially perilous for us. What helps us is our human factors partner has decades long connections with NASA, FAA, and DoD which gets us a foot in the door, and allows us adopt a low volume/high margin strategy. This approach is letting us refine our technology and then we can focus on standardizing it, reducing costs, and perhaps refocus on medical applications. I do think your suggestion is a good approach, but what's key is finding smart investment that has experience in developing medical devices.
- arketyp 3y agoTobii have been doing eye-tracking since 2001 and have a product for that. https://www.tobiidynavox.com/ https://www.tobiidynavox.com/
- mercurialsolo 3y agoEye tracking is essentially a model of visual attention. Visual attention is part of the overall attention space and big companies and use-cases are built around visual attention. Today we track attention by explicit interactions, if we can model around implicitly observable interactions - then we have a much larger observable data space around the user.,
- mcbutterbunz 3y agoThis comment makes me think we’ve gone too far and I should shut off my phone.
- guerrilla 3y agoIndeed, it's time to re-learn how to live without the Internet and such advanced technology.
- acyou 3y agoYes, the ALS/disability angle is noble. Viewed another way, the entire human race is afflicted by the disability of not having access to eye-tracking (and other) technologies. Paul Graham and co. are also invested in companies that are going to be highly enabled and boosted by the growth of eye-tracking and related technologies. I don't view his statement of motivation related to ALS as insincere, I just also notice that it's accessible, easily understandable, and also in line with other aspects of Paul's motivation (and that's a good thing). I would also recommend Jean-Dominique Bauby's Le Scaphandre et le Papillon to anyone interested in this topic. Typing using eye movements was used in that book in a slow, inefficient manner. In the book's case, the question one should ask is, was his UI paced at the exact correct speed? I was and still am deeply emotionally moved by what the author was able to accomplish and convey. I am unsure if a faster keyboard would have made a meaningful and positive difference in that particular case, to the author's quality of life. I'll need to give that book another read with that question in mind. Happily, I expect eye tracking to find fascinating, novel and unexpected applications. As others have stated, UI/UX design is an interesting part of this puzzle. For example, if you ask an LLM to output short branches of text and have a writer look at the words that he wants to convey. It's definitely blurring the line between reading and writing. Myself, finding writing to be a tactile exercise, I think that emotional state comes into play. That's what I'm interested in. Yes, can you literally read someone's eyes and tell what they are thinking?
- 6stringmerc 3y agoJason Becker is a great subject because if you can help him compose with his eyes the world can use his music he’s a genius.
- tmalsburg2 3y agoSolving eye-tracking keyboards is not so much a task for a company with eye-tracking expertise but for one with expertise in large language models.
- jfrbfbreudh 3y agoPlease elaborate.
- tmalsburg2 3y agoIt is easy to detect what people are looking at. Solved problem. It’s difficult to effectively use such a low—bandwidth signal, and that’s where LLM come in. You need an LLM to anticipate what the user wants to say. An LLM fine-tuned on the user’s needs may have sufficiently low entropy that the small number of bits/s you’re getting from the eye-tracker can reliably identify the intended message.
- deleted 3y ago[deleted]
- rsync 3y agoSo you're saying there's a final frontier in the mapping of intention and the discernment of preferences ... and you'd like some bright young things to explore that possibility space under the guise of (unimpeachable good cause) ? Go to hell. Unless, of course, you'd like to commit the funded work to the free commons, unencumbered by patents and copyrights, and free to use by any entity for any purpose. That's what we'd do for ALS, right ?
- bingemaker 3y agoDoes pg visit HN to give explanations? HN is his baby afterall
- deleted 3y ago[deleted]
- pmichaud 3y agoI don't understand the hostility?
- supremearkitect 3y agoIm something of a conspiracy theorist myself, but this is irrational and unwarranted: if they wanted to research something strategic, they could easily summon the brain power (see OpenAI and the creation of near-general intelligence, and then ask where Facebooks $40B on metaverse smokescreen went to.) PG is a good guy - he sides with entrepreneurs, and created an industry-wide standard for a founder friendly seed stage investment deal (called a SAFE) in a world of hugely predatory deal makers. And from his twitter its pretty clear that he's biased towards fairness and humanity in general. Read some of PG's essays. People dont share deep insights and knowledge like that for free if they are made of the wrong stuff.
- rsync 3y ago"... this is irrational and unwarranted ..." Agreed. My comment violates the HN guideline to "respond to the strongest plausible interpretation of what someone says" and I beg the op's pardon. A low (or no) return investment in basic technology to help the disabled better express their intentions - for their own private uses - is an unalloyed good. Of my two interpretations, I will set aside my initial, very cynical reaction and wait to see what actually comes from this.
- joshm93 3y agoI can make an eye tracking keyboard with tensor flow, if anyone is interested in this problem. It would be great to hear from paul about how his friend uses the keyboard and what kind of tasks he’d love to do but can’t with current solutions. It seems like a throughput problem to me. How can you type quickly using only your eyes? Have people explored using small phonetic alphabets or Morse code style encoding? Once I got tensorflow working, I’d start mapping different kinds of ux. Throughput is king.
- deleted 3y ago[deleted]
- supremearkitect 3y ago[flagged]
- sillysaurusx 3y agoHmm. I was curious and started looking for the comments you mention, and was pleasantly surprised to find zero pg hate comments, downvoted or otherwise. People seem pretty on board with billionaires solving problems for disabled people in exchange for mutual upside. I used to be bothered by those kinds of sentiments too, by the way. The way I got over it was to realize how many people are just bitter, and not because of pg or YC. This is different than having an actual issue with pg or YC — it’s random noise rather than points worth listening to. Weirder than the haters are the people who reply to his tweets. Some of them post bizarre things. I find it fascinating when people project their own feelings on him, whether it’s hate, admiration, or (my favorite, having been a victim of it myself) misplaced ambition.
- peter_retief 3y agoI would like to look at the problem more deeply, the eyes can be tracked but what about facial movement, the more data the better training for machine learning
- fit2rule 3y ago[dead]
- bacon_waffle 3y agoEdit: Check out Dasher for a much better interface to enter text with a cursor, compared to a virtual keyboard. https://dasher.acecentre.net/ https://dasher.acecentre.net/ , source at https://github.com/dasher-project/dasher https://github.com/dasher-project/dasher --- I remember seeing a program years ago, which used the mouse cursor in a really neat way to enter text. Seems like it would be far better than clicking on keys of a virtual keyboard, but I can't remember the name of this program nor seem to find it... Will probably get some of this wrong, but just in case it rings a bell (or someone wants to reinvent it - wouldn't be hard): The interface felt like a side-scrolling through through a map of characters. Moving left and right controlled speed through the characters; for instance moving to the left extent would backspace, and moving further to the right would enter more characters per time. Up and down would select the next character - in my memory these are presented as a stack of map-coloured boxes where each box held a letter (or, group of letters?), say 'a' to 'z' top-to-bottom, plus a few punctuation marks. The height of each box was proportional to the likelihood that letter would be the next you'd want, so the most likely targets would be easier+quicker to navigate to. Navigating in to a box for a character would "type" it. IIRC, at any instant, you could see a couple levels of letters, so if you had entered c-o, maybe 'o' and 'u' would be particularly large, and inside the 'o' box you might see that 'l' and 'k' are bigger so it's easy to write "cool" or "cook". (I do hardware+firmware in Rust and regularly reference Richard Hamming, Fred Brooks, Donald Norman, Tufte. Could be up for a change)
- Avshalom 3y agoDasher https://dasher.acecentre.net/ https://dasher.acecentre.net/
- bacon_waffle 3y agoWow that was quick, thanks! Have imagined a music theory/improvisation practice program that looks something like Dasher + Guitar Hero.
- follower 3y agoOkay, so, your comment about a "Dasher + Guitar Hero music theory/improvisation practice program" just sent me down a huge rabbit hole... Well, rabbit hole(s) plural, I guess, most not directly related. :D Largely because I made the "mistake" of looking at your HN profile & discovering you're also in NZ & we seem to have somewhat overlapping interests (and an affinity for "bacon" in account names, apparently), so, some thoughts[0]... :) # Topic 1: Nissan Leaf VSP hacking After reading your recent posts (https://ianrrees.github.io//2023/07/03/vsp-hacking.html https://ianrrees.github.io//2023/07/03/vsp-hacking.html & https://ianrrees.github.io//2023/08/05/voltage-glitch-injection.html https://ianrrees.github.io//2023/08/05/voltage-glitch-inject...) on this topic & noting your remark about wanting to try reverse engineering a firmware image, I found the following thesis PDF (via a brief google search for `"reverse engineer" "firmware" "Renesas"`): * "AUTOMOTIVE FIRMWARE EXTRACTION AND ANALYSIS TECHNIQUES" by Jan Van den Herrewegen https://etheses.bham.ac.uk/id/eprint/11516/1/VandenHerrewegen2021PhD.pdf https://etheses.bham.ac.uk/id/eprint/11516/1/VandenHerrewege... Not really what I was anticipating finding but seems relevant to your interests--I don't think it was already in your resource list. While the thesis addresses the Renesas 78K0 rather than the Renesas 78K0R, from a brief look at the "Flash Protection" PDF Application Note in your resource list it seems there's a large overlap. Perhaps most significantly the author presents "novel methods" that combine bootloader binary analysis with constraint-based power glitching in an effort to improve on the results described in "Shaping the Glitch". While I haven't read the entire 186 pages :D they theorize that using their approach extracting 8kB firmware might only take ~10 hours. And, most helpfully, they even published their source code under the GPL here: https://github.com/janvdherrewegen/bootl-attacks https://github.com/janvdherrewegen/bootl-attacks So, an interesting adjacent read even if it turns out not to be directly applicable to your situation. Given I have an interest in & a little experience with firmware reversing my original thought was to maybe provide some hopefully helpful references that more generically related to firmware reversing but more specific is good too, I guess. :) In terms of reverse engineering tooling, I've used Rizin/Cutter/radare2 previously: https://rizin.re https://rizin.re https://cutter.re https://cutter.re On the CAN tooling/info front, you might be interested in taking a look at my "Adequate CAN" list which I originally wrote-up for a client a couple years ago: https://gitlab.com/RancidBacon/adequate-can https://gitlab.com/RancidBacon/adequate-can Some other probably outdated reverse engineering tooling links of mine: https://web.archive.org/web/20200119074540/http://www.labradoc.com/i/follower/p/notes-reverse-engineering https://web.archive.org/web/20200119074540/http://www.labrad... In terms of how to approach RE, other than just "getting started & digging in" & learning by doing, I've sometimes found it informative to read other people's firmware reverse engineering write-ups to learn about potentially useful approaches/tools. Anyway, hopefully some of this is helpful! [0] I have a tendency to be a little... "verbose" and/or "thorough" (depending on one's POV :) ) so I'll probably split this over a couple of comments, in case I run out of steam while writing and for topic separation.
- MasterYoda 3y agoPG mention that the solution his friend used wasn't any good. How does the best system there is out today work? And what different solutions are there?
- amelius 3y agoI hear the Apple Vision Pro has a good implementation. If this were Microsoft, you'd be able to find the details in the Microsoft Research website.
- archo 3y agohttps://archive.is/TiCun https://archive.is/TiCun
- maccard 3y agoHuh. I wrote a paper for my undergraduate dissertation on eye tracking using a laptop camera, and it ended up published and I won a scholarship award (for €150, imagine that). I wonder if it's time to dust off that project
- Firmwarrior 3y agoYeah dude, if you can figure out a good eye tracking UX you could be a billionaire
- ricardobayes 3y agoI thought this was solved long time ago, I wrote a program many years ago using kinect that tracks the center of the eye pretty precisely, using color gradients. The pupil is pretty uniform in every human being (it's black) surrounded by some color and then white. Even just a few pixels are enough to do it.
- yyyk 3y agoI suspect the real moneymakers for such startups have very little to do with ALS. ALS demand is fortunately small, and can't lead to VC desired growth curve. Imagine instead using it in a classroom to ensure the kids pay attention. Or making sure you see the advertisement.
- DoingIsLearning 3y ago> Imagine instead using it in a classroom to ensure the kids pay attention. Imagine being a parent and being ok with this? The real "moneymakers" in eye-tracking have always been and will continue to be Defense applications for better or worse.
- yyyk 3y ago>>Imagine instead using it in a classroom to ensure the kids pay attention. >Imagine being a parent and being ok with this? I'm sure there are many tiger-style parents which would be perfectly ok, nay, thrilled!, with this. --- I didn't describe a utopian scenario. More like a dystopian one. The Defense applications do exist, but I expect the advertisers to dominate as they usually do.
- epi3 3y ago[dead]
- lostdog 3y agoI would love to hear pg's analysis of the business case for this company. By my math, 5k people in the US are diagnosed per year, and if your keyboard costs $1k, then your ARR is $5m, and maybe the company valuation is $50m. Numerically, this is pretty far from the goal of a typical YC company. I hate to be so cold-hearted about the calculations, but I've had a few friends get really passionate about assistive tech, and then get crushed by the financial realities. Just from the comments, you can see how many startups went either the military route or got acquired into VR programs. The worst I've seen, btw, is trying to build a better powered wheelchair. All the tech is out there to make powered wheelchairs less bulky and more functional, but the costs of getting it approved for health insurance to pay the price, combined with any possible risk of them falling over, combined with the tiny market you are addressing makes it nearly impossible to develop and ship an improvement. I do hope that we reach a tipping point in the near future where a new wheelchair makes sense to build, because something more nimble would be a big improvement to people's lives.
- sokoloff 3y agoI have absolutely no idea of the state of play of eye-based input devices, but I wouldn't expect an entire eye-input keyboard system to be anywhere near as cheap as $1K, but rather a figure 20-100x that.
- anymouse123456 3y agoThey're currently in the $1,000-$3,000 range.
- eoerl 3y agoWe (The Eye Tribe folks) sold one at 99$ years ago. 1k-3k is mostly lack of competition I believe.
- anymouse123456 3y agoAlso post-covid, it's incredibly difficult to convince a domestic CM to assemble a couple hundred devices per year.
- sam_goody 3y agoSo, guy who has deployed eye-scanning machines all over Africa and has found that many of them have been hacked and are giving incorrect responses suddenly has a friend with ALS and is willing to fund better quality eye tracking? Either: - Part of the whole world-coin thing was privately trying to get the data to help his friend - He doesn't want to say "looking to develop eye tracking tech for my world-coin scam", since most devs won't touch that thing. Conveniently found a "friend" with ALS. Saying, on behalf of a friend, that he doesn't believe PG.
- losvedir 3y agoI dunno, they seem like pretty different domains to me, despite both looking at the eye. Generating an image of the eye that's consistent across different scans in different environments but different from person to person, is one challenging problem. Figuring out where two pupils are in tandem while handling saccades and different focal depths is a different challenging problem. The former is a detailed static view of an eye. The latter is a less detailed picture of the eye itself (probably pretty easy to look at an eye and identify the pupil) but a complex dynamic algorithmic problem.
- sam_goody 3y agoHe currently has to put up with fakers, dupes that are not picked up, etc. Once he can require you to focus on a series of letters with your eyes, it will be harder to fake, easier to pick up dupes. Aside for the fact that getting a "typing cadence" with each individual's eyes is probably valuable in and of itself.
- MasterYoda 3y agoWhat company is it that?
- tikkun 3y agoI believe you may be mixing up Paul Graham and Sam Altman. Easy to do because they both ran YC at different times, but WorldCoin is from Sam Altman and this tweet is from PG.
- dimask 3y agoWhat do you think are some challenges that an eyetracker in this specific context has to face? What is your friend mostly struggling with the current solutions? Are there tracking specific challenges related to ALS? Is it mostly a UI/"better prediction" interface issue? With my group we are developing an eyetracker for studying developmental and clinical populations, which typically present challenges to conventional eyetrackers. It is a spin off from our academic work with infants, and we already have a study almost done that uses it. We are still into the very beginning phase in terms of where this may lead us, but we are interested in looking into contexts where eyetracking for different reasons may be more challenging.
- claytongulick 3y agoI did a fun project a few years ago with eye tracking. We built a prototype for roadside sobriety checks. The idea was to take race/subjectivity out of the equation in these traffic stops. We modified an oculus quest and added IR LEDs and cameras with small PI zero's. I wrote software for the quest that gave instructions and had a series of examinations where you'd follow a 3D ball, the screen would brighten and darken, and several others while I looked for eye jerks (saccades) and pupil dilation. The officer was able to see your pupil (enlarged) on a laptop in real time and we'd mark suspicious times on the video timeline for review. It was an interesting combination of video decoding, OpenCV and real-time streams with a pretty slick UI. The Pi Zero was easily capable of handling real-time video stream decoding, OpenCV and Node. Where I ran into performance problems I wrote node -> c++ bindings. We did it all on something silly like a 50k budget. Neat project.
- dennis_jeeves1 3y agoAnd now if only someone funded an ALS cure... As far I know, I don't don't think mainstream medicine is close to solving _any_ chronic condition, except managing it.
- tim333 3y agoA lot of past chronic diseases have had cures found. Leprosy and tuberculosis for example. ALS is a tricky one though, not well understood and quite likely prion like.
- imranq 3y agoI found this tool to be interesting to play with and seems to work pretty well assuming you stay in the same position: https://gazerecorder.com/ https://gazerecorder.com/ I'm guessing a combination of projection mapping, built in lighting, and some crowdsourced data will get accuracy to very usable levels
- zefzefzef 3y agoI'm very interested in eye-tracking and see a lot of potential in this tech. For inspiration, check out the Vocal Eyes Becker Communication System: https://jasonbecker.com/archive/eye_communication.html https://jasonbecker.com/archive/eye_communication.html A system invented for ALS patient Jason Becker by his dad: https://www.youtube.com/watch?v=wGFDWTC8B8g https://www.youtube.com/watch?v=wGFDWTC8B8g Also already mentioned in here, EyeWriter ( https://en.wikipedia.org/wiki/EyeWriter https://en.wikipedia.org/wiki/EyeWriter ) and Dasher ( https://en.wikipedia.org/wiki/Dasher_(software) https://en.wikipedia.org/wiki/Dasher_(software) ) are two interesting projects to look into.
- kubi07 3y agoThere is a Turkish ALS patient, he has a youtube channel, he is creating youtube videos, podcasts, streams on twitch thanks to eye tracker. He is using tobii eye tracker. There is a video he made about the eye tracker. It's in Turkish but you can see how he uses it. https://www.youtube.com/watch?v=pzSXyiWN_uw https://www.youtube.com/watch?v=pzSXyiWN_uw Here is a article about him in English: https://www.dexerto.com/entertainment/twitch-streamer-with-als-beats-the-odds-by-using-eye-tracker-to-make-content-2073988/ https://www.dexerto.com/entertainment/twitch-streamer-with-a...
- TheGuyWhoCodes 3y agohttps://www.eyecontrol.co.il/ https://www.eyecontrol.co.il/ was founded exactly to solve this problem
- mrwnmonm 3y ago[dead]
- DoingIsLearning 3y agoIs there any solution out there that does not use IR + dark pupil segmentation? Seems like all the solutions out there are some flavour or variation of this.
- Schwolop 3y agoI'm consulting with an Australian group called Control Bionics. They have a US company & office, with CTO and sales team in Ohio, but software engineering is done in AU. Their primary product is an electromyography and accelerometer hardware device to detect muscle activations or movements, and then most commonly used as a mouse-click substitute in conjunction with third-party eye-gaze hardware proving the cursor. (I've also designed them an autonomous wheelchair module, but that's another story...) @pg - If your friend has not tried adding a mouse-click via something they can activate other than eye-gaze, this would be worth a shot. We have a lot of MND patients who use our combination to great success. If they can twitch an eyebrow, wiggle a toe or a finger, or even flex their abdomen, we can put electrodes there and give them a way forward. Also, my contact details are in my profile. I'd be happy to put you in touch with our CEO and I'm confident that offers of funding would be of interest. The company is listed on the Australian stock exchange, but could likely go much further with a direct injection of capital to bolster the engineering team. Cheers, Tom
- caspar 3y agoAs someone who suffered some severe mobility impairment a few years ago and relied extensively on eye tracking for just over a year, https://precisiongazemouse.org/ https://precisiongazemouse.org/ (Windows) and https://talonvoice.com/ https://talonvoice.com/ (multiplatform) are great. In my experience the hardware is already surprisingly good, in that you get accuracy to within an inch or half an inch depending on your training. Rather, it's all about the UX wrapped around it, as a few other comments have raised. IMO Talon wins* for that by supporting voice recognition and mouth noises (think lip popping), which are less fatiguing than one-eye blinks for common actions like clicking. The creator is active here sometimes. (* An alternative is to roll your own sort of thing with https://github.com/dictation-toolbox/dragonfly https://github.com/dictation-toolbox/dragonfly and other tools as I did, but it's a lot more effort)
- anupamchugh 3y agoDoes something like a blink, wink tracker for swiping UI pique your interest? I built a PoC a while back: https://github.com/anupamchugh/BlinkPoseAndSwipeiOSMLKit https://github.com/anupamchugh/BlinkPoseAndSwipeiOSMLKit