15 ms·
Turning a MacBook into a Touchscreen with $1 of Hardware (2018)
- moondev 7y agoWow! This is really creative. Gets my gears turning on all the other applications for using opencv as an interface!
- ethanpil 7y agoCurious what kind of CPU usage the OpenCV code requires on the backend...
- mighty_bander 7y agoThere is some well-informed speculation about latency up-thread, but I've tinkered with OpenCV from time to time. Specifically, I used ROS and OpenCV to do face detection on a netbook in about 2012. That worked out pretty well - it would track multiple faces in realtime. Edge detection is a different algorithm, but isn't super likely to be the origin of the latency issues they're having. It's just a threshold check over a window of a few (e.g. 3 or 5) pixels, multiplied by the dimensions of the image.
- pcr910303 7y agoWow, this is a great idea! Looking at the video, looks like the latency isn't great; what kind of problems should be solved for this to have a ~100ms latency? Rewriting it in C/C++/Rust instead of Python would be one... Any ideas?
- drchewbacca 7y agoIt looks like it's just applying a bunch of filters/convolutions to the image. You could write it in C++ to be pretty quick, in CUDA you could get it down to microseconds I think. Though that does not take account of the latency of the webcam itself.
- fortran77 7y agoToo bad you can't get CUDA easily on a recent Macintosh.
- tantalor 7y agoHuh? https://docs.nvidia.com/cuda/cuda-installation-guide-mac-os-x/index.html https://docs.nvidia.com/cuda/cuda-installation-guide-mac-os-...
- RussianCow 7y agoMacBooks haven't had Nvidia GPUs for a few years.
- coldtea 7y agoThey have also supported eGPUS for years though...
- codetrotter 7y agoYou’ll also need an NVIDIA GPU. All of the MacBooks that came with NVIDIA GPUs are from 2010 and earlier. That’s quite a while ago in computer age. https://support.apple.com/en-us/HT204349#nvidia https://support.apple.com/en-us/HT204349#nvidia
- nacs 7y agoYou could use one via the eGPU support in most newer Macs: https://support.apple.com/en-us/HT208544 https://support.apple.com/en-us/HT208544
- codetrotter 7y agoYeah but then you haven’t turned a MacBook into a Touchscreen with $1 of Hardware. You’ve turned a MacBook into a Touchscreen with $price_of_GPU + $price_of_eGPU_adapter + $1. And additionally you’ve made your MacBook a lot less portable.
- snazz 7y agoWebcams usually have somewhat poor latency, at least in my experience. And I think the performance critical code is all in OpenCV, not Python.
- megous 7y agoWebcams don't have memory, so they can't have latency. It's all in the software. In fact, I've done some apps on Linux that just did readout of the frame from the camera sensor to ram, and displayed the buffer on the next framebuffer flip with zero copies. There was no perceived latency. It just takes some work.
- monocasa 7y agoWeb cams do have memory, and do a remarkable amount of post processing on the frames, hence the latency. Source: have used used webcam cube cams in a robotics application where latency needed to be accounted for.
- throwaway5752 7y agoNot to drag this too far off topic, but what kind of post-processing? And is it standardized across different types of webcams?
- jrowley 7y agoWhile not post-processing, often you can configure the exposure duration and frame rate which will make things feel more or less responsive.
- simias 7y agoNoise reduction is a big one. You usually need to interpolate over several frames to do that. Also depending on the bus being used cameras often compress the image using JPEG to reduce the bandwith (very common with USB cameras, I doubt that the Mac camera does that). Then on a computer you have the latency of whatever pipeline is being used and how much buffering is involved. For instance since typically the display doesn't run exactly synchronized with the camera you'll have some frames that'll end up "stuck" waiting for the monitor's vsync. If the app then does some internal buffering on top of that you can very easily reach a noticeable amount of latency. In general you can consider than anything above 100ms is easily noticeable and that's only 3 frames at 30fps and 6 at 60, it's really not that much when you consider the complexity of modern video capture pipelines.
- Liquix 7y agoYMMV, but I've experienced slight webcam lag on Macbooks even with Apple-optimized applications such as Photo Booth and FaceTime. Perhaps the biggest bottleneck is at the hardware / driver level instead of at the CPU-cycles level?
- JustSomeNobody 7y ago> Rewriting it in C/C++/Rust instead of Python would be one... Would it? It's using libraries for the heavy lifting already.
- collyw 7y agoI had a laptop with a touchscreen and the only time I ever used it was when the linux mouse drivers were playing up. So I guess that would be one use.
- ghostbrainalpha 7y agoThey are asking what kind of problems need to be solved in order to speed up the latency. They are not asking what kind of problems having a touch screen on a laptop would solve.
- sdegutis 7y agoSlightly off topic, but C/C++/Rust are always the go-to languages when talking about high performance. People love the speed of C but it's so bare-bones that they move to C++, or it's too dangerous so they move to Rust. But those are much too complicated, so people came up with Zig and Nim and D instead, but those are very far from C, even though they're not as far as C++ and Rust. So I'm wondering, are there any languages almost as close to C but slightly safer and slightly less bare-bones? Because I like C a lot, I think it's a great language with an excellent simplicity, except for some slight rough edges.
- coldtea 7y agoUnfortunately no. We could have a better/safer C but alas we don't. It's either C or all the way to Zig/Rust/etc territory (or using C++ like C).
- nicoburns 7y agoZig seems very close to C to me. What makes you think it's different?
- blattimwind 7y agoThe gap between C's model and a somewhat memory safe language is unfortunately very large. You can either limit semantics a lot, use GC (mostly-D, Go, ...) or have relatively complex ways to describe life times (Rust).
- eclat 7y agoCrystal? Hasn't reached 1.0 though.
- snazz 7y agoI think Crystal is garbage-collected, right?
- rafaelvasco 7y agoYeah, the whole language is designed around it. Impossible to remove without rewriting it from scratch: https://github.com/crystal-lang/crystal/wiki/FAQ#language-x-has-feature-y-why-dont-you-have-such-feature https://github.com/crystal-lang/crystal/wiki/FAQ#language-x-...
- TYPE_FASTER 7y agoI'd be interested in a version that used the Vision Framework. Back when I looked at the Apple frameworks for iOS, they utilized the Neural Engine hardware available on iPhones while OpenCV didn't (that may have changed). The laptop hardware is different of course, but it would be an interesting performance comparison. Here's an interesting overview of using Core frameworks with Python: https://developer.apple.com/videos/play/wwdc2018/719/ https://developer.apple.com/videos/play/wwdc2018/719/
- duxup 7y agoThat is amazing. I really wouldn't have expected this to work but once you read about it.... of course it would. This is one of those things that while there maybe wasn't a 'problem' shows how knowing a few things critical, you can really solve problems / come up with solutions that nobody would have ever thought of (well I wouldn't have...).
- macspoofing 7y ago>I really wouldn't have expected this to work but once you read about it.... of course it would. It's a fun DIY project and you can get it almost working. I suspect edge cases (stemming from different lighting conditions and imperfect finger detection) would render this method unworkable in general.
- kreetx 7y agoI wonder if this has ever been tentatively tested by the laptop makers, perhaps even by Apple. Surely thought about -- but tested?
- dharma1 7y agoThere are plenty of Windows laptops with a touchscreen. In Apple's case MacOS isn't really designed for touch
- kreetx 7y agoNo-no - I mean using the webcam and a mirror. I.e there would be a little mirror that would fold out of the screen.
- that_jojo 7y agoI think his point was that it would be a bit silly to put this kind of solution into a product -- which is, in a word: jank -- when capacitive touchscreens are already a tried and true solution in production devices.
- guelo 7y agoThat is great! On the topic of touchscreen Macbooks, I've been using my Mac a lot less since I've spent a lot of time on a touchscreen Windows laptop. It is super handy! A lot of times when using the Mac I'll absentmindedly drag something on the screen only to be disappointed. Apple's resistance to the idea reminds me of their stubbornness on the single button mouse. Admitting you're wrong is not a weakness.
- grecy 7y agoWhat hardware do you have? All the touchscreen windows laptops I've tried have the responsiveness and accuracy of a touchscreen from 1990.
- shawnz 7y agoResistive touchscreens which were dominant in 1990 typically have higher accuracy than the capacitive touchscreens of today. But they have terrible usability because you have to apply pressure to the screen to use them.
- socksy 7y agoHave you tried the Dell XPS? Even with shitty linux touchscreen drivers/experience I end up using it almost every day when scrolling. On Windows it felt very natural.
- robohoe 7y agoI think Apple is trying to push folks to iPad Pros since they're touch capable and now support a mouse.
- nomy99 7y agohow does the battery life compare to the touch enabled windows laptops?
- spiderfarmer 7y agoI don't miss it. While working I hate finger grease on my MacBook screen much more than I do on my iPhone or iPad. Also, my wife has a Surface laptop with touch screen and she never uses it.
- ideasRgood 7y agoPrototypes are cool, but I think this is dead. Anything that requires tons of visual calibration is serious work. Both for the programmer and the computer. It worked for a planned demonstration, but I imagine this would be impossible for Mass production. And I wonder if the users found it useful due to latency and accuracy problems.
- snazz 7y agoI don’t think it was intended for mass production, and that’s okay. It’s a really cool and impressive hack.
- duxup 7y agoI think you may be approaching it differently than the creators are. They hacked something cool together, I suspect that is the point, and it is awesome. This isn't a product.
- Liquix 7y agoThe authors didn't publish thinking "this is going to rock the world, it's ready for mass production"... More along the lines "hey fellow hackers, we found a way to emulate a touch screen at 1% of the price".
- ideasRgood 7y agoBut if the latency and accuracy isn't great, did they solve anything?
- jankotek 7y agoLong time ago there was an ad PC vs Mac. PC guy had usb camera tapped on top of his head, claiming how PC are entering new modern age. Mac had camera build in. This kind of feels similar.
- punnerud 7y agoDUPE https://news.ycombinator.com/item?id=16745998 https://news.ycombinator.com/item?id=16745998
- kusha 7y ago>Are reposts ok? >If a story has had significant attention in the last year or so, we kill reposts as duplicates. If not, a small number of reposts is ok. That post is from over a year ago, and didn't get much traction.
- minimaxir 7y agoAlthough it was indeed posted over a year ago, 239 points is certainly traction.
- soegaard 7y agoMissed the old post. Thought reposts were rejected automatically (like on Reddit).
- minimaxir 7y agoFWIW it does, although it's fussy and only works on posts in the last few days. tbh Reddit's repost detector causes more problems than it solves (it still triggers if the original post was removed).
- deleted 7y ago[deleted]
- nsxwolf 7y agoThis could be extended for multitouch too, yeah?
- robotresearcher 7y agoYes, but it would have a possibly annoying limitation of occlusion making the multi-touch fail at some positions. Fingers very close to the camera would appear huge and occlude everything else.
- mkl 7y agoSort of. Actual optical multitouch systems use multiple cameras, so fingers blocked by other fingers can be seen from other angles.
- melling 7y agoMaybe next someone will build a “camera keyboard“ for my iPad. Turn any surface into a keyboard with a camera and machine vision.
- smeyer 7y agoDo you mean similar to a projection keyboard[0] but without actually projecting a keyboard onto the surface and using the integrated camera? [0] https://en.wikipedia.org/wiki/Projection_keyboard https://en.wikipedia.org/wiki/Projection_keyboard
- melling 7y agoSure. Ideally the keyboard wouldn’t have to look exactly like a standard keyboard. It could be split, and perhaps tented. Found this Kickstarter project that’s now a company. http://serafim-tech.com/products/serafim-keybo-world-s-most-advanced-projection-keyboard.html http://serafim-tech.com/products/serafim-keybo-world-s-most-...
- LyndsySimon 7y agoInteresting, thank you for the link! I'm not at all interested in this as a keyboard per se, but the fact that it functions as a piano keyboard as well gives me ideas. I wonder if the latency and resolution would allow for things like automatic transcription of written documents as you write, a mouse replacement, or other cool things that I haven't thought of yet :) They claim they have an SDK. I'll have to explore this some more when I have time later this evening.
- johnday 7y agoOoh, how about this? 1. Point a camera at a surface (be it a piece of paper or whatever). 2. Write/print a word on that paper and use CV/OCR to parse it. 3. Detect when a person has pressed on that word and exec the command specified by the text. Instant macro pad?
- cgriswald 7y ago
- bomanbot 7y agoVery neat! I am wondering whether it could also recognize multiple fingers and thus sort of emulate multitouch gestures, although my guess is latency is not great with one finger already, so this might not work well. Als interested to see how it works with different lighting and how dirty the screen can get before recognition fails because the reflection is too „muddled“ ;)
- newaccoutnas 7y agoThis is great. I notice it's Python 2, could this be ported to Python 3 easily or some inherent issue in upstream deps?
- jpeanuts 7y agoWhile I understand the desire for a unified Python ecosystem as much as anyone - this kind of pettifogging is not going to convince anyone to switch to Py3. Consider if the authors of the article had used (for example) Ruby - you probably wouldn't ask why they didn't do it in Python3, so why do you do it if they write it in Python2?
- deleted 7y ago[deleted]
- 40four 7y agoPut Python 2 on the dunzo list!!! https://www.youtube.com/watch?v=acLp0S3K6mk https://www.youtube.com/watch?v=acLp0S3K6mk
- codetrotter 7y agoI think they asked in a respectful manner, so I don’t see anything wrong with their comment really. Python 2 will be EOLed soon. https://pythonclock.org/ https://pythonclock.org/ 4 months 25 days 5 hours left until Python 2 is retired. After that point it will no longer be maintained by the Python project. Of course Linux distros and companies will keep Python 2 running for years to come because of all of the tools that are Python 2 only. New projects ought to be written in Python 3. However, in this case my guess would be that OP probably used Python 2.7 because macOS ships with Python 2.7 preinstalled. So if we could, Apple are the ones that we ought to speak to. Convince Apple to ship macOS with Python 3 pre-installed.
- rerx 7y agoThe switch from Python 2 to 3 was slowed down tremendously because major libraries did not fully support Python 3 immediately. The question regarding the dependencies was legitimate.
- innerop 7y agohttps://www.playosmo.com/en/ https://www.playosmo.com/en/ <-- this also uses a mirror to refocus the built-in camera but on another surface, not the screen.
- cvaidya1986 7y agoWell done!
- cletus 7y agoSo I was expecting one of those sensors that can turn a TV into a touchscreen with like an IR sensor or something so I checked this wondering how they were so cheap. I was delighted to see this is so much better. Figuring out touch events from the reflection is really cool. Well done.
- Aperocky 7y agoWhat they did not factor in was the cost of the software. You can make junkyard bound stuff do amazing things when you program them right, but you pay for that programming or time spent.
- nullandvoid 7y agoThe software is a "one time cost" in quotes due to maintenance etc but talking in broad strokes which wouldn't impact the claim of $1 touch Or to put it another way you wouldnt invent a new way to desalinate seawater and then say the cost is $10/litre instead of 2 cent because of r&d on the device you made..
- Sharlin 7y agoIn economic terms, the marginal cost is low.
- musingsole 7y ago> invent a new way to desalinate seawater and then say the cost is $10/litre Depends on the marketing strategy, most likely. But I like GP's point about undervaluing programmer time. It is non-free and everyone down to the programmers themselves tend to grossly underestimate its cost. Humans are kind of expensive to keep alive and happy enough to program willingly /That last statement had more caveats than I wanted...
- tripzilch 7y agoIt really depends on their intention, whether they consider that part of the work DIY or not. If I paint my wall, I would generally say it costs the price of the paint, not whatever I'd have to pay to have someone do it for me. Or what about the people who tinker with their own smart home solutions with arduinos and raspberries. If you factor in the price of work/research/coding, it would be very hard to justify not buying an off-the-shelf solution instead.
- TheRealPomax 7y agoThis probably needs [2018] in the title?
- cs02rm0 7y agoI wonder what the field of view is like. The demos only show it being used towards the middle of the lower half of the screen - can the camera really see the top left or right corners?
- dfee 7y agoFrom the conclusion: > With some simple modifications such as a higher resolution webcam (ours was 480p) and a curved mirror that allows the webcam to capture the entire screen, Sistine could become a practical low-cost touchscreen system.
- shireboy 7y agoNow, that is just smart.
- ph33t 7y agoStunning out of the box thinking. Fantastic idea and implementation. Color me impressed.
- stcredzero 7y agoThe basic principle behind Sistine is simple. Surfaces viewed from an angle tend to look shiny, and you can tell if a finger is touching the surface by checking if it’s touching its own reflection. This is especially true of MacBook screens!
- shadofx 7y agoNext step is to gauge the amount of blood underneath the fingernail to tell the amount of pressure being applied.
- rafaelvasco 7y agoHaha, acid.
- mkagenius 7y agoOr track your eye/ eye lens
- macca321 7y agoI had an idea years ago using a webcam and the 'blood under fingernail' to have a trackpadless-trackpad (i.e. same as this project but on your table). Maybe some time-rich person will fork it and do so.
- atoav 7y agoOne issue I imagine is coverage. How do you get decent horizontal resolution on top and on bottom of the screen? Either you have a mirror that converts your webcam into ultra wide angle so you cover the top edge of the screen (which you pay for by poor resolution on the bottom) or you decide not to cover the whole top area (which is where most window control elements are). The video action all happens on the lower half of the screen for a reason. A solution would be some sort of orthographic lense, that basically splits the sensor into multiple distributed lenses to avoid the FOV problems. This however would be not trivial to get right without serious tools.
- regularfry 7y agoUse a pinball, not a flat mirror, and map back to something like a flat plane before processing. The registration is a bit harder, but not by an intractable amount.
- Aperocky 7y agoIt's a good idea and great implementation. However I do strongly feel that touchscreen is the opposite to productivity, maybe usability to the end users, but definitely not what developers need/want.
- s_y_n_t_a_x 7y agoIt helps older users who have trouble with the mouse/touchpad.
- sbradford26 7y agoA touch screen is definitely a more approachable method to interact with computers for people who are not as tech literate. My grandma had a computer for years and only kind of used it. Then she got an ipad from a relative and she uses everyday, from casting netflix to her chromecast to reading ebooks she rented from the library. I may not prefer touch screens but they can be a complete game changer for some users.
- donarb 7y agoThere is nothing wrong with touch screens when they are horizontal like a tablet. Vertical touch screens like on laptops or desktops are not ergonomic, and especially so for the elderly.
- WorldMaker 7y agoEntirely horizontal touch screens have the same problems as trackpads/keyboard/mice (or piano playing) that they can have in terms of wrist strain/RSIs (including carpal tunnel), and sometimes elbow movement strain/RSIs (including things like "tennis elbow"). Vertical touch screens trade the more traditional wrist/elbow strain/RSIs for other arm/elbow/shoulder strain/RSIs (including things like "gorilla arm"). Given the nature of repetitive stress, the growing consensus seems to be not that one orientation is better than the other, but that a range of orientations, and a mixture of movement types / input methods is better. For instance, move your arm some to touch a vertical screen, to mix things up from mouse movements and keyboard movements.
- deleted 7y ago[deleted]
- _hardwaregeek 7y agoThis is exactly what a hackathon project should be. Delightful, innovative and scoped small enough that they produce results without too much bullshit. Not the usual "world changing" hacks that end up abandoned the moment the event concludes. If only events encouraged more hacks like these.
- fucking_tragedy 7y agoYep, this is truly awesome. These types of hacks are what got me into computing as a kid. Reminds me of early Hack-A-Day content before most of their posts became Arduino/Raspberry Pi centric.
- fireattack 7y ago>that end up abandoned the moment the event concludes What makes this one different in this regard? (I'm not trying to say it's a meaningless project.)
- jrh206 7y agoThe difference is that when this is abandoned, it is complete
- rafaelvasco 7y agoBingo.
- deleted 7y ago[deleted]
- sli 7y agoThe last hackathon I attended, the winners just built something from an Instructable that existed well before the hackathon. My friends and I built a time machine radio (i.e. you tune it to a decade, with a knob and everything, and radio news from the decade was included if we could find it) and won best hardware hack, so it wasn't a total loss. Had a blast either way.
- kizer 7y agoThat... is a _hack_! Awesome :D
- chrisweekly 7y agoThis is really f'ing cool. Nice work!!
- daveheq 7y agoYay now their software can see everything on your screen.
- notimetorelax 7y agoAs opposed to a driver that may have greater permissions?
- dkonofalski 7y agoNo it can't. Look at the photos. The glare is part of what makes this method work in the first place.
- ryan93 7y agoDo you think this MIT grad student wants to steal your data?
- bitL 7y agoHmm, given camera is on top, computing homography and rectifying the perspective image would give fairly good precision around the camera, falling off quickly with distance, likely not being very useful at the bottom of the screen. Also, it's likely sensitive to both ambient light and the dominant light on the screen, in dark rooms with dark themes it might be pretty off, as most computer vision algorithms need constant tuning for changing conditions.
- mbreese 7y agoMaybe you could add an IR LED that could be observed by the webcam, but not in the visible spectrum. I’m not sure if the MacBook webcam includes an IR filter or not. It wouldn't help the precision, but might make the camera less susceptible to changes in ambient light.
- tigershark 7y agoEven 5$ webcam include a IR filter, the image quality without is pretty awful, so I’m pretty sure that apple webcams have it.
- kevinsundar 7y agoI recently shined a remote control into my late 2013 macbook pro's camera and I totally saw the ir light. Not saying that you're wrong, I don't know if there are filters but only for certain specific wavelengths or something like that.
- mkl 7y agoRemote controls are pretty bright IR, so it could be just that it only blocks up to a certain brightness.
- kevinsundar 7y agoMakes sense. Direct IR light is probably harder to block.
- aidos 7y agoI love this. When I was young I saw a hack for an Amstrad which added light pen support. If I recall it effectively synced with the scans on the crt which tells you your coordinates. I wanted to build it so much as a kid, but never could. It’s the sort of thing that really fired my imagination and helped me fall in love with computers.
- thomasfl 7y agoI hope this hack get’s so popular that the folks at Apple finally makes a MacBook with a proper touch screen.
- 188201 7y agoThis project makes laptop vendors selling touchscreen laptop with premium price cry.
- lifeformed 7y agoThe hovering aspect of this makes it something quite special, a feature normal touchscreens don't have. I imagine css ::hover elements would feel nice with this.
- redthrowaway 7y agoAnd they called it Sistine, which flabbergastingly shows that coming up with a good name for something is indeed possible.
- rammy1234 7y agowhoa! amazing. out of box thinking
- broken_symlink 7y agoWhat sorts of improvements could you do if you used a stylus instead?
- quickthrower2 7y agoIs this something he should have patented?
- mkl 7y agoThe main idea was probably patented decades ago. There have been optical touch screens around for ages (that use multiple cameras for good multitouch, and edge IR illumination to make detection more reliable and precise).
- tiku 7y agoCould we just add some shiny window tint foil to normal laptop screens to achieve this effect?
- mkl 7y agoThis is a normal laptop screen. What do you mean?
- chickenchaser 7y agoThis is super interesting, I'm about to undergo a small DIY smart mirror project and wanted touchscreen capabilities but all the hardware solutions seemed impractical and expensive. This might be a super easy way to make that work.
- innocentoldguy 7y agoI normally hate touchscreen computers, but this project and its implementation are really cool.
- stackzero 7y agoIncredible. And here I thought a good assortment of lollies at the local corner store was the best $1 could do
- eric_khun 7y agoIs anyone found a stylus that works with the macbook's trackpad? The usecase is to be able to draw some diagrams during video calls/screencasst. Trying to avoid the wacom/tablet path to keep it lightweight :)
- elSidCampeador 7y agoI bought a case in India for my Kindle Paperwhite which came with a stylus [1]. That stylus works, and works quite well with my 2015 MBP. [1] - http://www.proelite.co.in/Amazon-Kindle/All-Amazon-Kindle-10th-Generation-2019-Slim-Smart-Flip-case-Cover--Deep-Blue--with-Stylus-Pen--Not-Fit-Paperwhite--id-3984513.html http://www.proelite.co.in/Amazon-Kindle/All-Amazon-Kindle-10...
- mavidser 7y agoSome accompanying software would definitely be required so that it ignores/reduces the velocity factor, and has a way to differentiate movement with pen-up vs movement with pen-down.
- zoom6628 7y agoPure brilliance. While many commenters seem to focus on computer applications this could be very viable solution for industrial environments for touch controls.
- AzzieElbab 7y agoBrilliant