14 ms·

Facebook confirms that it tracks mouse movements

345 points by shahocean 8y ago

MRonaldhino77 8y ago
thats great step towards security and privacy
sofaofthedamned 8y ago
I assume from this they can infer if you're left or right handed and store that in your profile.
amelius 8y ago
> Facebook said that it tracks mouse movements to help its algorithm distinguish between humans and bots. Stupid cat and mouse game. How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult. Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability? > to also determine if the window is foregrounded or backgrounded Shouldn't there be an API for that?
- lucb1e 8y ago
  As for the former, I'm pretty sure I'd trigger as robot since I use keyboard mostly (using some special features in Firefox) just because they make it a lot faster and nicer. As for the latter, iirc there are blur/focus(-like) events for the window object. Maybe mouse movements gives them better confidence? Because of course you want to make absolutely sure your users are seeing all the ads.
  - zingmars 8y ago
    I wouldn't be surprised if they tracked keyboard input as well.
    - lucb1e 8y ago
      Yeah that's why I think they'd classify me as a robot.
- apexalpha 8y ago
  >Stupid cat and mouse game. How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult. http://idlewords.com/talks/website_obesity.htm http://idlewords.com/talks/website_obesity.htm An interesting read on this cat and mouse game.
- TekMol 8y ago
  How difficult would it be for a bot to simulate a human's mouse movements? Very very hard. Because the bot author does not have the giant database that FB has to analyze how humans move the mouse around. Also the bot author does not know which aspects FB looks at to determine if it's a human. And even if the bot author had all that information, it would still be super hard to write an AI that accomplishes a given task in a way that mimicks a human successfully. It would mean to win a 'mouse turing test'. Shouldn't there be an API for that? What the API returns is under the control of the user. So the API does not help FB to fingerprint you. This issue touches on the real privacy problem the net is facing. It's not the wrong cookies or privacy policies. It's fingerprinting. There is no technical solution to it.
  - amelius 8y ago
    > Very very hard. Because the bot author does not have the giant database that FB has to analyze how humans move the mouse around. Don't forget that Facebook's false positive rate should be very low. There are lots of humans on their platform, and they should all pass the test. This makes it easier to construct a bot that will pass the test.
    - Klathmon 8y ago
      It won't hurt if humans fail the test every so often, as long as it's under a threshold that humans regularly can overcome. I can imagine it would be easy to trick the system a few times (either as a bot pretending to be human, or a human acting like a bot), but tricking it consistently over months or years is going to be damn near impossible.
      - amelius 8y ago
        Also don't forget that Facebook probably has to do all detection in Javascript on the client, i.e. with limited resources. I suspect they don't send every mouse-movement to the server. This also means they probably don't have fine-grained historical data.
        8y ago
- luigi23 8y ago
  Google's already doing it with captcha: https://security.stackexchange.com/questions/78807/how-does-googles-no-captcha-recaptcha-work https://security.stackexchange.com/questions/78807/how-does-... But does Facebook track user during everyday session or just during some validation-action?
- AznHisoka 8y ago
  I actually think its difficult to simulate mouse movements. Is there even a way to do so using PhantomJS or Headless chrome?
  - jhowell 8y ago
    I would imagine that FB is not using these libraries but instead using their engineering teams to develop these "solutions." Perhaps some of our friends are actually "working against" our privacy.
- ajnin 8y ago
  > Shouldn't there be an API for that? There is one : https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibi... Although I think it can be a privacy concern and somewhat of an anti-feature. For instance, Youtube uses that API to stop playback on mobile when the page or browser is not in the foreground. Of course, there's an extension for that ...
- learnstats2 8y ago
  > How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult. I also suppose not very difficult in principle (imitate any nearby human's movements and Facebook should not complain), but it is not within the focus of a general bot developer and therefore makes the whole project exponentially more difficult.
- taurine 8y ago
  > Stupid cat and mouse game. I suspect this is part of covering their ass for GDPR. Pose everything as a security problem, so you can claim you have legit interest in tracking all of that. > How difficult would it be for a bot to simulate a human's mouse movements? Simulating? Extremely difficult. Perturbing pre-recorded paths is a bit easier to do, but requires pre-recording of a lot of paths. One of the fastest ways to get your Poker bot banned is to not fix this one way or the other. > Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability? It still tracks the mouse movements, just now being able to classify their users as disabled or using an arcane device (both of which are interesting tidbits to add to your advertisement profile).
vladharbuz 8y ago
This should be no surprise — received chat messages are marked as read on Facebook if you wiggle your mouse after a few minutes of inactivity. To be fair, this makes sense from a UX standpoint. You don't want messages to be marked as read if you have your window open but have walked away from the computer.
- proto-n 8y ago
  This one is quite a big quality of life improvement IMO. It used to be that I sometimes left my browser accidentally open on facebook, and people would get offended because I 'read' their messages and didn't respond.
  - jhowell 8y ago
    Maybe telling your friends that you didn't actually see their message and that instead the algorithm FB uses to record my mouse activity sometimes reports that I have read messages that I have not. This sounds dangerous and could be harmful to a person's credibility if another with bad intentions acts.
- jhowell 8y ago
  From a UI perspective it makes sense to persist my mouse activity to a database for eternity?
throwawayxxx12 8y ago
Does this allow detecting drug addicts and recommending treatment?
- zerostar07 8y ago
  "We think you may have Parkinson's, would you like to visit the doctor?"
  - Applejinx 8y ago
    "We think you may have Parkinson's, your driver's license has been suspended until you can confirm your fitness to drive. Also you're no longer registered to vote"
sunseb 8y ago
This is beyond absurd. Why would you do that? What could go wrong? Is this a social network or spy agency?
- johnnyfaehell 8y ago
  To be able to detect robots. Makes sense since most users will move their mouse in a massively different way than default robots. There is also probably some product reason like being able to see if people had their mouse around an ad before clicking it, etc.
  - palerdot 8y ago
    If robots can mimic human conversations, this should also be trivial. This not only looks like a lame excuse, but also massively underestimates the ability of automatons that mimic humans.
- SmellyGeekBoy 8y ago
  What!? Plenty of websites do this.
- andrewmackrodt 8y ago
  Lots of websites you would not expect do this; it's not for spying on users but to improve the user experience. Being able to aggregate data or inspect individual sessions is a useful tool to learn how users navigate with a site. Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details. I don't really like this form of monitoring either but I've seen it in several companies.
  - pmlnr 8y ago
    > Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details. Sure. Except they are not.
  - orwin 8y ago
    > Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details. Citation needed. But i did not realize that before, so thank you very much for this information, i will desactivate js on every page with a password field from now on.
cbcoutinho 8y ago
It isn't just Facebook that has been doing this. Web development tools are able to track scroll and mouse movements as well, and have been used to test website usage
- angott 8y ago
  There is a big difference between using this kind of technology to run usability studies or to provide more targeted advertising.
  - kaybe 8y ago
    Not really. If you record and save that data, the reason is secondary.
    - jhowell 8y ago
      Tracking an individuals mouse movement across devices for perpetuity, packaging said data as a product offering to potential advertisers, seems to be a more extensive undertaking than simply UI/UX improvements.
mattbessey 8y ago
as does every person here using Mouseflow, LogRocket, or a dozen other off the shelf solutions. is this really that surprising?
- sti 8y ago
  Seriously my thought.. Pretty much every company does that. Not especially for detecting if its a Human or AI, but for understanding if the User has problems with the Usability of the Product..
  - princekolt 8y ago
    Yes, of course, I'm sure Facebook only uses this data for usability studies. I'm also sure they could pretext all other tracking they do to some other marginally positive use case.
- skummetmaelk 8y ago
  "Everyone does it" != "it's the right thing to do"
zerostar07 8y ago
"Facebook confirms it receives your IP address!!!" We need a new science of hysterias. The internet provides ample data for robust analysis and prediction. Any takers?
- badwolf 8y ago
  Seriously. The hysteria is getting a bit stale that this is what people are now being outraged about? We've been using CrazyEgg, or any other number of different mouse and scroll tracking tools for ages. "But Facebook is using it to signal advertising!" Well. Everything you do on Facebook is used to signal advertising. I thought we were kind of all aware that was what happens with this free service. /shrug
ckastner 8y ago
> We collect information from and about the computers, phones, connected TVs [emphasis mine] Personally, I find this even more disturbing. Does this "only" apply to TVs where a user is logged on, or are they also building shadow profiles for any smart TV that comes with the Facebook App preloaded and is connected to the internet? At least for Samsung and Sony, I can easily see them cooperating with Facebook for a negligible fee.
gkya 8y ago
I have a network meter on my xfce taskbar, and it shows uploads and downloads whenever I move my cursor on a website I view with a JS enabled browser. It's almost all of them.
- zimmund 8y ago
  Correlation doesn't imply causation. (You can always open the browser inspector and check network traffic for each page or, if you are using Chrome, dive into chrome://net-internals/ )
  - gkya 8y ago
    Yes, certainly. But I don't have background programs that connect randomly, and I have observed this enough times that I can say this.
    - seba_dos1 8y ago
      As somebody who worked in the past on a piece of software that generated heatmaps from cursor movements on websites, I can confirm that it's a very widespread thing. Well, it was ~5 years ago, so I'd guess it's even worse now.
      - gkya 8y ago
        How did you use this data? Well, except for bot detection that is. I can't think of a particularly useful way this data can be utilised.
        burger_moon 8y ago
        For a non-nefarious use case, it can be used to iterate on the UI to create a better user experience because it can expose areas that people aren't seeing on the webpage. Your site might have the important content or useful navigation in a place that users aren't noticing which causes them to leave the site in frustration.
akerro 8y ago
According to lcamtuf[1] how people move mouse and use keyboard are unique per person, it's your personal digital-fingerprint that can be transferred to track you elsewhere. [1] From Book "Silence on the Wire"
- dvfjsdhgfv 8y ago
  Unfortunately, it's not just JS, but several other indicators. In practice, if you don't use the Tor Browser, it's as if you decided to leave tracks everywhere, almost identifying yourself. Yet, this is not common knowledge among web users. These are the things that children need to be taught in schools. The society as a whole needs to be aware and learn how to protect itself.
  - yoz-y 8y ago
    That is victim blaming, just because you do not know that everybody out there is tracking you does not mean that you have decided to let them. I agree that this is something people should be more aware of and school is a good place to start. However it is up to browser manufacturers to fix this, not users.
    - code_duck 8y ago
      The comment says "it's as if" you are intentionally leaving traces everywhere. So, the effect is the same as doing it intentionally, not that users are to blame somehow.
    - giancarlostoro 8y ago
      Wonder if the Facebook Containers for Firefox can stop this type of fingerprinting too.
  - pavel_lishin 8y ago
    True, but I'd wager that if Facebook is using mouse movements to track users, it means those other methods aren't foolproof.
  - 8y ago
methodover 8y ago
Um. This isn’t nefarious. We’ve used inspectlet from time to time to help figure out where there are problems in our UI. It tracks mouse movements as part of a complete session. It’s been really helpful.
- p49k 8y ago
  You should still only do this when the user is aware you’re tracking these behaviors and you have their consent. That’s the problem here.
  - reustle 8y ago
    I'm not so worried about this, as long as it's only on their site and when I'm logged in. They're using it to test features and engagement, and can't really learn much about me as a person with it.
- em3rgent0rdr 8y ago
  The average user is unaware their mouse is being tracked. Without this knowledge or without given an informed choice, it could be argued that they can't act with full consent. They might be less willing to use a site which does this if they did know.
StanAngeloff 8y ago
I'm not sure if this is actually how reCAPTCHA v2 works, but I've found moving my mouse and highlighting text immediately after ticking the box almost certainly passes the tests for human. I very rarely get asked to recognise images or pick X out of Y. When I don't do this, i.e., I don't move my mouse or highlight text on a page at random, I most certainly have to sit through a couple of screens of tests (I'm behind a shared IP with lots of users). All this leads me to think mouse movements tracking is much more widespread.
- plopilop 8y ago
  I have a similar experience. Filling in fields and switching with the TAB button then pressing ENTER always brings me to a visual recognition test, while manually clicking on the fields (and adding a bit of sloppiness) is most of the time an immediate validation from reCAPTCHA2. However I'm pretty sure this was advertised or at least acknowledged by Google in the launching of reCAPTCHA v2.
- orwin 8y ago
  Oh, that's probably why i always have to go through those tests, reCAPTCHAv2 is not vimium-friendly.
- danso 8y ago
  This is correct, mouse movement is one factor for Google's one-click Captcha: https://www.wired.com/2014/12/google-one-click-recaptcha/ https://www.wired.com/2014/12/google-one-click-recaptcha/ > IP addresses and cookies provide evidence that the user is the same friendly human Google remembers from elsewhere on the Web. And Shet says even the tiny movements a user’s mouse makes as it hovers and approaches a checkbox can help reveal an automated bot.
- craftyguy 8y ago
  I rarely use the mouse, so this might explain why recrapcha v2 always thinks I am a robot
Rjevski 8y ago
Is there even anything left that Facebook doesn't track?
- ilikehurdles 8y ago
  They claim they don’t record and track what you say. My spouse claims otherwise, saying she’s gotten hyper-specific advertising for medications that she’s discussed with her patients with the phone in the room.
  - doubleunplussed 8y ago
    Facebook didn't claim they don't record what you say. They said very specifically they don't record what you say for advertising. It was specific enough that it almost seems an admission that they use it for something else. However, there are plausible explanations for your spouse getting ads for things she's spoken with her phone in the room - for example, the patient googled the medication later or wrote about it on facebook, and facebook knows thay your spouse and the patient likely had a conversation based on location data showing that they walked down a hallway together.
    - Karunamon 8y ago
      I still call maximum shenanigans on that whole concept. You'd think there'd be one hacker out there with a packet dump or disassembly of the application (yknow.. concrete evidence) rather than lame anecdotes that reek of confirmation bias. Furthermore, I don't believe that Google and Apple (well.. less Google, more Apple) are in cahoots with Facebook to give them a backdoor to device permissions.
      - dirkgently 8y ago
        > (well.. less Google, more Apple) That's funny, because very recently, there was a discovery that fb gave most of the device makers deep access to user data, including Apple. Against all evidences, Apple still gets a pass.
code_duck 8y ago
I have always assumed that Facebook uses heat maps to track what is under your pointer. Doesn't every serious site do that to gauge user behavior, interest and interface utilization? I guess the difference is FB is putting it in an available data set along with everything else about consumers' individual lives. More interesting is "a patent held by the company states that the Facebook app uses voice recognition algorithm, which uses audio recorded by the microphones, to modify the ranking scores of stories in users News Feed." and their speculation that Facebook could soon reveal details about their use of surreptitiously recorded user audio. Facebook makes a curiously specific denial about audio, which is that it is not used for advertising. Considering their entire business is basically advertising, what does that leave? But all they mean is ad selection. When they were found to be recording audio during the posting of statuses, I believe they claimed it it was so they could recognize the music you were listening to, and know something about your mood. So for a long time, I have thought that they use audio to select other content, like friend suggestions, or to inform the selection of stories that appear on your newsfeed.
- kreetx 8y ago
  Perhaps a noob question but doesn't the browser ask for permission to use the microphone? And hence: they can't really listen in if you haven't given it explicitly.
  - vincentmarle 8y ago
    If you attempted to record a video once with the Facebook app, it then has the microphone permissions forever.
    - princekolt 8y ago
      Plus: At least Instagram and Messenger will force you to enable the microphone to let you use the camera even if just to take pictures. Now I always take pictures from the Camera app and simply load it into the app instead of giving them permission to use the microphone.
throwaway122378 8y ago
At this point Facebook can come out and say they track any and everything. They might as well because no one will do a single thing.
benevol 8y ago
Lol @ the Facebook army astro-turfing the comment section. "But it's not nefarious..." "But everybody else is doing it too..." "It's not for surveillance, profiling and shadow profiling, pinky promise, trust us..."
- malmsteen 8y ago
  yeah it's not like your mouse movements is a serious privacy issue
  - benevol 8y ago
    Your ignorance is astounding.
  - dest 8y ago
    I guess mouse movements are a very good way to identify people across websites. It's also an efficient way to distinguish people from machines. Think about the click-only reCaptcha for example.
    - Applejinx 8y ago
      Change mouse brand to surreptitiously become another person without telling a single lie or changing any normal user data :)
      - dest 8y ago
        How often would you be ready to change mouse brand? Would you change mouse each time you change browser tabs, keeping a mouse for Facebook and a mouse for other websites? Also, keep in mind mouse tracking is just another technique deployed to identify you. It's not supposed to be a silver bullet.
      - philipodonnell 8y ago
        It's about the pattern of movement, not the brand of mouse. Identity identification algorithms are more robust than that.
deleted 8y ago
[deleted]
ameister14 8y ago
""We collect information from and about the computers, phones, connected TVs and other web-connected devices users use that integrate with our Products, and we combine this information across different devices users use," Facebook wrote in the document adding that the collected information is used to "give better personalize the content (including ads), to measure whether they took an action in response to an ad we showed them on their phone"." In other news, Google Tag Manager and crazyegg exist.
gambiting 8y ago
I remember just few years ago there was an article here about a website tracking mouse movement to predict what the user will click and pre-loading the content in the background. It was seen as a huge improvement to the experience and the comments section was full of people saying how cool it is. But hey, now facebook is doing it, so it's suddenly the worst thing in the world?
- pavel_lishin 8y ago
  There's a difference between pre-loading content to improve user experience, and tracking mouse movements to gather information about users that's then used to show them advertisements, or potentially track them across the web[0]. There's also the difference between announcing "hey, look at this cool tech we have to make the web faster!" vs "we are legally required to admit that we've been watching you like a hawk, for... reasons." [0] https://news.ycombinator.com/item?id=17301769 https://news.ycombinator.com/item?id=17301769
  - anthonybullard 8y ago
    I can't speak to Facebook's intentions or policys(I don't use big blue app at all), but many sites use tools like Full story to discover bugs or watch a customers user journey to discover flows that aren't working well. Some portions of the page are automatically filtered(inputs with type=password), and the rest depends on the team being very thoughtful about marking sensitive portions of the screen as such. It's a very manual process, but probably one of the most powerful tools for improving user experience I've ever seen. And typically for most businesses, you are keeping these sessions for 2 weeks or thirty days at most.
    - pavel_lishin 8y ago
      I absolutely agree - I used something similar at my previous job. But neither of us know Facebook's policy (although we can easily guess their intentions, based on past behavior), and nobody ought to be expected to cut them any slack.
- deleted 8y ago
titzer 8y ago
I'm not surprised. Honestly, I stopped trusting them years ago, so pretty much all the words out of their mouths I consider to be BS. I'm not even sure why people apologize for creepy-ass multi-billion dollar companies that treat their users like products and study them like labrats. I'm looking forward to technical solutions to verifiably disable and prevent this kind of tracking. To those talking about "tracking UI usage": do a UX study. Sit some people down and watch how they use the site. Ask them questions about what works and what doesn't. Stop spying. You got all this damn money and you can't be bothered to actually lift a finger to do some difficult work that involves interacting with all those "dirty" people out there. FFS most people would probably be happy to fill out a survey if it actually would impact the product in a positive way. Creepiness is creepy.
- brlewis 8y ago
  I am not affiliated with Facebook myself; I barely use it, but given how big it is I bet they do user studies in addition to metrics collection. Metrics alone leave out important parts of the human element, but user studies alone are a biased sample.
  - rohit2412 8y ago
    Given the speed and comfort of not doing user studies, I expect "move fast and break things" Facebook to not care about user studies at all
    - jartelt 8y ago
      I know several UX researchers at facebook. Yes, they do in person user studies.
learnstats2 8y ago
Presumably Facebook can do this because Javascript gives up this information. What's the easiest way to hack your browser to give no information or dummy information here?
g105b 8y ago
BREAKING NEWS: A free service on the web monitors how you use it.
mlthoughts2018 8y ago
Earnest question from someone without much front-end knowledge: how is browser-based mouse tracking performed in a way that doesn't significantly degrade performance? Do they use some type of client-side library that caches data for a while and asynchronously uploads it occasionally? Or occasionally try to asynchronously sample the mouse position and just get a coarser set of data? It seems like real-time requests that respond to mouse changes would create huge performance problems and/or be easily stopped with browser extensions.
- noiv 8y ago
  Capturing, collecting and forwarding mouse movement events can be done at almost no cost. 10 locs max.
lend000 8y ago
Is there a more legitimate source for this announcement? Seems like HN has fallen prey to fake news.
- gvb 8y ago
  https://www.judiciary.senate.gov/imo/media/doc/Zuckerberg%20Responses%20to%20Judiciary%20Committee%20QFRs.pdf https://www.judiciary.senate.gov/imo/media/doc/Zuckerberg%20... Quick search indicates the tracking information starts on page 84. Page 86: "Device operations: information about operations and behaviors performed on the device, such as whether a window is foregrounded or backgrounded, or mouse movements (which can help distinguish humans from bots)."
GarrisonPrime 8y ago
Ironically, the more interested I am in part of a page the less likely my mouse is going to be over or even anywhere near it. I find the pointer distracting. If I'm interested in the content of a page, I either swipe the mouse off to the edge of the screen or put it on an area of whitespace so I can scroll with the scroll wheel. The thought that people are hovering their pointers over stuff they're actively looking at strikes me as odd. Oh well.
codedokode 8y ago
Using mouse movements to detect that tab is inactive is an old trick. Compared to other facts, I think that tracking mouse movements is absolutely not a problem. Collecting things like IMEI or IP addresses needs more attention. For example, I think it is not necessary to keep IP addresses longer than week or two if you respect your users' privacy.
zer0faith 8y ago
Where I work we track: mouse movements, how long you spend on a specific page, the path that you took to get to a specific page, what you clicked, ip addresses, and basically all information included in the header of the request. We also screen shot everything from when you login to when you logout.
erickj 8y ago
document.body.addEventListener('mouseenter', e => { console.log('this just in, erickj tracks mouse movements'); })
cartercole 8y ago
its just for security like he said to congress right?
theweb1 8y ago
Mark is tracking lots of user data
diehunde 8y ago
This is hardly interesting. A lot of websites have been doing this for a while. Just open your network console while you are browsing and you'll see all the things they are tracking
hmate9 8y ago
Every website that uses google analytics does this.
IshKebab 8y ago
This seems like a reasonable thing to do to detect bots.
err4nt 8y ago
Does Facebook just grab everything they can? Does anybody know if they grab your device's battery status, or gyroscopic data about how you are tilting/holding your device?
- taormina 8y ago
  I mean, likely. Given the number of devices they are on, someone has to be using the Battery Historian or similar tools to optimize for power usage.
- cristiantyping 8y ago
  it's stated right in this article that Facebook "also admitted that it collects information about operating systems, hardware, software versions, battery levels, signal strength, available storage space, Bluetooth signals, file names and types, device Ids, browser and browser plugins" ;
goldhand 8y ago
Every company and their mother is tracking mouse movements. How is this trending???
fori1to10 8y ago
The really sad thing is that all this data is not available for academic research. Instead of understanding human nature to make a better world, it is being used to make better targeted ads.

345 points by shahocean 8y ago

MRonaldhino77 8y ago

thats great step towards security and privacy

sofaofthedamned 8y ago

I assume from this they can infer if you're left or right handed and store that in your profile.

amelius 8y ago

> Facebook said that it tracks mouse movements to help its algorithm distinguish between humans and bots. Stupid cat and mouse game. How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult. Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability? > to also determine if the window is foregrounded or backgrounded Shouldn't there be an API for that?

lucb1e 8y ago
As for the former, I'm pretty sure I'd trigger as robot since I use keyboard mostly (using some special features in Firefox) just because they make it a lot faster and nicer. As for the latter, iirc there are blur/focus(-like) events for the window object. Maybe mouse movements gives them better confidence? Because of course you want to make absolutely sure your users are seeing all the ads.
- zingmars 8y ago
  I wouldn't be surprised if they tracked keyboard input as well.
  - lucb1e 8y ago
    Yeah that's why I think they'd classify me as a robot.
apexalpha 8y ago
>Stupid cat and mouse game. How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult. http://idlewords.com/talks/website_obesity.htm http://idlewords.com/talks/website_obesity.htm An interesting read on this cat and mouse game.
TekMol 8y ago
How difficult would it be for a bot to simulate a human's mouse movements? Very very hard. Because the bot author does not have the giant database that FB has to analyze how humans move the mouse around. Also the bot author does not know which aspects FB looks at to determine if it's a human. And even if the bot author had all that information, it would still be super hard to write an AI that accomplishes a given task in a way that mimicks a human successfully. It would mean to win a 'mouse turing test'. Shouldn't there be an API for that? What the API returns is under the control of the user. So the API does not help FB to fingerprint you. This issue touches on the real privacy problem the net is facing. It's not the wrong cookies or privacy policies. It's fingerprinting. There is no technical solution to it.
- amelius 8y ago
  > Very very hard. Because the bot author does not have the giant database that FB has to analyze how humans move the mouse around. Don't forget that Facebook's false positive rate should be very low. There are lots of humans on their platform, and they should all pass the test. This makes it easier to construct a bot that will pass the test.
  - Klathmon 8y ago
    It won't hurt if humans fail the test every so often, as long as it's under a threshold that humans regularly can overcome. I can imagine it would be easy to trick the system a few times (either as a bot pretending to be human, or a human acting like a bot), but tricking it consistently over months or years is going to be damn near impossible.
    - amelius 8y ago
      Also don't forget that Facebook probably has to do all detection in Javascript on the client, i.e. with limited resources. I suspect they don't send every mouse-movement to the server. This also means they probably don't have fine-grained historical data.
      - 8y ago
luigi23 8y ago
Google's already doing it with captcha: https://security.stackexchange.com/questions/78807/how-does-googles-no-captcha-recaptcha-work https://security.stackexchange.com/questions/78807/how-does-... But does Facebook track user during everyday session or just during some validation-action?
AznHisoka 8y ago
I actually think its difficult to simulate mouse movements. Is there even a way to do so using PhantomJS or Headless chrome?
- jhowell 8y ago
  I would imagine that FB is not using these libraries but instead using their engineering teams to develop these "solutions." Perhaps some of our friends are actually "working against" our privacy.
ajnin 8y ago
> Shouldn't there be an API for that? There is one : https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibi... Although I think it can be a privacy concern and somewhat of an anti-feature. For instance, Youtube uses that API to stop playback on mobile when the page or browser is not in the foreground. Of course, there's an extension for that ...
learnstats2 8y ago
> How difficult would it be for a bot to simulate a human's mouse movements? I suppose not very difficult. I also suppose not very difficult in principle (imitate any nearby human's movements and Facebook should not complain), but it is not within the focus of a general bot developer and therefore makes the whole project exponentially more difficult.
taurine 8y ago
> Stupid cat and mouse game. I suspect this is part of covering their ass for GDPR. Pose everything as a security problem, so you can claim you have legit interest in tracking all of that. > How difficult would it be for a bot to simulate a human's mouse movements? Simulating? Extremely difficult. Perturbing pre-recorded paths is a bit easier to do, but requires pre-recording of a lot of paths. One of the fastest ways to get your Poker bot banned is to not fix this one way or the other. > Also, doesn't this conflict with rare types of input devices? Or people with a motor function disability? It still tracks the mouse movements, just now being able to classify their users as disabled or using an arcane device (both of which are interesting tidbits to add to your advertisement profile).

vladharbuz 8y ago

This should be no surprise — received chat messages are marked as read on Facebook if you wiggle your mouse after a few minutes of inactivity. To be fair, this makes sense from a UX standpoint. You don't want messages to be marked as read if you have your window open but have walked away from the computer.

proto-n 8y ago
This one is quite a big quality of life improvement IMO. It used to be that I sometimes left my browser accidentally open on facebook, and people would get offended because I 'read' their messages and didn't respond.
- jhowell 8y ago
  Maybe telling your friends that you didn't actually see their message and that instead the algorithm FB uses to record my mouse activity sometimes reports that I have read messages that I have not. This sounds dangerous and could be harmful to a person's credibility if another with bad intentions acts.
jhowell 8y ago
From a UI perspective it makes sense to persist my mouse activity to a database for eternity?

throwawayxxx12 8y ago

Does this allow detecting drug addicts and recommending treatment?

zerostar07 8y ago
"We think you may have Parkinson's, would you like to visit the doctor?"
- Applejinx 8y ago
  "We think you may have Parkinson's, your driver's license has been suspended until you can confirm your fitness to drive. Also you're no longer registered to vote"

sunseb 8y ago

This is beyond absurd. Why would you do that? What could go wrong? Is this a social network or spy agency?

johnnyfaehell 8y ago
To be able to detect robots. Makes sense since most users will move their mouse in a massively different way than default robots. There is also probably some product reason like being able to see if people had their mouse around an ad before clicking it, etc.
- palerdot 8y ago
  If robots can mimic human conversations, this should also be trivial. This not only looks like a lame excuse, but also massively underestimates the ability of automatons that mimic humans.
SmellyGeekBoy 8y ago
What!? Plenty of websites do this.
andrewmackrodt 8y ago
Lots of websites you would not expect do this; it's not for spying on users but to improve the user experience. Being able to aggregate data or inspect individual sessions is a useful tool to learn how users navigate with a site. Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details. I don't really like this form of monitoring either but I've seen it in several companies.
- pmlnr 8y ago
  > Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details. Sure. Except they are not.
- orwin 8y ago
  > Keyboard keystrokes get captured too but the systems are intelligent enough to filter out passwords and payment details. Citation needed. But i did not realize that before, so thank you very much for this information, i will desactivate js on every page with a password field from now on.

cbcoutinho 8y ago

It isn't just Facebook that has been doing this. Web development tools are able to track scroll and mouse movements as well, and have been used to test website usage

angott 8y ago
There is a big difference between using this kind of technology to run usability studies or to provide more targeted advertising.
- kaybe 8y ago
  Not really. If you record and save that data, the reason is secondary.
  - jhowell 8y ago
    Tracking an individuals mouse movement across devices for perpetuity, packaging said data as a product offering to potential advertisers, seems to be a more extensive undertaking than simply UI/UX improvements.

mattbessey 8y ago

as does every person here using Mouseflow, LogRocket, or a dozen other off the shelf solutions. is this really that surprising?

sti 8y ago
Seriously my thought.. Pretty much every company does that. Not especially for detecting if its a Human or AI, but for understanding if the User has problems with the Usability of the Product..
- princekolt 8y ago
  Yes, of course, I'm sure Facebook only uses this data for usability studies. I'm also sure they could pretext all other tracking they do to some other marginally positive use case.
skummetmaelk 8y ago
"Everyone does it" != "it's the right thing to do"

zerostar07 8y ago

"Facebook confirms it receives your IP address!!!" We need a new science of hysterias. The internet provides ample data for robust analysis and prediction. Any takers?

badwolf 8y ago
Seriously. The hysteria is getting a bit stale that this is what people are now being outraged about? We've been using CrazyEgg, or any other number of different mouse and scroll tracking tools for ages. "But Facebook is using it to signal advertising!" Well. Everything you do on Facebook is used to signal advertising. I thought we were kind of all aware that was what happens with this free service. /shrug

ckastner 8y ago

> We collect information from and about the computers, phones, connected TVs [emphasis mine] Personally, I find this even more disturbing. Does this "only" apply to TVs where a user is logged on, or are they also building shadow profiles for any smart TV that comes with the Facebook App preloaded and is connected to the internet? At least for Samsung and Sony, I can easily see them cooperating with Facebook for a negligible fee.

gkya 8y ago

I have a network meter on my xfce taskbar, and it shows uploads and downloads whenever I move my cursor on a website I view with a JS enabled browser. It's almost all of them.

zimmund 8y ago
Correlation doesn't imply causation. (You can always open the browser inspector and check network traffic for each page or, if you are using Chrome, dive into chrome://net-internals/ )
- gkya 8y ago
  Yes, certainly. But I don't have background programs that connect randomly, and I have observed this enough times that I can say this.
  - seba_dos1 8y ago
    As somebody who worked in the past on a piece of software that generated heatmaps from cursor movements on websites, I can confirm that it's a very widespread thing. Well, it was ~5 years ago, so I'd guess it's even worse now.
    - gkya 8y ago
      How did you use this data? Well, except for bot detection that is. I can't think of a particularly useful way this data can be utilised.
      - burger_moon 8y ago
        For a non-nefarious use case, it can be used to iterate on the UI to create a better user experience because it can expose areas that people aren't seeing on the webpage. Your site might have the important content or useful navigation in a place that users aren't noticing which causes them to leave the site in frustration.

akerro 8y ago

According to lcamtuf[1] how people move mouse and use keyboard are unique per person, it's your personal digital-fingerprint that can be transferred to track you elsewhere. [1] From Book "Silence on the Wire"

dvfjsdhgfv 8y ago
Unfortunately, it's not just JS, but several other indicators. In practice, if you don't use the Tor Browser, it's as if you decided to leave tracks everywhere, almost identifying yourself. Yet, this is not common knowledge among web users. These are the things that children need to be taught in schools. The society as a whole needs to be aware and learn how to protect itself.
- yoz-y 8y ago
  That is victim blaming, just because you do not know that everybody out there is tracking you does not mean that you have decided to let them. I agree that this is something people should be more aware of and school is a good place to start. However it is up to browser manufacturers to fix this, not users.
  - code_duck 8y ago
    The comment says "it's as if" you are intentionally leaving traces everywhere. So, the effect is the same as doing it intentionally, not that users are to blame somehow.
  - giancarlostoro 8y ago
    Wonder if the Facebook Containers for Firefox can stop this type of fingerprinting too.
- pavel_lishin 8y ago
  True, but I'd wager that if Facebook is using mouse movements to track users, it means those other methods aren't foolproof.
- 8y ago

methodover 8y ago

Um. This isn’t nefarious. We’ve used inspectlet from time to time to help figure out where there are problems in our UI. It tracks mouse movements as part of a complete session. It’s been really helpful.

p49k 8y ago
You should still only do this when the user is aware you’re tracking these behaviors and you have their consent. That’s the problem here.
- reustle 8y ago
  I'm not so worried about this, as long as it's only on their site and when I'm logged in. They're using it to test features and engagement, and can't really learn much about me as a person with it.
em3rgent0rdr 8y ago
The average user is unaware their mouse is being tracked. Without this knowledge or without given an informed choice, it could be argued that they can't act with full consent. They might be less willing to use a site which does this if they did know.

StanAngeloff 8y ago

I'm not sure if this is actually how reCAPTCHA v2 works, but I've found moving my mouse and highlighting text immediately after ticking the box almost certainly passes the tests for human. I very rarely get asked to recognise images or pick X out of Y. When I don't do this, i.e., I don't move my mouse or highlight text on a page at random, I most certainly have to sit through a couple of screens of tests (I'm behind a shared IP with lots of users). All this leads me to think mouse movements tracking is much more widespread.

plopilop 8y ago
I have a similar experience. Filling in fields and switching with the TAB button then pressing ENTER always brings me to a visual recognition test, while manually clicking on the fields (and adding a bit of sloppiness) is most of the time an immediate validation from reCAPTCHA2. However I'm pretty sure this was advertised or at least acknowledged by Google in the launching of reCAPTCHA v2.
orwin 8y ago
Oh, that's probably why i always have to go through those tests, reCAPTCHAv2 is not vimium-friendly.
danso 8y ago
This is correct, mouse movement is one factor for Google's one-click Captcha: https://www.wired.com/2014/12/google-one-click-recaptcha/ https://www.wired.com/2014/12/google-one-click-recaptcha/ > IP addresses and cookies provide evidence that the user is the same friendly human Google remembers from elsewhere on the Web. And Shet says even the tiny movements a user’s mouse makes as it hovers and approaches a checkbox can help reveal an automated bot.
craftyguy 8y ago
I rarely use the mouse, so this might explain why recrapcha v2 always thinks I am a robot

Rjevski 8y ago

Is there even anything left that Facebook doesn't track?

ilikehurdles 8y ago
They claim they don’t record and track what you say. My spouse claims otherwise, saying she’s gotten hyper-specific advertising for medications that she’s discussed with her patients with the phone in the room.
- doubleunplussed 8y ago
  Facebook didn't claim they don't record what you say. They said very specifically they don't record what you say for advertising. It was specific enough that it almost seems an admission that they use it for something else. However, there are plausible explanations for your spouse getting ads for things she's spoken with her phone in the room - for example, the patient googled the medication later or wrote about it on facebook, and facebook knows thay your spouse and the patient likely had a conversation based on location data showing that they walked down a hallway together.
  - Karunamon 8y ago
    I still call maximum shenanigans on that whole concept. You'd think there'd be one hacker out there with a packet dump or disassembly of the application (yknow.. concrete evidence) rather than lame anecdotes that reek of confirmation bias. Furthermore, I don't believe that Google and Apple (well.. less Google, more Apple) are in cahoots with Facebook to give them a backdoor to device permissions.
    - dirkgently 8y ago
      > (well.. less Google, more Apple) That's funny, because very recently, there was a discovery that fb gave most of the device makers deep access to user data, including Apple. Against all evidences, Apple still gets a pass.

code_duck 8y ago

I have always assumed that Facebook uses heat maps to track what is under your pointer. Doesn't every serious site do that to gauge user behavior, interest and interface utilization? I guess the difference is FB is putting it in an available data set along with everything else about consumers' individual lives. More interesting is "a patent held by the company states that the Facebook app uses voice recognition algorithm, which uses audio recorded by the microphones, to modify the ranking scores of stories in users News Feed." and their speculation that Facebook could soon reveal details about their use of surreptitiously recorded user audio. Facebook makes a curiously specific denial about audio, which is that it is not used for advertising. Considering their entire business is basically advertising, what does that leave? But all they mean is ad selection. When they were found to be recording audio during the posting of statuses, I believe they claimed it it was so they could recognize the music you were listening to, and know something about your mood. So for a long time, I have thought that they use audio to select other content, like friend suggestions, or to inform the selection of stories that appear on your newsfeed.

kreetx 8y ago
Perhaps a noob question but doesn't the browser ask for permission to use the microphone? And hence: they can't really listen in if you haven't given it explicitly.
- vincentmarle 8y ago
  If you attempted to record a video once with the Facebook app, it then has the microphone permissions forever.
  - princekolt 8y ago
    Plus: At least Instagram and Messenger will force you to enable the microphone to let you use the camera even if just to take pictures. Now I always take pictures from the Camera app and simply load it into the app instead of giving them permission to use the microphone.

throwaway122378 8y ago

At this point Facebook can come out and say they track any and everything. They might as well because no one will do a single thing.

benevol 8y ago

Lol @ the Facebook army astro-turfing the comment section. "But it's not nefarious..." "But everybody else is doing it too..." "It's not for surveillance, profiling and shadow profiling, pinky promise, trust us..."

malmsteen 8y ago
yeah it's not like your mouse movements is a serious privacy issue
- benevol 8y ago
  Your ignorance is astounding.
- dest 8y ago
  I guess mouse movements are a very good way to identify people across websites. It's also an efficient way to distinguish people from machines. Think about the click-only reCaptcha for example.
  - Applejinx 8y ago
    Change mouse brand to surreptitiously become another person without telling a single lie or changing any normal user data :)
    - dest 8y ago
      How often would you be ready to change mouse brand? Would you change mouse each time you change browser tabs, keeping a mouse for Facebook and a mouse for other websites? Also, keep in mind mouse tracking is just another technique deployed to identify you. It's not supposed to be a silver bullet.
    - philipodonnell 8y ago
      It's about the pattern of movement, not the brand of mouse. Identity identification algorithms are more robust than that.

deleted 8y ago

[deleted]

ameister14 8y ago

""We collect information from and about the computers, phones, connected TVs and other web-connected devices users use that integrate with our Products, and we combine this information across different devices users use," Facebook wrote in the document adding that the collected information is used to "give better personalize the content (including ads), to measure whether they took an action in response to an ad we showed them on their phone"." In other news, Google Tag Manager and crazyegg exist.

gambiting 8y ago

I remember just few years ago there was an article here about a website tracking mouse movement to predict what the user will click and pre-loading the content in the background. It was seen as a huge improvement to the experience and the comments section was full of people saying how cool it is. But hey, now facebook is doing it, so it's suddenly the worst thing in the world?

pavel_lishin 8y ago
There's a difference between pre-loading content to improve user experience, and tracking mouse movements to gather information about users that's then used to show them advertisements, or potentially track them across the web[0]. There's also the difference between announcing "hey, look at this cool tech we have to make the web faster!" vs "we are legally required to admit that we've been watching you like a hawk, for... reasons." [0] https://news.ycombinator.com/item?id=17301769 https://news.ycombinator.com/item?id=17301769
- anthonybullard 8y ago
  I can't speak to Facebook's intentions or policys(I don't use big blue app at all), but many sites use tools like Full story to discover bugs or watch a customers user journey to discover flows that aren't working well. Some portions of the page are automatically filtered(inputs with type=password), and the rest depends on the team being very thoughtful about marking sensitive portions of the screen as such. It's a very manual process, but probably one of the most powerful tools for improving user experience I've ever seen. And typically for most businesses, you are keeping these sessions for 2 weeks or thirty days at most.
  - pavel_lishin 8y ago
    I absolutely agree - I used something similar at my previous job. But neither of us know Facebook's policy (although we can easily guess their intentions, based on past behavior), and nobody ought to be expected to cut them any slack.
deleted 8y ago

titzer 8y ago

I'm not surprised. Honestly, I stopped trusting them years ago, so pretty much all the words out of their mouths I consider to be BS. I'm not even sure why people apologize for creepy-ass multi-billion dollar companies that treat their users like products and study them like labrats. I'm looking forward to technical solutions to verifiably disable and prevent this kind of tracking. To those talking about "tracking UI usage": do a UX study. Sit some people down and watch how they use the site. Ask them questions about what works and what doesn't. Stop spying. You got all this damn money and you can't be bothered to actually lift a finger to do some difficult work that involves interacting with all those "dirty" people out there. FFS most people would probably be happy to fill out a survey if it actually would impact the product in a positive way. Creepiness is creepy.

brlewis 8y ago
I am not affiliated with Facebook myself; I barely use it, but given how big it is I bet they do user studies in addition to metrics collection. Metrics alone leave out important parts of the human element, but user studies alone are a biased sample.
- rohit2412 8y ago
  Given the speed and comfort of not doing user studies, I expect "move fast and break things" Facebook to not care about user studies at all
  - jartelt 8y ago
    I know several UX researchers at facebook. Yes, they do in person user studies.

learnstats2 8y ago

Presumably Facebook can do this because Javascript gives up this information. What's the easiest way to hack your browser to give no information or dummy information here?

g105b 8y ago

BREAKING NEWS: A free service on the web monitors how you use it.

mlthoughts2018 8y ago

Earnest question from someone without much front-end knowledge: how is browser-based mouse tracking performed in a way that doesn't significantly degrade performance? Do they use some type of client-side library that caches data for a while and asynchronously uploads it occasionally? Or occasionally try to asynchronously sample the mouse position and just get a coarser set of data? It seems like real-time requests that respond to mouse changes would create huge performance problems and/or be easily stopped with browser extensions.

noiv 8y ago
Capturing, collecting and forwarding mouse movement events can be done at almost no cost. 10 locs max.

lend000 8y ago

Is there a more legitimate source for this announcement? Seems like HN has fallen prey to fake news.

gvb 8y ago
https://www.judiciary.senate.gov/imo/media/doc/Zuckerberg%20Responses%20to%20Judiciary%20Committee%20QFRs.pdf https://www.judiciary.senate.gov/imo/media/doc/Zuckerberg%20... Quick search indicates the tracking information starts on page 84. Page 86: "Device operations: information about operations and behaviors performed on the device, such as whether a window is foregrounded or backgrounded, or mouse movements (which can help distinguish humans from bots)."

GarrisonPrime 8y ago

Ironically, the more interested I am in part of a page the less likely my mouse is going to be over or even anywhere near it. I find the pointer distracting. If I'm interested in the content of a page, I either swipe the mouse off to the edge of the screen or put it on an area of whitespace so I can scroll with the scroll wheel. The thought that people are hovering their pointers over stuff they're actively looking at strikes me as odd. Oh well.

codedokode 8y ago

Using mouse movements to detect that tab is inactive is an old trick. Compared to other facts, I think that tracking mouse movements is absolutely not a problem. Collecting things like IMEI or IP addresses needs more attention. For example, I think it is not necessary to keep IP addresses longer than week or two if you respect your users' privacy.

zer0faith 8y ago

Where I work we track: mouse movements, how long you spend on a specific page, the path that you took to get to a specific page, what you clicked, ip addresses, and basically all information included in the header of the request. We also screen shot everything from when you login to when you logout.

erickj 8y ago

document.body.addEventListener('mouseenter', e => { console.log('this just in, erickj tracks mouse movements'); })

cartercole 8y ago

its just for security like he said to congress right?

theweb1 8y ago

Mark is tracking lots of user data

diehunde 8y ago

This is hardly interesting. A lot of websites have been doing this for a while. Just open your network console while you are browsing and you'll see all the things they are tracking

hmate9 8y ago

Every website that uses google analytics does this.

IshKebab 8y ago

This seems like a reasonable thing to do to detect bots.

err4nt 8y ago

Does Facebook just grab everything they can? Does anybody know if they grab your device's battery status, or gyroscopic data about how you are tilting/holding your device?

taormina 8y ago
I mean, likely. Given the number of devices they are on, someone has to be using the Battery Historian or similar tools to optimize for power usage.
cristiantyping 8y ago
it's stated right in this article that Facebook "also admitted that it collects information about operating systems, hardware, software versions, battery levels, signal strength, available storage space, Bluetooth signals, file names and types, device Ids, browser and browser plugins" ;

goldhand 8y ago

Every company and their mother is tracking mouse movements. How is this trending???

fori1to10 8y ago

The really sad thing is that all this data is not available for academic research. Instead of understanding human nature to make a better world, it is being used to make better targeted ads.