11 ms·
Googlebot’s JavaScript random() function is deterministic
- amelius 9y agoNice to see fingerprinting used against Google (instead of by Google). But this is easy to fix, so I don't expect this to work for much longer.
- xstartup 9y agoI don't think it's particularly easy to fix without fixing a lot of other things. Let's see how much time it takes them. :)
- sergz 9y agoPerhaps for performance or testing reasons they compiled V8 with a predefined seed, keywords are "V8", "mksnapshot", "--random-seed". It should not be too difficult to fix it if there are no artificial restrictions by use cases.
- codedokode 9y agoIt is much easier to detect Googlebot by IP address or User-Agent though.
- p49k 9y agoI’d assume that they occasionally send requests with different IPs and user-agents to detect cloaking.
- chrisper 9y agoYes, but their IPs are registered to Google Inc.
- codedokode 9y agoI think that cloaking is very difficult to detect automatically without using humans. You have to distinguish for example between a paywall and real cloaking.
- leni536 9y agoMaybe they change the behavior of their random() function as well. The deterministic random() function can very well be a red herring.
- lucideer 9y agoI'd guess if they're crawling purely for the purpose of detecting cloaking (which would be a much smaller-scale job than the standard Googlebot indexing), they'd just use Chrome Headless[0][1] [0] https://intoli.com/blog/not-possible-to-block-chrome-headless/ https://intoli.com/blog/not-possible-to-block-chrome-headles... [1] http://antoinevastel.github.io/bot%20detection/2018/01/17/detect-chrome-headless-v2.html http://antoinevastel.github.io/bot%20detection/2018/01/17/de...
- thecatspaw 9y agoI doubt that they use chrome at all. They probably just fetch the page over http and let it run in a JS sandbox. hence the deterministic random function (which I assume is not there in chrome?)
- DamonHD 9y ago2018/01/18: Chrome 41: The Key to Successful Website Rendering: "Google updated their Search Guides and announced that they use Chrome 41 for rendering." https://www.elephate.com/blog/chrome-41-key-to-website-rendering/ https://www.elephate.com/blog/chrome-41-key-to-website-rende...
- codedokode 9y agoIt is interesting that when I use Chromium 46 (which is newer than Googlebot) I get warnings from Github and Google Docs about an outdated or unsupported browser, and I get plain HTML Github pages without JS. But Google uses even older browser than I. So even Google cannot cope with browser version race and keep their browsers updated although they require this from the users.
- jrochkind1 9y agoI think how easy it is for Google to fix to make detecting googlebot this way harder depends on why Googlebot is doing it in the first place, which we don't really know. If it's done for performance reasons, or for predictability reasons (rendering the same page twice guaranteed or at least more likely to produce the same result), it might be difficult to change without cost. But I believe Googlebot always faithfully sends it's user-agent. Is there a reason Google would care about 'fixing' this to make Googlebot harder to detect via random() predictability, when you can always just detect it via user-agent anyway? I'm not sure, curious if others have thoughts!
- Kesty 9y agoGoogle has checks in place to see if someone is serving things to GoogleBot differently that the rest of the users. So it almost definitely has bots that double checks pages without the user-agent. If the "disguised" googlebot is the same as the actual one, chances are it is since it would want to be as close as possible to not flag false positives, and use the same seed for consistency then you might be able to use that to avoid detection on the fact that you are serving google something different than normal users. Newspaper used/do that to be able to have their full article content indexed while serving a paywall to everyone else.
- kmbriedis 9y agoEvery random() function is kind of deterministic, but still very interesting discovery!
- TomAnthony 9y agoYeah, as was pointed out in /r/programming, it is perhaps an imperfect title in hindsight!
- KyeRussell 9y agoTechnically true, but pointless pedantry when I'm sure that most people are on the the same page wrt pseudorandomness. Typically what I'd expect of /r/programming.
- skrebbel 9y agoTotally down nitpick alley yeah, but: Not if fed by an external source of randomness (eg cosmic rays).
- philbarr 9y ago> (eg cosmic rays). YouTube copyright violations? The chance of getting a response from Google support?
- skrebbel 9y agoI love those. A bit easy to game if you have the right job at Google, but still this is cool. I'd also assume that the Twitter firehose could be a great source of randomness.
- Godel_unicode 9y agoSince we're nitpicking; deterministic effectively means same input produces same output. Input includes the seed. All PRNGs are deterministic.
- 9y ago
- est 9y agoThis makes me wonder if Chrome Headless has enough entropy for random() if running on a Linux server.
- meirelles 9y agoEntropy is not a problem, surely! The choice to return deterministic values for random() is a feature, not a bug. When you are crawling pages, looking for meaningful content changes and new links you probably don't want your random() creating noise.
- zuzun 9y agoMore than enough. Chromium takes only 128 bits from /dev/urandom to seed a V8 isolate. You can also use a fixed seed like Googlebot by passing the command line flag --js-flags="--random_seed=12345" to Chromium.
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- h000per 9y agoThe google queries for "roll a dice" and "flip a coin" aren't actually random either. They seem to be based off the current time.
- anc84 9y agoCould you add a lot more detail than just a blank statement like this?
- nukeop 9y agoWhat's a plausible real world use case for this if one wanted to exploit this to game SEO? Can it even be exploited in any way?
- TomAnthony 9y agoOP here. I made a silly function that identifies Googlebot, but has 'plausible deniability' built in: http://www.tomanthony.co.uk/fun/googlebot_puzzle.html http://www.tomanthony.co.uk/fun/googlebot_puzzle.html The idea being you could use a function to always send Googlebot down one execution path and users down another, and make it look like you are doing an AB test on a small set of traffic. You could then do something nefarious, such as add spammy content to the page for Googlebot. However, in reality it is not likely to be a viable tactic for any decent quality site, and unlikely to have much of an impact on lower quality sites that may be willing to risk it. It is an interesting idea though, and we research this sort of thing such that we can better identify such behaviours in competitors of our clients.
- codedokode 9y agoYou can detect Googlebot by IP address or User-Agent which is much easier.
- lucb1e 9y agoWell, presumably some people want to pose as Google to view content which is reserved to logged in members or bots, in which case the user agent is not advisable.. but indeed, IP address.
- TomAnthony 9y agoYes you can - but it lacks the 'plausible deniability'. Especially if you have it in your frontend code. :)
- realPubkey 9y agoAfter doing seo for 10 years, I assure you that plausible deniability will not give you an advantage. Google will punish your site and not listen to your arguments.
- onion2k 9y agoI guess they used the XKCD method of random number generation: https://xkcd.com/221/ https://xkcd.com/221/ It is actually truly random if you have a fair dice.
- PeterisP 9y ago... assuming that this function is called at most once.
- na85 9y agoIn an infinite string of random integers, 4 appears consecutively an arbitrary number of times. There's no way to prove, of which I am aware, that a string of 24 fours is not random.
- PeterisP 9y agoRandomness describes a process, not its results. You can't prove that a string of 24 fours is not random, and the fact that some process returned 24 fours once is not proof (but is some evidence) that it's not random - however, a process that always returns 4 is not random. A process that returns a single fixed number (which was chosen by a fair dice roll) once is a random process. Using that same process twice or more is not.
- jameshart 9y agoBut given a pseudorandom number generator with a known amount of internal state (eg a 32 bit seed) there are hard upper limits on the lengths of sequences like that that can plausibly be produced. Of course, to get 4 out of a PRNG you probably have to ask it for a random number in a range - if you’re requesting numbers in the range 0-1000 then you would expect fewer long sequences of fours than if you request in the range 0-5. And you can get arbitrarily many 4s by requesting numbers in the range 4-4...
- thanatropism 9y agoTangential but related: Brownian motion (i.e. continuous-time random walk) is recurrent in one and two dimensions -- it hits zero infinite times, but in three dimensions it is not. The way my professor explained: a drunkard can always find his way back to his building, but not to his apartment.
- lolc 9y agoMaybe Googlebot forks its JS-engine from a pre-initialized image. That would explain the unchanging seed.
- simias 9y agoRegardless of other technical limitations I'm guessing that it's actually done on purpose, as point #3 in TFA states: >Predictable – Googlebot can trust a page will render the same on each visit It's probably important for Google's crawler to identify whether a page changed or not, if some elements in a page are randomly generated they may want to limit the impact. I mean, after all they seem to use a real, changing value for their Date, so if they wanted they could just seed their RNG with that.
- geocar 9y agoI wrote about something like this previously[1] Ads often have something like this attached to the end: load("https://adserver/track.gif?{stuff}&_=" + Math.random()) My ad server collected multiple tracking pixels -- for various events and after five pixels I could fingerprint the browser (Firefox, Chrome, MSIE, etc) and identify someone fiddling with the user agent string, or using a proxy server to mask this information. [1]: http://geocar.sdf1.org/browser-verification.html http://geocar.sdf1.org/browser-verification.html
- na85 9y agoHow would a person defend against this fingerprinting?
- josteink 9y agoDisable javascript and use the web as a hypertext document-store.
- osteele 9y agoUse distinct browsers (or profiles) for browsing web-of-documents, and using apps hosted on the web-as-application-distribution-platform. They're each (for many of us) legitimate uses, but have different requirements and threat models. Cons: interactive infographics and courseware don't fit neatly in either.
- megous 9y agoUse curl. Disable javascript. Replace Math.random with your own function via extension, etc.
- thanatropism 9y agoLawyer up, delete Facebook, hit the gym?
- geocar 9y agoBuy an iPhone. • Did proper random before anybody else • Active countermeasures against cookie-based retargeting • Popular enough market that merely having an iPhone in a geographic area doesn't single you out There's a guy who downloads every page with curl. I see him on web logs. I think he must have some script that parses the amp pages out and does something with it, but because he's the only person in that geographic region who browses the web with curl, he's very easy to spot from a tracking perspective. On the other hand, because he's using curl, I don't think anyone wants to bother trying to show him an ad.
- endymi0n 9y agoThis is known for a while and used to cause huge problems for our tracking: https://github.com/snowplow/snowplow-javascript-tracker/issues/499 https://github.com/snowplow/snowplow-javascript-tracker/issu...
- lotyrin 9y agoUnder the assumption that the same event fired from the same IP at the same time, with the same environment, etc. would be considered a duplicate in your system, I'd have designed this system to be idempotent-insert-only and use content-addressing instead of nonces for identity (event ID = suitably large hash of event data to avoid collision). If that assumption doesn't hold, then add your nonce to the event data (and thereby modify the hash).
- Shoothe 9y agohttps://developer.mozilla.org/en-US/docs/Web/API/RandomSource/getRandomValues https://developer.mozilla.org/en-US/docs/Web/API/RandomSourc... can be used to generate cryptographically strong random numbers. It's not as cheap as Math.random though.
- phoneposter123 9y agoI assume this is a joke or that someone inside google pulled a prank: >The first time Googlebot calls Math.random() the result will always be 0.14881141134537756, the second call will always be 0.19426893815398216 1488 is a white supremacist number known as "the 14 words" / "heil hitler", and 1942 is part of world war 2.
- some1else 9y agoI employed a seed-based deterministic random function in a WebWorker once, to noisily but predictably drive an animation. I suppose one could use the same approach to have a decent non-deterministic source alongside Google's patched seedrandom.
- sevensor 9y agoSeems like the expectation on PRNGs being expressed in this thread is a bit unrealistic. They're always deterministic. The fact that this PRNG is also always seeded the same makes it easier to fingerprint, but that has no bearing on whether the PRNG is deterministic.
- yndoendo 9y agoRandom should be undetermined. While PRNG should only be for cryptography. Both serve two different purposes.
- sevensor 9y agoThat would be terrible! I rely on the deterministic behavior of PRNGs all the time. For instance, I often generate random test vectors. If I have a failure, I want it to be reproducible so I can fix it. And it is, as long as I supply the same seed.
- yndoendo 9y agoSo you found another use for PRNGs. Random should be close to real world multi-side dice with seed entropy used from multiple sources such as video card, nic, and storage buffers. PRNG != Random.
- sevensor 9y agoThere are lots more examples: heuristic optimization, discrete event simulation, sampling... I could go on. Deterministic RNGs are much better in all of these applications, where reproducibility of results is important. I'm sure nondeterministic RNGs have important uses too. Perhaps you'd care to describe some of them.
- vlovich123 9y agoI know you're being facetious because the OP isn't correct that deterministic PRNGs aren't useful, but any cryptographic application of a PRNG should be non-deterministic.
- jchw 9y agoSeems reasonable to me. My guess is that it's not performance, but rather predictability, that matters here. Being able to detect when a page meaningfully changes is probably useful for Google, and a good implementation of Math.random() would potentially thwart that. Especially seeing how many pages have the magic constant in them... Also, probably useful for determining two pages are the same, which may be needed to help prevent the crawler from crawling a million paths into a SPA that don't actually exist, for example.
- taf2 9y agoI wonder if they also reimplement or adjust crypto api since it offers better random numbers
- jchw 9y agoIf I had my guess, my guess would be Googlebot simply disables the API. It's new enough that this would be reasonable, and executing real crypto in context of Googlebot is probably rarely desirable.
- herodotus 9y agoFor those (like me) who are not that familiar with Javascript, the Javascript spec for Math.random says: "....The implementation selects the initial seed to the random number generation algorithm; it cannot be chosen or reset by the user." Furthermore, the seed usually changes. It seems that Google has modified their Javascript library, perhaps by allowing an explicit Math.Seed function.
- partycoder 9y agoPRNGs are deterministic. Even the PRNG shipped in processors available through the RDSEED/RDRND instructions is deterministic. Unless you are using some form of entropy, e.g: dedicated hardware, that will be the case.
- Kesty 9y agoThe article doesn't simply say it's deterministic but the seed doesn't change.
- yueq 9y agodef roll(): return 4
- nimell 9y agoPython version of: https://xkcd.com/221/ https://xkcd.com/221/ ?
- moonbug22 9y agoOf course it is. Why'd you think it would be otherwise?