6 ms·
How Simple Analytics calculates unique visits without cookies or fingerprinting
- narad 7y agoI am wondering how they will try to compensate when the browser is actively blocking referrer information. In that case the data will be polluted like the website has received more unique visitors, but it is not the actual. In earlier times, this kind of tracking was the norm. Big Analytic companies tried to distill this information and add value to the existing data to make sense. So the cookies and fingerprinting were added to distill that data.
- donohoe 7y agoGiven you want discount internal site traffic from being unique, you could add a query param to URLs as the user navigates.
- __ka 7y agoI think a more robust (and privacy-preserving) way to calculate uniques ( per time-frame as well) is to make use of the local storage / indexDB in the visitor's browser. This is how it would work: - Visitor X lands on your website. - With some JavaScript you check local storage if `dailyUnique` for today's date is set. - if yes, send `visit` signal - if not set, send `unique visit` signal and set `dailyUnique` to local storage. You can apply this to any analytics use case. Its is private and does not rely on referrers. We have been using this goal-attainment approach to do analytics at my company for quite some years.
- AdriaanvRossum 7y agoAuthor here. Our tool is GDPR compliant out of the box. We can't use local storage (it's legal wise the same as a cookie). Although your suggestion will provide more accurate stats, it's a trade-off we are willing to accept.
- bouncycastle 7y agoHow about referring from a https site? AFAIK, browsers don't send a referrer from s https site.
- michaelbuckbee 7y agoIf the referring site is http and the site being referred to it won't send the referrer header. http --> https = no referrer header https --> https = referrer header
- sm4rk0 7y agoActually: https --> http = no referrer header
- sm4rk0 7y ago> HTTPS sites will never transmit referer information to non-HTTPS sites https://developer.mozilla.org/en-US/docs/Web/Security/Referer_header:_privacy_and_security_concerns https://developer.mozilla.org/en-US/docs/Web/Security/Refere...
- AdriaanvRossum 7y agoThat is not really relevant because if a user comes from a different website and the referrer is not present, it will be counted as unique. Having a referrer of that other website or having no referrer does not change anything. We both treat them as unique. The only referrer we actually use is the one from the same site to the same site. Those requests are always with the same protocol (and if you not you should change that). In that case we see both the hostname and the referrer are equal thus being non unique.
- __ka 7y ago> it's legal wise the same as a cookie I am not sure that is the case. GDPR makes provisions for personal data that can uniquely identify users. Blanket statements like: Local storage is not allowed I think are misleading. The state is persisted in the client's machine. Unlike cookies, which get attached to all requests in the specified path, local storage items are not transmitted with the request. Furthermore, in the approach I recommended earlier, no unique identifiers are being sent with the request at all. I am pretty sure that is GDPR compliant, but would love to be pointed to legal provisions that would suggest otherwise. > it's a trade-off we are willing to accept. Referrers, in my opinion, are not reliable enough to derive uniques, and I would assume (although I would not have any numbers to back it up), that the margin of error is very significant when you consider every condition under which referrers would not be sent (some very good cases when that happens are mentioned by other people in this very thread)
- jedimastert 7y agoHow is this different from using a cookie, from a privacy perspective?
- thaumasiotes 7y agoThere's no difference between a cookie and local storage -- that's what a cookie is. But there is a difference between using an identifier of 2019-12-14 vs an identifier of 175c6328-0699-4577-8364-3db5071c2173.
- grsmvg 7y agoWould this pass GDPR without a cookie notice?
- thaumasiotes 7y ago> ‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person; > ‘processing’ means any operation or set of operations which is performed on personal data or on sets of personal data, whether or not by automated means, such as collection, recording, organisation, structuring, storage, adaptation or alteration, retrieval, consultation, use, disclosure by transmission, dissemination or otherwise making available, alignment or combination, restriction, erasure or destruction; As far as the English text of the regulation goes, it's clear that the generation, detection, and use of this record count as "processing", as long as the record itself is "personal data". It's not clear that this is the case. A record stating "this browser has visited XXXX website today" does nothing, in the absence of other records, to identify the person providing the record. But this is open to some interpretation. In particular, you might be getting this record from web requests that already identify the user by other means (perhaps they're logged in). In that case, someone could argue that the fact of the user having visited (or not) your site before on the same day is data that pertains to them specifically ("personal data"), and that your making note of it is prohibited by default under article 6 of the GDPR. The counterargument would be that when the data is reified in your use and your records, it has already become impossible to relate to any individual person. You'd have to rely on that counterargument, because article 6 won't help you at all: > 1. Processing shall be lawful only if and to the extent that at least one of the following applies: > (a) the data subject has given consent to the processing of his or her personal data for one or more specific purposes; > (b) processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract; > (c) processing is necessary for compliance with a legal obligation to which the controller is subject; > (d) processing is necessary in order to protect the vital interests of the data subject or of another natural person; > (e) processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller; > (f) processing is necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data, in particular where the data subject is a child. > Point (f) of the first subparagraph shall not apply to processing carried out by public authorities in the performance of their tasks. None of these will apply unless you are an agent of the government. ----- Thought experiment: as a hostile website operator, you decide to attack your users by filling their local storage. You generate random bytes and store them under random keys to the limit of what their browser will allow. You don't yourself know what the keys are. Is this a GDPR violation? Those random bytes are highly entropic identifiers which you processed and associated with individual users.
- donohoe 7y agoSorry, wouldn't work. Directive 2009/136/EC (aka EU Cookie Law) is not limited to "cookies". The law itself doesn't even mention the word "cookie"! It covers localStorage and any similar features that may come along in the future.
- ricardobeat 7y agoThe GDPR law concerns data which can be used to identify / track a person. It does mention cookies once! Quote: > Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags. This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them. This would not apply to the above method since it is not an identifier. The problem is, there is another regulation called the ePrivacy directive (from 2009, the original 'cookie law') that states you need user consent before setting any kind of cookies other than the strictly necessary. In the privacy dialog, these are allowed to be pre-selected, unlike identifying cookies, but you still need to acquire consent explicitly. Not a lawyer, this is not legal advice.
- dennisy 7y agoVery excited by the title, but very disappointed by the content... This is not true unique visit counting. Which can be achieved using non unique values being set in browser storage, which is privacy friendly in my eyes.
- AdriaanvRossum 7y agoI agree with you. It's privacy friendly to have a cookie stored on the browser with a boolean only. Unfortunately it's not allowed by GDPR without asking for consent first. Would be open to other ways without cookies and fingerprinting. I think there is no other way.
- grsmvg 7y agoSo even if it’s obvious that there is no unique ID in there, you still need to show the notice? We really need to lobby for a smarter law.
- raverbashing 7y agoI'm just glad someone is focusing on obeying the spirit of the law rather than tacking an ironic "we value your privacy" popup to the website and continue selling my data.
- fhars 7y agoYeah, I always read these popups as “we value your privacy, therefore we take it away from you and sell it to the highest bidder” and hit the back button.
- marcusjt 7y agoThere's nothing "unique" about those visits, just using the referrer isn't enough to make them unique.
- tyingq 7y agoYeah, I had to keep re-reading it to make sure I didn't miss something clever. The "trick" is that they use an unrelated metric instead? Heh.
- sm4rk0 7y agoIt's always quasi-unique, because people might use some privacy protection measures that skew the results. This is less precise but more privacy friendly (for those who care).
- glitcher 7y agoThe author's comment post discusses how the analytics are going to be flawed, but is emphasizing that the trends in the numbers over time can be very useful, even when we know the exact numbers are not precise. I think it's a great trade-off to respect user privacy while removing annoying popup garbage from the user's screen about consent. Aren't we all sick and tired of these consent buttons everywhere online now???
- jypepin 7y agoThe article says > When a user lands on your website without visiting another website (direct visit) we will record it as a non unique visit But then the image below says the contrary > This will be tracked as a unique visit as the current website name is different from referer (not available). Which is it? Anyhow, this seems like this approach unfortunately makes the "unique visits" data completely wrong, since users might tend to visit your website multiple times a day.
- AdriaanvRossum 7y agoThank you. It was a typo, it's now fixed. Unique visits data is never 100% accurate. What if a user switches from device or browser?
- thaumasiotes 7y agoWell, your site could require authentication. 100% accurate unique visits data isn't really a good reason to do that, but it is a side effect if you already do.
- AdriaanvRossum 7y agoAuthor here. Just to shine some light on the problem we are having and how this solves that problem. We get more and more customers that are required by law to have an active opt-in for cookies and fingerprinting. They basically can't use those techniques before a visitor actually ticks a checkbox (which has to unchecked by default). This results in a lot of missed data. They can't run Google Analytics for the visitors that don't check this box. For them it's a big issue because they can't see how their visitors are behaving before checking that box. Simple Analytics wants to fix this issue by not requiring consent. Customers can use our tool for every visitor that lands on their website. One other thing that is a common misconception is that unique visits with cookies are accurate. They also have flaws: - What if a user uses a different browser - What if a user uses a different device - What if a user blocks (third-party) cookies The same goes for using IP based fingerprinting. All techniques have flaws. Although they are flawed they are used to get a number. It's not an accurate number but it's a number. That's also the case with using our technique. It's less accurate then using cookies, yes, but it's a number you can use. You can compare it to a previous period, you can see the unique views of a page, etc. If you don't need to comply with privacy regulations you can use all the tools you want. We just focus on the companies who do care about those regulations and give them the big picture they are now missing.
- digitalengineer 7y ago> They can't run Google Analytics for the visitors that don't check this box I don’t understand. They can. Just anonymize the ip. You can track all their activities. Am I missing something?
- AdriaanvRossum 7y agoGoogle Analytics tracks your visitors with a cookie [1]. You need consent for placing that cookie under the GDPR and other privacy regulations. [1] https://developers.google.com/analytics/devguides/collection/analyticsjs/cookie-usage https://developers.google.com/analytics/devguides/collection...
- 7y ago
- buremba 7y agoA user can land your page from multiple referrers (or the same) more than once and some search engines including Google will hide the referrer information so the calculation will be off IMO. The regulations are not strict if you're not identifying the individual people from your visitor/user data. If you're collecting user event data into your servers and can prove that you're using the data just for the analytics purposes (i.e. providing a better user experience to your users) there is nothing you can afraid of using cookies. That may not be the case for third party analytics providers though because if you're using Google Analytics, you're responsible from how Google are using your customer data.
- maweki 7y agoAs I understand it, this is the method Webalizer has been using to calculate unique visitors for over two decades now. It is mindboggling that we have come full circle and this is now a business model again.
- rwmj 7y agoIt's one of the methods we used when doing logfile analytics in the early 2000s. At the time it just appeared obvious to me. One problem with it (which is why we combined it with other techniques too) is that some browsers even then did not send Referer headers reliably, neither between pages on the same site nor between third party sites, so what they say in the article about "direct visit" is not reliable.
- giancarlostoro 7y agoThis explains that obnoxious URL google has for results. To be able to fully ensure they know who referred you. I use DDG at home but Google at work.
- avian 7y ago> some browsers even then did not send Referer headers reliably Apart from https/http thing mentioned elsewhere in this thread, there's also Referrer-Policy [1]. I don't know how wide-spread its use is, but I find that most requests I get have no referrer header these days. [1] https://developer.mozilla.org/en-US/docs/Web/Security/Referer_header:_privacy_and_security_concerns#How_can_we_fix_this https://developer.mozilla.org/en-US/docs/Web/Security/Refere...
- lucb1e 7y agoCompletely true, but I'm fine with this being posted because I do think people need reminding of how much we can do without Google tracking.
- tiku 7y agoYou could just add a unique value to every link between pages?
- AdriaanvRossum 7y agoThis way a user is still unique when he/she closes the tab and reopens, right?
- kbsletten 7y agoWhat if they share the link on social? All the new traffic would seem to be non-unique.
- founderling 7y agoThis article uses the terms "visit" and "pageview" in unusualy ways. They give the example that when a user clicks through from yourwebsite.com/ to yourwebsite.com/page then the first is a unique visit while the second is a non-unique visit. I would neither call a "visit" but rather a "pageview". In my book, the visit is what started from the first pageview on yourwebsite.com site and continues to the last pageview.
- glandium 7y agoI haven't touched analytics in 15 years, but back then what you describe for visit was called a session. That seems less ambiguous to me.
- AdriaanvRossum 7y agoAuthor here. It's technically more correct to call those visits "unique page views" and "page views". I think visit is a very simple word to understand for most people. My tool is targeted at people who want a simple tool, that's why I want to keep the wording also simple.
- Terretta 7y agoHow we used these words 20 years ago, when server logs were cool: Uniques are visitors not pageviews. Hits, page views, and sessions are decreasingly granular content stats, not people stats. Visits and visitors are people stats. Sessions tend to be the matrix of page views (containing multiple hits) per visitor visit.
- grodes 7y agoThe same user will trigger many unique visits if coming from, for example, google search
- AdriaanvRossum 7y agoThis is true and that's a trade-off for not using fingerprinting or cookies.
- kimsant 7y agoPWA 1px image on top with cache-first show single visits. A cookie without a cookie
- pdkl95 7y ago> This referrer is very useful to figure out where traffic is coming from. This is exactly why sending the referrer header is insane[1]. The header betrays the user's privacy by design. Remove it ASAP. If your app breaks, that's unfortunate but necessary. [1] https://news.ycombinator.com/item?id=20468115 https://news.ycombinator.com/item?id=20468115
- noman-land 7y agoWe have an eslint rule at work that forces you to add `rel="noreferrer noopener"`to any outbound link you write precisely because it's a security concern.
- dessant 7y agoIt's enough to add rel="noreferrer" to links, noopener is implied. https://html.spec.whatwg.org/multipage/links.html#link-type-noreferrer https://html.spec.whatwg.org/multipage/links.html#link-type-...
- deleted 7y ago[deleted]
- liquidise 7y agoNecessary for whom? Privacy purists seem to be surprised that website and app owners want/need analytics to meaningfully improve their app's usability. I praise SA's privacy oriented approach as a meaningful alternative to GA. This sort of overly-critical hyperbole doesn't help the privacy conversation. If follow SA loses a feature it likely needs to remain competitive, and customers would have another reason to go back to Google Analytics. Hardly a necessary privacy win a my book.
- pdkl95 7y ago> Necessary for whom? Necessary for anyone interested in preserving democracy[1]. Unless we actively fight to preserve privacy, surveillance capitalism will continue to "disrupt" traditional forms of power. > If follow SA loses a feature it likely needs to remain competitive, and customers would have another reason to go back to Google Analytics. Continuing the betrayal of user privacy in the hope of appeasing the surveillance capitalists won't be "[privacy] for our time". Advertisers and other eavesdroppers have consistently demonstrated they will maximally exploit any tool available to them. [1] http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and-a-the-age-of-surveillance-capital.html http://nymag.com/intelligencer/2019/02/shoshana-zuboff-q-and...
- skizm 7y ago> Other analytics businesses use this technique (for example based on IP address). This seems privacy friendly but is considered fingerprinting. For which you need consent. Do you need consent to count unique IPs that visit your site in the UK under GDPR? Aren't there laws that say unique IPs do not correspond directly to an individual (in the context of piracy), and yet for simple website analytics you count IP as uniquely identifying a user? Am I required by law to change the default config of apache or nginx to not log the IPs? Or is it enough if I promise not to analyze the logs?
- deleted 7y ago[deleted]
- WoefullyInept 7y agoIt really doesn't solve any problems surrounding web analytics, user privacy or user experience. I think it shows the creators lack of industry experience in ecommerce, advertising and user tracking. It's like someone has sprinkled a flawed 20 year old method of tracking in glitter and saas'd it. I think there is a very small subset of website owners that would be willing to pay for this. Then again he only needs to get a few people that aren't sales focused and are deluded enough to think this is the pinnacle of 'privacy'.
- jgalt212 7y agoGDPR is just a mess. The bureaucrats who wrote it don't understand it, neither do the enforcers, or the folks under enforcement. Brussels should just shut itself down.