12 ms·
How we built a GDPR-compliant website analytics platform without using cookies
- pauljarvis 7y agoWe are incredibly open to any ideas, comments or concerns on how we're doing this. This is a big step up from what we had previously, but there’s always room for improvement. Happy to hear thoughts in the comments.
- abtestdev 7y agoHow do you guard against hash collision?
- JackWritesCode 7y agohttps://crypto.stackexchange.com/a/47810 https://crypto.stackexchange.com/a/47810
- abtestdev 7y agoBased on the blog - anyone who shares a IP address (such as inside a company network) would effectively look the same.
- proaralyst 7y agoAnd given most companies run managed browsers on managed systems, the user agent is going to collide too
- JackWritesCode 7y agoIndeed. This is one of the limitations to our approach. We're thinking of ways to workaround it. Please shout if you have any ideas ;)
- cameronbrown 7y agoIf/When IPv6 is widely used the problem should fade away.
- _-___________-_ 7y agoNot just inside a company network. CGNAT is pretty commonplace nowadays.
- lmkg 7y agoHi Paul, thanks for being open about this. I have a big, important question. ICO, the agency in charge of enforcing GDPR and related legislation in England, released guidance earlier this month on the topics of cookies. One of the most notable parts of this guidance is that "device fingerprinting" is treated the same as a cookie[1]. And also that website analytics requires consent to use cookies or similar technologies[2] ("similar technologies" including device fingerprinting). Now, the above guidance is related to PECR rather than GDPR, which is what your post is about. But, given the above, do you think that your software is compliant/exempt from PECR or do you think that organizations will still have to take extra steps to be compliant with privacy legislation? [1] https://ico.org.uk/for-organisations/guide-to-pecr/guidance-on-the-use-of-cookies-and-similar-technologies/what-are-cookies-and-similar-technologies/#cookies5 https://ico.org.uk/for-organisations/guide-to-pecr/guidance-... [2] https://ico.org.uk/for-organisations/guide-to-pecr/guidance-on-the-use-of-cookies-and-similar-technologies/how-do-we-comply-with-the-cookie-rules/#comply15 https://ico.org.uk/for-organisations/guide-to-pecr/guidance-...
- deleted 7y ago[deleted]
- orra 7y agoI think that's a fair question. AFAICT, v1 of PECR awkwardly applies whenever the cookie is not functionally directly necessary for the service that the user is using. PECR applies even if, like here, the cookie is just for counting unique numbers of visitors, and is not used for fingerprinting individuals. The draft v2 of PECR contains an exemption for first party analytics. I think this maybe strikes a nice balance: explicit consent would still be required for the more-harmful third party analytics. Not sure when v2 of PECR will happen. It is years overdue. Perhaps it is a priority for the newly elected European Parliament and the new Commission?
- krageon 7y agoWhat makes analytics first-party? When the first party serves them, or only if the data never leaves machines under the direct and exclusive control of the first party?
- atonse 7y agoI trust that there's enough questions and scrutiny about your anonymization, so I don't have any questions about that. Mine is more about implementation. If I want to integrate this with a single page app (like Ember or React), are there enough API hooks on how I can track click events and page load events, etc in the JS? We threw together Google Analytics for our launch just so we would have SOME data, but we want to move away from it ASAP for privacy reasons. Anymore about how it would compare to something like Piwik? (the product we're looking at).
- JackWritesCode 7y agoWe <3 EmberJS. Fathom's Dashboard is built using it. You'd want to built it into the Router and call fathom('trackPageview') to log it :)
- billabul 7y agosorry but, isn't that a (unnecessarily complex) cookie?
- JackWritesCode 7y agoWe don't store a cookie but, yes, the hash doesn't need the date item in it since it has the salt. We'll change that in the next release
- Spivak 7y agoLook, entire industries exist for being complaint with the letter but not the spirit of the law so I'm sure that this in no way meets the definition of a cookie as far as the GPDR is concerned. However, this is absolutely a cookie. Scraping just enough information from the browser to create a unique but stable hash and then having the browser compute it every request isn't at all different from that browser information acting as the cookie.
- Nasrudith 7y agoTechnically no - cookies are stored by the user. If anything it is technically worse in some ways because it moves it to the back end outside of user control.
- st3ve445678 7y agoLooks decent, but pricing is insanely high for the extremely limited set of stats.
- pauljarvis 7y agoTotally fair, as that's an opinion :) Luckily our customers are happy with the price, and other folks use the open-source (100% free) version. Cheers!
- paulgb 7y agoI actually think your prices are pretty reasonable for business use, but I wonder if you've considered a "personal" plan of, say, $6/month for up to 10k hits, non-commercial use only. I've been looking for a privacy-first tracker for some personal sites for a while, but nobody is offering pricing that makes sense at the lower end.
- soared 7y agoPricing looks comically cheap for SMB and above.
- LeonidBugaev 7y agoexactly, expected to be 5-10 times higher.
- cm2012 7y agoThe price is equivalent to one hour of developer time per year.
- CHsurfer 7y agoI think the GDPR was enacted into law not to prevent cookies, but to prevent collecting data on regular people. This seems to circumvent the technicalities of the law but not the spirit. The risk is that they enact a new law that puts even further restrictions on website operators. I'm not sure this is a good idea.
- deleted 7y ago[deleted]
- JackWritesCode 7y agoThanks for the concern here. We are GDPR compliant (and may be exempt from it). See here: https://usefathom.com/data/ https://usefathom.com/data/
- jfk13 7y agoYou might like to edit the line on that policy page that refers to "the most privacy-focused manor"... while a privacy-focused manor is an interesting idea, I suspect you meant "manner". :)
- pauljarvis 7y agololololol
- i_anon 7y agoEqually, I'm not sure what it means to be GDPR "complaint" but I'm thinking it's probably supposed to be "compliant" ;)
- noxToken 7y agoTell me if I get this correct: Alice visit a site and gets the hash 1234. The analytics data is stored and associated with hash 1234, but soon after, hash 1234 is removed. However the aggregate visitor analytic that was associated with hash 1234 data persists. Then another user (say Alice again) returns and gets hash 5678. Analytic data is tracked, stored with hash 5678 for the 30 minutes (or less), and then hash 5678 is again removed. However the analytic data that was associated with 5678 is aggregated with the rest?
- AndrewStephens 7y agoMost schemes of this kind are just more complicated cookies that people hope will avoid the GDPR provisions by dint of being obfuscated. What the article is discussing looks (at first brush) to be a sensible way of aggregating users up-front before it hits the database, rather than later. So no personal data is stored. Does this meet the requirements for a site to avoid notifying users under the GDPR? I have no idea. Even with the best of intentions, if you use a service like this then you are relying on them a) doing what they claim, and b) not screwing up (by leaving logs around, etc). If I use this service and data from my users gets leaked by Fathom, who gets blamed? The users were on my site, so I guess it is I that gets fined. Maybe the risk is worth it, maybe it isn't.
- gjs278 7y ago>If I use this service and data from my users gets leaked by Fathom, who gets blamed? The users were on my site, so I guess it is I that gets fined. fathom would get fined
- michaelbuckbee 7y agoThe GDPR (and other data privacy legislation) uses the concept of a "data controller" and a "data processor". Data controllers use a variety of data processors to deliver a service to their users. If you ran a SAAS and used Fathom analytics, a SMTP email provider (to send password resets), a newsletter provider (for your monthly newsletter), a blog host etc. each of those would be data processors as you (the SAAS) are making the determination of where user data is going on the backend. As a data controller, it's your responsibility to make sure that each of those services you are using is handling the data in an appropriate and safe way. As to who gets fined if there is a data breach: the answer is likely nobody. I say this because it's not like you have a credit card on file and breach automatically means a fine. What really matters is what actions you're taking before and after a breach. - As a data controller did you notify your users with no undue delay? - As a data processor did you notify the SAAS with no undue delay? - Did you identify the source of the breach? Did you take steps to remedy it? For almost all these privacy regulations if you: 1. Take steps to protect user data/privacy like https, encryption etc. 2. Provide a mechanism to allow users to make data requests for their own identifying data 3. Notify users if there's a data breach You would be in compliance.
- moose333 7y agoAs a user of the open source version of Fathom, I'm a little concerned by the lag in publishing this update to the community edition. I assumed development work was happening in the open on Github, but I guess that's not the case?
- JackWritesCode 7y agoWhole new language & codebase. Old developer left, we don't write Go.
- pauljarvis 7y agoSo it's not as easy as just pushing the update to the repo. We are still committed to open-source, but we also have a business to run and need to make a living here (we're two dudes who care about privacy, not a huge company with deep pockets) :) The community version is getting a full update soon. We just have to focus on profit a bit (this keeps us in business and able to update the repo).
- joekrill 7y agoThere's certainly nothing wrong with that. But this sounds a little defensive. And if you're going to promote the "Community Edition"/open-source nature of the codebase, this should probably be made more clear to your users/contributors. (Maybe it is made clear and I'm just not seeing it, though - this is my first time hearing about this product).
- JackWritesCode 7y agoI think the defensiveness comes when people put comments such as "I'm a little concerned by the lag in publishing this update". I've never spoken like this in my life to an OS contributor and I use open-source software every day. We're working hard to get the software OS but we were originally planning on keeping the codebases separate (since they were different languages). We only recently pivoted on this after speaking to some wonderful people in the OSS community. Perhaps our communication has been poor, so we'll work on that, but we are working hard at this.
- mrweasel 7y agoCouldn’t people just parse the log files from their webservers?
- JackWritesCode 7y agohttps://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- bashy 7y agoDefinitely. Helps with people who disable JS as well. I've used this before https://goaccess.io https://goaccess.io
- JackWritesCode 7y agobashy from laracasts? Hey there! And I've heard the disabled JS piece before. How many people are disabling javascript?
- bashy 7y agoYes that's me. Hi there. I use fathom on one of my sites and I like it. I was just replying to someone who said people could just use access logs instead. Some don't like adding JS to their site I guess. Ad blockers will also block requests to the tracker.js if you use analytics.domain for example.
- felixfbecker 7y agoIt's so exciting that thanks to GDPR we are now seeing innovate analytics solutions that respect privacy.
- pauljarvis 7y agoAgreed!
- EGreg 7y agoI am not sure what exactly they did here. How do they persist the hash between requests? My guess is they use localStorage and sending the hash to their servers with each request. So we are talking about a mechanism that’s just like a cookie. As long as they don’t have any PII and can’t figure out who the user was, then I think the GDPR gives them an exception. But “without cookies” claim is dubious!
- deleted 7y ago[deleted]
- JackWritesCode 7y agoWe have been GDPR compliant for many months but our aim here was to meet E-Privacy demands. We don't use localStorage... Read the blog post, we don't use cookies.
- EGreg 7y agoOh, apologies, you are right! You don't use any kind of cookie mechanism. I just read it again: Random SHA256 String (daily regenerated) IP Address User Agent Site ID Day of the year My only question is about this "Random SHA256 String", where is it stored between requests?
- JackWritesCode 7y agoRedis Cache. It's a Fathom-wide random string that is used to prevent rainbow table attacks. The salt is refreshed at midnight every day.
- EGreg 7y agoThanks, makes total sense. So basically the only drawback I see is that all employees from behind one corporate NAT using same browser will count as one user. But don’t see a way around that if you can only use IP and User Agent strings.
- jacquesm 7y ago
- itronitron 7y agowhy are you calling it an analytics platform when it isn't one?
- kitchenkarma 7y agoThis is very weak reasoning, because you cannot identify an individual by IP either. This project looks like trying to exploit loopholes. The idea behind GDPR is to make sure companies log only data they need. This project looks into logging the data but without expressing why this is even necessary. Therefore I don't think this is compliant with GDPR.
- JackWritesCode 7y agoGDPR is for protection of personal data and we store no personal data. Please take a read of this: https://usefathom.com/data/ https://usefathom.com/data/
- kitchenkarma 7y agoI don't believe you have understanding what personal data and GDPR is. You are capturing user behaviour and that is very personal regardless if it is "anonymised" or not - and that is without clear need for doing that. That is pretty much against GDPR.
- JackWritesCode 7y agoYou come across as somewhat hostile but I'm going to assume good intent on your part, so thank you for the challenges on our stance. So if you take a look at Recital 26 (https://gdpr-info.eu/recitals/no-26/ https://gdpr-info.eu/recitals/no-26/): > To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly. > To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments. > The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. > This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes. So the piece about the principles of data protection not applying to personal data rendered anonymous is crucial. We believe that GDPR does not apply to us because of that. But even if GDPR did apply to us (we'll assume it does, that's always the best way to be), then our legal basis is that there's legitimate interest. As a website owner, it is in your legitimate business interest to understand how your website is performing - e.g. the most popular pages, the pages where people linger for longer, the pages where people bounce.
- jacquesm 7y agoAt first glance this appears to be a well thought out solution. It will never be able to give you some of the stuff that GA can give you but that is by design. The problem that I see is that as long as GA is able to claim they are GDPR compliant there will be very few websites that will see this as a necessity and so adoption will be relatively low. But, and this is just an idea, one of the things that company could do is to proudly present a 'zero retention' button or logo assuming they do not have other trackers on their pages. That way it might become a distinguishing factor for the adopters and that might drive further adoption. Thanks for building this, I will promote it.
- JackWritesCode 7y agoA fantastic idea. We've recently designed a button for websites to show users that they care but I really love this idea of a 'zero retention' button :)
- saagarjha 7y ago> Tracking page views alone, without visits, is completely useless and means that you won’t have insight into how many people visit your site / pages each day. What's the difference between a page view and a visit?
- JackWritesCode 7y agoIf I came onto your website and refreshed one of your pages 500 times, that would count as 500 page views but only 1 visit.
- ares2012 7y agoThis is a common solution to the problem of PII, but without any information on returning users I would argue that it's value as an analytics platform is limited. Few are the tools where you can grow the business without knowing the difference between a first-time and return user which is the reason cookies were invented in the first place. However, since such businesses already need to collect personal info as part of your account creation it shouldn't be hard to build analytics on top of that existing PII. If they are already collecting PII it doesn't seem to save much to have their analytics tool avoid it?
- JackWritesCode 7y agoFathom Analytics is intentionally limited, and the limitations you point out are 100% intentional. There are many businesses who can't use our product, but millions that can :)
- SCLeo 7y agoLooking at their live demo (https://stats.usefathom.com/#!p=1w&g=hour https://stats.usefathom.com/#!p=1w&g=hour), I can see a lot of traffic is coming from ycombinator. So... (I mean, I don't have a point here but I find it pretty interesting. xD)
- ricardobeat 7y agoThat's exactly what they are selling - the ability to get relevant business metrics while preserving user privacy.
- hedora 7y agoDuck Duck Go traffic is 10% of the Google traffic. Is that typical for privacy-focused tech sites? I would have expected a lower percentage. (DDG fan here, but people look at me funny when they see me using it...)
- vmlpvf 7y agoThe data is not anonymous. Anonymity is actually very hard to claim (read k-anonymity, differential privacy, etc). Nevertheless, the chances of identifying someone are probably pretty low, and it´s a good effort to make analytics more privacy friendly.
- JackWritesCode 7y agoSo it's practically anonymous. Nobody has enough computing power to brute it & the data is deleted, typically, in 30 minutes.
- nybble41 7y agoTo de-anonymize the data you don't actually need to brute force the 256-bit hash. If the other pieces of data are known (salt, site, page, day of year) and you can make a shrewd guess at the user agent then you'd only need to brute force the 32-bit IP address(). () Assuming IPv4. Obviously IPv6 addresses would be much harder to brute force, but still easier than a 256-bit hash.
- JackWritesCode 7y agoI do take your point but when we get into this area, it becomes a big question of trust. Because if I gave you a hash, you would need to guess the 256 bit salt in addition to all the other possibilities. I mean, hey, here's a hash: 26246226167b9f190d3a1ce726efe07ae18bbf0480a78d19390b9aaf13f25cb0 Imagine you just got hold of it through a data breach. I'll give you $1,000 if you can get it de-hashed before midnight Chicago time ;)
- nybble41 7y agoIt's true that you'd need to know the salt first, but the server does know the salt, and if you obtained the hash through a data breach then you probably obtained the current day's salt as well.
- 7y ago
- unilynx 7y agoWhy all the trouble with hashes - can't you just do it on the client and not having to store any data at all? "For tracking unique page views" if(!sessionStorage[location.href]) { sessionStorage[location.href]=1; navigator.sendBeacon("/unique-pagehit?" + encodeURIComponent(location.href)); } "For tracking unique site views" if(!sessionStorage["Hi!"]) { sessionStorage["Hi!"]=1; navigator.sendBeacon("/unique-sitehit"); } "For tracking previous requests" I'm not sure I fully understand what is being measured (is it session-only?). For the duration someone watched a page, you can use sendBeacon in onBeforeUnload. To detect a bounce, set a Math.random() in a session variable, send it at the start of the page, and have every page load send the previously stored random variable. Then count the unique random keys you received on the server - those are the bounces. I know, in practice you'll need to trim sessionStorage, sanitize URLs, use something less-colliding than Math.random, dealing with new tabs, some polyfills and other robustness, etc... but I don't yet see why the tracking mentioned needs any user ids or hashing at all.
- JackWritesCode 7y agoGreat idea but we can't use sessionStorage under PECR, which is why we made this move. Plus we got rid of anything being stored on the user's machine.
- draw_down 7y ago> can't you just This is how you know you're about to make a really good point. Heck, I could code this up in a weekend!!
- ricardobeat 7y agoThis client-based scheme would be totally vulnerable to spamming / general messing about. Doing it in the server requires an actual request, and should still allow them to filter out attempts at gaming the system.
- gcbw2 7y agoThis is still logging everything the GDPR says you can't without asking for consent, but you made your search convoluted (but not less efficient if you have all the pieces) to (suggest|lie?) that you need to break the hash and that's why you don't need consent. None of the information you are using on the hash wouldn't be in the search query itself! ip, user agent, path, date, etc. So there is no way to reverse the hash. You just hash your search query and compare in O(1) time. The only piece of information that realistically makes the hash slightly difficult to get is the random number refreshed every day. But either you store it (and i have no reason to believe you do not) or it make the brute force effort trivial as I only need to generate the hash with that variable now.
- JackWritesCode 7y agoYou're focused mostly on Recital 26, which was only a theory of mine, outside of that we are GDPR compliant anyway. I likely shouldn't have included it since that isn't our primary ground for processing. Please see: https://usefathom.com/data/ https://usefathom.com/data/ And yes the daily hash gets stored until midnight. But what are you talking about with 'search query' containing IP, user agent etc.?
- gcbw2 7y agoIf a search query on your data would contain all the components of the original hash, i don't have to walk backwards and break the hash. i just have to hash my query terms in the same way. Also I suggested you store the daily hash forever. But even if you really erase it every day, as you say, If you or an attacker makes the same request every day at a predetermined time, when you/they get your logs, you/they can use that predictable request to get the daily secret too. I consider the information to be stored in plain text, and that you would have to have requested permission just the same. You pretty much have an identifiable user (via IP/UA/access time) stored in your logs. Anonymization is removal of information, not encoding it in a convoluted hash.
- JackWritesCode 7y agoSo that needs to be our next target point (access logs). We want to move to a position to keep no access lgos. And a hacker could indeed "win" if they broke into our system, got the salt and exported the DB. We didn't focus on this in our article, as it's unbelievably unrealistic, but it's still possible. Our next step is to address that. Without the hash, it's practically impossible to brute force.
- tomp 7y agoMaybe I'm missing something but (1) I don't think this is GDPR compliant, and (2) why so complicated? Regarding (1), > Brute forcing a 256 bit hash would cost 10^44 times the Gross World Product (GWP). [...] > We have rendered the data anonymous to the point where we could not identify a natural person from the hash. > It's possible that GDPR does not apply to Fathom since data is made completely anonymous. Even if GDPR did still apply, we reiterate the stance that there is legitimate business interest to understand how your website is performing. This seems to imply a profound confusion between the difference of hashing vs. anonymity. Just because it's hashed doesn't mean it's anonymous! You don't need to "brute-force" the hash, you just need to find a user that matches your hash... which is 1 in 7 billion (or so), much more tractable. This is also the principle e.g. MD5 rainbow tables are based on... They claim to change the hash every 24 hours, so it's equivalent to having a session cookie with 24-hour expiration (session cookies are "anonymous" by their definition, they don't have any user information and they're impossible to "brute force", they "just" enable tracking). I've no idea if 24-hour session cookies are GDPR-compliant... Regarding (2), given that this seems (again, I might be misunderstanding) equivalent to a 24-hour session cookie, why not just do that? However, then you're ... drumroll ... giving control to the user. Why not just give control to the user, period?! For example, by storing the list of pages visited in Local Storage, and only pinging the server once for each page(view) every 24 hours?
- JackWritesCode 7y agoFor GDPR compliance notes please see: https://usefathom.com/data/ https://usefathom.com/data/ > You don't need to "brute-force" the hash, you just need to find a user that matches your hash... which is 1 in 7 billion (or so), much more tractable. This is also the principle e.g. MD5 rainbow tables are based on... Not quite. We use a SHA256 hash as our salt, and that changes each day, so you'd need to brute force that. In terms of how many possible combinations there are for this salt, please see: https://stackoverflow.com/a/49520766 https://stackoverflow.com/a/49520766 - you would need to brute force it and try each possible combination with every single possible IP / User Agent / Site combination to break a hash. This is why it's not theoretically impossible but it's practically impossible. We would love to approach things in an easier way but PECR doesn't want cookies, even anonymous ones. Now, one thing that we have uncovered thanks to someone on here is that we need to increase our resistance to data breaches. If someone had complete, unlimited access to all our data / servers, including the daily salt, then they could de-hash page views from the last 30 minutes. I have no idea how long that would take. There are 4,294,967,296 (?) possible IP addresss, and then over 3M (?) user agents, so it'd be an absurd, pointless exercise.... Anyway, we're going to be bringing in multiple salts that depend on the user IP address, meaning that, in the event of a data breach, a hacker won't know which salt has been used for a hash :) Perhaps we base the salts on the first 3 digits of an IP address? That would mean we'd have 720 possible SHA256 salts!
- i_anon 7y agoHi Jack and Paul - love what you're doing! This solution is so needed. I wondered whether you could explain what makes your hashing different from the hashing used by Facebook for their custom audiences tool which was deemed unsuitable for anonymisation as per https://www.spiritlegal.com/en/news/details/e-commerce-retail-facebook-custom-audience-not-allowed-without-consent.html https://www.spiritlegal.com/en/news/details/e-commerce-retai...
- hellenwaweru 7y agoi like the idea. the privacy is now guaranteed. https://finance.uonbi.ac.ke/node/128950 https://finance.uonbi.ac.ke/node/128950
- labawi 7y agoIf visits expire after 30 min, why not rotate the salt every 30 min? Keep current and previous salt, update as needed. I would have more faith in privacy, if you didn't store the salt in the DB or permanent storage. If you manage to statically load-balance the users (e.g. hash site, ip, user-agent, don't forget site), the hash could be in-memory only. Sessions would break on server restart, but that's more of a feature. To move thing further, you might not even need to store the hashes in the DB. Keep them in server memory only and (real-time) update aggregate data in DB.
- JackWritesCode 7y agoThe visit expires 30 minutes after the visitor lands. The expiration isn't generic. It's an interesting idea. We have multiple servers under the load balancers, so we'd be able to store them in Redis, but that is no better than permanent storage, since Redis could still be breached and you'd see it with ease.