11 ms·
Replace Google Analytics with a shell script
- williesleg 6y agoWow! Using that technique I replaced nuclear fusion with a rubber band and some python code!
- chalst 6y agoI rather dislike the surveillance capitalism model Google promotes, but the article is a little unfair. As a payload, the snippet of code that calls the GA library is pretty lightweight and there are many ways to make this asynchronous: this really does not need to have a significant impact on user experience. As an aside, I find Google's model to be less bad than Facebook's attention capitalism model because it permits a diverse, unoptimised and decentralised web, amp notwithstanding.
- HWR_14 6y agoI think it's a tad disingenuous to focus only on the initial loading of Google Analytics. Presumably there is some ongoing cost to the user regardless of load time and it's probable that the blog was using the synchronous version. There may be many ways to make this asynchronous, but they chose the easiest way: removing it.
- eli 6y agoThe google analytics tags have defaulted to loading asynchronously for many years now.
- anonydsfsfs 6y agoGoogle Analytics by itself is lightweight, but if you're at a large company with a marketing department, my experience has been that it never stops there. We've got pages with literally dozens of third-party Javascript trackers, and I've had to continually push back against requests from marketing to add more. While each one individually isn't a big performance hit, they add up fast. Same principle applies to security: each of these trackers has basically full control over the page, and if one of them gets compromised and replaced by malicious code, it would be disastrous. The probability of any individual tracker getting compromised is low, but what you should be worried about is the probability of at least one being compromised, and that probability rises exponentially as you add more.
- datastoat 6y agoThe probability that at least one tracker is compromised actually grows sub-linearly in the number of trackers. Let n be the number of trackers, let p be the probability that an individual tracker is compromised, and assume compromises are independent. Then: Prob(at least one comp.) = 1 - (1-p)^n ~~ np. In other words, linear in n. This approximation is fine for small n*p. For large n*p, the probability has to be <= 1 of course, so the real probability is sublinear.
- anonydsfsfs 5y agoThanks for the correction!
- d0gbread 5y agoFrom my experience this is a function of client side libraries not having an "owner" in place to refactor. Marketing often doesn't know or care about performance, so of course they'll just want to add over time if it makes them more effective at their job. When businesses invest in caring about this (and it can even be just one headcount for the biggest of businesses) there's opportunity to reign it in and maximize the value from a given platform, reduce it the need to add others. That helps with security, governance, and adoption of what's there. Nevermind more forward looking solutions like server side which gets these injected scripts off your site completely and greatly improves security.
- sneak 6y agoI run a self-hosted analytics tool called Matomo (via CapRover, a selfhosted PaaS) that I include on my static site pages. No third parties are involved unless you count Hetzner where I get my bare metal. Best of both worlds.
- ev1 6y agoI wish Matomo/Piwik worked well on larger sites, or supported Postgres. It seems to start falling apart on my joke sites that get a few million PV/day.
- cpach 6y agoHave you tried adding more servers for Matomo? https://matomo.org/docs/requirements/#tracking-100-million-page-views-per-month-or-less https://matomo.org/docs/requirements/#tracking-100-million-p...
- mraza007 6y agoThis is a very simple and unique approach to gather metrics of your site while maintaining the privacy of the user. I’m just curious how can you use this solution with a static site generator like jekyll
- slimsag 6y agoThe article approach requires access to apache access logs, so you'd need to host your static site on an apache (or other) server with logs you can access.
- slimsag 6y agoI just deployed Plausible.io[1] for my blog $6/mo on a droplet, it's self-hosted and 100% open source. It seems very privacy focused, and provides basically the same minimalist data described in this article. I always find it weird though that privacy advocates still go for "I have this hidden treasure trove of analytics about my site that you cannot access" Analytics should always be 100% anonymous and aggregated. And if you've met those two bars, making them open and public should be a non-issue. If you can't make your analytics public, that's called spying. There should be no issue making your analytics public. Mine are[2] [1] https://plausible.io/docs/self-hosting https://plausible.io/docs/self-hosting [2] https://devlog.hexops.com/about#open-analytics-data https://devlog.hexops.com/about#open-analytics-data
- kbenson 6y ago> $6/mo on a droplet I wonder how many extra dollars cumulatively DO makes from the $1 premium on getting a better Intel CPU or AMD CPU and/or NVMe up sell on the smallest droplets. I pay the extra dollar as well. I doubt I actually see the difference for what I use it for. I feel no desire to pay a dollar less though, so they are making another 20% from me on that product than they would have.
- slimsag 6y agoI didn't pay the extra dollar, I got the $5 droplet - but the backups are $1/mo extra
- kbenson 6y agoAh, interesting. Was it offered at the time and you didn't choose it, or is it an older droplet that didn't have the choice? I've had a lot of different droplets of different sizes over the last 6-7 years. Recently when I created a new one for a new project, I noticed that option and chose it. I'm not sure I would if buying a larger droplet, or a lot of small ones (I've done that for certain projects), but $1/mo is the point where it feels like nothing.
- 6y ago
- loloquwowndueo 6y agoBefore google analytics juggernauted the problem space, there were a bunch of log analysis tools which produced metrics similar to what the article describes, and more, locally and based on nothing but server logs. A ton of them still exist : things like webalizer, goaccess, and the venerable analog. See the self-hosted category here (https://en.m.wikipedia.org/wiki/List_of_web_analytics_software https://en.m.wikipedia.org/wiki/List_of_web_analytics_softwa...) Like the article, I recently realized that, while I did use it, I didn’t take advantage of google analytics at all so decided to nuke it from my sites. I had additionally been running webalizer for over a decade, but recently replaced it with goaccess (took about 10 minutes to set up) which is more actively maintained and have been quite happy with the basic metrics it produces. Sure it won’t stalk a user as they browse my site but I don’t care, just having an idea of how many hits I get, from which countries and which browsers is enough for my few personal and hobby sites.
- ximm 6y agoI was deeply amused by the title. In my world using shell scripts to analyze access logs is the obvious solution and using google analytics is the exotic approach that you might come up with in very few special cases.
- mobilemidget 5y agoAWK & SED :) one of the better books I bought when studying. Often parsing a logs to find some things that other logging don't record/show for a manager. Often you hear the same phrase 'You can check logs like that, and you don't need a browser? How... magic....'
- eli 6y agoI get a LOT of bot traffic with bogus user agents. Google Analytics does an okayish job dealing with it. Looking at web server logs and trying to reconstruct traffic patterns from just real users would be real chore.
- Seirdy 6y ago
- kevinyew 6y agoI wish there was something like this for JAMstack/JS only websites that you host on eg Netlify. I know Netlify has their own analytics but they're paid, which is fair enough, but hard to bite the bullet when Google Analytics is free.
- nindalf 6y agoI believe Cloudflare’s analytics are free. I use it because they don’t set cookies and don’t share the collected data with anyone.
- hanniabu 5y agoI'm using netlify by building directly from github. How would I go about inserting cloudflare into this?
- nindalf 5y agoI think you can sign up for a Cloudflare account and add the JS snippet they mention to your site. It’ll look like <script defer src='https://static.cloudflareinsights.com/beacon.min.js https://static.cloudflareinsights.com/beacon.min.js' data-cf-beacon='{"token": "$SITE_TOKEN"}'></script>
- hanniabu 5y agoAhhh okay thanks thought it had to be done by using cf as a cdn or something.
- nindalf 5y agoTotally understandable that you would think that. Cloudflare also provides server analytics if you use them as a CDN. That doesn't work for single page apps which don't necessarily make server requests when the URL changes, so they introduced a new product ("Web Analytics") to handle this.
- shubhamjain 6y agoIf you want more out of your access logs, you can also use two free IP Tools we have recently created at IPInfo. This maybe less useful if your aim is to understand the website usage, since we are only creating a report based on IPs. But it can help you understand the type of traffic you're getting. Summarize IPs takes in a list of IPs to create a report, like this[1], about location of IPs, if they are VPN/Proxy/Tor, and type of traffic. There's a limit of 1000 IPs here, but that can easily be circumvented by creating a free account. Map IPs plots them on a world map, like this [2]. Both of them are free and can be used from command line. We will also filter out the text to find IPs, so you can just pass raw access logs. [1]: https://ipinfo.io/summarize-ips/result?id=320d5003-a75f-42c3-98c1-c2558b8d0918 https://ipinfo.io/summarize-ips/result?id=320d5003-a75f-42c3... [2]: https://ipinfo.io/map/5b49e994-ec7a-416c-8140-b311364f6160 https://ipinfo.io/map/5b49e994-ec7a-416c-8140-b311364f6160
- 29athrowaway 6y agoIn these scripts... /usr/bin/awk '{print $7}' /home/logs/access_log.old | /usr/bin/sort | /usr/bin/uniq -c | /usr/bin/sort -r 1) If your PATH is well configured, you can omit the path to the binaries (i.e.: "/usr/bin/sort" becomes just "sort") 2) Instead of using awk, you can just use cut. # Delimit this line by the space character # and get the 7th field cut -d " " -f 7 /home/logs/access_log.old 3) Then, there's this: # This is not correct (lexicographic sort) sort | uniq -c | sort -r # This is correct (sort numerically) sort | uniq -c | sort -rn Why? because otherwise you get 1, 10, 2, 3, 4, 5, 7, 8, 9 instead of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Try it yourself: seq 1 10 | sort # lexicographic sort seq 1 10 | sort -n # numeric sort 4) If you are just interested in the top 10, you can add sort | uniq -c | sort -rn | head -n 10
- wiredfool 6y agoAwk is friendlier than cut in that it can easily collapse double spaces into one deliminator. Cut is much more painful to use.
- thegeekpirate 5y agoFull paths are probably used to avoid malicious $PATH issues (https://cwe.mitre.org/data/definitions/426.html https://cwe.mitre.org/data/definitions/426.html). It should just be `sort -rn | uniq`, no point in sorting the list twice.
- 29athrowaway 5y agoThat's incorrect. "uniq -c" adds a column with the count of unique elements. For the purposes of returning the most frequent elements, you can only do the final sort once that column has been added.
- dybber 6y agoAn alternative to rolling your own scripts is to use GoAccess for web lig analysis, which can both generate HTML statistics or you can open its ncurses based UI: https://goaccess.io/ https://goaccess.io/
- pugworthy 6y agoThis is basically a mid 90's approach to analytics. Glad it works for the author.
- je42 6y agorunning posthog, which is open source and can be selfhosted ;) stack requires a server instance + redis + postgres for the opensource deployment.
- 00deadbeef 6y agoSorry to hijack but I have a slightly related question: anyone know a good privacy-focussed analytics tool for apps that works nicely in React Native? Sort of like plausible.io but for apps
- marvinblum 6y agoOne of our [0] customers does something like that. He has developed a .NET framework [1] to send clicks to our API and monitor how his apps are used from our dashboard. [0] https://pirsch.io/ https://pirsch.io/ [1] https://github.com/Brouilles/pirsch-dotnet-sdk https://github.com/Brouilles/pirsch-dotnet-sdk
- feistypharit 6y agoCheckout country: https://count.ly/ https://count.ly/. can be pretty easily self host, supposed web and apps.
- jerrygoyal 6y agoIf you don't want to pay for privacy-focused analytics (in case it's some non-commercial project/personal blog) then recently launched cloudflare analytics is the way to go.
- FalconSensei 5y agoThat's what I did a couple days ago. Moved my blog from GH pages to cloudflare pages (and DNS). So far so good, and I'm seeing more views than before, when using goatcounter and seems more accurate.
- tarkin2 6y agoYou can also log requests and filter out obvious bots. Reading the logs, and using basic filters, gives you a decent idea very quickly. I suspect Google Analytics became so popular because people stopped running their own servers. And these days it's ridiculously easy to run your own server.
- bgrgndzz 5y agoGoogle Analytics has many more use cases than looking at vanity metrics to be replaced by a shell script. Something like https://hockeystack.com https://hockeystack.com gets closer.
- coderager32 5y agois this cookieless?
- bogwog 5y agoThe landing page claims that it doesn't, but their own site has this message: > Hey! We use cookies to track you through our website and use this data to optimize your experience! Thanks! So apparently, their own software isn't good enough to optimize their own site. At least, not without cookies.
- bgrgndzz 5y agoOther tracking scripts like facebook's require cookies unfortunately, so a banner is a must
- tux 5y agoI've been doing similar but with much more advanced bash script since about 2010 :-) I haven't used Google Analytics, after I learned how they use this data. I just never even thought about sharing this, because I thought no one would care. But I find like minded people on HN all the time ^_^ It's much cooler if you use this as a live script to watch for traffic. You can filter out only the pages you want to see for example. If there is any interest, I can share my scripts ;-)
- samstave 5y agoPlease share
- fidesomnes 5y ago2nd share request
- zeeshanqureshi 5y agoJoining the line. Please do share.
- tux 5y agoNothing special really, but you asked for it ^_^ https://github.com/tuxys/bash/blob/main/tail.sh https://github.com/tuxys/bash/blob/main/tail.sh This script is for nginx web server, but I'm sure with some modification it will work on other web servers like apache. EDIT: Later after some rewrite, I will also publish "stats.sh" a tiny script that generates daily/monthly/yearly static logs.
- zeeshanqureshi 5y agoThanks!
- jeffreygoesto 5y agohttps://i.pinimg.com/originals/18/6a/26/186a26ae732311033c6725e7fdf92710.jpg https://i.pinimg.com/originals/18/6a/26/186a26ae732311033c67...
- tootie 5y agoHalf the point of GA is tracking users across sites for ad targeting. It is simply not possible without 3rd party cookies. First-party analytics can always be done with log analyzers or even just a CNAME on a hosted tool. That's just not what most marketers want or need.
- deleted 5y ago[deleted]
- ohyash 5y agoReplaced GA with goat counter and called it a day. I'm not disappointed.
- FalconSensei 5y agoI used goatcounter for a while, it's very nice. Def. a great replacement for google analytics if you only want to see if there are visitors.
- bernardand 5y agoI’d actually argue that this would have detrimental effects. What’s stopping a company from visiting your site, seeing what’s popular, and stealing it all for themselves?
- zeeshanqureshi 5y agoLots of paid recomendations on this thread. There's also Umami[1] which is free, open source and self hosted. I've not used it yet, just have it bookmarked for my next project. [1] https://umami.is https://umami.is
- mikojp 5y agoYep, sounds like a lot of the authors found a cheap way to market their thing. I've been reading a bunch of the same type of Google 'replacements' promoting the same thing over and over. TBH, it gets annoying over time. HN should do something about this.
- bbkane 5y agoThis works well if you run a server, but for a static site, you need client-side JS. I recently found 2 free AND open source services (though I did donate): GoatCounter[0] and Counter.dev[1]. I've added them to my static site[2]. Naturally they disagree on my (low in any case) traffic, but they're still fun to look at :) [0] https://www-bbkane-com.goatcounter.com/ https://www-bbkane-com.goatcounter.com/ [1] https://counter.dev/dashboard.html?user=bbkane&token=MEzACRAJKeLPGaij https://counter.dev/dashboard.html?user=bbkane&token=MEzACRA... [2] https://www.bbkane.com/ https://www.bbkane.com/
- _wldu 5y agoThe blog is generated by Hugo and rsynced to NFS. It is a static website.