17 ms·
Reliably Send an HTTP Request as a User Leaves a Page
- justWells 5y agoI genuinely don’t understand why you wouldn’t use a product like Heap or Amplitude to do this type of tracking
- bigyikes 5y agoEven if you use Amplitude, this could still be helpful information. I’m not sure what kind of APIs Amplitude provides, but we use it at my job. In my work, the details of Amplitude are abstracted away such that this information is still highly relevant. At a minimum, this kind of information is relevant to the people creating products like Heap or Amplitude.
- justWells 5y agoYou can just pass props to events like any other object data structure. I work at Heap and am just curious why you would still want to implement manually coded tracking.
- bigyikes 5y agoAt my company, the analytics APIs are provided by a platform team. I think there’s some utility in a company owning an abstraction away from a third party. Supposing if Amplitude (or Heap) were to ever go under, or if we were to want to change Analytics providers for whatever reason, it would be extremely helpful if the code didn’t depend directly on these third parties. On a smaller app, I would mostly likely directly use the Heap or Amplitude APIs. (Late reply, apologies)
- zmxz 5y agoUBlock blocks Amplitude. Nextdns blocks Amplitude. You get incomplete, inaccurate data. It's also fun and good for curiosity to do something on your own rathern than rely on a service and never think about what browsers can do.
- XCSme 5y agoIs this related to the article? How do you track time spend on page for a single page-visit using Heap? Don't they have to use a similar mechanism as described in the article?
- justWells 5y agoThat’s a tougher one. But you can just send the app time stamps and do a bit of math between events.
- bhaney 5y ago> Browser support is good, but not great. At the time of this writing, Firefox specifically doesn’t have it enabled by default Little sad that Firefox not supporting an API only downgrades that API's browser support to "good" instead of "unusable" these days
- danuker 5y agoThis is a feature built specifically for tracking. Firefox takes anti-tracking measures. Another example is trimming referrer URLs (just keeping the domain).
- bhaney 5y agoI wasn't commenting on Firefox's decision to not enable the API, I was commenting on how the author appears to consider Firefox to not be very significant.
- dspillett 5y agoThough according to caniuse, beacons is fully implemented which has much the same effect from a tracking PoV. As a side note, I was somehow completely unaware of that feature (and I don't think in a "heard about it and forgot" sort of way). Every day is a school day...
- Phelinofist 5y agoSo actually I would want to disable beacons and keepalive flagged fetch requests? Is this possible?
- rasz 5y agoinject navigator.sendBeacon = undefined before any JS has a chance to run. You can also trap "beforeunload" listeners, or just delete them document.body.onbeforeunload=null document.onbeforeunload=null
- zelon88 5y agoInteresting. I am curious what browsers were tested in this experiment, and if the different flavors behave differently. From my casual observation of Firefox (I use the network debugging tools in Firefox frequently) there isn't a column for priority. Firefox also does not show a status of pending, rather it just omits a pending status until a status becomes available.
- shlosky 5y agoI remember struggling with this a couple of years ago. Pretty incredible the myriad of reliable options in modern web development...
- jbverschoor 5y ago> I’ve needed to send off an HTTP request with some data to log when a user does something like navigate to a different page or submit a form. No need for that. Stop hoarding data!
- capableweb 5y agoWhat's wrong with doing something like this? If it contains PII, then I agree, the additional hassle of dealing with everything that comes with handling PII (like GDPR) becomes too much, easier to just don't do it. But if it doesn't contain PII, it can be useful to see how many people drop off a form VS submitting it for example.
- Moru 5y agoRecord how many opens the form and compare to how many actually submit the form?
- capableweb 5y agoYeah, I'm not saying this is a silver-bullet that works in all cases, but some use cases do work better with it. For example, if said form shows one question at a time without any backend requests being done between questions, you could do one last try to log which question the user dropped off at. But again, no silver-bullet.
- vntok 5y agoHow many users submitted the form vs how many users reached the form's page won't tell you anything about why they left or how long they struggled before submitting. A beacon sent for each click on links on the form's page will tell you how many users left for this or that other page, which is very useful to know and optimize by removing the link and/or bringing its contents to the page itself. A beacon sent for each form submission attempt will tell you how many users make mistakes while filling in the form, how good the error messages are, and whether the users tend to fix their input or leave. Bonus points for incrementing an "attempt" hidden field on each submit button click so that you can see how many attempts it takes for all fields to be filled correctly. Tracking what works and what doesn't can drive a lot of UX improvements, for the immediate and measurable benefit of the user.
- TheAceOfHearts 5y agoWhat use-cases are there for this feature other than tracking users? Anyway, the best solution is to pass any links through a redirect which is responsible for logging the visit. That's how Google tracks what search results get clicked on. And it doesn't require any JavaScript. I'm surprised they didn't even mention it as a possibility.
- capableweb 5y ago> What use-cases are there for this feature other than tracking users? Social services where users want to signal if they are online/offline Users who want to know how much time they've spent on a platform. Closing down sessions for things that can be expensive to just run "forever". Think VMs and similar that gets started when the user visits the page, that you want to turn off when they leave. The use cases are many, and there are also many other alternative solutions, this is obviously not the only one. But as always, all of them come with their own tradeoffs, so this technique might be the best for some of them.
- adhesive_wombat 5y agoYou will already need to handle the case when you don't get the last request, because there are more ways the user can just vanish on you: suspending the computer, killing the browser process, crashing the browser, turning off WiFi or going out of range, etc, etc. The best you can hope for is a "happy path" where you can clear up a little sooner in most cases, but you can't rely on it. If a user being on a webpage is holding open a resource, that might be an architectural issue.
- dspillett 5y ago> might be an architectural issue That definitely is an architectural issue. Unfortunately it is essentially a cache management matter (cache invalidation considers when something becomes useless/wasteful, as well as when it becomes incorrect) which is a hard problem to solve. There are legitimate circumstances where a user causes a long running prices to start, that it is wasteful to continue if incomplete when they vanish (because the result isn't going to be useful to another user, or even that same user if they return). While you can't catch everything (the user's machine losing power or all connectivity, etc.) you can catch a deliberate exit. And in the case of a long running process, if the user's exit was accidental it is more likely that they'll be back and having continued the process is useful.
- userbinator 5y agoFor those wondering --- despite the domain of the site seeming to imply so, this does not use CSS. On the other hand, the more I read about stuff like this, the more disenchanted I am with the "modern web". When something as seemingly innocent as a link can "stealthily" do something else when it's clicked, but otherwise looks exactly the same as a benign one (and moreover, behaves entirely as expected with JS off), it really brings into question whose interests such features are designed to serve.
- capableweb 5y agoEvery time I hear some argument against "the modern web", I always try to imagine if you could apply the same argument to anything in computing. And for this one, you could. Native applications might seem like you'll "submit the form when you click on the submit button, but something else entirely happens" because it's a general computing environment, just like the web. So in reality, it seems you're just against "modern" computing in general, not specifically the web. Because you can have that behavior anywhere, not just in the web platform.
- _moof 5y agoTake "the modern web" as a cultural descriptor then, not a technical one. There has been a huge shift in the last decade-plus. Software does not serve the user's interest anymore, and it sucks. Yes, this sort of thing has been technically possible across a range of platforms for a while. But the difference is that people weren't doing this kind of garbage before, and now it's everywhere. Casual gaming is an excellent example of how this has changed. The whole purpose of the software is different. It used to be that you would buy a game—a piece of software purpose-built to entertain you. Now you are no longer buying a game; you are buying a value extraction device that uses a game as bait. Software just isn't built for users anymore.
- Nextgrid 5y ago> It used to be that you would buy a game—a piece of software purpose-built to entertain you. Now you are no longer buying a game; you are buying a value extraction device that uses a game as bait. This applies to a lot of things besides games nowadays. A lot of software (and certain hardware - "smart" devices, etc) are built/sold at a loss with the primary objective being to generate "engagement" and/or collect personal data. The functionality of the device is purely there as a bait to get the mark to buy/adopt the software or device and would not be there if there was a way to get the marks to "engage" anyway.
- eloisius 5y ago> I swear. What happened here? Like why do things feel sluggish than software over 20 years ago? I just can't wrap my mind around it. We have SSDs and infinite CPU and RAM now compared to those days, yet there is always this feeling the computer is doing something more than I asked for behind the scenes. Even on websites, you can feel when you click on a link, it is doing something more. It's a very uncomfortable feeling. https://news.ycombinator.com/item?id=30875539 https://news.ycombinator.com/item?id=30875539
- OtomotO 5y agoGreed happened. It poisons whatever it touches. Mammon is the only thing that matters to a lot of primates.
- chiefalchemist 5y agoKeep in mind, "free" pairs with greed. The Market even pre-internet gravitated to free. I'm certainly not defending greed, but the general lack of market pushback doesn't discourage it either.
- atoav 5y agoI think it is more. I see people who make a simple website for a small family guesthouse voluntarily add all kinds of scripts that give them not a lot but send a ton of data to the big players. Typically the reason is, that they don't know how to do it any other way.
- hansworst 5y agoThere is no market pushback because free services hide negative externalities. So when users choose something that’s free over something they’d have to pay for, I’d say they’re generally unaware of the negative consequences (either to them or to the rest of society) because by claiming a service is free of charge, the provider is effectively hiding those negative effects, and making their revenue in some opaque way (e.g. advertising) There will never be market pushback as long as those externalities aren’t priced in. And Imho the only for that to happen is for governments to introduce regulation, e.g. by taxing the externalities so that “free” service providers are forced to be more upfront about the actual cost of their service.
- hans1729 5y ago[meta/feedback on the website] reader-view doesn't show the images, and for me the webdesign is too hostile to read the content without reader-mode. Please don't build websites that are attention-expensive to consume, everything about it is terrible. If your content is text, ship it on a neutral background, everything else is not acceptable.
- parasti 5y agoAll I see is black text on a white background? Hard to see how much more neutral can you get.
- hans1729 5y agohttps://imgur.com/a/kbGCf2k https://imgur.com/a/kbGCf2k
- ShortStretto 5y agoI fail to see the problem with this page, the large header is just announcing the topic, scrolling down there is no more distraction.
- hans1729 5y agoTo me, the semi-transparent header and border are absolutely unacceptable. Imagine opening a book and 20% of the page are decor, decor with text no less. In clear words: this fucks with my attention, and the reason is that this is design that serves no actual purpose. If you're building a house or a product, anything that's not necessary is detrimental to the users experience, I don't even understand what can be argued about this
- shakna 5y ago> Imagine opening a book and 20% of the page are decor, decor with text no less. Regularly happens in "book world". Chapter headings are often an entire page, and sometimes even a double spread. The average whitespace around headings on the three closest books at hand were all 1/3rd of the page, making that decor even greater than the 20% you're describing.
- someweirdperson 5y agoWhat happens if the user has more than one tab/window open an closes one? Weirdest case I've observed in the wild is a site that works perfectly without referer header, except for logout. As usual for anything requiring referer it fails without a user-readable message without referer. If referer is sent, the logout procedure switches through multiple redirects and some waiting while js is doing something, server backend probably, too. Some sites make weird assumptions about the network, the browser, and the user.
- Aeolun 5y agoNow try doing it when you close a tab/page. IE11 is the last browser that can reliably send a request before the tab closes (by using synchrous). I swear telling the business guys that we had no way to run an action when someone closed the page had them nearly running back to IE. And for that specific functionality, I find it hard to disagree. In our case, we lock entities when someone starts working on them, but now we have no way to unlock them if they fat-finger the close button, or press one too many backspaces.
- vesinisa 5y agoHow about if someone just forgets the document open on their browser? Or their network connection is down when they close the browser window? You eventually have to somehow deal with documents that are left locked, no matter what (google CAP theorem.) On a general level, locking is a subpar solution for distributed systems. You should try to build support for automatic resolution of (most) edit conflicts instead, and forego locking altogether. If that's not possible, try to make the locks smart, e.g. so they auto expire unless periodically renewed.
- Aeolun 5y agoThey do auto expire, but it’s not a great user experience to tell someone to ‘just wait an hour’.
- Rizz 5y agoYou can use web sockets for this sort of functionality, the socket is closed as soon as the browser is closed.
- Aeolun 5y agoYeah, I’d thought about that after reading a few comments here, but it feels a bit like using a shotgun to kill a fly.
- OtomotO 5y agoStuff like that is the reason I have disabled JS by default. Access to client side scripts on my machine has to be earned in trust. Most websites never get that far, because they are broken without js. Doesn't matter, saves me a lot of time ;)
- PUSH_AX 5y agoThis is an interesting take, I would assume at some point it would end up costing you time, because for all its faults, the modern web has some great tools (apps) that can improve your life in a number of ways. I’m also unsure how they’d earn your trust without giving them that essential unblocking js trust in the first place.
- beebeepka 5y agoI've been browsing without js for a decade. The alternative is to install blockers or live with ads. I have zero interest in "apps that can improve my life". People have different interests. Disclaimer: I write us for a living.
- XCSme 5y ago> Access to client side scripts on my machine has to be earned in trust I agree that it's all about trust. Why would you ever enter information on a website you don't trust? What extra information can be gathered using JS that can't be otherwise? Or what is the exact harm done if they send extra requests to a tracking API? My point is that once you access a specific website you consent with that website sending HTML/CSS/JS to your computer and executing it. The biggest problem those days is when that website is sending your information to other entities, 3rd parties, for other purposes than improving your experience. I think having "tracking" to detect errors, loading time issues and improving the user experience is perfectly reasonable if implemented in a proper way.
- OtomotO 5y ago> My point is that once you access a specific website you consent with that website sending HTML/CSS/JS to your computer and executing it Just HTML and CSS in my case.
- janci 5y agoWeird window.onbeforeunload was not mentioned
- openplatypus 5y agoProbably because this is discouraged. Check here: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon#sending_analytics_at_the_end_of_a_session https://developer.mozilla.org/en-US/docs/Web/API/Navigator/s... > However, this is extremely unreliable. In many situations, especially on mobile, the browser will not fire the unload, beforeunload, or pagehide events.
- XCSme 5y agoBecause it is synchronous/thread-blocking.
- ninkendo 5y agoWill any of these techniques inform the site owners that I close the tab the instant they put up a big “sign up for our mailing list” modal? And maybe that would help train them to not design sites that way? Maybe that could be one good outcome of such an API.
- panphora 5y agoI think popups on page load are more of a symptom of short-term, myopic thinking than anything else. This type of thinking is strongly encouraged by always trying to live up to quarterly projections - "Did Sales meet their targets?" Popups exchange long-term trust in your brand for short-term profits. Of course it's the case that if you pop a modal in front of everyone, your conversion stats will go up. But potential life-time customers who have faith in your brand and believe in your mission will lose trust in you quickly and probably never sign up in the first place. For me, it always comes down to the primary rule of UX: if you wouldn't do it IRL, don't do it online. You can ask for someone's email if they look like they're interested in something you're selling, but don't ask for it the moment they enter your store!
- jasonlotito 5y agoLong term trust doesn’t pay the bills. See every underfunded open source project. Think of it this way: if the site that you have long term trust for does this, it means the previous methods weren’t working. The ‘trust’ was broken. Would love to see your advice applied to your own life.
- jeltz 5y agoLong term trust pays the bills quite often, it all depends on what product or service your company is selling. And, yes, I apply this in my own projects, unless I am doing some ugly cash grab project.
- jasonlotito 5y agoNo. Long term trust never pays the bills. You are confusing long term trust with money. Sure, money might come from long term trust, but long term trust itself, literally, does not pay the bills.
- jzer0cool 5y agoIs there a reliable way when someone exits your site?
- arbuge 5y agoNavigator.sendBeacon() doesn't reliably follow 30x redirects in my experience. About 30% of the time, for no obvious reason, it stops following redirect chains after hitting the first redirect in the chain, even though it's supposed to follow those for analytics purposes.
- vbezhenar 5y agoI'm surprised that browsers nowadays include specific API for better user tracking. At first thought I don't like it. But at second thought I think that I like it. Issue is, webmasters will still track you, like it or not. But with those dedicated APIs tracking will be less annoying and those who can control their browsers, can disable that kind of tracking more easily (provided that they're even aware of those features). Anyway I don't like the direction web standards are evolving. Why have so many custom APIs and attributes? There's already fetch priority flag, that should be enough. This is exponential explosion of APIs. Modern API designers don't know about concept of orthogonal APIs (or I don't know something). Also <a ping attribute seems to track user even if he disabled JavaScript. Bad bad thing.
- jeroenhd 5y agoLuckily, you can disable anchor pings in Firefox: https://bgstack15.wordpress.com/2019/07/01/firefox-disable-a-ping/ https://bgstack15.wordpress.com/2019/07/01/firefox-disable-a... Oh wait, not on mobile, unless you're willing to deviate from stable releases or download a fork. I feel tempted to write an addon that will catch tracking like this and just repeat every ping a thousand times across the rest of the day, perhaps with some permutations of the data these sites are trying to collect. If I can't stop you from collecting my data, I'll make my data useless to you instead.
- kevindong 5y agoKeep in mind that a substantial portion of users now use ad blockers such that a lot of URLs used for analytics like this are blocked. Consequently, you can't actually expect to capture 100% of these analytics events nor even expect the percentage captured to stay the same over time since the filter lists are very regularly updated and users enable/disable different ad blockers over time. More broadly speaking, once you have sent a webpage to the user, you should not expect anything from the user's browser. They may or may not allow whatever arbitrary JS you have on the page. They may even intentionally give you bad data (e.g. hijack the payload to give you intentionally malformed data). edit: even more broadly speaking, there's additional reasons why you can't expect to receive these kinds of callbacks: consider what happens if a user loses connectivity between loading the page and them navigating away (e.g. their phone loses service because they went into an elevator before navigating away)
- dfas23 5y agoNot always - more sophisticated analytics will proxy these requests through the websites own domain.
- ForHackernews 5y ago> they may even intentionally give you bad data (e.g. hijack the payload to give you intentionally malformed data). Interesting. Are there tools we can install to send malicious payloads to surveillance companies?
- HWR_14 5y agoThere are plugins that do this, but I think trying to choke off the signal is more effective than just adding noise.
- deleted 5y ago[deleted]
- capableweb 5y ago> Keep in mind that a substantial portion of users now use ad blockers such that a lot of URLs used for analytics like this are blocked. How sure are we about this? I'm pretty sure it depends on which market specifically you're in, and the data I'm about to show is of course not perfect, but it seems that not so many users actually do use adblockers today. Although I don't know a single developer who doesn't, and in some web applications I'm running, the majority of users do use adblockers as they are focused on developers. Chrome is assumed to be the most popular browser (by a large margin last time I checked, so I won't bother to check again) and a quick search puts the user base around 2-3 billion users. Searching for "adblock" in the Chrome Web Store (https://chrome.google.com/webstore/search/adblock?hl=en&_category=extensions https://chrome.google.com/webstore/search/adblock?hl=en&_cat...) shows that the most popular adblocker has a user base of ~300,000 users. That makes 0.015% to 0.01% of Chrome users having the "AdBlock" extension installed. Not that substantial. If someone has some more accurate numbers than my slightly-educated guess, I'd be happy to be proven wrong. Edit: The above user base of the adblock extension is wrong. As Jabbles pointed out, I was seeing the number of reviews, not number of users. So instead, the page lists "10,000,000+ users" so we can assume the true number to be above that, but below "100,000,000+ users" users. That would put the amount of Chrome users using the "AdBlock" extension between 0.3% and 5% more of less. Closer to "substantial", but not sure if it would impact businesses choice regarding ads/tracking or not.
- taf2 5y agoThis is a really nice write up I found https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon https://developer.mozilla.org/en-US/docs/Web/API/Navigator/s... to be a good source for insight as well...
- z3t4 5y agohttps://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon https://developer.mozilla.org/en-US/docs/Web/API/Navigator/s...
- jtchang 5y agoWow there are a lot of new features in web browsers lately just targeting analytics and ads. Makes me wonder what other use cases have beacons as the solution.
- glitcher 5y agoWe have attempted to use sendBeacon as a last resort signal for releasing an exclusive lock granted to a user for editing a record. Only one user can be granted this at a time for a given record to avoid contention, but there are so many ways a user might abandon an edit-in-progress in web applications (close browser, inactivity, loss of network or power). I realize there are a lot of different strategies for addressing this situation, and I'd be curious to hear how others have solved this problem. In the specific case of using sendBeacon, it is a little disheartening that once a basic programming tool gets abused by the analytics/advertising/tracking behemoths then the ad blockers' only recourse at times is to disable it completely. I completely support the efforts made by the ad blockers, but also find it sad how legitimate uses of certain tools end up getting blocked in the process.
- Resonance1584 5y agoI'd just open a websocket and send regular ping messages, release the lock on disconnect
- flatline 5y agoThe industry has, for the most part, been migrating to live collaborative edits via CRDTs and the like rather than exclusive access with non-deterministic terminating conditions. You could argue that the whole field of distributed computing is geared toward solving this type of problem.
- scottlamb 5y agoMy first preference would be to avoid it by using optimistic concurrency control rather than locking. Maybe display a hint that someone else is editing but allow the user to ignore it.
- thomastay 5y agoAs others have mentioned, this is very well studied in the field of distributed computing, where servers can acquire a lock and then immediately disconnect due to a power failure. Other people have recommended CRDTs and Optimistic Concurrency control, which is great if you are designing an app from the ground up. If you have an existing large app that just needs locks, though, I recommend following the Chubby paper: https://research.google.com/archive/chubby-osdi06.pdf https://research.google.com/archive/chubby-osdi06.pdf Section 2.4 Locks addresses how Chubby thinks about locking in a distributed system: It is costly to introduce sequence numbers into all the interactions in an existing complex system. Instead, Chubby provides a means by which sequence numbers can be introduced into only those interactions that make use of locks. At any time, a lock holder may request a sequencer, an opaque byte-string that describes the state of the lock immediately after acquisition. It contains the name of the lock, the mode in which it was acquired (exclusive or shared), and the lock generation number. The client passes the sequencer to servers (such as file servers) if it expects the operation to be protected by the lock. The recipient server is expected to test whether the sequencer is still valid and has the appropriate mode; if not, it should reject the request. The validity of a sequencer can be checked against the server’s Chubby cache or, if the server does not wish to maintain a session with Chubby, against the most recent sequencer that the server has observed. The sequencer mechanism requires only the addition of a string to affected messages, and is easily explained to our developers. Although we find sequencers simple to use, important protocols evolve slowly. Chubby therefore provides an imperfect but easier mechanism to reduce the risk of delayed or re-ordered requests to servers that do not sup- port sequencers. If a client releases a lock in the normal way, it is immediately available for other clients to claim, as one would expect. However, if a lock becomes free because the holder has failed or become inaccessible, the lock server will prevent other clients from claiming the lock for a period called the lock-delay. Clients may specify any lock-delay up to some bound, currently one minute; this limit prevents a faulty client from making a lock (and thus some resource) unavailable for an arbitrarily long time. While imperfect, the lock-delay protects unmodified servers and clients from everyday problems caused by message delays and restarts
- LorenDB 5y agoMan, I had no idea that this stuff could happen. Having a built-in, non-disableable pingback method in the browser engine is scary stuff for privacy-oriented people. I don't know why more people don't know about this. Happily, I found and installed https://github.com/vijithassar/pingkiller https://github.com/vijithassar/pingkiller to help mitigate the `ping` attribute (in case my ad blocker isn't doing that already).
- nokya 5y agoWho needs this, except privacy toxic companies?