I’d like to see a comparison of the server log information with the hit endpoint information: my feeling is that the reasons for separating it don’t really hold

I’d like to see a comparison of the server log information with the hit endpoint information: my feeling is that the reasons for separating it don’t really hold water, and that the initial request server logs could fairly easily be filtered to acceptable quality levels, obviating the subsequent request. The basic server logs include declared bots, undeclared bots pretending to use browsers, undeclared bots actually using browsers, and humans. The hit endpoint logs will exclude almost all declared bots, almost all undeclared bots pretending to use browsers, and some humans, but will retain a few undeclared bots that search for and load subresources, and almost all humans. About undeclared bots that actually use browsers, I’m uncertain as I haven’t inspected how they are typically driven and what their initial mouse cursor state is: if it’s placed within the document it’ll trigger, but if it’s not controlled it’ll probably be outside the document. (Edit: actually, I hadn’t considered that bearblog caps the body element’s width and uses margin, so if the mouse cursor is not in the main column it won’t trigger. My feeling is that this will get rid of almost all undeclared bots using browsers, but significantly undercount users with large screens.) But my experience is that reasonably simple heuristics do a pretty good job of filtering out the bots the hit endpoint also excludes. • Declared bots: the filtration technique can be ported as-is. • Undeclared bots pretending to use browsers: that’s a behavioural matter, but when I did a little probing of this some years ago, I found that a great many of them were using unrealistic user-agent strings, either visibly wonky or impossible or just corresponding to browsers more than a year old (which almost no real users are using). I suspect you could get rid of the vast majority of them reasonably easily, though it might require occasional maintenance (you could do things like estimate the browser’s age based on their current version number and release cadence, with the caveat that it may slowly drift and should be checked every few years) and will certainly exclude a very few humans. • Undeclared bots actually using browsers: this depends on the unknown I declared, whether they position their mice in the document area. But my suspicion is that these simply aren’t worth worrying about because they’re not enough to notably skew things. Actually using browsers is expensive, people avoid it where possible. And on the matter of humans, it’s worth clarifying that the hit endpoint is worse in some ways, and honestly quite risky: • Some humans will use environments that can’t trigger the extra hit request (e.g. text-mode browsers, or using some service that fetches and presents content in a different way); • Some humans will behave in ways that don’t trigger the extra hit request (e.g. keyboard-only with no mouse movement, or loading then going offline); • Some humans will block the extra hit request; and if you upset the wrong people or potentially even become too popular, it’ll make its way into a popular content blocker list and significant fractions of your human base will block it. This, in my opinion, is the biggest risk. • There’s also the risk that at some point browsers might prefetch such resources to minimise the privacy leak. (Some email clients have done this at times, and browsers have wrestled from time to time with related privacy leaks, which have led to the hobbling of what properties :visited can affect, and other mitigations of clickjacking. I think it conceivable that such a thing could be changed, though I doubt it will happen and there would be plenty of notice if it ever did.) But there’s a deeper question to it: if you don’t exclude some bots; or if the URL pattern gets on a popular content filter list: does it matter? Does it skew the ratios of your results significantly? (Absolute numbers have never been particularly meaningful or comparable between services or sources: you can only meaningfully compare numbers from within a source.) My feeling is that after filtering out most of the bots in fairly straightforward ways, the data that remains is likely to be of similar enough quality to the hit endpoint technique: both will be overcounting in some areas and undercounting in others, but I expect both to be Good Enough, at which point I prefer the simplicity of not having a separate endpoint. (I think I’ve presented a fairly balanced view of the facts and the risks of both approaches, and invite correction in any point. Understand also that I’ve never tried doing this kind of analysis in any detail, and what examination and such I have done was almost all 5–8 years ago, so there’s a distinct possibility that my feelings are just way off base.)