8 ms·
We built a page logger. Any time a page was loaded, we'd keep track of it. They reinvented the access log? By hand? Are people really that detached from how
by seiji 14y ago
We built a page logger. Any time a page was loaded, we'd keep track of it.
They reinvented the access log? By hand? Are people really that detached from how servers work?
- hobonumber1 14y agoThis is besides the point.
- seiji 14y agoIs it though? It seems they spent a lot of time throwing more and more javascript at the problem until they realized they couldn't figure out what was happening. If they had real logging from the beginning, it would have unconfused them much sooner.
- spaghetti 14y agoThey were taking orthogonal measurements to see if the pattern would appear over and over. Also it sounds like each successive measurement technique was at a lower level than the previous technique. Sounds pretty thorough. I'd love to see some actual data though.
- crygin 14y agoHow would you use the access log to determine if client-side javascript was being executed?
- mike-cardwell 14y agoCheck if it's even being fetched for a start. Assuming it's not inline.
- alttab 14y agoBrowser-cache my friend. You'll need to poll the server on page load (or even better in this case, after you load the ad)
- mapgrep 14y agoFor the page in question, count total hits in the server access log and subtract total hits reported by the Javascript analytics software.
- waivej 14y agoUse a javascript function to poll the server for something.
- SoftwarePatent 14y agothanks for coining the word "unconfused"
- wpietri 14y agoThis could be a businessperson's interpretation of what happened. Or they might have indeed reinvented it. Kids these days and their Heroku! Back in my day we'd process each HTTP request by hand. And we'd do it uphill both ways in the snow.
- vampirechicken 14y agousing awk.
- jlgreco 14y agoHey now, awk is simply lovely. It's ridiculous how popular it is to slight it these days.
- dredmorbius 14y agoIndeed. Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon. Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures). It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.
- vampirechicken 14y agoHow does 'using awk' count as a slight? It was the proper add-on to the previous comment kids today don't learn awk. Becuase they have Perl, Ruby, and Python which are supersets of the unix command line tool set. Awk was the king in the years before Perl. I never really learned to use cut, so I sitll use awk for one-liners the deal with columns in delimited-data text-files.
- jlgreco 14y agoSorry, I read it as a slight because it came right after "uphill both ways in the snow".
- tlrobinson 14y agoI assumed this just meant they built something on top of access logs that told them what percentage of page loads also loaded JavaScript. I don't know about your webserver, but mine doesn't tell me this kind of thing out of the box. Anyway, it's pretty irrelevant whether they added logging in their app (one line of code?) or enabled their webserver's built in logging.
- brador 14y agoSurprised no ones mentioned the noscript plugin. Since the user is on facebook they'd have facebook unblocked, but noscript still blocking everything else. How would you discern a noscript user from bot?
- Zak 14y agoYou wouldn't, but if the vast majority of your ad click visitors have javascript disabled, and only a tiny percentage of your other visitors do, what conclusion would you draw?
- takluyver 14y agoYou don't, but it's very unlikely that there were enough noscript users to match the stats. From the post: "...in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook."
- brador 14y agoIt's a different marketing block. For example, target something for the Reddit.com users and you'll see 90%+ block ads. With facebook granular targetting you could very specifically end up with a target segment with a large proportion of noscript type plugin users. Without more details on their campaign I would not be so quick to place blame purely on bots.
- takluyver 14y agoBut this is about users who click on ads on Facebook. The sort of users who have javascript disabled are going to self-select out of that group pretty strongly. And even among the most technically literate groups, I'd be amazed if you saw 80% use of noscript. So many sites require javascript that it's easier to browse with it enabled.
- mattmanser 14y agoAccess logs can't record if javascript is enabled. There are other things I would be interested in if I were doing this too, I'd check the browser's TZ, check if flash was enabled, etc. and do it across sessions. Depends how far they went profiling their visitors, it clearly says in the article they were tracking if js was enabled, so yes, you'd need to roll something or another by hand. GA relies solely on js for example.
- cientifico 14y agoeasy: < script > (new Image()).src = document.location.href + "Hey, i have javascript < / script > And then compare in the log.
- wildtype 14y agogenius
- Someone 14y agoThe way I understand this is that they want to log who accesses their pages on FaceBook's servers. I am not familiar with how things work on FaceBook, but I would think that FaceBook users do not have access to the logs that FaceBook's servers write.