6 ms·

Tell me about it. Awhile back as a reason to learn some new things I setup a Tor hidden service/page crawler and a basic search engine. For troubleshooting purp

• belorn 11y ago
  I suspect this is where the myth of 99% of the Internet is made out of porn comes from. It is very easy to get misleading statistics, especially if you have small sample size or where a minority of users are creating the majority of the collected data points. Add to this the amount of bots out there, and "false" data generated by police, federal investigators and interest groups doing the very thing people pay them to do, and you would really have to design the data collecting system in such a way that it identified who is doing the searching and what their usage patterns are. For example, If they return every hour on the hour, such result would likely be made by a bot, or if every search is extremely specific, it could very likely be an investigator doing work in a specific case. If I did this kind of research, I would contact professional investigators in order to work together and to collect search patterns so to exclude that kind of traffic.
• gambiting 11y ago
  Bear in mind that at least some of the "14 year old sex" searches are done by 14 year old children who do not know any better.