n1xis10t

6 ms·

61.
by n1xis10t 10mo ago
This would require new laws though, wouldn’t it?
62.
by n1xis10t 10mo ago
Apparently Google publishes their crawler’s IPs, this was mentioned somewhere in this same thread
63.
by n1xis10t 10mo ago
“as acknowledged by OP”: that’s funny, if you hadn’t added that to your comment I was about to point it out
64.
by n1xis10t 10mo ago
At this point it seems like the problem isn’t internet bandwidth, but just expensive for a server to handle all the requests because it has to process them. Does that seem correct?
65.
by n1xis10t 10mo ago
See this thread from yesterday or so: https://news.ycombinator.com/item?id=46302496#46306025
66.
by n1xis10t 10mo ago
Gotcha. Which search engine(s) do you source results from?
67.
by n1xis10t 10mo ago
It would be interesting if someone made a map that depicts the locations of the ip addresses that are sending so many requests, over the course of a day maybe.
68.
by n1xis10t 10mo ago
Interesting. When it was just normal search engines I didn’t hear of people having this problem, so this either means that there are a bunch of people pretending to be bing google and yandex, or those companies have gotten a lot more aggres
69.
by n1xis10t 10mo ago
Good to know thanks
70.
by n1xis10t 10mo ago
Fair point
71.
by n1xis10t 10mo ago
Really? I haven’t started to see that yet. Weird
72.
by n1xis10t 10mo ago
Crazy
73.
by n1xis10t 10mo ago
Imagine if all this scraping was going into a search engine with a massive index, or a bunch of smaller search engines that a meta-search engine could be made for. This’d be a lot more cool in that case
74.
by n1xis10t 10mo ago
Here’s one from yesterday: https://news.ycombinator.com/item?id=46302496#46306025
75.
by n1xis10t 10mo ago
Seems like, but there are tons of things that forge request headers all the time, and I don’t think I’ve heard of anyone getting in legal trouble for it. Now I think most of these are scrapers pretending to be browsers, so it might be diffe
76.
by n1xis10t 10mo ago
Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing? If so, I suppose it’s like those magazines that say ”free cd”.
77.
by n1xis10t 10mo ago
Interesting note, thank you.
78.
by n1xis10t 10mo ago
Yes probably bad. Also smooth criminals.
79.
by n1xis10t 10mo ago
You can’t deny that it’s fun though. Personally I generally feel like more people should be coming up with creative (if not entirely necessary) solutions to problems.
80.
by n1xis10t 10mo ago
Oh good idea!
81.
by n1xis10t 10mo ago
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but
82.
by n1xis10t 10mo ago
*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!
83.
by n1xis10t 10mo ago
Yeah I guess I don’t know if you can sue someone for using your headers, would be interesting to see how that goes.
84.
by n1xis10t 10mo ago
Is it based on your own index? If so how many pages are in the index?
85.
by n1xis10t 10mo ago
Nice! Reminds me of “Piracy as Proof of Personhood”. If you want to read that one go to Paged Out magazine (at https://pagedout.institute/ ), navigate to issue #7, and flip to page 9. I wonder if this will start making porn
86.
by n1xis10t 10mo ago
A cronjob, that’s cool. A little rudeness is fine, you could probably say I’ve been a little rude. Thank for making a cool product!
87.
by n1xis10t 10mo ago
I noticed that, but that page makes it sound like it can only be unbranded if you pay for the commercial version. It looks like Anubis is open source though, so I suppose you might be able to download the source and switch out images for yo
88.
by n1xis10t 10mo ago
Another option to consider (which marginalia-search.com uses) is Anubis (anubis.techaro.lol). The operator of Marginalia told me that he was getting lots of people spamming the same queries over and over, which he thought might be them tryi
89.
by n1xis10t 11mo ago
Pretty cool, nice job. What’s “buh” though?
90.
by n1xis10t 11mo ago
Yes, there are lots more results now, cool!

More ›