7 ms·

Thanks for the mention. Mojeek CEO here and happy to answer any questions

AnbeSivam 6y ago
Does your spider face issues with cloudflare mentioned by Gigablast founder here. https://www.gigablast.com/blog.html https://www.gigablast.com/blog.html
- ColinHayhurst 6y ago
  We are not aware that we have any problem like that. This might explain why GigaBlast has a problem: Because of bugs in the original Gigablast spidering code, the Findx crawler ended up on a blacklist in Project Honeypot as being “badly behaved” (fixed in our fork). That meant quite a bit of trouble for us because CDN providers, which are a very powerful hubs for internet traffic, put a lot of weight on this blacklist. Some of the most popular websites and services on the internet run through services like Cloudflare and other CDNs – so if you are in bad standing with them, suddenly a large part of the internet is not available, and we weren’t able index it. extract from: https://web.archive.org/web/20190921180535/https://privacore.github.io/ https://web.archive.org/web/20190921180535/https://privacore...
  - AnbeSivam 6y ago
    Thanks for that link, I haven't come across it before.
  - deepstack 6y ago
    any insight as to what is consider "Badly behaved crawlers"? Or is it something that you work out with CDN so they don't blacklist your ip?
    - ricardo81 6y ago
      Hello, I work on the technical side of Mojeek. Mojeek follows the robots.txt protocol so if a site doesn't want to be crawled by MojeekBot we respect that wish. There is also a generous crawl delay between pages on the same host. Generally a 'badly behaved bot' will ignore robots.txt or hit a site too hard with requests. Our bot uses a specific user agent which you can verify via DNS. https://www.mojeek.com/bot.html https://www.mojeek.com/bot.html
      - wyldfire 6y ago
        > There is also a generous crawl delay between pages on the same host. What's the order of magnitude of this delay? milliseconds? hundreds of milliseconds? seconds? I'm curious what's considered 'polite' in this realm and how the various parties come to form opinions on this.
        ricardo81 6y ago
        A minimum of 4 seconds.
        YeGoblynQueenne 6y ago
        I just had a look and there's a non-standard "crawl-delay directive" extension to robots.txt that can be used to ask a spider to take some time between page visits: User-agent: bingbot Allow : / Crawl-delay: 10 https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
        ricardo81 6y ago
        Hello, MojeekBot doesn't observe the crawl-delay directive but thanks for the reminder of it as it's beneficial for us to know if site owners require more grace between requests.
        YeGoblynQueenne 6y ago
        Hey. Good job with Mojeek. It seems the crawl-delay directive is not part of the robots.txt standard. It probably should be but that's not up to you!
  - mikkom 6y ago
    > fixed in our fork Does this mean your spider is a fork of Gigablast? Is there some additional interesting technical information about how your code/infrastructure is set up?
    - ColinHayhurst 6y ago
      No, we have our own spider
    - kevsim 6y ago
      The post was from Findx not Mojeek
    - ColinHayhurst 6y ago
      I realise this is not addressing your second question but you might find it interesting. Post below on server expansion one year ago. We are adding another 100 servers over Christmas and early new year. https://blog.mojeek.com/2019/12/100-server-build-and-install.html https://blog.mojeek.com/2019/12/100-server-build-and-install... We'll be writing about our tech stack in our next FAQs series; 3 of 4, this is 1 of 4: https://blog.mojeek.com/2020/11/frequently-asked-questions-about-mojeek-objectives-company-and-team.html https://blog.mojeek.com/2020/11/frequently-asked-questions-a...
Zhyl 6y ago
No need to go into details, but how are you for usage and profitability? I remember Gabe Weinberg mentioning on here that DDG became profitable pretty early on, so I'd be interested how easy it is to meet that threshold and how sustainable it is to create search competitors!
- mikkom 6y ago
  I think it's worth mentioning that DDG does not do it's own crawling, they are using Bing API and therefore don't need that much infrastrcuture (except the presentation layer). This means that they just need to sell enough advertising (which I think bing provides / forces them to use) so basically for DDG, more visitors = more revenue and breakeven means that the traffic pays for the costs they have.
  - Zhyl 6y ago
    Understandable, and also a reason why I'd be interested to know if the infra is already being covered or is being stood up with runway. If I were to create a privacy first search engine, I would likely not go general purpose and would instead try to focus on specific use cases first for specific audiences (that would be willing to pay). In practice this would look more like an extended version of DDG's instant answers than a Google-alike, but I think the world is convinced that general purpose web search is a) necessary and b) the only valid approach to finding answers to questions.
    - monsieurbanana 6y ago
      How did you come to 'b' conclusion? Many search engines already exists for specific use cases: - www.bailii.org for UK law related searches like court verdicts - tachiyomi.org for searching manga Etc. Generally speaking, search engines that cater for a specific use case are simply called "aggregators".
      - Zhyl 6y ago
        b) is an oversimplification, but I'd consider it to be a thought experiment whenever a story about Google or Duckduckgo comes up. You will always find: ><Search Engine> is <bad/good> and the results are way <worse/better> than <Another search engine>. ><Search Engine> is only a frontend for <Parent search engine> <but/and> does(n't) do their own indexing. >Results for <Topic (usually technical)> (in <country>)? on <search engine> are garbage. Search results for <purpose> are poor. What you will see as a response: >You should use <search engine> instead! It uses <different parent search> instead so the results are better! >I agree, I didn't like <search engine> so I had to go back to Google. What you won't see: >Nowadays I only use <search engine> for some search, most of the time I use <specific source(s)>. >Maybe you should use specific searches for specific purposes? The two that you cited are very valid, and I'm sure many people use specific sources for specific things (most doing it unknowingly via apps). The main thing seems to be that (on especially desktop) the single purpose all-encompassing web portal seems to be the way to find something. The phrase "Google it" is a catch all for 'do your research'.
deleted 6y ago
[deleted]
bogidon 6y ago
Wow this is a terrific answer to the exact question of "what's a good unbiased web craweler" I asked yesterday in another thread [0]. Do you disclose what goes into your ranking algorithm? I think having that transparency and perhaps being able to tweak some ranking parameters would be go a long way to being able to verify it's built in the best interests of the users. [0] https://news.ycombinator.com/item?id=25382703 https://news.ycombinator.com/item?id=25382703
- ColinHayhurst 6y ago
  Agreed and yes, we will.
  - bdibs 6y ago
    Are you worried at all about sites abusing the transparency to rank higher, given they would know what to focus on to do so?
    - ColinHayhurst 6y ago
      That's not a challenge right now, but yes could become more of one. A balance in ranking for those seeking information and those looking to promote content will always be an issue; at any level of transparency. Done right, more transparency can benefit seekers too.
Groxx 6y ago
Odd / possibly concerning error fwiw: searching for "respondsToSelector:, performSelector:," (with quotes) gives me a 403 Forbidden error.
- ricardo81 6y ago
  Thank you for the feedback, it's very much appreciated. This is a case of our bot blocking being overly aggressive, where bots tend to use quotes. We will rectify this in the coming days, thanks again.
1024core 6y ago
How to you handle spam/fake sites?
- ColinHayhurst 6y ago
  That's a big question and the details would require a long answer. A short one is that it's mostly algorithmic aided by human feedback to improve; manual intervention is rare.
stanislavb 6y ago
Hey mate, first, kudos for all the work you've put into developing Mojeek up to this level. That seems like an enormous achievement for such a small team as yours. Then, I'm wondering do you guys have the notion of "product brands" or something like that? For example, when I search for "SaaSHub" - a product (I work on) that has been online since 2014 and have quite a few mentions around the web, the first result is some "random" Wordpress theme. i.e. if I'm searching for a particular product, I'd expect the product homepage to be the first result.
- ColinHayhurst 6y ago
  Thank you for your kind words. The feedback is appreciated and your example is one for a generic challenge that we are actively working on. I note that your site shows as link #2 so someone looking for your site/brand should see it; but still your point stands.
yuhong 6y ago
I wonder if you are using SSDs instead of DRAM for the search index as Google did in 1998. Actually, a more detailed comparison of Mojeek with Google in 1998 would be interesting.
lsiebert 6y ago
Hi Colin, Given that you run your own spider, presumably you provided it one or more urls to get started spidering. I'm curious what those were.
- ColinHayhurst 6y ago
  It was originally seeded with URLs from DMOZ, when Marc first stared working on Mojeek.

Does your spider face issues with cloudflare mentioned by Gigablast founder here. https://www.gigablast.com/blog.html https://www.gigablast.com/blog.html

ColinHayhurst 6y ago
We are not aware that we have any problem like that. This might explain why GigaBlast has a problem: Because of bugs in the original Gigablast spidering code, the Findx crawler ended up on a blacklist in Project Honeypot as being “badly behaved” (fixed in our fork). That meant quite a bit of trouble for us because CDN providers, which are a very powerful hubs for internet traffic, put a lot of weight on this blacklist. Some of the most popular websites and services on the internet run through services like Cloudflare and other CDNs – so if you are in bad standing with them, suddenly a large part of the internet is not available, and we weren’t able index it. extract from: https://web.archive.org/web/20190921180535/https://privacore.github.io/ https://web.archive.org/web/20190921180535/https://privacore...
- AnbeSivam 6y ago
  Thanks for that link, I haven't come across it before.
- deepstack 6y ago
  any insight as to what is consider "Badly behaved crawlers"? Or is it something that you work out with CDN so they don't blacklist your ip?
  - ricardo81 6y ago
    Hello, I work on the technical side of Mojeek. Mojeek follows the robots.txt protocol so if a site doesn't want to be crawled by MojeekBot we respect that wish. There is also a generous crawl delay between pages on the same host. Generally a 'badly behaved bot' will ignore robots.txt or hit a site too hard with requests. Our bot uses a specific user agent which you can verify via DNS. https://www.mojeek.com/bot.html https://www.mojeek.com/bot.html
    - wyldfire 6y ago
      > There is also a generous crawl delay between pages on the same host. What's the order of magnitude of this delay? milliseconds? hundreds of milliseconds? seconds? I'm curious what's considered 'polite' in this realm and how the various parties come to form opinions on this.
      - ricardo81 6y ago
        A minimum of 4 seconds.
        YeGoblynQueenne 6y ago
        I just had a look and there's a non-standard "crawl-delay directive" extension to robots.txt that can be used to ask a spider to take some time between page visits: User-agent: bingbot Allow : / Crawl-delay: 10 https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
        ricardo81 6y ago
        Hello, MojeekBot doesn't observe the crawl-delay directive but thanks for the reminder of it as it's beneficial for us to know if site owners require more grace between requests.
        YeGoblynQueenne 6y ago
        Hey. Good job with Mojeek. It seems the crawl-delay directive is not part of the robots.txt standard. It probably should be but that's not up to you!
- mikkom 6y ago
  > fixed in our fork Does this mean your spider is a fork of Gigablast? Is there some additional interesting technical information about how your code/infrastructure is set up?
  - ColinHayhurst 6y ago
    No, we have our own spider
  - kevsim 6y ago
    The post was from Findx not Mojeek
  - ColinHayhurst 6y ago
    I realise this is not addressing your second question but you might find it interesting. Post below on server expansion one year ago. We are adding another 100 servers over Christmas and early new year. https://blog.mojeek.com/2019/12/100-server-build-and-install.html https://blog.mojeek.com/2019/12/100-server-build-and-install... We'll be writing about our tech stack in our next FAQs series; 3 of 4, this is 1 of 4: https://blog.mojeek.com/2020/11/frequently-asked-questions-about-mojeek-objectives-company-and-team.html https://blog.mojeek.com/2020/11/frequently-asked-questions-a...

Zhyl 6y ago

No need to go into details, but how are you for usage and profitability? I remember Gabe Weinberg mentioning on here that DDG became profitable pretty early on, so I'd be interested how easy it is to meet that threshold and how sustainable it is to create search competitors!

mikkom 6y ago
I think it's worth mentioning that DDG does not do it's own crawling, they are using Bing API and therefore don't need that much infrastrcuture (except the presentation layer). This means that they just need to sell enough advertising (which I think bing provides / forces them to use) so basically for DDG, more visitors = more revenue and breakeven means that the traffic pays for the costs they have.
- Zhyl 6y ago
  Understandable, and also a reason why I'd be interested to know if the infra is already being covered or is being stood up with runway. If I were to create a privacy first search engine, I would likely not go general purpose and would instead try to focus on specific use cases first for specific audiences (that would be willing to pay). In practice this would look more like an extended version of DDG's instant answers than a Google-alike, but I think the world is convinced that general purpose web search is a) necessary and b) the only valid approach to finding answers to questions.
  - monsieurbanana 6y ago
    How did you come to 'b' conclusion? Many search engines already exists for specific use cases: - www.bailii.org for UK law related searches like court verdicts - tachiyomi.org for searching manga Etc. Generally speaking, search engines that cater for a specific use case are simply called "aggregators".
    - Zhyl 6y ago
      b) is an oversimplification, but I'd consider it to be a thought experiment whenever a story about Google or Duckduckgo comes up. You will always find: ><Search Engine> is <bad/good> and the results are way <worse/better> than <Another search engine>. ><Search Engine> is only a frontend for <Parent search engine> <but/and> does(n't) do their own indexing. >Results for <Topic (usually technical)> (in <country>)? on <search engine> are garbage. Search results for <purpose> are poor. What you will see as a response: >You should use <search engine> instead! It uses <different parent search> instead so the results are better! >I agree, I didn't like <search engine> so I had to go back to Google. What you won't see: >Nowadays I only use <search engine> for some search, most of the time I use <specific source(s)>. >Maybe you should use specific searches for specific purposes? The two that you cited are very valid, and I'm sure many people use specific sources for specific things (most doing it unknowingly via apps). The main thing seems to be that (on especially desktop) the single purpose all-encompassing web portal seems to be the way to find something. The phrase "Google it" is a catch all for 'do your research'.

deleted 6y ago

[deleted]

bogidon 6y ago

Wow this is a terrific answer to the exact question of "what's a good unbiased web craweler" I asked yesterday in another thread [0]. Do you disclose what goes into your ranking algorithm? I think having that transparency and perhaps being able to tweak some ranking parameters would be go a long way to being able to verify it's built in the best interests of the users. [0] https://news.ycombinator.com/item?id=25382703 https://news.ycombinator.com/item?id=25382703

ColinHayhurst 6y ago
Agreed and yes, we will.
- bdibs 6y ago
  Are you worried at all about sites abusing the transparency to rank higher, given they would know what to focus on to do so?
  - ColinHayhurst 6y ago
    That's not a challenge right now, but yes could become more of one. A balance in ranking for those seeking information and those looking to promote content will always be an issue; at any level of transparency. Done right, more transparency can benefit seekers too.

Groxx 6y ago

Odd / possibly concerning error fwiw: searching for "respondsToSelector:, performSelector:," (with quotes) gives me a 403 Forbidden error.

ricardo81 6y ago
Thank you for the feedback, it's very much appreciated. This is a case of our bot blocking being overly aggressive, where bots tend to use quotes. We will rectify this in the coming days, thanks again.

1024core 6y ago

How to you handle spam/fake sites?

ColinHayhurst 6y ago
That's a big question and the details would require a long answer. A short one is that it's mostly algorithmic aided by human feedback to improve; manual intervention is rare.

stanislavb 6y ago

Hey mate, first, kudos for all the work you've put into developing Mojeek up to this level. That seems like an enormous achievement for such a small team as yours. Then, I'm wondering do you guys have the notion of "product brands" or something like that? For example, when I search for "SaaSHub" - a product (I work on) that has been online since 2014 and have quite a few mentions around the web, the first result is some "random" Wordpress theme. i.e. if I'm searching for a particular product, I'd expect the product homepage to be the first result.

ColinHayhurst 6y ago
Thank you for your kind words. The feedback is appreciated and your example is one for a generic challenge that we are actively working on. I note that your site shows as link #2 so someone looking for your site/brand should see it; but still your point stands.

yuhong 6y ago

I wonder if you are using SSDs instead of DRAM for the search index as Google did in 1998. Actually, a more detailed comparison of Mojeek with Google in 1998 would be interesting.

lsiebert 6y ago

Hi Colin, Given that you run your own spider, presumably you provided it one or more urls to get started spidering. I'm curious what those were.

ColinHayhurst 6y ago
It was originally seeded with URLs from DMOZ, when Marc first stared working on Mojeek.