4 ms·

I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling https://github.com/mwmbl/mwmbl#crawling Also,

I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling https://github.com/mwmbl/mwmbl#crawling Also, your own posting appears to be missing from the index: https://mwmbl.org/?q=mwmbl+ycombinator https://mwmbl.org/?q=mwmbl+ycombinator (and, yes, another vote for changing the domain name; you can have a quirky project name, but if I can't remember the cat-walking-on-keyboard domain, I'm not going to use it)

marc_abonce 3y ago
> We now have a distributed crawler that runs on our volunteers' machines! If you have Firefox you can help out by installing our extension. This is a very interesting idea that other search engines have tried before. Actually, the Brave search engine is built over Cliqz[6] that implemented this same idea but *without* the user's consent. Copy pasting from an old comment I made about this "human web" crawler idea: Both PeARS[1] and Cliqz[2] tried to do that. Both got direct support from Mozilla[3][4] but it looks like neither really kicked off. PeARS was meant to be installed voluntarily by users who would then choose to share their indexes only to those they personally trusted, so the idea is very privacy conscious but also very hard to scale. Cliqz, on the other hand, apparently tried to work around that issue by having their add-on bundled by default in some Firefox installations[5] which was obviously very controversial because of its privacy and user consent implications. I still think the idea has potential, though, even if it's in a more limited scope. [1] https://github.com/PeARSearch/PeARS-orchard https://github.com/PeARSearch/PeARS-orchard [2] https://cliqz.com/en/whycliqz/human-web https://cliqz.com/en/whycliqz/human-web [3] https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-385000-to-open-source-projects-as-part-of-moss-mission-partners-program/ https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-3... [4] https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-strategic-investment-in-cliqz-to-enable-privacy-focused-search-innovation/ https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-s... [5] https://www.zdnet.com/article/firefox-tests-cliqz-engine-which-slurps-user-browsing-data/ https://www.zdnet.com/article/firefox-tests-cliqz-engine-whi... [6] https://www.theregister.com/2021/03/03/brave_buys_a_search_engine/ https://www.theregister.com/2021/03/03/brave_buys_a_search_e...
- Proven 3y ago
  [dead]
- daoudc 3y ago
  Thanks, I didn't know this history! We don't use any user data when crawling, just bandwidth and compute. We tell the extension what to crawl.
eviks 3y ago
You don't need to remember it, just bookmark and tag however you like (it's anyway a waste of keystrokes to manual type the full domain for such a frequently used site like a search engine)
- DandyDev 3y ago
  That is not how a large part of the citizens on the internet works. Hell, a not insignificant number of people will still "search" for Google in their address bar before they get to the actual googling
  - eviks 3y ago
    Except I'm not talking to a large part of the citizens, but to a single one. Do you type 'google.com' in your address bar to search?
    - tux1968 3y ago
      You're being argumentative for no good reason. He was suggesting a name change to improve the likelihood of a large userbase, not a change for his own convenience.
- Brian_K_White 3y ago
  This is 100% wrong. One's normal phone and laptop is actually only a fraction of uses, and even one's normal device isn't just one thing that needs to be done one time in grade school and then set for life. It's a dozen different things, and they are all perpetually rotating, and most people are not highly optimized with profiles they actually export and import. This idea is great but it's going absolutely nowhere without a better understanding of actual humans.
  - eviks 3y ago
    Name at least half of those perpetually rotating things and quantify "only a fraction" I'm an actual human, I use alternative search engines, I don't memorize their full names, and the only thing perpertually rotating is the planet
robinduckett 3y ago
I’m from Wales and it almost seems like a transliteration of the word “Mumble” - actual translation is “mwmial”
- dmurray 3y ago
  Welsh has the unfortunate combination of being unfamiliar to most English speakers, and not exotic enough to score diversity points.
  - melx 3y ago
    On that topic I love the Welsh-English encounter of civil servants thinking they understood each other[0] [0] http://news.bbc.co.uk/1/hi/7702913.stm http://news.bbc.co.uk/1/hi/7702913.stm
    - zx8080 3y ago
      It surprisingly resembles Swedish language a bit.
    - mdaniel 3y ago
      Heh, I look forward to the future version of "en tant que modèle linguistique, je ne peux pas traduire ce texte" or in this specific case "fel model iaith, ni allaf gyfieithu'r testun hwnnw"
- toastal 3y ago
  But Mumble brings back fond memories https://www.mumble.info/ https://www.mumble.info/
- remram 3y ago
  So not only is it based on an obscure word in Welsh, but it's not even spelled correctly?
- daoudc 3y ago
  Yeah, pretty much, I named it after Mumbles or Mwmbwls where I live, but it's also a play on the word mumble.

marc_abonce 3y ago

> We now have a distributed crawler that runs on our volunteers' machines! If you have Firefox you can help out by installing our extension. This is a very interesting idea that other search engines have tried before. Actually, the Brave search engine is built over Cliqz[6] that implemented this same idea but *without* the user's consent. Copy pasting from an old comment I made about this "human web" crawler idea: Both PeARS[1] and Cliqz[2] tried to do that. Both got direct support from Mozilla[3][4] but it looks like neither really kicked off. PeARS was meant to be installed voluntarily by users who would then choose to share their indexes only to those they personally trusted, so the idea is very privacy conscious but also very hard to scale. Cliqz, on the other hand, apparently tried to work around that issue by having their add-on bundled by default in some Firefox installations[5] which was obviously very controversial because of its privacy and user consent implications. I still think the idea has potential, though, even if it's in a more limited scope. [1] https://github.com/PeARSearch/PeARS-orchard https://github.com/PeARSearch/PeARS-orchard [2] https://cliqz.com/en/whycliqz/human-web https://cliqz.com/en/whycliqz/human-web [3] https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-385000-to-open-source-projects-as-part-of-moss-mission-partners-program/ https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-3... [4] https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-strategic-investment-in-cliqz-to-enable-privacy-focused-search-innovation/ https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-s... [5] https://www.zdnet.com/article/firefox-tests-cliqz-engine-which-slurps-user-browsing-data/ https://www.zdnet.com/article/firefox-tests-cliqz-engine-whi... [6] https://www.theregister.com/2021/03/03/brave_buys_a_search_engine/ https://www.theregister.com/2021/03/03/brave_buys_a_search_e...

Proven 3y ago
[dead]
daoudc 3y ago
Thanks, I didn't know this history! We don't use any user data when crawling, just bandwidth and compute. We tell the extension what to crawl.

eviks 3y ago

You don't need to remember it, just bookmark and tag however you like (it's anyway a waste of keystrokes to manual type the full domain for such a frequently used site like a search engine)

DandyDev 3y ago
That is not how a large part of the citizens on the internet works. Hell, a not insignificant number of people will still "search" for Google in their address bar before they get to the actual googling
- eviks 3y ago
  Except I'm not talking to a large part of the citizens, but to a single one. Do you type 'google.com' in your address bar to search?
  - tux1968 3y ago
    You're being argumentative for no good reason. He was suggesting a name change to improve the likelihood of a large userbase, not a change for his own convenience.
Brian_K_White 3y ago
This is 100% wrong. One's normal phone and laptop is actually only a fraction of uses, and even one's normal device isn't just one thing that needs to be done one time in grade school and then set for life. It's a dozen different things, and they are all perpetually rotating, and most people are not highly optimized with profiles they actually export and import. This idea is great but it's going absolutely nowhere without a better understanding of actual humans.
- eviks 3y ago
  Name at least half of those perpetually rotating things and quantify "only a fraction" I'm an actual human, I use alternative search engines, I don't memorize their full names, and the only thing perpertually rotating is the planet

robinduckett 3y ago

I’m from Wales and it almost seems like a transliteration of the word “Mumble” - actual translation is “mwmial”

dmurray 3y ago
Welsh has the unfortunate combination of being unfamiliar to most English speakers, and not exotic enough to score diversity points.
- melx 3y ago
  On that topic I love the Welsh-English encounter of civil servants thinking they understood each other[0] [0] http://news.bbc.co.uk/1/hi/7702913.stm http://news.bbc.co.uk/1/hi/7702913.stm
  - zx8080 3y ago
    It surprisingly resembles Swedish language a bit.
  - mdaniel 3y ago
    Heh, I look forward to the future version of "en tant que modèle linguistique, je ne peux pas traduire ce texte" or in this specific case "fel model iaith, ni allaf gyfieithu'r testun hwnnw"
toastal 3y ago
But Mumble brings back fond memories https://www.mumble.info/ https://www.mumble.info/
remram 3y ago
So not only is it based on an obscure word in Welsh, but it's not even spelled correctly?
daoudc 3y ago
Yeah, pretty much, I named it after Mumbles or Mwmbwls where I live, but it's also a play on the word mumble.