22 ms·

Mwmbl: Free, open-source and non-profit search engine

209 points by marcodiego 3y ago

Andrew018 3y ago
[dead]
dang 3y ago
Related: Show HN: I'm building a non-profit search engine - https://news.ycombinator.com/item?id=29690877 https://news.ycombinator.com/item?id=29690877 - Dec 2021 (199 comments)
deleted 3y ago
[deleted]
romwell 3y ago
OK, the obvious question: Why go with an unpronounceable name? I mean, great that it was made, but I can't even tell people I'm using... mwumble? But it's spelled em-doubleyou-em-bee-el dot org.
- thelastparadise 3y ago
  I think it's mimblewimble.
  - kylecazar 3y ago
    That's worse than I thought, suspected it was a play on mumble.
- BLKNSLVR 3y ago
  It's pronounced mumble. An explanation is at the very bottom of the github Readme, quoting: > How do you pronounce "mwmbl"? > Like "mumble". I live in Mumbles, which is spelt "Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in "don't search, just mwmbl!"
  - xpe 3y ago
    Ok, I'll think of it like this. Since the name of "w" is "double u" that means "mwmbl" is also "muumbl"! UUho knows, maybe the name can uuork after all!
  - sdf4j 3y ago
    really? like mumble? [0] [0] https://mumble.info https://mumble.info
    - BLKNSLVR 3y ago
      According to government records, the only names not yet trademarked are "Popplers" and "Zittzers"
      - lionkor 3y ago
        Not for long! Someone ought to make a Popplers fastfood chain.
  - 9dev 3y ago
    Marketing 101: don’t try to be clever with your brand name :)
    - xpe 3y ago
      Philosophy 101: Think deeply. Use reasoning. Things depend on other things. The marketing claim above is so far from universal truth. Choosing a "clever" versus a straightforward brand name often depends on the brand strategy, target audience, and market conditions. But, sorry, for this example, I personally think the current brand name is atrocious.
  - Brian_K_White 3y ago
    It's pronounced "google mumble".
  - romwell 3y ago
    >I live in Mumbles, which is spelt "Mwmbwls" in Welsh. Ah Welsh, the golden standard of phonetic spelling and easy pronunciation! In other words, one needs a manual just to learn how to pronounce the name of that thing. Off to a great start, aren't we? >An explanation is at the very bottom of the github Readme AKA the first place anyone visiting the website would look at. NOT. Did writing "pronounced Mumble" on the landing page hurt puppies or something? >"don't search, just mumble!" I think half of the country is already doing that when it comes to fact-checking, and they certainly don't need a search engine for that. And here I was thinking GIMP was a horrible name.
- evolve2k 3y ago
  +1000 change the name Make it so you can use it in a sentence to replace “Just google it”
- dabluecaboose 3y ago
  Its a tech startup, vowels are not chic
- Brian_K_White 3y ago
  They are begging to be either ignored or forked. There are no other outcomes if they don't already understand why everyone is telling them this is unusable.
based-nerd 3y ago
[flagged]
evolve2k 3y ago
UI feedback: On my iPhone the search box shows two magnifine glasses, make it just one.
- retrofuturism 3y ago
  Google only has 1. You gotta 1-up the competition to win.
evolve2k 3y ago
Love that you folks are working on this. We desperately need more diversity in search options. Much is at stake in this arena.
- Proven 3y ago
  [dead]
- marginalia_nu 3y ago
  I'd love to see more competition in search. Feels like everyone right now gets tripped up on trying to emulate Google, which is a trap even if you succeed. Nobody is going to out-Google Google. ChatGPT's recent huge success in performing a specific tasks previously within the domain of Google by doing something other than they are is a good example of this.
  - kristopolous 3y ago
    Chatgpt is so much more useful for things that are specific and complex than Google is. Google used to be good at it but it's now utterly befuddled by specificity and returns such garbage that I had given up. But the form of "I'm doing this, I'm seeing this and I'm wondering if X is possible" chatgpt is solid on that - basically a personal stack overflow
    - marginalia_nu 3y ago
      Question-answering is something Google pivoted toward with great enthusiasm but never quite nailed down. They'd sometimes get some questions right, but it was more of a broken clock sort of a deal.
      - kristopolous 3y ago
        Most implementations of this have a race towards generalities. The biggest problem used to be when seemingly the whole internet was satisfied with an answer that is extremely wrong and broken when you do it. Chatgpt can work though this without getting into a weird markov cycle maybe half the time which is great. Patterns like "Hey I tried that. It still doesn't work, can you give me another option"
imachine1980_ 3y ago
still don't understand ""
mdaniel 3y ago
I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling https://github.com/mwmbl/mwmbl#crawling Also, your own posting appears to be missing from the index: https://mwmbl.org/?q=mwmbl+ycombinator https://mwmbl.org/?q=mwmbl+ycombinator (and, yes, another vote for changing the domain name; you can have a quirky project name, but if I can't remember the cat-walking-on-keyboard domain, I'm not going to use it)
- marc_abonce 3y ago
  > We now have a distributed crawler that runs on our volunteers' machines! If you have Firefox you can help out by installing our extension. This is a very interesting idea that other search engines have tried before. Actually, the Brave search engine is built over Cliqz[6] that implemented this same idea but *without* the user's consent. Copy pasting from an old comment I made about this "human web" crawler idea: Both PeARS[1] and Cliqz[2] tried to do that. Both got direct support from Mozilla[3][4] but it looks like neither really kicked off. PeARS was meant to be installed voluntarily by users who would then choose to share their indexes only to those they personally trusted, so the idea is very privacy conscious but also very hard to scale. Cliqz, on the other hand, apparently tried to work around that issue by having their add-on bundled by default in some Firefox installations[5] which was obviously very controversial because of its privacy and user consent implications. I still think the idea has potential, though, even if it's in a more limited scope. [1] https://github.com/PeARSearch/PeARS-orchard https://github.com/PeARSearch/PeARS-orchard [2] https://cliqz.com/en/whycliqz/human-web https://cliqz.com/en/whycliqz/human-web [3] https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-385000-to-open-source-projects-as-part-of-moss-mission-partners-program/ https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-3... [4] https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-strategic-investment-in-cliqz-to-enable-privacy-focused-search-innovation/ https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-s... [5] https://www.zdnet.com/article/firefox-tests-cliqz-engine-which-slurps-user-browsing-data/ https://www.zdnet.com/article/firefox-tests-cliqz-engine-whi... [6] https://www.theregister.com/2021/03/03/brave_buys_a_search_engine/ https://www.theregister.com/2021/03/03/brave_buys_a_search_e...
  - 3y ago
38 3y ago
slashes get eaten by the page. not cool.
- dotcoma 3y ago
  So do vowels ;)
bobse 3y ago
What a terrible name! Into the trash.
- jw_cook 3y ago
  While it doesn't refute your point, the Frequently Asked Question section does give an explanation for the consonant soup: it's Welsh. > How do you pronounce "mwmbl"? Like "mumble". I live in Mumbles, which is spelt "Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in "don't search, just mwmbl!"
  - seanthemon 3y ago
    I highly recommend grabbing something simpler to say and remember to redirect to your site. You're going to need a large amoung of inertia to get people to comfortably use an odd domain name.
    - AlphaCerium 3y ago
      Arguably, Google was probably a odd name for a search engine to people in the 90s that weren't maths-savvy.
      - brettermeier 3y ago
        But it's a normal word, unlike "mwmbl" (I had to look it up, couldn't remember where the "b" and "w" goes after some seconds).
        marginalia_nu 3y ago
        It's a normal word now. In 1998, it was pretty weird. How many o:s does it have, is it -el or -le? etc.
        arghwhat 3y ago
        No it's not, it is an intentional misspelling of "googol" and means nothing - not in English or any other language. "Spotify" is also not a "normal word" in any language. And for those not native in English (there's supposedly only some 400 million of those), it's just a random sound sequence like any other. mwmbl is a shortening of the welsh writing of https://en.wikipedia.org/wiki/Mumbles https://en.wikipedia.org/wiki/Mumbles. Only tricky part is knowing that the w is pronounced as a u. Maybe it would be slightly easier if one followed the fad of leaving out vowels, but guessing a vowel and having a tricky vowel does not seem much different.
Black616Angel 3y ago
Okay, name aside, because I instantly got that and englisch isn't my first language. But the crawler seems to be lacking quite a bit. For my first search (current work problem) "rust json diff" it only found 6 links, only one of which was a rust crate. Unfortunate. Second Search: "black sabbath sleeping village lyrics" only gave 2 results, only one of which was correct. Also the repo is missing the SearXNG[1] search engine. [1] https://github.com/searxng/searxng https://github.com/searxng/searxng
- marginalia_nu 3y ago
  SearXNG isn't really a search engine. It's just a unified front-end for other search engines, doesn't do any actual crawling or indexing as far as I'm aware.
marginalia_nu 3y ago
I was happy to notice these guys a while back, but the git repo seems very dormant. I wonder if they backpedaled on the open source side of things, or if the project is asleep. Either would be sad, because the world needs more open source search engines.
- daoudc 3y ago
  Yes, I took a break for a while, my fourth child was just born! Still committed to the project though and working on it when I get time. Thanks for your encouragement. Would love to have a chat some time.
  - marginalia_nu 3y ago
    Feel free to shoot me an email, I love to talk search engines :D
mdtrooper 3y ago
I remember https://yacy.net/ https://yacy.net/ but the big problem of this project was java and had not implementations in others languages. I mean it as imagine torrent was only in perl.
- marginalia_nu 3y ago
  YaCy's big problem is that distributed search is a bad idea that will never perform well. Search is as fast as the data is local.
  - kristopolous 3y ago
    There was an effort in the early 90s to have search as a protocol so you could have a query and then select the domains you want to run it on and return an aggregate result. It was 100% abandoned and I think that's a mistake. It'd be nice to explore some of those ideas again
    - marginalia_nu 3y ago
      I think a big part of the problem is that domains in isolation don't provide the best search results. Out-of-band information like (global) anchor texts or click data makes search perform so much better. If I want to learn how to do an INNER JOIN in MariaDB, this is the authoritative source: https://mariadb.com/kb/en/join-syntax/ https://mariadb.com/kb/en/join-syntax/ The problem being that INNER JOIN isn't particularly important to that page using most IR measures of importance, it's also primarily in a <code>-block which is typically further de-prioritized. To learn that this is an important link, you need to look outside of mariadb.com.
      - kristopolous 3y ago
        There's more to it than that. What if instead of crawling the php generation of database rows with a bunch of cruft, the administrator published some kind of schema with scraping and querying rules and you could alternatively make a single call to capture all of the data in a sematic schema. You can still do all the stuff you're talking about but it could make search more coherent. An entry for that humans and an entry for the computers. You can't trust everybody like this sure, but say imdb, discogs, wikipedia, all of which provide database dumps anyways (eg: https://datasets.imdbws.com/ https://datasets.imdbws.com/). That's what I'm advocating for revisiting. Lots of legit sites such as universities, newspapers, public records offices... You could even have a search toggle "screened sources" or whatever for the ones that make the cut
xenodium 3y ago
If keen on some minor feedback (specially for mobile), you can likely cut down on landing page text: From: MWMBL [Search on mwmbl...] Welcome to mwmbl, the free, open-source and non-profit search engine. You can start searching by using the search bar above! Find more on [Github] [Wiki] To: MWMBL [Search on mwmbl...] A free, open-source and non-profit search engine. [Github] [Wiki]
- daoudc 3y ago
  Thanks, feel free to send a PR!
krishadi 3y ago
This and the other engines seem to implement all the components of crawling, indexing, and searching strung together. Is there a reason for this? Wouldn't an option of, let's say, crawling + indexing made available separately, where others could built a search algorithm on top of, or just the crawling as a service made available. Are there stuff like these already available? Or is it just not a viable option?
- ddorian43 3y ago
  There is common crawl: https://github.com/commoncrawl https://github.com/commoncrawl
  - marginalia_nu 3y ago
    Kinda sucks that it's stuck in AWS with no easy way of exfiltrating the data from the Amazon ecosystem. Last I tried I got like 100 Kb/s on their HTTP mirror. At that rate, the download would take 12 years.
    - ddorian43 3y ago
      I just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth): wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz
- deleted 3y ago
  [deleted]
- marginalia_nu 3y ago
  Crawling can be done collaboratively, but there's not a lot of point to doing this. Crawling is the cheap and easy part. As for the rest, in order to perform well, the indexer needs to be built specifically tailored to the what the search engine is doing. Often you're scrounging for places to cram in individual bits to encode some additional piece of information about the term. If a DBMS tries to support every use case, a search engine index does the opposite, it supports a singular use case and cuts every corner imaginable and then some to make that happen with as much resource frugality as possible.
nonrandomstring 3y ago
> This website requires you to support/enable scripts. Bye bye. You do not need "scripts" to turn the text string I'll supply into a list of candidate links. How can you not understand this basic accessibility foundation?
- daoudc 3y ago
  The API is open so feel free to write your own front end that doesn't need js, or send a PR to add support for no js.
BlackLotus89 3y ago
Gigablast (linked in the faq) is dead for some time now. Had some sort of collab with freenode and then suddenly disappeared (not implying causality)
carlsborg 3y ago
Sub-100 ms search results, nicely typed python codebase, good project. How many 4096 byte pages do you currently store?
- worksonmine 3y ago
  Why 4096 bytes?
  - carlsborg 3y ago
    From the github: "Our design is a giant hash map. We have a single store consisting of a fixed number N of pages. Each page is of a fixed size (currently 4096 bytes to match a page of memory), and consists of a compressed list of items."
- daoudc 3y ago
  10240000 - see https://github.com/mwmbl/mwmbl/blob/18dc760a3402c74803823a94310ef058a60d57b4/Dockerfile#L50 https://github.com/mwmbl/mwmbl/blob/18dc760a3402c74803823a94...
Reticularas 3y ago
Don't know if the index isn't complete, but the results with this are quite poor
kwhitefoot 3y ago
A lot of the terms I searched for returned no hits. The Firefox add-on crawls pages linked from Hacker News which is amusing perhaps but seems unlikely to crawl a representative selection of the web. Perhaps the user should be able to suggest pages to be crawled. But when it does find something it is very quick! So I'll give it a go.
- hk__2 3y ago
  Same experience: it’s quick at finding irrelevant links. For some reason, it seems to have indexed a lot of spammy websites: search for "Trastevere" on Google, and you get Wikipedia and pages about the district in Rome. Search it on Mwmbl and you only get links from a random *.it-romehotels.com website. Other random examples: search for "2023" and the very first link is "2023 Pomeroy College Basketball Ratings". Search for "iphone", and the 5th link is a page about iPhone 6s that was last updated in 2021. Typos don't work: "haker news" has only one result, a hungarian press article.
  - marginalia_nu 3y ago
    Even Google is kinda not great for "Trastevere". I'd like to see results like these in favor of the sort of travel industry spam that's 90% of the search results page. https://www.romeartlover.it/Vasi60.htm https://www.romeartlover.it/Vasi60.htm https://www.maquettes-historiques.net/P19b.html https://www.maquettes-historiques.net/P19b.html
1vuio0pswjnm7 3y ago
"Welcome to mwmbl, the free, open-source and non-profit search engine. This website requires you to support/enable scripts." JSON results, no Javascript https://api.mwmbl.org/?search=search+the+web+without+javascript https://api.mwmbl.org/?search=search+the+web+without+javascr...
- 1vuio0pswjnm7 3y ago
  Correction: https://api.mwmbl.org/search?s=disable+javascript https://api.mwmbl.org/search?s=disable+javascript
davidebaldini 3y ago
If I understand, having only 4096 bytes of data per term causes multiple terms in the same query to intersect to little or no results. The purpose seems to cut cost in compromise of completeness.
- marginalia_nu 3y ago
  Yeah. That seems like a design decision that will scale poorly. For reference, even in my dinky 100M index I have individual terms with several gigabytes of associated document references. In general hash map table index designs don't tend to be very efficient. If you use a skip list or something similar, you can calculate the intersection between sets in sublinear time.
  - daoudc 3y ago
    We actually just take the union and then re-rank. Because the lists are all small, this is cheap.
    - marginalia_nu 3y ago
      Point is, with a skip list (or similar), the lists don't need to be small. You can intersect data sets that are enormous very quickly using this algo[1] where a single linear read of both lists is the worst case scenario. [1] https://nlp.stanford.edu/IR-book/html/htmledition/faster-postings-list-intersection-via-skip-pointers-1.html https://nlp.stanford.edu/IR-book/html/htmledition/faster-pos...
- daoudc 3y ago
  Yes, you're correct on the purpose. We mitigate it a little by also indexing on bigrams.
Fnoord 3y ago
When I enter Kamil Galeev I get directed to a Nitter post by him (and only that), but when I enter Kamil Kazani (which was the mentioned nickname of said Nitter post) I get returned nothing at all.
- daoudc 3y ago
  That seems to be because it's written kamilkazani, and automatically splitting such names is a hard problem
tamimio 3y ago
I searched first test “best business banks in Canada” and it showed no results saying it couldn’t find any “We could not find anything for your search..”, I can also see two redundant lenses icons.
- marginalia_nu 3y ago
  What sort of result would you expect for such a query?
  - tamimio 3y ago
    Primarily a list of options to choose from, preferably that from a non-affiliated site, asking the same in GPT-4 I get the following: >Tangerine Business Savings Account: This account offers a high interest rate of 2.65% to 3.25% on your balance, no monthly fees, no minimum balance requirement, unlimited transactions, free e-transfers, and access to over 3,000 ATMs. >Wise Business Account: This account offers low-cost international payments in over 50 currencies, no monthly fees, no minimum balance requirement, free local transfers, free debit card, and access to over 10 million ATMs. >BMO eBusiness Plan: This account offers no monthly fees, no minimum balance requirement, unlimited transactions, free e-transfers, free cheque deposits, and access to over 3,500 ATMs. >RBC Digital Choice Business Account: This account offers no monthly fees for the first three months ($5 per month thereafter), unlimited electronic transactions, 10 free debit transactions per month ($1.25 each thereafter), free e-transfers, free cheque deposits, and access to over 4,200 ATMs.
    - marginalia_nu 3y ago
      Well if we imagine a search engine as a document retrieval machine, who would publish such a document?
      - Brian_K_White 3y ago
        Before I kagi this (not even google, just kagi!) shall we wager on whether there is or is not at least one, likely several such documents? Come on.
_el1s7 3y ago
For fun and learning is good but don't think it's practical... not even close to functionality from search engines in the 90s
- daoudc 3y ago
  The more people that join, and help us crawl, the better it gets.
TheExplorer 3y ago
Typing 'Debian' and getting some results, adding 'gdm' results 0. lol
crtasm 3y ago
It's much easier to read after changing --bold-font-weight to 500 in the CSS.
joshxyz 3y ago
man these names are making me dyslexic. love it though.
starstripe 3y ago
Searched "Littler Books" and nothing came up. Would be awesome if this worked as expected.
- daoudc 3y ago
  It's easy to crawl specific sites using the command line crawler
daoudc 3y ago
Hi, creator here, happy to see this posted! Feel free to ask any questions.
luc_rnz 3y ago
Thank you for sharing this, this is very interesting. I will give it a try, although I don't think it can replace my current engine (DuckDuckGo/Searx), but rahter complement it maybe (by having a smaller, more curated set of data). Particularly I am having a great time reading the crawler extension source-code: https://github.com/mwmbl/crawler-extension https://github.com/mwmbl/crawler-extension
IYasha 3y ago
White screen, no content, 30 errors in the console. Firefox 50.1 @Ubuntu Too much JS for one page... :-|
rstreefland 3y ago
I was intrigued by the name and was very pleasantly surprised to confirm the Welsh influence when I clicked though. The creator lives very close to home for me. Dal ati! We really need open source alternatives to Google.

209 points by marcodiego 3y ago

[dead]

dang 3y ago

Related: Show HN: I'm building a non-profit search engine - https://news.ycombinator.com/item?id=29690877 https://news.ycombinator.com/item?id=29690877 - Dec 2021 (199 comments)

deleted 3y ago

[deleted]

romwell 3y ago

OK, the obvious question: Why go with an unpronounceable name? I mean, great that it was made, but I can't even tell people I'm using... mwumble? But it's spelled em-doubleyou-em-bee-el dot org.

thelastparadise 3y ago
I think it's mimblewimble.
- kylecazar 3y ago
  That's worse than I thought, suspected it was a play on mumble.
BLKNSLVR 3y ago
It's pronounced mumble. An explanation is at the very bottom of the github Readme, quoting: > How do you pronounce "mwmbl"? > Like "mumble". I live in Mumbles, which is spelt "Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in "don't search, just mwmbl!"
- xpe 3y ago
  Ok, I'll think of it like this. Since the name of "w" is "double u" that means "mwmbl" is also "muumbl"! UUho knows, maybe the name can uuork after all!
- sdf4j 3y ago
  really? like mumble? [0] [0] https://mumble.info https://mumble.info
  - BLKNSLVR 3y ago
    According to government records, the only names not yet trademarked are "Popplers" and "Zittzers"
    - lionkor 3y ago
      Not for long! Someone ought to make a Popplers fastfood chain.
- 9dev 3y ago
  Marketing 101: don’t try to be clever with your brand name :)
  - xpe 3y ago
    Philosophy 101: Think deeply. Use reasoning. Things depend on other things. The marketing claim above is so far from universal truth. Choosing a "clever" versus a straightforward brand name often depends on the brand strategy, target audience, and market conditions. But, sorry, for this example, I personally think the current brand name is atrocious.
- Brian_K_White 3y ago
  It's pronounced "google mumble".
- romwell 3y ago
  >I live in Mumbles, which is spelt "Mwmbwls" in Welsh. Ah Welsh, the golden standard of phonetic spelling and easy pronunciation! In other words, one needs a manual just to learn how to pronounce the name of that thing. Off to a great start, aren't we? >An explanation is at the very bottom of the github Readme AKA the first place anyone visiting the website would look at. NOT. Did writing "pronounced Mumble" on the landing page hurt puppies or something? >"don't search, just mumble!" I think half of the country is already doing that when it comes to fact-checking, and they certainly don't need a search engine for that. And here I was thinking GIMP was a horrible name.
evolve2k 3y ago
+1000 change the name Make it so you can use it in a sentence to replace “Just google it”
dabluecaboose 3y ago
Its a tech startup, vowels are not chic
Brian_K_White 3y ago
They are begging to be either ignored or forked. There are no other outcomes if they don't already understand why everyone is telling them this is unusable.

based-nerd 3y ago

[flagged]

evolve2k 3y ago

UI feedback: On my iPhone the search box shows two magnifine glasses, make it just one.

retrofuturism 3y ago
Google only has 1. You gotta 1-up the competition to win.

evolve2k 3y ago

Love that you folks are working on this. We desperately need more diversity in search options. Much is at stake in this arena.

Proven 3y ago
[dead]
marginalia_nu 3y ago
I'd love to see more competition in search. Feels like everyone right now gets tripped up on trying to emulate Google, which is a trap even if you succeed. Nobody is going to out-Google Google. ChatGPT's recent huge success in performing a specific tasks previously within the domain of Google by doing something other than they are is a good example of this.
- kristopolous 3y ago
  Chatgpt is so much more useful for things that are specific and complex than Google is. Google used to be good at it but it's now utterly befuddled by specificity and returns such garbage that I had given up. But the form of "I'm doing this, I'm seeing this and I'm wondering if X is possible" chatgpt is solid on that - basically a personal stack overflow
  - marginalia_nu 3y ago
    Question-answering is something Google pivoted toward with great enthusiasm but never quite nailed down. They'd sometimes get some questions right, but it was more of a broken clock sort of a deal.
    - kristopolous 3y ago
      Most implementations of this have a race towards generalities. The biggest problem used to be when seemingly the whole internet was satisfied with an answer that is extremely wrong and broken when you do it. Chatgpt can work though this without getting into a weird markov cycle maybe half the time which is great. Patterns like "Hey I tried that. It still doesn't work, can you give me another option"

imachine1980_ 3y ago

still don't understand ""

mdaniel 3y ago

I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling https://github.com/mwmbl/mwmbl#crawling Also, your own posting appears to be missing from the index: https://mwmbl.org/?q=mwmbl+ycombinator https://mwmbl.org/?q=mwmbl+ycombinator (and, yes, another vote for changing the domain name; you can have a quirky project name, but if I can't remember the cat-walking-on-keyboard domain, I'm not going to use it)

marc_abonce 3y ago
> We now have a distributed crawler that runs on our volunteers' machines! If you have Firefox you can help out by installing our extension. This is a very interesting idea that other search engines have tried before. Actually, the Brave search engine is built over Cliqz[6] that implemented this same idea but *without* the user's consent. Copy pasting from an old comment I made about this "human web" crawler idea: Both PeARS[1] and Cliqz[2] tried to do that. Both got direct support from Mozilla[3][4] but it looks like neither really kicked off. PeARS was meant to be installed voluntarily by users who would then choose to share their indexes only to those they personally trusted, so the idea is very privacy conscious but also very hard to scale. Cliqz, on the other hand, apparently tried to work around that issue by having their add-on bundled by default in some Firefox installations[5] which was obviously very controversial because of its privacy and user consent implications. I still think the idea has potential, though, even if it's in a more limited scope. [1] https://github.com/PeARSearch/PeARS-orchard https://github.com/PeARSearch/PeARS-orchard [2] https://cliqz.com/en/whycliqz/human-web https://cliqz.com/en/whycliqz/human-web [3] https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-385000-to-open-source-projects-as-part-of-moss-mission-partners-program/ https://blog.mozilla.org/press-uk/2016/06/22/mozilla-gives-3... [4] https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-strategic-investment-in-cliqz-to-enable-privacy-focused-search-innovation/ https://blog.mozilla.org/press-uk/2016/08/23/mozilla-makes-s... [5] https://www.zdnet.com/article/firefox-tests-cliqz-engine-which-slurps-user-browsing-data/ https://www.zdnet.com/article/firefox-tests-cliqz-engine-whi... [6] https://www.theregister.com/2021/03/03/brave_buys_a_search_engine/ https://www.theregister.com/2021/03/03/brave_buys_a_search_e...
- 3y ago

38 3y ago

slashes get eaten by the page. not cool.

dotcoma 3y ago
So do vowels ;)

bobse 3y ago

What a terrible name! Into the trash.

jw_cook 3y ago
While it doesn't refute your point, the Frequently Asked Question section does give an explanation for the consonant soup: it's Welsh. > How do you pronounce "mwmbl"? Like "mumble". I live in Mumbles, which is spelt "Mwmbwls" in Welsh. But the intended meaning is "to mumble", as in "don't search, just mwmbl!"
- seanthemon 3y ago
  I highly recommend grabbing something simpler to say and remember to redirect to your site. You're going to need a large amoung of inertia to get people to comfortably use an odd domain name.
  - AlphaCerium 3y ago
    Arguably, Google was probably a odd name for a search engine to people in the 90s that weren't maths-savvy.
    - brettermeier 3y ago
      But it's a normal word, unlike "mwmbl" (I had to look it up, couldn't remember where the "b" and "w" goes after some seconds).
      - marginalia_nu 3y ago
        It's a normal word now. In 1998, it was pretty weird. How many o:s does it have, is it -el or -le? etc.
      - arghwhat 3y ago
        No it's not, it is an intentional misspelling of "googol" and means nothing - not in English or any other language. "Spotify" is also not a "normal word" in any language. And for those not native in English (there's supposedly only some 400 million of those), it's just a random sound sequence like any other. mwmbl is a shortening of the welsh writing of https://en.wikipedia.org/wiki/Mumbles https://en.wikipedia.org/wiki/Mumbles. Only tricky part is knowing that the w is pronounced as a u. Maybe it would be slightly easier if one followed the fad of leaving out vowels, but guessing a vowel and having a tricky vowel does not seem much different.

Black616Angel 3y ago

Okay, name aside, because I instantly got that and englisch isn't my first language. But the crawler seems to be lacking quite a bit. For my first search (current work problem) "rust json diff" it only found 6 links, only one of which was a rust crate. Unfortunate. Second Search: "black sabbath sleeping village lyrics" only gave 2 results, only one of which was correct. Also the repo is missing the SearXNG[1] search engine. [1] https://github.com/searxng/searxng https://github.com/searxng/searxng

marginalia_nu 3y ago
SearXNG isn't really a search engine. It's just a unified front-end for other search engines, doesn't do any actual crawling or indexing as far as I'm aware.

marginalia_nu 3y ago

I was happy to notice these guys a while back, but the git repo seems very dormant. I wonder if they backpedaled on the open source side of things, or if the project is asleep. Either would be sad, because the world needs more open source search engines.

daoudc 3y ago
Yes, I took a break for a while, my fourth child was just born! Still committed to the project though and working on it when I get time. Thanks for your encouragement. Would love to have a chat some time.
- marginalia_nu 3y ago
  Feel free to shoot me an email, I love to talk search engines :D

mdtrooper 3y ago

I remember https://yacy.net/ https://yacy.net/ but the big problem of this project was java and had not implementations in others languages. I mean it as imagine torrent was only in perl.

marginalia_nu 3y ago
YaCy's big problem is that distributed search is a bad idea that will never perform well. Search is as fast as the data is local.
- kristopolous 3y ago
  There was an effort in the early 90s to have search as a protocol so you could have a query and then select the domains you want to run it on and return an aggregate result. It was 100% abandoned and I think that's a mistake. It'd be nice to explore some of those ideas again
  - marginalia_nu 3y ago
    I think a big part of the problem is that domains in isolation don't provide the best search results. Out-of-band information like (global) anchor texts or click data makes search perform so much better. If I want to learn how to do an INNER JOIN in MariaDB, this is the authoritative source: https://mariadb.com/kb/en/join-syntax/ https://mariadb.com/kb/en/join-syntax/ The problem being that INNER JOIN isn't particularly important to that page using most IR measures of importance, it's also primarily in a <code>-block which is typically further de-prioritized. To learn that this is an important link, you need to look outside of mariadb.com.
    - kristopolous 3y ago
      There's more to it than that. What if instead of crawling the php generation of database rows with a bunch of cruft, the administrator published some kind of schema with scraping and querying rules and you could alternatively make a single call to capture all of the data in a sematic schema. You can still do all the stuff you're talking about but it could make search more coherent. An entry for that humans and an entry for the computers. You can't trust everybody like this sure, but say imdb, discogs, wikipedia, all of which provide database dumps anyways (eg: https://datasets.imdbws.com/ https://datasets.imdbws.com/). That's what I'm advocating for revisiting. Lots of legit sites such as universities, newspapers, public records offices... You could even have a search toggle "screened sources" or whatever for the ones that make the cut

xenodium 3y ago

If keen on some minor feedback (specially for mobile), you can likely cut down on landing page text: From: MWMBL [Search on mwmbl...] Welcome to mwmbl, the free, open-source and non-profit search engine. You can start searching by using the search bar above! Find more on [Github] [Wiki] To: MWMBL [Search on mwmbl...] A free, open-source and non-profit search engine. [Github] [Wiki]

daoudc 3y ago
Thanks, feel free to send a PR!

krishadi 3y ago

This and the other engines seem to implement all the components of crawling, indexing, and searching strung together. Is there a reason for this? Wouldn't an option of, let's say, crawling + indexing made available separately, where others could built a search algorithm on top of, or just the crawling as a service made available. Are there stuff like these already available? Or is it just not a viable option?

ddorian43 3y ago
There is common crawl: https://github.com/commoncrawl https://github.com/commoncrawl
- marginalia_nu 3y ago
  Kinda sucks that it's stuck in AWS with no easy way of exfiltrating the data from the Amazon ecosystem. Last I tried I got like 100 Kb/s on their HTTP mirror. At that rate, the download would take 12 years.
  - ddorian43 3y ago
    I just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth): wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz
deleted 3y ago
[deleted]
marginalia_nu 3y ago
Crawling can be done collaboratively, but there's not a lot of point to doing this. Crawling is the cheap and easy part. As for the rest, in order to perform well, the indexer needs to be built specifically tailored to the what the search engine is doing. Often you're scrounging for places to cram in individual bits to encode some additional piece of information about the term. If a DBMS tries to support every use case, a search engine index does the opposite, it supports a singular use case and cuts every corner imaginable and then some to make that happen with as much resource frugality as possible.

nonrandomstring 3y ago

> This website requires you to support/enable scripts. Bye bye. You do not need "scripts" to turn the text string I'll supply into a list of candidate links. How can you not understand this basic accessibility foundation?

daoudc 3y ago
The API is open so feel free to write your own front end that doesn't need js, or send a PR to add support for no js.

BlackLotus89 3y ago

Gigablast (linked in the faq) is dead for some time now. Had some sort of collab with freenode and then suddenly disappeared (not implying causality)

carlsborg 3y ago

Sub-100 ms search results, nicely typed python codebase, good project. How many 4096 byte pages do you currently store?

worksonmine 3y ago
Why 4096 bytes?
- carlsborg 3y ago
  From the github: "Our design is a giant hash map. We have a single store consisting of a fixed number N of pages. Each page is of a fixed size (currently 4096 bytes to match a page of memory), and consists of a compressed list of items."
daoudc 3y ago
10240000 - see https://github.com/mwmbl/mwmbl/blob/18dc760a3402c74803823a94310ef058a60d57b4/Dockerfile#L50 https://github.com/mwmbl/mwmbl/blob/18dc760a3402c74803823a94...

Reticularas 3y ago

Don't know if the index isn't complete, but the results with this are quite poor

kwhitefoot 3y ago

A lot of the terms I searched for returned no hits. The Firefox add-on crawls pages linked from Hacker News which is amusing perhaps but seems unlikely to crawl a representative selection of the web. Perhaps the user should be able to suggest pages to be crawled. But when it does find something it is very quick! So I'll give it a go.

hk__2 3y ago
Same experience: it’s quick at finding irrelevant links. For some reason, it seems to have indexed a lot of spammy websites: search for "Trastevere" on Google, and you get Wikipedia and pages about the district in Rome. Search it on Mwmbl and you only get links from a random *.it-romehotels.com website. Other random examples: search for "2023" and the very first link is "2023 Pomeroy College Basketball Ratings". Search for "iphone", and the 5th link is a page about iPhone 6s that was last updated in 2021. Typos don't work: "haker news" has only one result, a hungarian press article.
- marginalia_nu 3y ago
  Even Google is kinda not great for "Trastevere". I'd like to see results like these in favor of the sort of travel industry spam that's 90% of the search results page. https://www.romeartlover.it/Vasi60.htm https://www.romeartlover.it/Vasi60.htm https://www.maquettes-historiques.net/P19b.html https://www.maquettes-historiques.net/P19b.html

1vuio0pswjnm7 3y ago

"Welcome to mwmbl, the free, open-source and non-profit search engine. This website requires you to support/enable scripts." JSON results, no Javascript https://api.mwmbl.org/?search=search+the+web+without+javascript https://api.mwmbl.org/?search=search+the+web+without+javascr...

1vuio0pswjnm7 3y ago
Correction: https://api.mwmbl.org/search?s=disable+javascript https://api.mwmbl.org/search?s=disable+javascript

davidebaldini 3y ago

If I understand, having only 4096 bytes of data per term causes multiple terms in the same query to intersect to little or no results. The purpose seems to cut cost in compromise of completeness.

marginalia_nu 3y ago
Yeah. That seems like a design decision that will scale poorly. For reference, even in my dinky 100M index I have individual terms with several gigabytes of associated document references. In general hash map table index designs don't tend to be very efficient. If you use a skip list or something similar, you can calculate the intersection between sets in sublinear time.
- daoudc 3y ago
  We actually just take the union and then re-rank. Because the lists are all small, this is cheap.
  - marginalia_nu 3y ago
    Point is, with a skip list (or similar), the lists don't need to be small. You can intersect data sets that are enormous very quickly using this algo[1] where a single linear read of both lists is the worst case scenario. [1] https://nlp.stanford.edu/IR-book/html/htmledition/faster-postings-list-intersection-via-skip-pointers-1.html https://nlp.stanford.edu/IR-book/html/htmledition/faster-pos...
daoudc 3y ago
Yes, you're correct on the purpose. We mitigate it a little by also indexing on bigrams.

Fnoord 3y ago

When I enter Kamil Galeev I get directed to a Nitter post by him (and only that), but when I enter Kamil Kazani (which was the mentioned nickname of said Nitter post) I get returned nothing at all.

daoudc 3y ago
That seems to be because it's written kamilkazani, and automatically splitting such names is a hard problem

tamimio 3y ago

I searched first test “best business banks in Canada” and it showed no results saying it couldn’t find any “We could not find anything for your search..”, I can also see two redundant lenses icons.

marginalia_nu 3y ago
What sort of result would you expect for such a query?
- tamimio 3y ago
  Primarily a list of options to choose from, preferably that from a non-affiliated site, asking the same in GPT-4 I get the following: >Tangerine Business Savings Account: This account offers a high interest rate of 2.65% to 3.25% on your balance, no monthly fees, no minimum balance requirement, unlimited transactions, free e-transfers, and access to over 3,000 ATMs. >Wise Business Account: This account offers low-cost international payments in over 50 currencies, no monthly fees, no minimum balance requirement, free local transfers, free debit card, and access to over 10 million ATMs. >BMO eBusiness Plan: This account offers no monthly fees, no minimum balance requirement, unlimited transactions, free e-transfers, free cheque deposits, and access to over 3,500 ATMs. >RBC Digital Choice Business Account: This account offers no monthly fees for the first three months ($5 per month thereafter), unlimited electronic transactions, 10 free debit transactions per month ($1.25 each thereafter), free e-transfers, free cheque deposits, and access to over 4,200 ATMs.
  - marginalia_nu 3y ago
    Well if we imagine a search engine as a document retrieval machine, who would publish such a document?
    - Brian_K_White 3y ago
      Before I kagi this (not even google, just kagi!) shall we wager on whether there is or is not at least one, likely several such documents? Come on.

_el1s7 3y ago

For fun and learning is good but don't think it's practical... not even close to functionality from search engines in the 90s

daoudc 3y ago
The more people that join, and help us crawl, the better it gets.

TheExplorer 3y ago

Typing 'Debian' and getting some results, adding 'gdm' results 0. lol

crtasm 3y ago

It's much easier to read after changing --bold-font-weight to 500 in the CSS.

joshxyz 3y ago

man these names are making me dyslexic. love it though.

starstripe 3y ago

Searched "Littler Books" and nothing came up. Would be awesome if this worked as expected.

daoudc 3y ago
It's easy to crawl specific sites using the command line crawler

daoudc 3y ago

Hi, creator here, happy to see this posted! Feel free to ask any questions.

luc_rnz 3y ago

Thank you for sharing this, this is very interesting. I will give it a try, although I don't think it can replace my current engine (DuckDuckGo/Searx), but rahter complement it maybe (by having a smaller, more curated set of data). Particularly I am having a great time reading the crawler extension source-code: https://github.com/mwmbl/crawler-extension https://github.com/mwmbl/crawler-extension

IYasha 3y ago

White screen, no content, 30 errors in the console. Firefox 50.1 @Ubuntu Too much JS for one page... :-|

rstreefland 3y ago

I was intrigued by the name and was very pleasantly surprised to confirm the Welsh influence when I clicked though. The creator lives very close to home for me. Dal ati! We really need open source alternatives to Google.