39 ms·

Alexandria Search

277 points by nafnlj 5y ago

waynecochran 5y ago
In a nutshell, what is the fundamental difference with this search engine compared to others?
- stazz1 5y ago
  >About Alexandria Alexandria.org is a non-profit, ad free search engine. Our goal is to provide the best available information without compromise. The index is built on data from Common Crawl and the engine is written in C++. The source code is available here. We are still at an early stage of development and running the search engine on a shoestring budget. Please contact us at -email- if you want to get involved, want to support this initiative or have any questions.
  - waynecochran 5y ago
    But what is different in terms of its indexing algorithm? The original secret sauce for google was the pagerank algorithm which was mathematically genius. Are you using a similar algorithm.
    - josefcullhed 5y ago
      Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.
      - Seirdy 5y ago
        I'm curious as to which reasons/tradeoffs were involved in the decision to use harmonic centrality, if you wouldn't mind sharing.
- Minor49er 5y ago
  Their About page has you covered: > Alexandria.org is a non-profit, ad free search engine. Our goal is to provide the best available information without compromise. > The index is built on data from Common Crawl and the engine is written in C++. The source code is available (at https://github.com/alexandria-org# https://github.com/alexandria-org#). Edit: formatting
  - greenyoda 5y ago
    More about Common Crawl: https://en.wikipedia.org/wiki/Common_Crawl https://en.wikipedia.org/wiki/Common_Crawl > Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of petabytes of data collected since 2011. It completes crawls generally every month. ...
Rich_Morin 5y ago
I just tried out this search engine and was very favorably impressed. It was quite responsive (though that could be affected by demand) and gave good results. I really like the lack of goo (e.g., ads) and the spare, clean presentation. I think it might be a great search engine for visually disabled users who rely on screen readers.
Minor49er 5y ago
This is really fast and cool. Looking for music-related pages, I've already found some interesting websites, like Wall of Ambient, which caters to ambient labels (https://wallofambient.com/# https://wallofambient.com/#) I noticed that if a term can't be found, there will be a random number of results that it says were found, but nothing is actually displayed. Eg: https://www.alexandria.org/?c=&r=&q=moonmusiq https://www.alexandria.org/?c=&r=&q=moonmusiq I'll keep trying this out. It seems really promising
dimitar 5y ago
Unfortunately it seems it doesn't support Cyrillic or Bulgarian well. I googled the mayor of the city I live in and there are 5 results all irrelevant. Unfortunately the experience in 'minor' languages is consistently bad in all alternative search engines.
- tandr 5y ago
  We don't know resources behind this project. But even if they substantial, still - they have to start small. It will come, give it time.
- deleted 5y ago
  [deleted]
tandr 5y ago
I think the fact that after a long while there are new search engines (Kagi was introduced very recently on HN, now this) should be a wake up call for Google - their search has lost some shine for quite a while. Hopefully something will come out of this - competition is good.
- bmmayer1 5y ago
  My first search on Alexandria was "UTC time". Google gives me the current time in UTC, which is all I needed. Alexandria gave me...a lot of links to click to find what I'm looking for. Google search is a lot better than people give it credit for.
  - boomboomsubban 5y ago
    When I need a clock I'll be sure to use Google.
  - tokai 5y ago
    You still have the current UTC time in the top links. Googles knowledge graph is a part of the problem with their results.
  - marginalia_nu 5y ago
    Strictly speaking, that's more in the domain of BonziBuddy or Alexa than internet search engine. What Google arguably struggles with is surfacing relevant documents, that is... search.
  - aghilmort 5y ago
    we're exploring adding instant answers in clean way at Breeze; leaning towards using open-source library &/or external API to compute vs. building in-house also adding premium tier that's alerts + ad free + feeling lucky that would take user to top result, which is a UTC page, re: https://breezethat.com/?q=UTC+time https://breezethat.com/?q=UTC+time
    - NoahTheDuke 5y ago
julienreszka 5y ago
For a same query the number of results varies dramatically. So weird.
wilg 5y ago
which makes it very hard to follow-up on queries.The cursor moves back to the beginning of the search box after you search,
pmarreck 5y ago
This actually makes me want to build my own web crawler and search
- marginalia_nu 5y ago
  Do try, it's a very interesting problem domain.
- josefcullhed 5y ago
  Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare so you will be blocked pretty quickly. Crawling is a big issue and most of the issues are non-technical.
  - Seirdy 5y ago
    I've heard from other people who run engines (Right Dao, Gigablast) that this is a major problem; Common Crawl does look helpful, but it's not continuously updated. FWIW, Right Dao uses Wikipedia as a starting point for crawling. Kiwix makes pre-indexed dumps of Wikipedia, StackExchange, and other sites available. Some sort of partnership between crawlers could go a long way. Have you considered contributing content back towards the Common Crawl?
    - marginalia_nu 5y ago
      There seems to be a threshold where you get greylisted by cloudflare. Not sure if it's requests per day or what they're doing. But I've been able to mostly circumvent it by crawling at a modest rate.
  - pmarreck 5y ago
    First off, nice work! This seems like a reasonable fallback option but it's also a weaker one. By "most of the issues are non-technical", do you mean that you need special permission from someone like cloudflare to get "crawl rights"?
potatoman22 5y ago
It doesn't work well for programming queries :(
- xerox13ster 5y ago
  I searched fs js and the nodejs.org documentation was the first result.
drcongo 5y ago
> Found 0 results in 0.05s I have a test search string that I use to try out search engines, this one didn't do very well.
- reaperducer 5y ago
  Unless you were testing for speed!
- doctor_eval 5y ago
  Negative tests are still tests!
devmunchies 5y ago
The initial commit was 11 months ago and written in C++. I haven’t done C++ since college ~7 years ago. Is it a good language for greenfield projects these days, or would something like Go or Rust (or Crystal, Nim, Zig) be better for maintainability and acquiring contributors?
- marginalia_nu 5y ago
  Main thing I'd worry about with C++ is heap fragmentation over time. Something like TCMalloc or JEMalloc might help a bit but it's hard to get around doing this type of thing in C++.
- extheat 5y ago
  I don’t think the problem is there’s a shortage of developers. Much less the language. There’s a shortage of people with the experience in working with search engines and the necessary algorithms to make them work as intended reliably.
- Shorel 5y ago
  I would prefer D, but I ended up using C++ for one of my projects because it has all the libraries, and these smaller languages only have bindings for some of them. So my answer is: if you can do all you need in the base language, something more modern like D-lang is preferred, but if you need some particular library, you either have to add all the bindings yourself, or use C++.
endisneigh 5y ago
https://www.alexandria.org/?c=&r=&q=SPY+current+value https://www.alexandria.org/?c=&r=&q=SPY+current+value https://www.google.com/search?q=SPY+current+value&rlz=1C1ONGR_enUS974US974&oq=SPY+current+value&aqs=chrome..69i64.2491j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=SPY+current+value&rlz=1C1ONG... https://www.alexandria.org/?c=&r=&q=kggle https://www.alexandria.org/?c=&r=&q=kggle https://www.google.com/search?q=kggle&rlz=1C1ONGR_enUS974US974&oq=kggle&aqs=chrome..69i57j69i60.508j0j9&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=kggle&rlz=1C1ONGR_enUS974US9... Search is hard.
- marginalia_nu 5y ago
  I'd argue "kggle" should surface this result: https://stackoverflow.com/questions/44077294/encounter-this-indexerror-list-index-out-of-range-error-while-i-fetch-image-n https://stackoverflow.com/questions/44077294/encounter-this-...
- deleted 5y ago
  [deleted]
blinding-streak 5y ago
Does this support phrase match? (ie "a query in quotes"). A few tries seems to show that it doesn't. Or, if it does, its corpus is tiny.
version_five 5y ago
There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial. That was different when search began. Google is now almost a convenience. If I have a coding question, I search for "turn list of tensors into tensor" or whatever but I'm really looking for SO or the pytorch documentation, and I'll ignore the geeksforgeeks and other seo spam that finds it's way it. It's almost like google is a statistical "portal" page, like Yahoo or one of those old cluttered sites were, that lets me quickly get through the menus by just searching. That's different from a blank slate search like we might have done 25 years ago. I think what's really lacking now is uncorrupted search for anything that can be monetized. Like I tried to search for a frying pan once on google and it was unusable. I'm not sure any better search engine can fix that, that's why everyone appends "reddit" to queries they are looking for a real opinion on, again, because they are optimizing for the current state of the web. Anyway, all that to say I think there are a lot of problems with (google dominated) search, but they are basically reflected in the current web overall, so just a better search engine, outside of stripping out the ads, can only do so much. Real improved search efforts need to somehow change the content that's out there at the same time as they improve they experience, and let us know how to, in a simple way, get the most out of it. I think google has a much deeper moat than most people realize
- danuker 5y ago
  > because they are optimizing for the current state of the web. I believe people will at least start looking for alternatives. For example, I have been collecting search engines, and whenever I encounter a page with too many commercial-laden SEO-porked results, I use a different search engine in Firefox. I have enabled the Search Bar, I can do Alt+D, Tab, Tab, enter my query, then click a different search engine, which searches instantly, unlike the main bar, where you have to press Enter once more after clicking. I just added this one also. See my collection: https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH97UUJ6h5+AWSeXxS https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH9...
bghfm 5y ago
How can we help improve the project? Usage, feedback?
- IvanHall 5y ago
  Hello Ivan here (one of the founders) At the moment we primarily need help with development and funding. But if you have suggestions or want to help in some other ways, please let us know!
hunter2_ 5y ago
The privacy settings are defaulting to unchecked, but the description above them suggests that they default to checked. This makes me wonder how the settings are actually being interpreted (i.e., what the actual initial state is).
byteski 5y ago
ive found several search engines/services besides G and ddg and the only thing that i cant figure out is do these search services have seo technique or its just random list of all resources? i mean how does it order search results
outcoldman 5y ago
Tried a few searches. https://www.alexandria.org/?c=&r=&q=real+estates+puerto+escondido https://www.alexandria.org/?c=&r=&q=real+estates+puerto+esco... - 3 results only :( If you correct it to "real estate puerto Escondido" - that works better https://www.alexandria.org/?c=&r=&q=real+estate+puerto+escondido https://www.alexandria.org/?c=&r=&q=real+estate+puerto+escon... A lot to improve. But a good start
pmontra 5y ago
I'll use it for general searches in the next days because it's the only way to do a fair evaluation. I just searched for python3 join string and I didn't get the Python docs in the first page. Both DDG and Google got them at position 9 which is way too low. At least I got a different set of random websites and not the usual tutorialpoint, w3schools, geeksforgeeks etc that I usually see in these cases.
qumpis 5y ago
Is there a web that pools multiple search engines results?
- mikkom 5y ago
  Search engines typically probihit this kind of usage via their TOS
- marginalia_nu 5y ago
  SearX?
- Seirdy 5y ago
  SearX and Searxng are the most common options, but instances often get blocked by the engines they use. Users need to switch between instances quite often. eTools.ch uses commercial APIs so it doesn't get blocked, but it might block you instead (very sensitive bot detection). Dogpile is one of the older metasearch engines, but I think it only uses Bing- and Google-powered engines.
- aghilmort 5y ago
  per other folks on thread, nearly impossible legally due to TOU/TOS or getting around those restrictions with rotating proxies, remote browsing + proxies, etc. also part of why going with topic filter approach at Breeze -- so if you search all the web, we'll give you option to open others with indie indexes in new tab - say Mojeek or Yandex -- similar to what airline search engines do if you switch to say code search, you can use ours or redirect to any one of say PublicWWW, Nerdy Data, or Builtwith pretty much no other legal way to do it on the main
jstx1 5y ago
Randomly picking a search that I needed for work today - searching for "pandas order by list" says that it has 44 results and it shows only 3: - a Github issue for dask - an article about panda populations - some coronavirus article that happens to have an unrelated snippet of pandas code Google obviously picks the relevant stackoverflow thread as the first response.
- bricemo 5y ago
  I searched for "pizza near me". Google shows me several pizza places within a 2 mile distance. Alexandria gave me the link to papajohns.com.
Xeoncross 5y ago
Thank you for building and sharing this. While many people rightly point out that this isn't a replacement for Google yet, the value of a shared working open source code base has been underestimated many times in the past. I hope this is a project that grows to solve real needs in this space. However, even if it never makes it past this point, there is a chance someone will be inspired by this to construct their own version. Maybe in a different language with a different storage format or a different way of ranking results. Thank you for sharing your work.
0xbadcafebee 5y ago
Why don't search engines have filters? Every single consumer retail website's search uses filters to help shoppers find something to buy. It is way more convenient than hoping the user can guess the magic search phrase to find the thing they're looking for (if they even know what that thing is).
kreeben 5y ago
I love the shortcut Alexandria takes by indexing Common Crawl instead of crawling the web themselves. It's how I would have bootstrapped a new search engine. In a future iteration they can start crawling themselves, if there is sufficient interest from the public. Searching is screamingly fast. The index seems stale, though. Alexandria, how old is your index? How long did it take you to create your current index? Is that your bottleneck, perhaps, that it takes you a long time (and lots of money?) to create a Common Crawl index?
- joshuamorton 5y ago
  > The index seems stale, though. Alexandria, how old is your index? Common crawl indexes about once every 40 days, the current crawl's data is through January 2022, so it's 1.5 months old at best.
rprenger 5y ago
For my first search of "GFlowNetworks" (which the search bar suggested) It said: Found 5,887 (or something) results, but showed no results For my second I searched my name and got a Wikipedia article about a show I've never heard of which didn't have my name anywhere in it. For my third I searched "GFlowNetworks" again and it said Found 2,656,844 results in 1.61s, but showed no results again
- marginalia_nu 5y ago
  I can't even find results for "GFlowNetworks" on google.
zeruch 5y ago
I was amused by the name (beyond the obvious reference to the ancient library, it also gender-switches on a heteronym of Fernado Pessoa, Alexander Search) https://www.brown.edu/Departments/Portuguese_Brazilian_Studies/ejph/pessoaplural/Issue10/PDF/I10A02.pdf https://www.brown.edu/Departments/Portuguese_Brazilian_Studi...
glitcher 5y ago
Really like the minimal UI and the speed! Great work. A few of my test searches came up with very useful results. However, one disappointment was searching for a javascript function, for example "javascript array splice", and the MDN site was not in the results. Adding "MDN" or "Mozilla" to the search did not help either.
moonshinefe 5y ago
Unfortunately it seems down for me right now.
- yosito 5y ago
  Yep, I'm getting a 502
unmole 5y ago
I'm getting a 502 :(
lukasb 5y ago
Excuse me while I get on a hobbyhorse - would love to use web search that lets me boost PageRank for certain sites (which then would carry over to sites they link to.) Could automatically boost PageRank for sites I subscribe to, for example. Expensive in terms of computation or storage? Charge me!
- marginalia_nu 5y ago
  That's easy to do (it's Personalized PageRank), but VERY expensive. Like just tossing them a few dollars doesn't cut it. You basically need your own custom index for that, as the way you achieve fast ranking is by sorting the documents in order of ranking within the index itself. That way you only need to consider a very small portion of the index to retrieve the highest ranking results. You might get away with having like a custom micro-index where your search basically does a hidden site:-search for your favorite domain and related domains, but that's not quite going to do what you want it to do.
  - lukasb 5y ago
    So uh ... 100 petabytes, then. Ah.
    - marginalia_nu 5y ago
      Realistically you could probably get away with something like a couple of terabytes, and then default to the regular index if it isn't found in the neighborhood close to your favored sites, but that's still anything but cheap especially this can't be some slow-ass S3 storage, this storage should ideally be SSDs or a RAID/JBOD-configuration of mechanical drives. That means you're also paying for a lot of I/O bandwidth and overall data logistics. If you try to rent that sort of compute, you're probably looking at like $100-200/month.
zander312 5y ago
getting a 502...
strongpigeon 5y ago
Slightly tangential, but does anyone know if there is a way to submit links to the Common Crawl (which Alexandria Search relies on)? I haven't seen any traffic from CCBot and my site doesn't seem to show up in Alexandria's results (compared to 2nd/3rd on Google for a bunch of queries).
- kreeben 5y ago
  You can verify whether or not your site exists in the CC data set by searching for it here: https://index.commoncrawl.org/ https://index.commoncrawl.org/
  - strongpigeon 5y ago
    Thanks for that! It does look like it's in there and got crawled in January. I probably didn't search back far enough in my logs...
andreygrehov 5y ago
How does it work? The GitHub page is not very descriptive. I tried to search "Putin" and the first link is the NYTimes homepage. Does that mean NYTimes covers the war more than the other publications, or is it backlink-driven?
- throwra620 5y ago
51Cards 5y ago
So, interesting thing, how when I visit this site for the first time (in Firefox) is the search box showing a drop down with a bunch of my previous searches? I can't tell where they are from but it is all stuff I have searched for in the past. I thought it might be the browser populating a list but that should be based on same domain. So where is it pulling this from? Some of the searche terms are months, perhaps more than a year old.
- tzot 5y ago
  I can only assume that Firefox associates filled-in data with the name of the input control; in this case, “q”, which is probably typical for a search inputbox.
josefcullhed 5y ago
Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourselves. We are indexing commoncrawl and the search engine is in a really early stage. We would be super happy to find more developers who want to help us.
- kreeben 5y ago
  Thanks for sharing this with the world. Did you manage to include all of a common crawl in an index? How long did that take you to produce such an index? Is your index in-memory or on disk? I'd consider contributing. Seems you have something here.
  - josefcullhed 5y ago
    The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It takes us a couple of days to build the index but we have been coding this for about 1 year. All the indexes are on disk.
    - kreeben 5y ago
      >> All the indexes are on disk. Love it. Makes for a cheaper infrastructure, since SSD is cheaper than RAM. >> It takes us a couple of days to build the index It's hard for me to see how that could be done much faster unless you find a way to parallelize the process, which in itself is a terrifyingly hard problem. I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure you use for indexing? According to you, what kind of data structure allows for the fastest indexing and how do you represent it on disk so that you can read your on-disk index in a forward-only mode or "as fast as possible"?
      - josefcullhed 5y ago
hadjian 5y ago
This is really lovely. I think the search results are useful, if you're looking for more static information. Very pleasant, to see only results. I think, I found a minor bug, while (of course) searching for my homepage: https://www.alexandria.org/?c=&r=&q=www.hadjian.com https://www.alexandria.org/?c=&r=&q=www.hadjian.com The status line below the search box says, that it found many results, but the results are empty. Also, when hitting F5 a couple of times, the number jumps around. Keep up the great work. I think there is a lot potential to Common Crawl and things built on top of it.
Seirdy 5y ago
I discovered Alexandria a month ago and was shocked by the quality of its results; while it's no Google/Bing replacement, it was leagues ahead of most other independent engines I'd collected and monitored for over a year[0]. [0]: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe... It's earned a spot in my search bookmarks alongside Right Dao and Marginalia for when I want a bit of serendipity in my results. I also appreciate the minimal UI, without any JavScript or other unnecessary bells and whistles.
charcircuit 5y ago
Terrible results for "minecraft any% world record"
svnpenn 5y ago
doesnt work https://alexandria.org?q=cannot+use+generic+function+without+instantiation https://alexandria.org?q=cannot+use+generic+function+without...
gverrilla 5y ago
People say the problem is google, but the real problem is content. All these new search engines are searching on the same space, filled with crap.
- bricemo 5y ago
  Agreed. When people say they Google is returning poor results, I can never get an answer of what specific URL people actually want. Just general unhappiness and some mythical, vague, ideal result.
  - blondin 5y ago
    google search often returns wrong, unrelated, or no result. i have been in that situation many times before and moved on to change my query. i am (and most of us are) trying to solve my own issue. not google's. that's why you get vague answers to your questions. it's not because it doesn't happen. it's because at that moment we care much more about solving our problem. that's what brought us to google search in the first place.
    - bricemo 5y ago
      thanks for engaging, I'm trying to understand. can you give a specific example of a search query, and the ideal URL you think you should get as a result (that google is missing)? what is one query<->result example of a gap these independent engines are filling?
pabs3 5y ago
Interesting, searching for "Debian", the third result is the rustc package and the eighth is the GitHub repo for the Debian packaging of bino. I wonder how this search engine does its ranking.
known 5y ago
perlwle 5y ago
already blocked by GFW. sigh..
deleted 5y ago
[deleted]

Alexandria Search

277 points by nafnlj 5y ago

waynecochran 5y ago

In a nutshell, what is the fundamental difference with this search engine compared to others?

stazz1 5y ago
>About Alexandria Alexandria.org is a non-profit, ad free search engine. Our goal is to provide the best available information without compromise. The index is built on data from Common Crawl and the engine is written in C++. The source code is available here. We are still at an early stage of development and running the search engine on a shoestring budget. Please contact us at -email- if you want to get involved, want to support this initiative or have any questions.
- waynecochran 5y ago
  But what is different in terms of its indexing algorithm? The original secret sauce for google was the pagerank algorithm which was mathematically genius. Are you using a similar algorithm.
  - josefcullhed 5y ago
    Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.
    - Seirdy 5y ago
      I'm curious as to which reasons/tradeoffs were involved in the decision to use harmonic centrality, if you wouldn't mind sharing.
Minor49er 5y ago
Their About page has you covered: > Alexandria.org is a non-profit, ad free search engine. Our goal is to provide the best available information without compromise. > The index is built on data from Common Crawl and the engine is written in C++. The source code is available (at https://github.com/alexandria-org# https://github.com/alexandria-org#). Edit: formatting
- greenyoda 5y ago
  More about Common Crawl: https://en.wikipedia.org/wiki/Common_Crawl https://en.wikipedia.org/wiki/Common_Crawl > Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of petabytes of data collected since 2011. It completes crawls generally every month. ...

Rich_Morin 5y ago

I just tried out this search engine and was very favorably impressed. It was quite responsive (though that could be affected by demand) and gave good results. I really like the lack of goo (e.g., ads) and the spare, clean presentation. I think it might be a great search engine for visually disabled users who rely on screen readers.

Minor49er 5y ago

This is really fast and cool. Looking for music-related pages, I've already found some interesting websites, like Wall of Ambient, which caters to ambient labels (https://wallofambient.com/# https://wallofambient.com/#) I noticed that if a term can't be found, there will be a random number of results that it says were found, but nothing is actually displayed. Eg: https://www.alexandria.org/?c=&r=&q=moonmusiq https://www.alexandria.org/?c=&r=&q=moonmusiq I'll keep trying this out. It seems really promising

dimitar 5y ago

Unfortunately it seems it doesn't support Cyrillic or Bulgarian well. I googled the mayor of the city I live in and there are 5 results all irrelevant. Unfortunately the experience in 'minor' languages is consistently bad in all alternative search engines.

tandr 5y ago
We don't know resources behind this project. But even if they substantial, still - they have to start small. It will come, give it time.
deleted 5y ago
[deleted]

tandr 5y ago

I think the fact that after a long while there are new search engines (Kagi was introduced very recently on HN, now this) should be a wake up call for Google - their search has lost some shine for quite a while. Hopefully something will come out of this - competition is good.

bmmayer1 5y ago
My first search on Alexandria was "UTC time". Google gives me the current time in UTC, which is all I needed. Alexandria gave me...a lot of links to click to find what I'm looking for. Google search is a lot better than people give it credit for.
- boomboomsubban 5y ago
  When I need a clock I'll be sure to use Google.
- tokai 5y ago
  You still have the current UTC time in the top links. Googles knowledge graph is a part of the problem with their results.
- marginalia_nu 5y ago
  Strictly speaking, that's more in the domain of BonziBuddy or Alexa than internet search engine. What Google arguably struggles with is surfacing relevant documents, that is... search.
- aghilmort 5y ago
  we're exploring adding instant answers in clean way at Breeze; leaning towards using open-source library &/or external API to compute vs. building in-house also adding premium tier that's alerts + ad free + feeling lucky that would take user to top result, which is a UTC page, re: https://breezethat.com/?q=UTC+time https://breezethat.com/?q=UTC+time
  - NoahTheDuke 5y ago

julienreszka 5y ago

For a same query the number of results varies dramatically. So weird.

wilg 5y ago

which makes it very hard to follow-up on queries.The cursor moves back to the beginning of the search box after you search,

pmarreck 5y ago

This actually makes me want to build my own web crawler and search

marginalia_nu 5y ago
Do try, it's a very interesting problem domain.
josefcullhed 5y ago
Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare so you will be blocked pretty quickly. Crawling is a big issue and most of the issues are non-technical.
- Seirdy 5y ago
  I've heard from other people who run engines (Right Dao, Gigablast) that this is a major problem; Common Crawl does look helpful, but it's not continuously updated. FWIW, Right Dao uses Wikipedia as a starting point for crawling. Kiwix makes pre-indexed dumps of Wikipedia, StackExchange, and other sites available. Some sort of partnership between crawlers could go a long way. Have you considered contributing content back towards the Common Crawl?
  - marginalia_nu 5y ago
    There seems to be a threshold where you get greylisted by cloudflare. Not sure if it's requests per day or what they're doing. But I've been able to mostly circumvent it by crawling at a modest rate.
- pmarreck 5y ago
  First off, nice work! This seems like a reasonable fallback option but it's also a weaker one. By "most of the issues are non-technical", do you mean that you need special permission from someone like cloudflare to get "crawl rights"?

potatoman22 5y ago

It doesn't work well for programming queries :(

xerox13ster 5y ago
I searched fs js and the nodejs.org documentation was the first result.

drcongo 5y ago

> Found 0 results in 0.05s I have a test search string that I use to try out search engines, this one didn't do very well.

reaperducer 5y ago
Unless you were testing for speed!
doctor_eval 5y ago
Negative tests are still tests!

devmunchies 5y ago

The initial commit was 11 months ago and written in C++. I haven’t done C++ since college ~7 years ago. Is it a good language for greenfield projects these days, or would something like Go or Rust (or Crystal, Nim, Zig) be better for maintainability and acquiring contributors?

marginalia_nu 5y ago
Main thing I'd worry about with C++ is heap fragmentation over time. Something like TCMalloc or JEMalloc might help a bit but it's hard to get around doing this type of thing in C++.
extheat 5y ago
I don’t think the problem is there’s a shortage of developers. Much less the language. There’s a shortage of people with the experience in working with search engines and the necessary algorithms to make them work as intended reliably.
Shorel 5y ago
I would prefer D, but I ended up using C++ for one of my projects because it has all the libraries, and these smaller languages only have bindings for some of them. So my answer is: if you can do all you need in the base language, something more modern like D-lang is preferred, but if you need some particular library, you either have to add all the bindings yourself, or use C++.

endisneigh 5y ago

https://www.alexandria.org/?c=&r=&q=SPY+current+value https://www.alexandria.org/?c=&r=&q=SPY+current+value https://www.google.com/search?q=SPY+current+value&rlz=1C1ONGR_enUS974US974&oq=SPY+current+value&aqs=chrome..69i64.2491j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=SPY+current+value&rlz=1C1ONG... https://www.alexandria.org/?c=&r=&q=kggle https://www.alexandria.org/?c=&r=&q=kggle https://www.google.com/search?q=kggle&rlz=1C1ONGR_enUS974US974&oq=kggle&aqs=chrome..69i57j69i60.508j0j9&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=kggle&rlz=1C1ONGR_enUS974US9... Search is hard.

marginalia_nu 5y ago
I'd argue "kggle" should surface this result: https://stackoverflow.com/questions/44077294/encounter-this-indexerror-list-index-out-of-range-error-while-i-fetch-image-n https://stackoverflow.com/questions/44077294/encounter-this-...
deleted 5y ago
[deleted]

blinding-streak 5y ago

Does this support phrase match? (ie "a query in quotes"). A few tries seems to show that it doesn't. Or, if it does, its corpus is tiny.

version_five 5y ago

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial. That was different when search began. Google is now almost a convenience. If I have a coding question, I search for "turn list of tensors into tensor" or whatever but I'm really looking for SO or the pytorch documentation, and I'll ignore the geeksforgeeks and other seo spam that finds it's way it. It's almost like google is a statistical "portal" page, like Yahoo or one of those old cluttered sites were, that lets me quickly get through the menus by just searching. That's different from a blank slate search like we might have done 25 years ago. I think what's really lacking now is uncorrupted search for anything that can be monetized. Like I tried to search for a frying pan once on google and it was unusable. I'm not sure any better search engine can fix that, that's why everyone appends "reddit" to queries they are looking for a real opinion on, again, because they are optimizing for the current state of the web. Anyway, all that to say I think there are a lot of problems with (google dominated) search, but they are basically reflected in the current web overall, so just a better search engine, outside of stripping out the ads, can only do so much. Real improved search efforts need to somehow change the content that's out there at the same time as they improve they experience, and let us know how to, in a simple way, get the most out of it. I think google has a much deeper moat than most people realize

danuker 5y ago
> because they are optimizing for the current state of the web. I believe people will at least start looking for alternatives. For example, I have been collecting search engines, and whenever I encounter a page with too many commercial-laden SEO-porked results, I use a different search engine in Firefox. I have enabled the Search Bar, I can do Alt+D, Tab, Tab, enter my query, then click a different search engine, which searches instantly, unlike the main bar, where you have to press Enter once more after clicking. I just added this one also. See my collection: https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH97UUJ6h5+AWSeXxS https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH9...

bghfm 5y ago

How can we help improve the project? Usage, feedback?

IvanHall 5y ago
Hello Ivan here (one of the founders) At the moment we primarily need help with development and funding. But if you have suggestions or want to help in some other ways, please let us know!

hunter2_ 5y ago

The privacy settings are defaulting to unchecked, but the description above them suggests that they default to checked. This makes me wonder how the settings are actually being interpreted (i.e., what the actual initial state is).

byteski 5y ago

ive found several search engines/services besides G and ddg and the only thing that i cant figure out is do these search services have seo technique or its just random list of all resources? i mean how does it order search results

outcoldman 5y ago

Tried a few searches. https://www.alexandria.org/?c=&r=&q=real+estates+puerto+escondido https://www.alexandria.org/?c=&r=&q=real+estates+puerto+esco... - 3 results only :( If you correct it to "real estate puerto Escondido" - that works better https://www.alexandria.org/?c=&r=&q=real+estate+puerto+escondido https://www.alexandria.org/?c=&r=&q=real+estate+puerto+escon... A lot to improve. But a good start

pmontra 5y ago

I'll use it for general searches in the next days because it's the only way to do a fair evaluation. I just searched for python3 join string and I didn't get the Python docs in the first page. Both DDG and Google got them at position 9 which is way too low. At least I got a different set of random websites and not the usual tutorialpoint, w3schools, geeksforgeeks etc that I usually see in these cases.

qumpis 5y ago

Is there a web that pools multiple search engines results?

mikkom 5y ago
Search engines typically probihit this kind of usage via their TOS
marginalia_nu 5y ago
SearX?
Seirdy 5y ago
SearX and Searxng are the most common options, but instances often get blocked by the engines they use. Users need to switch between instances quite often. eTools.ch uses commercial APIs so it doesn't get blocked, but it might block you instead (very sensitive bot detection). Dogpile is one of the older metasearch engines, but I think it only uses Bing- and Google-powered engines.
aghilmort 5y ago
per other folks on thread, nearly impossible legally due to TOU/TOS or getting around those restrictions with rotating proxies, remote browsing + proxies, etc. also part of why going with topic filter approach at Breeze -- so if you search all the web, we'll give you option to open others with indie indexes in new tab - say Mojeek or Yandex -- similar to what airline search engines do if you switch to say code search, you can use ours or redirect to any one of say PublicWWW, Nerdy Data, or Builtwith pretty much no other legal way to do it on the main

jstx1 5y ago

Randomly picking a search that I needed for work today - searching for "pandas order by list" says that it has 44 results and it shows only 3: - a Github issue for dask - an article about panda populations - some coronavirus article that happens to have an unrelated snippet of pandas code Google obviously picks the relevant stackoverflow thread as the first response.

bricemo 5y ago
I searched for "pizza near me". Google shows me several pizza places within a 2 mile distance. Alexandria gave me the link to papajohns.com.

Xeoncross 5y ago

Thank you for building and sharing this. While many people rightly point out that this isn't a replacement for Google yet, the value of a shared working open source code base has been underestimated many times in the past. I hope this is a project that grows to solve real needs in this space. However, even if it never makes it past this point, there is a chance someone will be inspired by this to construct their own version. Maybe in a different language with a different storage format or a different way of ranking results. Thank you for sharing your work.

0xbadcafebee 5y ago

Why don't search engines have filters? Every single consumer retail website's search uses filters to help shoppers find something to buy. It is way more convenient than hoping the user can guess the magic search phrase to find the thing they're looking for (if they even know what that thing is).

kreeben 5y ago

I love the shortcut Alexandria takes by indexing Common Crawl instead of crawling the web themselves. It's how I would have bootstrapped a new search engine. In a future iteration they can start crawling themselves, if there is sufficient interest from the public. Searching is screamingly fast. The index seems stale, though. Alexandria, how old is your index? How long did it take you to create your current index? Is that your bottleneck, perhaps, that it takes you a long time (and lots of money?) to create a Common Crawl index?

joshuamorton 5y ago
> The index seems stale, though. Alexandria, how old is your index? Common crawl indexes about once every 40 days, the current crawl's data is through January 2022, so it's 1.5 months old at best.

rprenger 5y ago

For my first search of "GFlowNetworks" (which the search bar suggested) It said: Found 5,887 (or something) results, but showed no results For my second I searched my name and got a Wikipedia article about a show I've never heard of which didn't have my name anywhere in it. For my third I searched "GFlowNetworks" again and it said Found 2,656,844 results in 1.61s, but showed no results again

marginalia_nu 5y ago
I can't even find results for "GFlowNetworks" on google.

zeruch 5y ago

I was amused by the name (beyond the obvious reference to the ancient library, it also gender-switches on a heteronym of Fernado Pessoa, Alexander Search) https://www.brown.edu/Departments/Portuguese_Brazilian_Studies/ejph/pessoaplural/Issue10/PDF/I10A02.pdf https://www.brown.edu/Departments/Portuguese_Brazilian_Studi...

glitcher 5y ago

Really like the minimal UI and the speed! Great work. A few of my test searches came up with very useful results. However, one disappointment was searching for a javascript function, for example "javascript array splice", and the MDN site was not in the results. Adding "MDN" or "Mozilla" to the search did not help either.

moonshinefe 5y ago

Unfortunately it seems down for me right now.

yosito 5y ago
Yep, I'm getting a 502

unmole 5y ago

I'm getting a 502 :(

lukasb 5y ago

Excuse me while I get on a hobbyhorse - would love to use web search that lets me boost PageRank for certain sites (which then would carry over to sites they link to.) Could automatically boost PageRank for sites I subscribe to, for example. Expensive in terms of computation or storage? Charge me!

marginalia_nu 5y ago
That's easy to do (it's Personalized PageRank), but VERY expensive. Like just tossing them a few dollars doesn't cut it. You basically need your own custom index for that, as the way you achieve fast ranking is by sorting the documents in order of ranking within the index itself. That way you only need to consider a very small portion of the index to retrieve the highest ranking results. You might get away with having like a custom micro-index where your search basically does a hidden site:-search for your favorite domain and related domains, but that's not quite going to do what you want it to do.
- lukasb 5y ago
  So uh ... 100 petabytes, then. Ah.
  - marginalia_nu 5y ago
    Realistically you could probably get away with something like a couple of terabytes, and then default to the regular index if it isn't found in the neighborhood close to your favored sites, but that's still anything but cheap especially this can't be some slow-ass S3 storage, this storage should ideally be SSDs or a RAID/JBOD-configuration of mechanical drives. That means you're also paying for a lot of I/O bandwidth and overall data logistics. If you try to rent that sort of compute, you're probably looking at like $100-200/month.

zander312 5y ago

getting a 502...

strongpigeon 5y ago

Slightly tangential, but does anyone know if there is a way to submit links to the Common Crawl (which Alexandria Search relies on)? I haven't seen any traffic from CCBot and my site doesn't seem to show up in Alexandria's results (compared to 2nd/3rd on Google for a bunch of queries).

kreeben 5y ago
You can verify whether or not your site exists in the CC data set by searching for it here: https://index.commoncrawl.org/ https://index.commoncrawl.org/
- strongpigeon 5y ago
  Thanks for that! It does look like it's in there and got crawled in January. I probably didn't search back far enough in my logs...

andreygrehov 5y ago

How does it work? The GitHub page is not very descriptive. I tried to search "Putin" and the first link is the NYTimes homepage. Does that mean NYTimes covers the war more than the other publications, or is it backlink-driven?

throwra620 5y ago

51Cards 5y ago

So, interesting thing, how when I visit this site for the first time (in Firefox) is the search box showing a drop down with a bunch of my previous searches? I can't tell where they are from but it is all stuff I have searched for in the past. I thought it might be the browser populating a list but that should be based on same domain. So where is it pulling this from? Some of the searche terms are months, perhaps more than a year old.

tzot 5y ago
I can only assume that Firefox associates filled-in data with the name of the input control; in this case, “q”, which is probably typical for a search inputbox.

josefcullhed 5y ago

Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourselves. We are indexing commoncrawl and the search engine is in a really early stage. We would be super happy to find more developers who want to help us.

kreeben 5y ago
Thanks for sharing this with the world. Did you manage to include all of a common crawl in an index? How long did that take you to produce such an index? Is your index in-memory or on disk? I'd consider contributing. Seems you have something here.
- josefcullhed 5y ago
  The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It takes us a couple of days to build the index but we have been coding this for about 1 year. All the indexes are on disk.
  - kreeben 5y ago
    >> All the indexes are on disk. Love it. Makes for a cheaper infrastructure, since SSD is cheaper than RAM. >> It takes us a couple of days to build the index It's hard for me to see how that could be done much faster unless you find a way to parallelize the process, which in itself is a terrifyingly hard problem. I haven't read your code yet, obviously, but could you give us a hint as to what kind of data structure you use for indexing? According to you, what kind of data structure allows for the fastest indexing and how do you represent it on disk so that you can read your on-disk index in a forward-only mode or "as fast as possible"?
    - josefcullhed 5y ago

hadjian 5y ago

This is really lovely. I think the search results are useful, if you're looking for more static information. Very pleasant, to see only results. I think, I found a minor bug, while (of course) searching for my homepage: https://www.alexandria.org/?c=&r=&q=www.hadjian.com https://www.alexandria.org/?c=&r=&q=www.hadjian.com The status line below the search box says, that it found many results, but the results are empty. Also, when hitting F5 a couple of times, the number jumps around. Keep up the great work. I think there is a lot potential to Common Crawl and things built on top of it.

Seirdy 5y ago

I discovered Alexandria a month ago and was shocked by the quality of its results; while it's no Google/Bing replacement, it was leagues ahead of most other independent engines I'd collected and monitored for over a year[0]. [0]: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe... It's earned a spot in my search bookmarks alongside Right Dao and Marginalia for when I want a bit of serendipity in my results. I also appreciate the minimal UI, without any JavScript or other unnecessary bells and whistles.

charcircuit 5y ago

Terrible results for "minecraft any% world record"

svnpenn 5y ago

doesnt work https://alexandria.org?q=cannot+use+generic+function+without+instantiation https://alexandria.org?q=cannot+use+generic+function+without...

gverrilla 5y ago

People say the problem is google, but the real problem is content. All these new search engines are searching on the same space, filled with crap.

bricemo 5y ago
Agreed. When people say they Google is returning poor results, I can never get an answer of what specific URL people actually want. Just general unhappiness and some mythical, vague, ideal result.
- blondin 5y ago
  google search often returns wrong, unrelated, or no result. i have been in that situation many times before and moved on to change my query. i am (and most of us are) trying to solve my own issue. not google's. that's why you get vague answers to your questions. it's not because it doesn't happen. it's because at that moment we care much more about solving our problem. that's what brought us to google search in the first place.
  - bricemo 5y ago
    thanks for engaging, I'm trying to understand. can you give a specific example of a search query, and the ideal URL you think you should get as a result (that google is missing)? what is one query<->result example of a gap these independent engines are filling?

pabs3 5y ago

Interesting, searching for "Debian", the third result is the rustc package and the eighth is the GitHub repo for the Debian packaging of bino. I wonder how this search engine does its ranking.

known 5y ago

perlwle 5y ago

already blocked by GFW. sigh..

deleted 5y ago

[deleted]