7 ms·

Only vaguely related: Is there any fully FOSS general purpose web search engine which gets close to DDG? It seems by now it should be possible to run a communit

by l0b0 9y ago

Only vaguely related: Is there any fully FOSS general purpose web search engine which gets close to DDG? It seems by now it should be possible to run a community supported completely transparent search engine with relatively limited means ($X00k/year).

jacquesm 9y ago
> It seems by now it should be possible to run a community supported completely transparent search engine with relatively limited means ($X00k/year). I'd argue the opposite, the time when such a thing would have been possible is long past. If you want to get anywhere near to the quality in results that the big engines offer then you're going to be spending some pretty big cash.
- fao_ 9y ago
  In my experience, the bar for quality has been rapidly dropping as of late. At this point, most of the things I type into google come back null or with random results -- even with results that used to return data that was relevant.
  - j_s 9y ago
    Can you please provide an example of this type of search?
    - godelski 9y ago
      I look for scientific papers a lot and have found that I'm getting a lot more sites like IFLS or Ars that are reporting on said paper. Or I'll get related studies, but not the one I'm looking for, when the related studies definitely don't contain a word I'm using. This is even expanding into my code search. Like I'll type "do something os related python linux" and get commands for windows as the first few hits. Clearly I don't want windows.
      - j_s 9y ago
        Like a link please, I want to see "null or with random results". Not sure how customized results affect things though.
        godelski 9y ago
        I can only account for my experiences. Only time I've ever had null results was when looking for really obscure things. I'm not the person that you originally replied to.
        j_s 9y ago
        Thanks for taking the time to provide a specific example where the results weren't up to your standards.
    - fao_ 9y ago
      I could spend a good hour or two of my time scouring through my web history for the things that you ask for proof of, or you could take what I said (Like anything on the internet, or based on recollection) with a pinch of salt. It happens just infrequently enough that it is both difficult to find proof of, and annoying when it happens. The null case does not happen so often as the case where nonsensical answers are returned. Needless to say I am not going to do the former, because it was intended to be an anecdote indicative of the general case -- so example-specific solutions are not likely to be transferable, because I don't hand portions of my web history to random people to prove something, and also because I have better things to do.
      - j_s 9y ago
        Thanks for 138 words to explain no proof of your original 43 (making what I personally consider to be a rather extraordinary claim) since you have better things to do? I'll go with several times more than a pinch and keep an eye out to document this phenomenon a bit more thoroughly should it ever happen to me. Success to you!
        fao_ 9y ago
        It's interesting to me that it took me one minute to write 138 words, thank you for that neat fact. And congratulations on a comment that is otherwise pretty useless. For the record, you do not have the right to demand that someone use their time to prove something that is only of interest to you, nor do you have the right to demand that someone debate you. It is extremely arrogant and self-righteous of you to state otherwise.
        j_s 9y ago
        Since we are going "for the record" (on a site where comments can't be deleted after an hour and a particular user record's karma column value is doubtless a part of consideration for ~$70,000 or whatever it is now) ... > Can you please provide an example of this type of search? My "arrogant and self-righteous" "demand that someone use their time to prove something that is only of interest to" me >> demand that someone debate you Not found. Apologies for any failures of communication or reading comprehension on my part? Peace!
  - StavrosK 9y ago
    I have noticed this too. Entire vast websites seem very poorly indexed (I can never find relevant reddit comments, Tweets, etc), and the web feels smaller and smaller, if I judge by how many useful results I get in a search. I have no idea why that is, but it's really hard to find relevant information for anything outside mainstream stuff. Just the other day I was looking for how to post JSON in a test with Flask, and I hadn't found anything after tens of searches. Surely, something must be referenced in some code on GitHub, or a reddit comment, or some blogpost somewhere, but it proved impossible for me to find.
    - j_s 9y ago
      I agree that there are problems searching large volumes of user-generated content when it isn't referenced much long term (Reddit, Twitter, etc.). If I don't keep record of Tweets myself I rarely find them again.
  - godelski 9y ago
    Well I'm glad it isn't just me. Though I've been seeing it on other sites like Netflix and YouTube. Hell, YouTube keeps suggesting me videos that I've already watched (search history is on). Like watch a video, go home, it is in the suggested. Sometimes Google amazes me (like when I typed in "mm unit" and it returned "molar mass" which is what I intended) but most of the time it feels like it is going down hill and I have to start using the macros.
- greglindahl 9y ago
  It's worth pointing out that blekko raised $63mm. That gives you an idea of what a multi-billion-page crawl and index requires.
  - carussell 9y ago
    That's only if you're trying to be competitive with Google et al as a business venture. If all you're going for is good enough, that's not indicative of your necessary costs. What's more, the FOSS movement has a long history of accepting these kinds of compromises. I call this the "Retina watermark fallacy"—when you equate something not being the latest and greatest with being unacceptable. When Apple introduced the "Retina" Hi-DPI display for the iPhone 4, it was good. But what's more, it was supposed to show that everything else was junk. And yet, if you looked at Apple marketing materials from ~5 years prior, you could find breathless ad copy about their then-latest displays that were (necessarily) not Retina quality. That means either one of two things are true: 1. either Apple was selling unusable junk prior to the introduction of the Retina displays, and they managed to mistakenly convince themselves and everyone else that this stuff was acceptable when it actually wasn't, or 2. pre-Retina displays were good enough, and Hi-DPI displays are simply better The truth is lies in 2. So, the takeaway as I understood it from the original question would be whether or not the FOSS world could produce today a search engine on par with, let's say, 2002-era Google. (I remember 2002-era Google, and not only did it work, but it was good!)
    - greglindahl 9y ago
      You can't build 2002-era google today, it's 2017 and the web is a much more hostile place. If there was any way the FOSS community could fund a reasonable search engine, I'd be happy to work at non-profit wages to make it happen. I don't see any way.
fghtr 9y ago
Check out http:/yacy.net. Do not expect the experience anywhere close to ddg though...
executesorder66 9y ago
There's YaCy, but as for achieving the results, there's a very large amount of network traffic and storage you'd need to have available to you.
Forbo 9y ago
You might be interested in searx.me
- l0b0 9y ago
  "Searx is a metasearch engine", so while it has many good uses any biases present in the results it retrieves for you are still going to be in the final result.
  - dalf 9y ago
    How it can about "for you" if the instance is shared between different users ?
ZenoArrow 9y ago
It's not exactly what you're looking for, but I'd recommend taking a look at Apache Solr: http://lucene.apache.org/solr/ http://lucene.apache.org/solr/ It allows you to run a document search engine that can be distributed over multiple machines. It could be adapted to create a web search engine. Interestingly, it looks like DDG uses Solr, though I'm not sure if it's used in their core search features or not.
- greglindahl 9y ago
  If you're up in the billion+ document size you'll probably want something more efficient like Xapian. Solr and Elastic are great at what they do, but web scale is bigger.
deleted 9y ago
[deleted]

by l0b0 9y ago

jacquesm 9y ago

> It seems by now it should be possible to run a community supported completely transparent search engine with relatively limited means ($X00k/year). I'd argue the opposite, the time when such a thing would have been possible is long past. If you want to get anywhere near to the quality in results that the big engines offer then you're going to be spending some pretty big cash.

fao_ 9y ago
In my experience, the bar for quality has been rapidly dropping as of late. At this point, most of the things I type into google come back null or with random results -- even with results that used to return data that was relevant.
- j_s 9y ago
  Can you please provide an example of this type of search?
  - godelski 9y ago
    I look for scientific papers a lot and have found that I'm getting a lot more sites like IFLS or Ars that are reporting on said paper. Or I'll get related studies, but not the one I'm looking for, when the related studies definitely don't contain a word I'm using. This is even expanding into my code search. Like I'll type "do something os related python linux" and get commands for windows as the first few hits. Clearly I don't want windows.
    - j_s 9y ago
      Like a link please, I want to see "null or with random results". Not sure how customized results affect things though.
      - godelski 9y ago
        I can only account for my experiences. Only time I've ever had null results was when looking for really obscure things. I'm not the person that you originally replied to.
        j_s 9y ago
        Thanks for taking the time to provide a specific example where the results weren't up to your standards.
  - fao_ 9y ago
    I could spend a good hour or two of my time scouring through my web history for the things that you ask for proof of, or you could take what I said (Like anything on the internet, or based on recollection) with a pinch of salt. It happens just infrequently enough that it is both difficult to find proof of, and annoying when it happens. The null case does not happen so often as the case where nonsensical answers are returned. Needless to say I am not going to do the former, because it was intended to be an anecdote indicative of the general case -- so example-specific solutions are not likely to be transferable, because I don't hand portions of my web history to random people to prove something, and also because I have better things to do.
    - j_s 9y ago
      Thanks for 138 words to explain no proof of your original 43 (making what I personally consider to be a rather extraordinary claim) since you have better things to do? I'll go with several times more than a pinch and keep an eye out to document this phenomenon a bit more thoroughly should it ever happen to me. Success to you!
      - fao_ 9y ago
        It's interesting to me that it took me one minute to write 138 words, thank you for that neat fact. And congratulations on a comment that is otherwise pretty useless. For the record, you do not have the right to demand that someone use their time to prove something that is only of interest to you, nor do you have the right to demand that someone debate you. It is extremely arrogant and self-righteous of you to state otherwise.
        j_s 9y ago
        Since we are going "for the record" (on a site where comments can't be deleted after an hour and a particular user record's karma column value is doubtless a part of consideration for ~$70,000 or whatever it is now) ... > Can you please provide an example of this type of search? My "arrogant and self-righteous" "demand that someone use their time to prove something that is only of interest to" me >> demand that someone debate you Not found. Apologies for any failures of communication or reading comprehension on my part? Peace!
- StavrosK 9y ago
  I have noticed this too. Entire vast websites seem very poorly indexed (I can never find relevant reddit comments, Tweets, etc), and the web feels smaller and smaller, if I judge by how many useful results I get in a search. I have no idea why that is, but it's really hard to find relevant information for anything outside mainstream stuff. Just the other day I was looking for how to post JSON in a test with Flask, and I hadn't found anything after tens of searches. Surely, something must be referenced in some code on GitHub, or a reddit comment, or some blogpost somewhere, but it proved impossible for me to find.
  - j_s 9y ago
    I agree that there are problems searching large volumes of user-generated content when it isn't referenced much long term (Reddit, Twitter, etc.). If I don't keep record of Tweets myself I rarely find them again.
- godelski 9y ago
  Well I'm glad it isn't just me. Though I've been seeing it on other sites like Netflix and YouTube. Hell, YouTube keeps suggesting me videos that I've already watched (search history is on). Like watch a video, go home, it is in the suggested. Sometimes Google amazes me (like when I typed in "mm unit" and it returned "molar mass" which is what I intended) but most of the time it feels like it is going down hill and I have to start using the macros.
greglindahl 9y ago
It's worth pointing out that blekko raised $63mm. That gives you an idea of what a multi-billion-page crawl and index requires.
- carussell 9y ago
  That's only if you're trying to be competitive with Google et al as a business venture. If all you're going for is good enough, that's not indicative of your necessary costs. What's more, the FOSS movement has a long history of accepting these kinds of compromises. I call this the "Retina watermark fallacy"—when you equate something not being the latest and greatest with being unacceptable. When Apple introduced the "Retina" Hi-DPI display for the iPhone 4, it was good. But what's more, it was supposed to show that everything else was junk. And yet, if you looked at Apple marketing materials from ~5 years prior, you could find breathless ad copy about their then-latest displays that were (necessarily) not Retina quality. That means either one of two things are true: 1. either Apple was selling unusable junk prior to the introduction of the Retina displays, and they managed to mistakenly convince themselves and everyone else that this stuff was acceptable when it actually wasn't, or 2. pre-Retina displays were good enough, and Hi-DPI displays are simply better The truth is lies in 2. So, the takeaway as I understood it from the original question would be whether or not the FOSS world could produce today a search engine on par with, let's say, 2002-era Google. (I remember 2002-era Google, and not only did it work, but it was good!)
  - greglindahl 9y ago
    You can't build 2002-era google today, it's 2017 and the web is a much more hostile place. If there was any way the FOSS community could fund a reasonable search engine, I'd be happy to work at non-profit wages to make it happen. I don't see any way.

fghtr 9y ago

Check out http:/yacy.net. Do not expect the experience anywhere close to ddg though...

executesorder66 9y ago

There's YaCy, but as for achieving the results, there's a very large amount of network traffic and storage you'd need to have available to you.

Forbo 9y ago