5 ms·

Everytime one of these threads show up, I find myself wondering why someone doesn't launch a technical and research focused competitor. There seems to be such b

Everytime one of these threads show up, I find myself wondering why someone doesn't launch a technical and research focused competitor. There seems to be such broad agreement that Google is now terrible for research that there's for sure a market for it. Edit: I'd love to see a some-of-the-web search engine like this. Start just with university sites, prepress archives, quality forums, public dev Slacks, etc.

aceazzameen 5y ago
Not only that, we just need one that works the way Google used to work. You know, something like the old algorithm that made Google famous and made competitors obsolete.
- alangibson 5y ago
  The hard part of what Google does is spam and abuse prevention. If you only indexed known-good sources I think the hard part would basically be done. Providing good search results, rich queries, sentiment analysis etc is all old hat by now.
  - jeffbee 5y ago
    "Known-good" sources have a serious habit of getting owned by malware and replaced by organized crime schemes. Anti-abuse can't be handled by whitelists, it needs to be an online system.
    - alangibson 5y ago
      Garbage sources have a serious habit of getting taken over. It's not that common with quality sites, but also not impossible. A simple downvoting system would be good enough.
      - marginalia_nu 5y ago
        The problem is that even quality sites die. And when they do, their domain gets poached and suddenly it's not a quality site anymore. But according to the ranking algorithms it's the same site, since its quality is measured by how many links it gets. Coopting quality sites is a bit of a cottage industry. There is also a significant amount of hacked wordpress sites. Their domain may be quite reputable, but they're unknowingly host to a ton of spam.
- temporaryi3 5y ago
  what you want is the internet with only a couple hundred million people connected, only a million or so content creators. Unless you have some pretty evil supervillain scheme, no moral compass, and succeed.... that world is never coming back.
  - pixelgeek 5y ago
    I think that if the company doing the search results wasn't also the company getting money from SEO spammers posting ads on their websites then the search results might be better.
- jeremyjh 5y ago
  That old algorithm would just hand you a bunch of SEO spam.
  - JohnWhigham 5y ago
    Google already does that though...
    - jeremyjh 5y ago
      Yes but the Google algorithm circa 2003 would be much, much worse. There has been an arms race for the last 20 years.
  - alok-g 5y ago
    I am not sure. It may be me lacking in understanding.
- mda 5y ago
  Old algorithm with todays internet content would be utterly garbage.
  - skinkestek 5y ago
    People keep saying that and yet the one person project search.marginalia.nu keeps delighting us. I can't imagine they use Googles newest algorithm.
    - marginalia_nu 5y ago
      I'm actually using Google's original algorithm, with a modification suggested in the original PageRank paper called Personalized PageRank that allows using a specific segment of the internet as a tastemaker in creating the ranking. It's a bit funny because the authors of the paper suggest this modification and explicitly states it makes the algorithm extremely resilient to manipulation from commercial interests.
- flenserboy 5y ago
  The spooks will take over that one, too.
vgeek 5y ago
I've pondered this, too. Why not a DMOZ 2.0 type concept, where you seed the index with high quality sites and grow from there. Freshness won't matter as much for most topics, so crawling X times per day probably won't be necessary. Maybe have user defined flags to indicate the types of results they want (large sites, small sites, high authority, technical sites-- maybe different sort by features) with backlinks/domain prominence as a small factor, but also using NLP to determine authoritativeness of said content so as to facilitate less-linked site discovery.
- alangibson 5y ago
  I think something that needs to be actively maintained like DMOZ is bound to burn out. I've been thinking about how you could passively maintain an index of good resources. Imagine a service that provides you with a personal search engine in exchange for a list of your bookmarks. Those bookmarks provide the signal for what sites to index for the public search engine.
- LargoLasskhyfv 5y ago
  That exists there: * https://curlie.org/ https://curlie.org/
NelsonMinar 5y ago
Neeva is worth looking at. I recently switched to it exclusively for a week and found it as good as Google for 90% of what I do. (FWIW, previous attempts doing this with Bing or DuckDuckGo last about a day before I get so mad I switch back.) Right now Neeva seems very good at navigational queries, which I do 90% of the time. It's still not as good at deep research queries for obscure things. Probably related: Neeva is relying on Bing for a big chunk of their queries. But they are building their own index.
- AlexCoventry 5y ago
  Yeah, I tried Neeva for a few days, but it was slowing me down.
- freediver 5y ago
  Neeva has the right business model (search as a product) which helps align interests. On the other hand some of their reasoning around the business execution left much to be desired: https://www.reddit.com/r/vivaldibrowser/comments/pol41p/comment/hd0ealo/ https://www.reddit.com/r/vivaldibrowser/comments/pol41p/comm...
- mpalczewski 5y ago
  Just tried this, even signed up. Immediately they started force feeding me "news". Looks like they added me to some (daily!) mailing list. It looks like more of the same, but this time you pay for it.
BitwiseFool 5y ago
I want a curated search engine so badly. I feel like some enterprising developer could craft one specifically geared towards developers. Imagine a search that doesn't ignore operators like '?.' and where you can set persistent conditions on every single query without having to type it out. This would help prevent you from needing to type in the language you are searching for in every query. Oh, and the cherry on top is completely abandoning the idea of Natural Language Processing. Go right back to keywords only.
- judge2020 5y ago
  Perhaps technical users are more likely to be running an ad blocker and/or are overall less likely to actually click on ads. If you're going to have 1/10th the CPM by making this search engine vs. one geared towards everyone, why would you spend the extra time and CPU cycles powering it? Maybe a subscription model would work but you'd have to show just how better it is since you're competing against good-enough free search engines.
- alangibson 5y ago
  I've considered taking a stab at it, but I didn't for likely the same reason no one else has: no idea how to monetize it. Sure you can run ads, but your conversion rates will likely stink because you'd be selecting for non-suckers.
  - eddieh 5y ago
    I’d love to take a stab at it. I think there are ways to monetize it that don’t involve advertising (at least enough to cover salaries and infrastructure). The only reason I haven’t is I don’t have access to the initial capital.
  - BitwiseFool 5y ago
    You might be able to monetize through closer partnerships with companies like Microsoft, Atlassian, Amazon, etc.. I'm envisioning an ad system that is basically geared towards selling their cloud and DevOps services. Because the engine is geared towards programming queries, you might also entice these companies by showing that developers have an easier time troubleshooting their systems. Perhaps, eventually, the engine could be bought out and act like a loss-leader or funnel to services. I know Microsoft is trying to be very developer friendly. Just some random thoughts. Monetization is a huge bootstrapping challenge for something like this.
jazzyjackson 5y ago
DevonAgent may do what you want, it felt very thorough and powerful but I never got in the habit of using it, there’s a learning curve to it https://www.devontechnologies.com/apps/devonagent#editions https://www.devontechnologies.com/apps/devonagent#editions mac os only. 5$ or 50$ for automation+archival
dna_polymerase 5y ago
> Everytime one of these threads show up, I find myself wondering why someone doesn't launch a technical and research focused competitor. There seems to be such broad agreement that Google is now terrible for research that there's for sure a market for it. Your bubble seems to agree, but the lack of serious competition, even in niches, is a sign that outside the HN bubble Google is in fact not seen as any worse.
- sidibe 5y ago
  Seems like for a decade several times a week this question shows up on HN and 99% of comments agree. Personally for my uses I don't remember it being significantly better, I'm not sure what I'm missing.
- alangibson 5y ago
  Not really. It's a sign that you can't profitably compete with Google.
marginalia_nu 5y ago
I did just that. Well not specifically aimed at research, but the types of websites I'm interested in, written by humans, blogs and such. It's got some quality issues right now because I fucked up some of the keyword extraction logic and it's taken a while to identify just what's gone wrong. Even though it's a bit broken, it has some lucid moments. Just compare: https://www.google.com/search?q=mechanical+keyboards https://www.google.com/search?q=mechanical+keyboards https://search.marginalia.nu/search?query=mechanical+keyboards https://search.marginalia.nu/search?query=mechanical+keyboar... The take-away I want to drive home is that it's absolutely possible to build something the scale of Google c.a. 2003 and run it on consumer hardware. I think, due to general difficulties in making these things profitable, the ideal approach is to make the operation so absurdly cheap it can be run non-profit instead. I'll gladly pay out of my own pocket to have a good search alternative.
- mrweasel 5y ago
  I love Marginalia, it's such a great project. You get really weird, but interesting results. Honestly I don't use it as much as I'd like, but it's still fun to go down a rabbit hole and read actual content.
  - marginalia_nu 5y ago
    It's probably quite far off being viable as a daily driver search engine, but maybe it doesn't need to be either. I built it to pick up the slack in the areas where the big search engines are floundering, and as long as it's doing that, I think that's quite sufficient.
- ycuser2 5y ago
  "random websites" is a great rabbit whole!
- skinkestek 5y ago
  Your project makes my life better both because it is already useful and because it gives me hope!
- gbrindisi 5y ago
wmil 5y ago
It's analogous to predatory pricing. Google has all the necessary tech to go back to doing good searches. They can easily pivot back in a few months. A new startup would need to dump loads of money into servers and building their own tech. So there's just no way to recoup the cost of building a competitor.
- systemvoltage 5y ago
  You might be surprised how much compute it takes. Not the same thing but, entire SO website runs on like 2 door sized racks. Search engine might be a different thing but if you have funding to get started, hardware costs aren’t going to be impossibly huge. Most is labor (engineering). I’m curious, how much compute power it takes to index the whole web? I presume queries are super fast.
  - alangibson 5y ago
    IMHO, indexing the whole web is an antifeature for technical and research topics. But there are projects like CommonCrawl that crawl a good portion of the web and give it away free.
  - estebarb 5y ago
    The monthly snapshots of CommonCrawl contains around 64k files of HTML, metadata and text only. The text only and metadata are around 250MB each, so in total the corpus would be around 16TB of text and 16TB of metadata. I don't know how much it costs to them, but when I used my own crawler I was able to download 500k webpages in around 4 hours (using the $80/month basic DigitalOcean server... let's say a total of $1/1M web pages, just the crawling process).
i-ning 5y ago
This site can help: http://teclis.com http://teclis.com
- alangibson 5y ago
  I just searched 'what is energy' and is see posts from Quora and Medium on the first page. Definitely not what someone doing real research would be looking for
  - CRConrad 5y ago
    Depends on how you define "real research". Would someone doing real research ask that question?
jpadkins 5y ago
what's the business model? Those verticals are probably not great for search ads. Do you think there is a large enough population of people willing to pay $X per month for a better technical + research search engine?

aceazzameen 5y ago

Not only that, we just need one that works the way Google used to work. You know, something like the old algorithm that made Google famous and made competitors obsolete.

alangibson 5y ago
The hard part of what Google does is spam and abuse prevention. If you only indexed known-good sources I think the hard part would basically be done. Providing good search results, rich queries, sentiment analysis etc is all old hat by now.
- jeffbee 5y ago
  "Known-good" sources have a serious habit of getting owned by malware and replaced by organized crime schemes. Anti-abuse can't be handled by whitelists, it needs to be an online system.
  - alangibson 5y ago
    Garbage sources have a serious habit of getting taken over. It's not that common with quality sites, but also not impossible. A simple downvoting system would be good enough.
    - marginalia_nu 5y ago
      The problem is that even quality sites die. And when they do, their domain gets poached and suddenly it's not a quality site anymore. But according to the ranking algorithms it's the same site, since its quality is measured by how many links it gets. Coopting quality sites is a bit of a cottage industry. There is also a significant amount of hacked wordpress sites. Their domain may be quite reputable, but they're unknowingly host to a ton of spam.
temporaryi3 5y ago
what you want is the internet with only a couple hundred million people connected, only a million or so content creators. Unless you have some pretty evil supervillain scheme, no moral compass, and succeed.... that world is never coming back.
- pixelgeek 5y ago
  I think that if the company doing the search results wasn't also the company getting money from SEO spammers posting ads on their websites then the search results might be better.
jeremyjh 5y ago
That old algorithm would just hand you a bunch of SEO spam.
- JohnWhigham 5y ago
  Google already does that though...
  - jeremyjh 5y ago
    Yes but the Google algorithm circa 2003 would be much, much worse. There has been an arms race for the last 20 years.
- alok-g 5y ago
  I am not sure. It may be me lacking in understanding.
mda 5y ago
Old algorithm with todays internet content would be utterly garbage.
- skinkestek 5y ago
  People keep saying that and yet the one person project search.marginalia.nu keeps delighting us. I can't imagine they use Googles newest algorithm.
  - marginalia_nu 5y ago
    I'm actually using Google's original algorithm, with a modification suggested in the original PageRank paper called Personalized PageRank that allows using a specific segment of the internet as a tastemaker in creating the ranking. It's a bit funny because the authors of the paper suggest this modification and explicitly states it makes the algorithm extremely resilient to manipulation from commercial interests.
flenserboy 5y ago
The spooks will take over that one, too.

vgeek 5y ago

I've pondered this, too. Why not a DMOZ 2.0 type concept, where you seed the index with high quality sites and grow from there. Freshness won't matter as much for most topics, so crawling X times per day probably won't be necessary. Maybe have user defined flags to indicate the types of results they want (large sites, small sites, high authority, technical sites-- maybe different sort by features) with backlinks/domain prominence as a small factor, but also using NLP to determine authoritativeness of said content so as to facilitate less-linked site discovery.

alangibson 5y ago
I think something that needs to be actively maintained like DMOZ is bound to burn out. I've been thinking about how you could passively maintain an index of good resources. Imagine a service that provides you with a personal search engine in exchange for a list of your bookmarks. Those bookmarks provide the signal for what sites to index for the public search engine.
LargoLasskhyfv 5y ago
That exists there: * https://curlie.org/ https://curlie.org/

NelsonMinar 5y ago

Neeva is worth looking at. I recently switched to it exclusively for a week and found it as good as Google for 90% of what I do. (FWIW, previous attempts doing this with Bing or DuckDuckGo last about a day before I get so mad I switch back.) Right now Neeva seems very good at navigational queries, which I do 90% of the time. It's still not as good at deep research queries for obscure things. Probably related: Neeva is relying on Bing for a big chunk of their queries. But they are building their own index.

AlexCoventry 5y ago
Yeah, I tried Neeva for a few days, but it was slowing me down.
freediver 5y ago
Neeva has the right business model (search as a product) which helps align interests. On the other hand some of their reasoning around the business execution left much to be desired: https://www.reddit.com/r/vivaldibrowser/comments/pol41p/comment/hd0ealo/ https://www.reddit.com/r/vivaldibrowser/comments/pol41p/comm...
mpalczewski 5y ago
Just tried this, even signed up. Immediately they started force feeding me "news". Looks like they added me to some (daily!) mailing list. It looks like more of the same, but this time you pay for it.

BitwiseFool 5y ago

I want a curated search engine so badly. I feel like some enterprising developer could craft one specifically geared towards developers. Imagine a search that doesn't ignore operators like '?.' and where you can set persistent conditions on every single query without having to type it out. This would help prevent you from needing to type in the language you are searching for in every query. Oh, and the cherry on top is completely abandoning the idea of Natural Language Processing. Go right back to keywords only.

judge2020 5y ago
Perhaps technical users are more likely to be running an ad blocker and/or are overall less likely to actually click on ads. If you're going to have 1/10th the CPM by making this search engine vs. one geared towards everyone, why would you spend the extra time and CPU cycles powering it? Maybe a subscription model would work but you'd have to show just how better it is since you're competing against good-enough free search engines.
alangibson 5y ago
I've considered taking a stab at it, but I didn't for likely the same reason no one else has: no idea how to monetize it. Sure you can run ads, but your conversion rates will likely stink because you'd be selecting for non-suckers.
- eddieh 5y ago
  I’d love to take a stab at it. I think there are ways to monetize it that don’t involve advertising (at least enough to cover salaries and infrastructure). The only reason I haven’t is I don’t have access to the initial capital.
- BitwiseFool 5y ago
  You might be able to monetize through closer partnerships with companies like Microsoft, Atlassian, Amazon, etc.. I'm envisioning an ad system that is basically geared towards selling their cloud and DevOps services. Because the engine is geared towards programming queries, you might also entice these companies by showing that developers have an easier time troubleshooting their systems. Perhaps, eventually, the engine could be bought out and act like a loss-leader or funnel to services. I know Microsoft is trying to be very developer friendly. Just some random thoughts. Monetization is a huge bootstrapping challenge for something like this.

jazzyjackson 5y ago

DevonAgent may do what you want, it felt very thorough and powerful but I never got in the habit of using it, there’s a learning curve to it https://www.devontechnologies.com/apps/devonagent#editions https://www.devontechnologies.com/apps/devonagent#editions mac os only. 5$ or 50$ for automation+archival

dna_polymerase 5y ago

> Everytime one of these threads show up, I find myself wondering why someone doesn't launch a technical and research focused competitor. There seems to be such broad agreement that Google is now terrible for research that there's for sure a market for it. Your bubble seems to agree, but the lack of serious competition, even in niches, is a sign that outside the HN bubble Google is in fact not seen as any worse.

sidibe 5y ago
Seems like for a decade several times a week this question shows up on HN and 99% of comments agree. Personally for my uses I don't remember it being significantly better, I'm not sure what I'm missing.
alangibson 5y ago
Not really. It's a sign that you can't profitably compete with Google.

marginalia_nu 5y ago

I did just that. Well not specifically aimed at research, but the types of websites I'm interested in, written by humans, blogs and such. It's got some quality issues right now because I fucked up some of the keyword extraction logic and it's taken a while to identify just what's gone wrong. Even though it's a bit broken, it has some lucid moments. Just compare: https://www.google.com/search?q=mechanical+keyboards https://www.google.com/search?q=mechanical+keyboards https://search.marginalia.nu/search?query=mechanical+keyboards https://search.marginalia.nu/search?query=mechanical+keyboar... The take-away I want to drive home is that it's absolutely possible to build something the scale of Google c.a. 2003 and run it on consumer hardware. I think, due to general difficulties in making these things profitable, the ideal approach is to make the operation so absurdly cheap it can be run non-profit instead. I'll gladly pay out of my own pocket to have a good search alternative.

mrweasel 5y ago
I love Marginalia, it's such a great project. You get really weird, but interesting results. Honestly I don't use it as much as I'd like, but it's still fun to go down a rabbit hole and read actual content.
- marginalia_nu 5y ago
  It's probably quite far off being viable as a daily driver search engine, but maybe it doesn't need to be either. I built it to pick up the slack in the areas where the big search engines are floundering, and as long as it's doing that, I think that's quite sufficient.
ycuser2 5y ago
"random websites" is a great rabbit whole!
skinkestek 5y ago
Your project makes my life better both because it is already useful and because it gives me hope!
gbrindisi 5y ago

wmil 5y ago

It's analogous to predatory pricing. Google has all the necessary tech to go back to doing good searches. They can easily pivot back in a few months. A new startup would need to dump loads of money into servers and building their own tech. So there's just no way to recoup the cost of building a competitor.

systemvoltage 5y ago
You might be surprised how much compute it takes. Not the same thing but, entire SO website runs on like 2 door sized racks. Search engine might be a different thing but if you have funding to get started, hardware costs aren’t going to be impossibly huge. Most is labor (engineering). I’m curious, how much compute power it takes to index the whole web? I presume queries are super fast.
- alangibson 5y ago
  IMHO, indexing the whole web is an antifeature for technical and research topics. But there are projects like CommonCrawl that crawl a good portion of the web and give it away free.
- estebarb 5y ago
  The monthly snapshots of CommonCrawl contains around 64k files of HTML, metadata and text only. The text only and metadata are around 250MB each, so in total the corpus would be around 16TB of text and 16TB of metadata. I don't know how much it costs to them, but when I used my own crawler I was able to download 500k webpages in around 4 hours (using the $80/month basic DigitalOcean server... let's say a total of $1/1M web pages, just the crawling process).

i-ning 5y ago

This site can help: http://teclis.com http://teclis.com

alangibson 5y ago
I just searched 'what is energy' and is see posts from Quora and Medium on the first page. Definitely not what someone doing real research would be looking for
- CRConrad 5y ago
  Depends on how you define "real research". Would someone doing real research ask that question?

jpadkins 5y ago

what's the business model? Those verticals are probably not great for search ads. Do you think there is a large enough population of people willing to pay $X per month for a better technical + research search engine?