8 ms·

Right Dao, a new independent search engine that doesn't track users

57 points by ivyabc 6y ago

ivyabc 6y ago
(Disclaimer: Self Promotion) Today, Google accounted for 92% of all Internet searches. Google and Bing together occupy 96% of the search market. Most other search engines, including the popular privacy focused ones, simply get the search results from Google or Bing and reorder them, because developing a search engine is neither easy nor cost effective. Consequently, the few big tech companies control what people see. Right Dao is different. We are a fully independent search engine, and we have the infrastructure and build the technology from the ground up. That enable us to show the search results free from search engine monopoly's manipulations. We invite you to have a try, while we are constantly improving the quality and adding more features. https://rightdao.com https://rightdao.com FAQ: open source? We think search engine's code is a bit sensitive in general. The engine depends on many subsystems, including storage, scheduling, indexing, etc. If the ranking code is seen by the spammers, they could push their spam websites. We are not sure if there's a viable way to open source a search engine.
- deleted 6y ago
  [deleted]
- DarthGhandi 6y ago
  > We are not sure if there's a viable way to open source a search engine. Yacy seems to manage just fine being open source and capable of self hosting.
- frongpik 6y ago
  Looks very decent. Make it configurable and I'll pay for a subscription. What I mean is instead of indexing the entire internet with an adhoc ranking, trying to guess what I want, let me whitelist and blacklist domains and let me configure the ranking. I'd begin with stackoverflow, hackernews and arxiv and probably blacklist pinterest and other paywalled gardens. From time to time your search engine could suggest search results from other sources, so I could update my whitelist. It could index pdf files or even show their summary with some ml model, perhaps for an additional fee. Another idea that's been bothering me for a while is searching for movies or songs. If you figure how to show me the most interesting (according to my filters) movies in 2020, I'd pay for that. Even more so for music.
  - corvuscorvid 6y ago
    This comment made me envision a tabbed results page with my results from my curated, first choice sources, and a second tab of general results based from the search engine's best guess algorithms.
- rendaw 6y ago
  This is super snappy! How are you funded? Is it right that you don't collect any user information, including ip address?
- m-i-l 6y ago
  Looks promising. But how do we know you are doing your own indexing and not simply buying in the results from Google or Bing like pretty much all of the other search startups? And if you are doing your own indexing, I see you include some very large sites like wikipedia which is going to cost a lot of money to index on a regular basis, so how are you going to pay for this in a sustainable way?
  - ivyabc 6y ago
    Our results are different from other search engines. Wikipedia has regular database dumps (https://dumps.wikimedia.org/ https://dumps.wikimedia.org/), which is a relatively low cost to index. Overall, our scale is currently small and the cost is manageable.
NonEUCitizen 6y ago
Re: "independent," how are you funded (so you can pay for infrastructure)? Are you independent of VCs? of governments?
- ivyabc 6y ago
  Currently our scale is small and the cost is manageable. We are not funded by venture capitals or governments.
  - permo-w 6y ago
    Yes, but how are you funded?
drannex 6y ago
Do you have your own indexer or are you relying on others?
- deleted 6y ago
  [deleted]
- ivyabc 6y ago
  We have our own crawler and indexing.
  - dpezely 6y ago
    Have you considered using Common Crawl [1], and if so, what was your assessment when compared to having your own spyders? Long-term, a combination of theirs and your own could be optimal. There are strengths and weaknesses with using their dumps: on one hand, benefits include them having crawled and having dealt with being throttled, etc. They offer monthly dumps for general content and daily dumps for news [2]. On the other hand, it's a huge pile of data to wade through, and their index format might not be your preferred method. The archive and index reside officially at AWS, so that may decide where to process it. (Not sure whether other providers maintain a copy as well or not.) By "huge", specifically: > October 2020 [...] contains 2.71 billion web pages or 280 TiB of uncompressed content. From our analysis a few years ago, that was to be the approach for the now-defunct Snagz.net [3] (which never fully launched because co-founders were unable to join due to extenuating circumstances). [1] https://CommonCrawl.org https://CommonCrawl.org [2] https://commoncrawl.org/2016/10/news-dataset-available/ https://commoncrawl.org/2016/10/news-dataset-available/ - this one can be hard to find unless you know to look for it [3] https://web.archive.org/web/20180320001756/http://snagz.net/ https://web.archive.org/web/20180320001756/http://snagz.net/
    - ivyabc 6y ago
      We think the quality of their crawled pages (both web and news) is not as good as ours. Our total dataset is larger than their monthly numbers.
jmcs 6y ago
"The Services are offered from the United States of America and, regardless of your place of residence or access location, your use of them is governed by the laws of the United State of America. Right Dao makes no representations that the Sites are appropriate for use in other locations or are legal in all jurisdictions. Those who access the Sites from other locations do so at their own risk and consent to the transfer and processing of their data in the United States of America and any other jurisdiction throughout the world." That's half the reason I don't use Google right there and they pretend to follow European laws. So, yeah, right, no thanks.
- indymike 6y ago
  Here's the problem with EU regulations: they really do bar small projects and hobbyists and assume that even the smallest website is backed by a corporation with resources to comply with pretty complex rules.
  - jiehong 6y ago
    Actually, that's not really true. If you don't store any user information, you're compliant. It's only if you start storing those that you have some rules to follow. Nowadays, it's the same if you are in California with the recent data protection laws. Also, Right Dao is under New York law, so it has to follow US law I guess.
    - indymike 6y ago
      "If you don't store any user information, you're compliant." So, how do I do business with people?
      - marenkay 6y ago
        That is a horribly wrong view point. Information required for business purposes, e.g. to write invoices or file taxes is considered user information you are fine to retain. It's not about not having information, it's about having consent before acquiring it.
rho4 6y ago
grammar and spelling on the about page are not very confidence inspiring (given this is a search engine)
jaclaz 6y ago
I don't know. Is it "English only"? I just searched on it for: CR2032 "due fili" ("due fili" means "two wires" in Italian) i.e. the button battery for RTC/CMOS on portables And results are essentially "completely random" pages without neither CR2032 nor "due fili", most notably second result is: https://en.wikipedia.org/wiki/United_States https://en.wikipedia.org/wiki/United_States and third is: https://en.wikipedia.org/wiki/Customer_relationship_management https://en.wikipedia.org/wiki/Customer_relationship_manageme...
- tkgally 6y ago
  I tried a few searches in Japanese. The results for search terms written in kanji were okay, but searches for terms written in kana—such as アメリカ or ぴかぴか—yielded no results at all.
benboughton1 6y ago
I like it, but how is it funded? Can I submit a site to be indexed?
- ivyabc 6y ago
  Currently our scale is small and the cost is manageable. Please send your site to the email address in the about us page. (We don't have incremental indexing yet, so we have to replace the entire index with newer results once a while.)
jhoechtl 6y ago
I want this to succeed. I have been searching for a while for a search engine which is independent from Google or Bing. First searches have been relevant and also refreshing different and new from the big ones. Battle-testing will tell.
- ColinHayhurst 6y ago
  I maintain this list of search engines: https://twitter.com/SearchEngineMap/lists https://twitter.com/SearchEngineMap/lists Disclosure: Mojeek team member Happy to add Right Dao but did not find it on Twitter.
  - ivyabc 6y ago
    Thanks! We have created our Twitter page. https://twitter.com/RightDao https://twitter.com/RightDao
rstarast 6y ago
Impressed by the speed of the results. Some of my more obscure tests didn't give relevant results, but I guess this is still early on. Do you have any numbers about the size of the index, and where you're aiming to go? My big question: What's behind the name? I find it a bit confusing and not very memorable at first sight, maybe an explanation would help.
- ivyabc 6y ago
  From Wikipedia: Dao is a Chinese word signifying the "way", "path", "route", "road"... In most belief systems, the word is used symbolically in its sense of 'way' as the 'right' or 'proper' way of existence... https://en.wikipedia.org/wiki/Tao https://en.wikipedia.org/wiki/Tao
rjknight 6y ago
"Dao" here is probably not a reference to Daoism, but to "decentralised autonomous organisations". Most often this is just a cyberpunky way of trying to avoid either tax or legal liability, but I'm not sure that either necessarily apply here. It would be interesting to know how the project is structured and funded.
pembrook 6y ago
Is this just a rebrand of bing’s results like DuckDuckGo? Or is there proprietary tech behind this?
- ivyabc 6y ago
  No, we use Kubernetes, grpc and protobuf as base, Prometheus and Grafana for monitoring. Our systems are mostly in C++, with some in Go such as crawler. No open source tool is used for indexing and search.
ralphc 6y ago
How often is this indexed? I have a great example for today, December 3rd. Yesterday, Salesforce announced a new product, Hyperforce. It's a new way to deploy their product, basically a big deal from a big company. Searching Salesforce Hyperforce gets no results in rightdao, plenty in duckduckgo.
- ivyabc 6y ago
  We don't have incremental indexing yet, so we have to replace the entire index with newer results once a while. Incremental index is on our roadmap. However, our news search is updated every hour, and Salesforce Hyperforce has new results. https://rightdao.com/search?q=Salesforce+Hyperforce&type=news https://rightdao.com/search?q=Salesforce+Hyperforce&type=new...
- ivyabc 6y ago
  By the way, thanks for your feedback. We have fixed a bug and now news stories are shown on the page. https://rightdao.com/search?q=Salesforce+Hyperforce https://rightdao.com/search?q=Salesforce+Hyperforce
kwhitefoot 6y ago
If I search for a person by given name and family name with quotation marks around the whole string then I would expect the hits that include the text to be at the top of the list. Google manages this for my name but Right Dao does not. If I search for my name like this: "kevin whitefoot" the first 27 hits on Google are directly relevant and my name appears in the link or in the extracted text. Right Dao on the other hand returns a list where the most of the hits do not include my name as quoted just the two words separately which means that the hits are completely irrelevant as they refer to a completely different person. So how does one search for a person by name?
StillBored 6y ago
Wow, the response times are crazy fast!
justaj 6y ago
Hmm, so when searching without JS, the links from results are behind a redirect. So in terms of tracking this is worse than DDG.
Fat_Thor 6y ago
Please add opensearch data so my browser can add to search list.
batrachom 6y ago
And yet, you haven't told us how you are funded or how you plan to be... And you retain data. I honestly don't see any reason to think you are any better than your comqetitors.
Jacksonkui 6y ago
Great
Nooshint 6y ago
Hello
Nooshint 6y ago
Parler
patgovender 6y ago
Yippee.the world was waiting for this search engine. Pat Govender
user711 6y ago
Golf

57 points by ivyabc 6y ago

(Disclaimer: Self Promotion) Today, Google accounted for 92% of all Internet searches. Google and Bing together occupy 96% of the search market. Most other search engines, including the popular privacy focused ones, simply get the search results from Google or Bing and reorder them, because developing a search engine is neither easy nor cost effective. Consequently, the few big tech companies control what people see. Right Dao is different. We are a fully independent search engine, and we have the infrastructure and build the technology from the ground up. That enable us to show the search results free from search engine monopoly's manipulations. We invite you to have a try, while we are constantly improving the quality and adding more features. https://rightdao.com https://rightdao.com FAQ: open source? We think search engine's code is a bit sensitive in general. The engine depends on many subsystems, including storage, scheduling, indexing, etc. If the ranking code is seen by the spammers, they could push their spam websites. We are not sure if there's a viable way to open source a search engine.

deleted 6y ago
[deleted]
DarthGhandi 6y ago
> We are not sure if there's a viable way to open source a search engine. Yacy seems to manage just fine being open source and capable of self hosting.
frongpik 6y ago
Looks very decent. Make it configurable and I'll pay for a subscription. What I mean is instead of indexing the entire internet with an adhoc ranking, trying to guess what I want, let me whitelist and blacklist domains and let me configure the ranking. I'd begin with stackoverflow, hackernews and arxiv and probably blacklist pinterest and other paywalled gardens. From time to time your search engine could suggest search results from other sources, so I could update my whitelist. It could index pdf files or even show their summary with some ml model, perhaps for an additional fee. Another idea that's been bothering me for a while is searching for movies or songs. If you figure how to show me the most interesting (according to my filters) movies in 2020, I'd pay for that. Even more so for music.
- corvuscorvid 6y ago
  This comment made me envision a tabbed results page with my results from my curated, first choice sources, and a second tab of general results based from the search engine's best guess algorithms.
rendaw 6y ago
This is super snappy! How are you funded? Is it right that you don't collect any user information, including ip address?
m-i-l 6y ago
Looks promising. But how do we know you are doing your own indexing and not simply buying in the results from Google or Bing like pretty much all of the other search startups? And if you are doing your own indexing, I see you include some very large sites like wikipedia which is going to cost a lot of money to index on a regular basis, so how are you going to pay for this in a sustainable way?
- ivyabc 6y ago
  Our results are different from other search engines. Wikipedia has regular database dumps (https://dumps.wikimedia.org/ https://dumps.wikimedia.org/), which is a relatively low cost to index. Overall, our scale is currently small and the cost is manageable.

NonEUCitizen 6y ago

Re: "independent," how are you funded (so you can pay for infrastructure)? Are you independent of VCs? of governments?

ivyabc 6y ago
Currently our scale is small and the cost is manageable. We are not funded by venture capitals or governments.
- permo-w 6y ago
  Yes, but how are you funded?

drannex 6y ago

Do you have your own indexer or are you relying on others?

deleted 6y ago
[deleted]
ivyabc 6y ago
We have our own crawler and indexing.
- dpezely 6y ago
  Have you considered using Common Crawl [1], and if so, what was your assessment when compared to having your own spyders? Long-term, a combination of theirs and your own could be optimal. There are strengths and weaknesses with using their dumps: on one hand, benefits include them having crawled and having dealt with being throttled, etc. They offer monthly dumps for general content and daily dumps for news [2]. On the other hand, it's a huge pile of data to wade through, and their index format might not be your preferred method. The archive and index reside officially at AWS, so that may decide where to process it. (Not sure whether other providers maintain a copy as well or not.) By "huge", specifically: > October 2020 [...] contains 2.71 billion web pages or 280 TiB of uncompressed content. From our analysis a few years ago, that was to be the approach for the now-defunct Snagz.net [3] (which never fully launched because co-founders were unable to join due to extenuating circumstances). [1] https://CommonCrawl.org https://CommonCrawl.org [2] https://commoncrawl.org/2016/10/news-dataset-available/ https://commoncrawl.org/2016/10/news-dataset-available/ - this one can be hard to find unless you know to look for it [3] https://web.archive.org/web/20180320001756/http://snagz.net/ https://web.archive.org/web/20180320001756/http://snagz.net/
  - ivyabc 6y ago
    We think the quality of their crawled pages (both web and news) is not as good as ours. Our total dataset is larger than their monthly numbers.

jmcs 6y ago

"The Services are offered from the United States of America and, regardless of your place of residence or access location, your use of them is governed by the laws of the United State of America. Right Dao makes no representations that the Sites are appropriate for use in other locations or are legal in all jurisdictions. Those who access the Sites from other locations do so at their own risk and consent to the transfer and processing of their data in the United States of America and any other jurisdiction throughout the world." That's half the reason I don't use Google right there and they pretend to follow European laws. So, yeah, right, no thanks.

indymike 6y ago
Here's the problem with EU regulations: they really do bar small projects and hobbyists and assume that even the smallest website is backed by a corporation with resources to comply with pretty complex rules.
- jiehong 6y ago
  Actually, that's not really true. If you don't store any user information, you're compliant. It's only if you start storing those that you have some rules to follow. Nowadays, it's the same if you are in California with the recent data protection laws. Also, Right Dao is under New York law, so it has to follow US law I guess.
  - indymike 6y ago
    "If you don't store any user information, you're compliant." So, how do I do business with people?
    - marenkay 6y ago
      That is a horribly wrong view point. Information required for business purposes, e.g. to write invoices or file taxes is considered user information you are fine to retain. It's not about not having information, it's about having consent before acquiring it.

rho4 6y ago

grammar and spelling on the about page are not very confidence inspiring (given this is a search engine)

jaclaz 6y ago

I don't know. Is it "English only"? I just searched on it for: CR2032 "due fili" ("due fili" means "two wires" in Italian) i.e. the button battery for RTC/CMOS on portables And results are essentially "completely random" pages without neither CR2032 nor "due fili", most notably second result is: https://en.wikipedia.org/wiki/United_States https://en.wikipedia.org/wiki/United_States and third is: https://en.wikipedia.org/wiki/Customer_relationship_management https://en.wikipedia.org/wiki/Customer_relationship_manageme...

tkgally 6y ago
I tried a few searches in Japanese. The results for search terms written in kanji were okay, but searches for terms written in kana—such as アメリカ or ぴかぴか—yielded no results at all.

benboughton1 6y ago

I like it, but how is it funded? Can I submit a site to be indexed?

ivyabc 6y ago
Currently our scale is small and the cost is manageable. Please send your site to the email address in the about us page. (We don't have incremental indexing yet, so we have to replace the entire index with newer results once a while.)

jhoechtl 6y ago

I want this to succeed. I have been searching for a while for a search engine which is independent from Google or Bing. First searches have been relevant and also refreshing different and new from the big ones. Battle-testing will tell.

ColinHayhurst 6y ago
I maintain this list of search engines: https://twitter.com/SearchEngineMap/lists https://twitter.com/SearchEngineMap/lists Disclosure: Mojeek team member Happy to add Right Dao but did not find it on Twitter.
- ivyabc 6y ago
  Thanks! We have created our Twitter page. https://twitter.com/RightDao https://twitter.com/RightDao

rstarast 6y ago

Impressed by the speed of the results. Some of my more obscure tests didn't give relevant results, but I guess this is still early on. Do you have any numbers about the size of the index, and where you're aiming to go? My big question: What's behind the name? I find it a bit confusing and not very memorable at first sight, maybe an explanation would help.

ivyabc 6y ago
From Wikipedia: Dao is a Chinese word signifying the "way", "path", "route", "road"... In most belief systems, the word is used symbolically in its sense of 'way' as the 'right' or 'proper' way of existence... https://en.wikipedia.org/wiki/Tao https://en.wikipedia.org/wiki/Tao

rjknight 6y ago

"Dao" here is probably not a reference to Daoism, but to "decentralised autonomous organisations". Most often this is just a cyberpunky way of trying to avoid either tax or legal liability, but I'm not sure that either necessarily apply here. It would be interesting to know how the project is structured and funded.

pembrook 6y ago

Is this just a rebrand of bing’s results like DuckDuckGo? Or is there proprietary tech behind this?

ivyabc 6y ago
No, we use Kubernetes, grpc and protobuf as base, Prometheus and Grafana for monitoring. Our systems are mostly in C++, with some in Go such as crawler. No open source tool is used for indexing and search.

ralphc 6y ago

How often is this indexed? I have a great example for today, December 3rd. Yesterday, Salesforce announced a new product, Hyperforce. It's a new way to deploy their product, basically a big deal from a big company. Searching Salesforce Hyperforce gets no results in rightdao, plenty in duckduckgo.

ivyabc 6y ago
We don't have incremental indexing yet, so we have to replace the entire index with newer results once a while. Incremental index is on our roadmap. However, our news search is updated every hour, and Salesforce Hyperforce has new results. https://rightdao.com/search?q=Salesforce+Hyperforce&type=news https://rightdao.com/search?q=Salesforce+Hyperforce&type=new...
ivyabc 6y ago
By the way, thanks for your feedback. We have fixed a bug and now news stories are shown on the page. https://rightdao.com/search?q=Salesforce+Hyperforce https://rightdao.com/search?q=Salesforce+Hyperforce

kwhitefoot 6y ago

If I search for a person by given name and family name with quotation marks around the whole string then I would expect the hits that include the text to be at the top of the list. Google manages this for my name but Right Dao does not. If I search for my name like this: "kevin whitefoot" the first 27 hits on Google are directly relevant and my name appears in the link or in the extracted text. Right Dao on the other hand returns a list where the most of the hits do not include my name as quoted just the two words separately which means that the hits are completely irrelevant as they refer to a completely different person. So how does one search for a person by name?

StillBored 6y ago

Wow, the response times are crazy fast!

justaj 6y ago

Hmm, so when searching without JS, the links from results are behind a redirect. So in terms of tracking this is worse than DDG.

Fat_Thor 6y ago

Please add opensearch data so my browser can add to search list.

batrachom 6y ago

And yet, you haven't told us how you are funded or how you plan to be... And you retain data. I honestly don't see any reason to think you are any better than your comqetitors.

Jacksonkui 6y ago

Great

Nooshint 6y ago

Hello

Nooshint 6y ago

Parler

patgovender 6y ago

Yippee.the world was waiting for this search engine. Pat Govender

user711 6y ago

Golf