5 ms·
> didn't realize Kagi had no aspirations to build their own general purpose index Kagi employee here. We're actively working on building our own indexes beyond
by nobodywasishere 4mo ago
> didn't realize Kagi had no aspirations to build their own general purpose index
Kagi employee here. We're actively working on building our own indexes beyond the limited ones we have now, not just a general index but also purpose built indexes for things like programming, etc.
- esperent 4mo agoHow broad will they be? Do you aim to ever have large scale indexing of the web?
- echelon 4mo agoHurry. Google might give up the ghost on its search product and maintaining indices on anything not geared for LLMs. I'm not sure antitrust will help you.
- jwr 4mo ago[flagged]
- redeeman 4mo agoyandex provides objectively better results for many things than google/ddg/bing
- pjc50 4mo agoThe results are kind of weird, but it does have one advantage: either nobody bothers to DMCA them, or they ignore it, or both.
- varispeed 4mo agoWestern companies also bought strategically important Nazi Germany industrial products in the 1930s because they were considered superior. Commercial convenience and technical quality are not moral or geopolitical absolution. Would you buy cheap quality gold if the Nazis were selling it knowing what it supports?
- account42 4mo agoI'm not sure about better but they do censor different topics and viewpoints which makes them an important source to aggregate.
- ragazzina 4mo agoWhat other countries do you want nothing to do with?
- virgil_disgr4ce 4mo agoThis is so funny, as though wanting to boycott specific entities is some kind of absurd notion, and as though saying "Sure, what ELSE don't you like?" is some kind of proof that it's an absurdity
- ljm 4mo agoIt kind of is though. Someone else will say "why are you sourcing results from an Israeli company?" and another will say "why are you sourcing results from a Chinese company?" and another one yet will say "why are you sourcing results from the US?". Why are the ethics of working with Yandex or Baidu any better or worse than the ethics of working with Google or Microsoft? Except that they're not western. The logical answer is that a person like this wants a very strong firewall, so ethically impure bits don't cross into their LAN.
- AlecSchueler 4mo ago> Why are the ethics of working with Yandex or Baidu any better or worse than the ethics of working with Google or Microsoft? A completely reasonable question that you should be able to answer if you're giving your money out to them.
- psychoslave 4mo agoMaybe not that ridiculous, since one can guess that the underlying thoughts are more about geostrategic concerns than favorite color of the day. The United States (I guess that's also the premise here, I'm not USA citizen myself) has notable rivalries with several countries, including China, Russia, Iran, North Korea, and Venezuela. These nations are often considered adversaries due to various geopolitical tensions and conflicts. I guess that Kahi is doing nothing illegal, so if people have that kind of question, it feels legitimate to reply with a demand of what is the extend this patriotism stance is going beyond the judicial requirements.
- severino 4mo agoWhat's the point in stopping paying money to Russia if Kagi is incorporated in Palo Alto, so it's paying money and will continue to do so to a country causing no less troubles than Russia?
- afavour 4mo agoTwo wrongs don’t make a right. Much of the HN audience lives in the US, giving money to the US is unavoidable in life. But in return we do have the democratic ability to try to alter the behavior of the country. It still makes sense to avoid giving money to other bad actors who are acting in direct opposition to your home country, and whom you have no control over, when you can.
- faangguyindia 4mo agoWhy that ability didn't result in less loss of lives in middle East for such a long time.
- severino 4mo agoThen one should ask for Kagi to move to a country that doesn't use taxes to invade other countries, and stop supporting them, instead of just asking to "stop paying money to Russia". Because fixing two wrongs is better than just fixing one, right? And I want to believe that living in the US doesn't make you agree with your country's foreign policy (which doesn't seem to change significantly even with your democratic ability to try to alter it) specially if you're worried about Russia.
- Vasbarlog 4mo agoA lot more one could argue.
- yehat 4mo agoIf a search engine starts censoring by whatever means, I'll not be using it, neither free or paid no matter how good it is (and by definition it can't be). Shills can provide a list of countries they don't want to see, hopefully some crooked search engine will satisfy their desire for censorship.
- flir 4mo agoThen you have to stop using search engines.
- mort96 4mo agoCensorship of search results and deciding which companies to do business with are two completely separate topics.
- account42 4mo agoIn this case it isn't since there is no alternative that covers the censorship holes that Yandex does.
- smallerize 4mo agoCensorship? This isn't about free speech, there's a war on.
- severino 4mo agoThere's not just one war on, there are many. Should Kagi exclude all the results from American companies because of the actions of the US government in Iran?
- deleted 4mo ago[deleted]
- smallerize 4mo agoDepends which side of the war you're on I guess? Or more to the point, whose money kagi wants to go after.
- itdoesntjust 4mo ago[dead]
- levhawk 4mo agoRu-speaking audience is ~2 times bigger than Russia, why would they cut this? Only because of SJWs like you? I'd much prefer one search engine that searches well on two languages, instead of using different engines for each language. Ru-net has huge amount of usefulness in it, cutting it out is like cutting a finger. Fun fact, before the RU-UKR war, most Ukranians contributed to the Runet, so that would cut their heritage too.
- mort96 4mo agoYandex isn't just a "Russian language search engine", it's a Russian company with quite close ties to the Kremlin. See: https://www.zois-berlin.de/en/publications/zois-spotlight/the-sad-fate-of-yandex-from-independent-tech-startup-to-kremlin-propaganda-tool https://www.zois-berlin.de/en/publications/zois-spotlight/th... I'm frankly a bit surprised that it's even legal for a US company like Kagi to do business with Yandex, considering it's sanctioned: https://sanctionssearch.ofac.treas.gov/Details.aspx?id=18711 https://sanctionssearch.ofac.treas.gov/Details.aspx?id=18711. Though in fairness, I don't know enough about how exactly sanction laws work so it might be legally okay even if I find it morally questionable.
- notpushkin 4mo agoYandex has quite a few international entities, which are probably not direct subsidiaries, which in turn probably helps with sanctions. Yandex Cloud seems to be sold by a UAE company internationally: https://yandex.cloud/en/about#impressum https://yandex.cloud/en/about#impressum
- levhawk 4mo agoYes, Yandex was effectively seized by Ru govt "friends" and turned into propaganda tool. And that's very unfortunate. But I'm not talking about getting the propaganda, only about search index. About money - don't think this income is even remotely comparable with oil/gas incomes, which EU passes to RU. Please don't start political debate. I do not like censorship of any kind, hence my initial response. I want to have available information in full.
- xnx 4mo ago> We're actively working on building our own indexes Lip service. You'll have some token index of Wikipedia or something so you can say your results are "a blend of our own index and other sources".
- freehorse 4mo agoWikipedia is prob in "other sources", as they actually say they have a direct license for it. https://blog.kagi.com/waiting-dawn-search#:~:text=Wikipedia,license https://blog.kagi.com/waiting-dawn-search#:~:text=Wikipedia,...
- account42 4mo agoLol everyone has a direct license for Wikipedia.
- nobodywasishere 4mo agoIt's funny you say that as we just switched the Wikipedia widget over to our own internal index. We don't intend to stop there either.
- ljm 4mo agoWhat are the challenges of doing that when so much of the internet has turned itself into SEO slop to fit Google's algorithms? I imagine there is still a whole load of stuff out there on the internet that Google would never surface because it doesn't have enough adsense or whatever. Are you finding that?
- nobodywasishere 4mo agoThat's what SlopStop is for :) developing methodologies that scale for detecting slop. I mean it sounds like that already has a lot of overlap with our Small Web indexing efforts, so that part of our indexing efforts could be an extension of that. A lot if this is still in development though so I can't speak on specifics just yet.
- account42 4mo agoThere is a very easy 90% solution to that: massively downrank everything with ads with some exceptions added as needed.
- rpdillon 4mo agoI did not intend to spread misinformation here, and would like to hear more about the general-purpose index Kagi is working on. I had based my comment on several Kagi pages, but mostly https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm..., which mentions Teclis as Kagi's own index, but https://teclis.com/ https://teclis.com/ makes it pretty clear that it's a "small web"-focused tool: > Teclis is an attempt to surface the less known web, the web of creativity and self expression, the more humane web. > Teclis includes its own crawl as well as results from Kagi Small Web index and results with permission from Marginalia Search. > Teclis works best with broad queries such as 'machine learning', 'vegan diet', 'religion' etc.. Is there another crawler doing the general-purpose stuff?
- nobodywasishere 4mo agoNot in production yet, so will have more to share in time.
- rpdillon 4mo agoFair enough, thanks for replying!
- Xunjin 4mo agoHey do you guys have posts or sharing about it? It would be awesome to see what you are trying to accomplish, maybe it's time to post on HN ;)
- Dwedit 4mo agoHow do you build a search index in the days of Anubis pages everywhere?
- account42 4mo agoAnubis is easy, just use a whitelisted user agent or a headless browser if some sites disable that - you need one to index web app abominations anyway. Cloudflare and Google reCaptcha are bigger problems.
- majorchord 4mo agohttps://roundproxies.com/blog/how-to-bypass-anti-bots/ https://roundproxies.com/blog/how-to-bypass-anti-bots/
- tokai 4mo agoSo you will stop buying Yandex data at some point?
- sph 4mo agoI care more from breadth of the dataset than politics in my search engine, thank you very much. For everybody else there’d Google I guess.