7 ms·

Gigablast Search Engine

235 points by SubiculumCode 5y ago

eatbitseveryday 5y ago
Source of this post likely from [1] [1] https://news.ycombinator.com/item?id=29417296 https://news.ycombinator.com/item?id=29417296
- SubiculumCode 5y ago
  Yep, I posted this today after the author of gigablast posted a comment in this https://news.ycombinator.com/item?id=29417061 https://news.ycombinator.com/item?id=29417061 today
superasn 5y ago
I used to donate my idle cpu to seti@home back in the day. Wonder if the same can be done for creating an open search engine to compete with Google. Also since the resources are crowd sourced it can make it easy to get around rate limits and anti scraping too.
- boyter 5y ago
  You can do this with Yacy right now, https://yacy.net https://yacy.net but it's not great for results generally. I have often wondered if something built on activity pub or like that would be an option allowing people to group servers with peers they like or trust. Its something I want to implement actually and may get around to doing one of these days.
  - zdkl 5y ago
    Well for starters, one could implement the API to return ActivityStreams formatted responses. That would be a good start to being compatible with the fediverse and stuff while not going insane in implementing a full and proper ActivityPub service. Been there, done that, way better tool for "lower level" features. [0] https://www.w3.org/TR/activitystreams-core/ https://www.w3.org/TR/activitystreams-core/
    - boyter 5y ago
      That roughly what I thought. I’m not familiar with activitypub at all. I will probably investigate this deeper. You see to have some knowledge in this area. Do you have any suggestions of places to look to achieve something like this?
- tigerlily 5y ago
  Perhaps not weirdly I had the same thought yesterday [1]. https://yacy.net https://yacy.net was suggested in response. [1] https://news.ycombinator.com/item?id=29417925 https://news.ycombinator.com/item?id=29417925
- SubiculumCode 5y ago
  That's a really cool idea!
- R0b0t1 5y ago
  I don't think it would be that easy. You need to distribute the indexing data. But you could federate search servers and have them send queries to others.
- SubiculumCode 5y ago
  I know the usual thing against crypto, but I wonder whether a gridcoin model would work. https://gridcoin.us/ https://gridcoin.us/
1cvmask 5y ago
I saw this in a thread earlier today. I couldn't understand why it has a login and account. They seem to be the anti-Google and and anti-personalization search engine.
- SubiculumCode 5y ago
  good question
- GistNoesis 5y ago
  You don't need an account to make searches though : https://gigablast.com/index.html https://gigablast.com/index.html If you have an account you can probably log your queries.
- aquarin 5y ago
  It looks, you need a account to add url-s. "You need to login to use the add url tool. " "Each added url is $0.25."
nixgeek 5y ago
“Gigablast has teamed up with Imperial Family Companies to create a next generation private search engine, private.sh.” Imperial Family Companies are the people who essentially destroyed [1] the freenode IRC network, aren’t they? [1] https://netsplit.de/networks/history/top10_2021u.png https://netsplit.de/networks/history/top10_2021u.png
- humanistbot 5y ago
  Yes, the same ones [1]. They are an investment firm that was formerly known as London Trust Media. You can see they have listed both "IRC" (links to irc.com) and "freenode" in their portfolio. [2] [1] https://lists.ubuntu.com/archives/ubuntu-irc/2021-May/001923.html https://lists.ubuntu.com/archives/ubuntu-irc/2021-May/001923... [2] https://imperialfamily.com/ https://imperialfamily.com/ (hover over "Technology)
  - mrtweetyhack 5y ago
    let's just make sure we destroy everything they invest in
- superkuh 5y ago
  It's hard to believe they'd take money from someone that attacked so many open source projects earlier this year by leveraging "donations". They should be careful.
  - ludamad 5y ago
    At the same time, they're in a position to consent to a transition, not sure there's a community of collective ownership here like with freenode
twofornone 5y ago
Well, pirate bay is returned in search results, so that's a good start...
bigyellow 5y ago
> Furthermore, the client-side javascript on private.sh encrypts any query done on private.sh so that only Gigablast can read it. Therefore, no single party has access to both the IP address and the query. This is something that is truly unique and truly powerful, and, right now, only private.sh can supply this level of privacy. Run proprietary Javascript for privacy - what a fallacious concept. Going to assume this service is a honeypot or run by incompetent staff - pass.
- gbmatt 5y ago
  the javascript is run by your browser, so you can fully audit it.
  - bigyellow 5y ago
    It's still served by the site and I doubt most are interested or capable in auditing software to perform routine online tasks.
    - rasengan 5y ago
      There's an extension as well [1]. This means that the code is not being served by the server in this use case. [1] https://private.sh/extension.html https://private.sh/extension.html
    - eftychis 5y ago
      I am not sure there are good solutions besides going off browser. P.S. I was involved in user authorization, attestation and privacy flows for a particular product recently and the browser was always where shit hit the fan. The web features are just not made with simplicity and privacy in mind. Then again we had more complex constraints.
- mrtweetyhack 5y ago
  Only Gigablast can read it means it is not private
boyter 5y ago
Matt Wells who wrote the majority of the code for gigablast is someone I have been following online for a long time. I used to live for http://gigablast.com/rants.html http://gigablast.com/rants.html updates. Gigablast being an amazing example of what one person can do given the time and effort. If you look around for articles and interviews by him you get some nice insights that would never come from the likes of Google, although Bing has some very good in depth technical discussions such as how bitfunnel works. It’s also nice to look through the code of it and see how things like porn filters were implemented. It’s also nice to know that for a while in internet history gigablast was mentioned in the same breath as google. An amazing achievement at the time for a single person against the core google product. I wish that someone with some design chops could work on it for a few weeks though. Or it was rolled back to the design from around 2006 with the rocket logo. I really liked that design. I really wish I had the courage to strike out on my own like Matt has. I have written a few search engines, but a general purpose one from scratch on my own hardware is unlikely to ever happen, as much as I would love to do it. That’s for inspiring me so much Matt if you do read this (notice me senpai!). Oh and sorry for abusing your XML api so much. I was poor at the time and needed some search results. A few choice articles, https://queue.acm.org/detail.cfm?id=988401 https://queue.acm.org/detail.cfm?id=988401 https://www.abc.net.au/news/science/2021-02-14/google-news-media-bargaining-code-build-search-engine/13143582 https://www.abc.net.au/news/science/2021-02-14/google-news-m...
- gbmatt 5y ago
  thanks ben, you are too kind.
  - samhw 5y ago
    I second this - thanks for building this. It's an unbelievably inspiring achievement. It's my default search engine, and I'm really glad it exists.
  - boyter 5y ago
    Wow even knows me by my first name too. Very humbled. Once again thanks for being so open with what you have done.
ohiovr 5y ago
it found this: https://gigablast.com/search?c=main&qlangcountry=en-us&q=how+to+resolve+missing+symbol+r+android https://gigablast.com/search?c=main&qlangcountry=en-us&q=how... Which is definitely a good sign of a competent search engine.
gbmatt 5y ago
hey thanks for the recognition, people. :) finally, all my problems are solved. this comment is here for hacker news karma points.
- kingcharles 5y ago
  *throws karma at the screen*
- ivanche 5y ago
  Hey Matt, would you consider making a search box (input with id="q") a bit wider? I can type only around 15 characters before the beginning of search query becomes "cut off".
  - InfiniteRand 5y ago
    The too small text box is also a pain when deleting the query in order to type a new one on mobile. A clear button would mitigate some of this pain although making the field larger would probably be sufficient
  - dzdt 5y ago
    Seconded. I went to check out a few example searches and the too-narrow search bar is the first annoyance I found. The next annoyance was that the crawled index seems much smaller than google's or bing's. I looked for things I know exist on twitter, on an old wordpress blog, on obscure websites I frequent: forcing terms to not be skipped using + I could see that none of my test cases were in the index.
- benwills 5y ago
  I noticed there were IPs in the source code that seemed to reference yours, and mabye others', home IP addresses. I'm curious if you run any parts of either the crawling, indexing, or searching from home networks? I'm asking since I'm working on similar/different crawling problems that would make some stuff easier to just handle from the hardware I have at home, and have always assumed the provider would shut it down. Have you had any issues with that?
webZero 5y ago
I cant go back to search results from private.sh.
deleted 5y ago
[deleted]
musicale 5y ago
I like the idea of a web search engine that works for searching the web.
SubiculumCode 5y ago
Honestly, I find this search engine pretty dang usable. I've thrown technical to frivolous at it, and i like the mix of results.
maverick74 5y ago
Matt is a great guy and it does not have the credit he should have! Same thing for Gigablast! It is amazing what one person alone can accomplish. Congratulations for that, Matt! You've done impossible things with few resources! It's a shame to have so many money given to so many projects and no one ever remembers Gigablast. (About private.sh: I think it would be nice to have image search on private.sh)
- jll29 5y ago
  > It is amazing what one person alone can accomplish I was also wondering how Matt did all this mostly alone until I discovered he joined HN only nine months ago. ;)
ronenlh 5y ago
Hi @gbmatt, amazing work! What is the business history of it? Did companies /investors show interest in acquiring it? What do you think needs to be done in terms of business development to extend the index to cover the modern internet, as well as get whitelisted (and shortlisted) properly by cdns?
lkramer 5y ago
Initial searches are very promising. Is there a good way to add this as my default search engine in Firefox?
- blobcode 5y ago
  You could give https://addons.mozilla.org/en-CA/firefox/addon/gigablast-search/ https://addons.mozilla.org/en-CA/firefox/addon/gigablast-sea... a try.
- avery42 5y ago
  If you don't want an extension, another option is to find it on Mycroft Project [0], choose Gigablast, and on the "Install plugin" page, right click the address bar and choose "Add Gigablast". Then you can set it as your default from the Firefox search settings. [0]: https://mycroftproject.com/search-engines.html?name=gigablast https://mycroftproject.com/search-engines.html?name=gigablas...
lepouet 5y ago
"Clients" --> Error = Not Found :')
dash2 5y ago
Competition in search would be great. * This needs to be quicker. Nobody wants to wait 3 seconds watching cogs spin. * It needs a UX designer. The search box jumps around the page when you type into it. The left-orientation of the search results is ugly and distracting. If this is a one-person project, then that is really cool, but if it wants to be a serious contender in consumer-facing search, then it is probably time to hire an employee with complementary skills.
- fibbberMEN 5y ago
  It's been around since early google days, and that seems to be when most of the HTML was written. Every single page has several HTML errors... so definitely needs help there. Basic things like HTML tables are not closed / switching <td> and <tr> around etc.
zandorg 5y ago
I tried to submit my website to Gigablast, but apparently it costs 25 cents. This doesn't make any sense to me for a search engine.

235 points by SubiculumCode 5y ago

Source of this post likely from [1] [1] https://news.ycombinator.com/item?id=29417296 https://news.ycombinator.com/item?id=29417296

SubiculumCode 5y ago
Yep, I posted this today after the author of gigablast posted a comment in this https://news.ycombinator.com/item?id=29417061 https://news.ycombinator.com/item?id=29417061 today

superasn 5y ago

I used to donate my idle cpu to seti@home back in the day. Wonder if the same can be done for creating an open search engine to compete with Google. Also since the resources are crowd sourced it can make it easy to get around rate limits and anti scraping too.

boyter 5y ago
You can do this with Yacy right now, https://yacy.net https://yacy.net but it's not great for results generally. I have often wondered if something built on activity pub or like that would be an option allowing people to group servers with peers they like or trust. Its something I want to implement actually and may get around to doing one of these days.
- zdkl 5y ago
  Well for starters, one could implement the API to return ActivityStreams formatted responses. That would be a good start to being compatible with the fediverse and stuff while not going insane in implementing a full and proper ActivityPub service. Been there, done that, way better tool for "lower level" features. [0] https://www.w3.org/TR/activitystreams-core/ https://www.w3.org/TR/activitystreams-core/
  - boyter 5y ago
    That roughly what I thought. I’m not familiar with activitypub at all. I will probably investigate this deeper. You see to have some knowledge in this area. Do you have any suggestions of places to look to achieve something like this?
tigerlily 5y ago
Perhaps not weirdly I had the same thought yesterday [1]. https://yacy.net https://yacy.net was suggested in response. [1] https://news.ycombinator.com/item?id=29417925 https://news.ycombinator.com/item?id=29417925
SubiculumCode 5y ago
That's a really cool idea!
R0b0t1 5y ago
I don't think it would be that easy. You need to distribute the indexing data. But you could federate search servers and have them send queries to others.
SubiculumCode 5y ago
I know the usual thing against crypto, but I wonder whether a gridcoin model would work. https://gridcoin.us/ https://gridcoin.us/

1cvmask 5y ago

I saw this in a thread earlier today. I couldn't understand why it has a login and account. They seem to be the anti-Google and and anti-personalization search engine.

SubiculumCode 5y ago
good question
GistNoesis 5y ago
You don't need an account to make searches though : https://gigablast.com/index.html https://gigablast.com/index.html If you have an account you can probably log your queries.
aquarin 5y ago
It looks, you need a account to add url-s. "You need to login to use the add url tool. " "Each added url is $0.25."

nixgeek 5y ago

“Gigablast has teamed up with Imperial Family Companies to create a next generation private search engine, private.sh.” Imperial Family Companies are the people who essentially destroyed [1] the freenode IRC network, aren’t they? [1] https://netsplit.de/networks/history/top10_2021u.png https://netsplit.de/networks/history/top10_2021u.png

humanistbot 5y ago
Yes, the same ones [1]. They are an investment firm that was formerly known as London Trust Media. You can see they have listed both "IRC" (links to irc.com) and "freenode" in their portfolio. [2] [1] https://lists.ubuntu.com/archives/ubuntu-irc/2021-May/001923.html https://lists.ubuntu.com/archives/ubuntu-irc/2021-May/001923... [2] https://imperialfamily.com/ https://imperialfamily.com/ (hover over "Technology)
- mrtweetyhack 5y ago
  let's just make sure we destroy everything they invest in
superkuh 5y ago
It's hard to believe they'd take money from someone that attacked so many open source projects earlier this year by leveraging "donations". They should be careful.
- ludamad 5y ago
  At the same time, they're in a position to consent to a transition, not sure there's a community of collective ownership here like with freenode

twofornone 5y ago

Well, pirate bay is returned in search results, so that's a good start...

bigyellow 5y ago

> Furthermore, the client-side javascript on private.sh encrypts any query done on private.sh so that only Gigablast can read it. Therefore, no single party has access to both the IP address and the query. This is something that is truly unique and truly powerful, and, right now, only private.sh can supply this level of privacy. Run proprietary Javascript for privacy - what a fallacious concept. Going to assume this service is a honeypot or run by incompetent staff - pass.

gbmatt 5y ago
the javascript is run by your browser, so you can fully audit it.
- bigyellow 5y ago
  It's still served by the site and I doubt most are interested or capable in auditing software to perform routine online tasks.
  - rasengan 5y ago
    There's an extension as well [1]. This means that the code is not being served by the server in this use case. [1] https://private.sh/extension.html https://private.sh/extension.html
  - eftychis 5y ago
    I am not sure there are good solutions besides going off browser. P.S. I was involved in user authorization, attestation and privacy flows for a particular product recently and the browser was always where shit hit the fan. The web features are just not made with simplicity and privacy in mind. Then again we had more complex constraints.
mrtweetyhack 5y ago
Only Gigablast can read it means it is not private

boyter 5y ago

Matt Wells who wrote the majority of the code for gigablast is someone I have been following online for a long time. I used to live for http://gigablast.com/rants.html http://gigablast.com/rants.html updates. Gigablast being an amazing example of what one person can do given the time and effort. If you look around for articles and interviews by him you get some nice insights that would never come from the likes of Google, although Bing has some very good in depth technical discussions such as how bitfunnel works. It’s also nice to look through the code of it and see how things like porn filters were implemented. It’s also nice to know that for a while in internet history gigablast was mentioned in the same breath as google. An amazing achievement at the time for a single person against the core google product. I wish that someone with some design chops could work on it for a few weeks though. Or it was rolled back to the design from around 2006 with the rocket logo. I really liked that design. I really wish I had the courage to strike out on my own like Matt has. I have written a few search engines, but a general purpose one from scratch on my own hardware is unlikely to ever happen, as much as I would love to do it. That’s for inspiring me so much Matt if you do read this (notice me senpai!). Oh and sorry for abusing your XML api so much. I was poor at the time and needed some search results. A few choice articles, https://queue.acm.org/detail.cfm?id=988401 https://queue.acm.org/detail.cfm?id=988401 https://www.abc.net.au/news/science/2021-02-14/google-news-media-bargaining-code-build-search-engine/13143582 https://www.abc.net.au/news/science/2021-02-14/google-news-m...

gbmatt 5y ago
thanks ben, you are too kind.
- samhw 5y ago
  I second this - thanks for building this. It's an unbelievably inspiring achievement. It's my default search engine, and I'm really glad it exists.
- boyter 5y ago
  Wow even knows me by my first name too. Very humbled. Once again thanks for being so open with what you have done.

ohiovr 5y ago

it found this: https://gigablast.com/search?c=main&qlangcountry=en-us&q=how+to+resolve+missing+symbol+r+android https://gigablast.com/search?c=main&qlangcountry=en-us&q=how... Which is definitely a good sign of a competent search engine.

gbmatt 5y ago

hey thanks for the recognition, people. :) finally, all my problems are solved. this comment is here for hacker news karma points.

kingcharles 5y ago
*throws karma at the screen*
ivanche 5y ago
Hey Matt, would you consider making a search box (input with id="q") a bit wider? I can type only around 15 characters before the beginning of search query becomes "cut off".
- InfiniteRand 5y ago
  The too small text box is also a pain when deleting the query in order to type a new one on mobile. A clear button would mitigate some of this pain although making the field larger would probably be sufficient
- dzdt 5y ago
  Seconded. I went to check out a few example searches and the too-narrow search bar is the first annoyance I found. The next annoyance was that the crawled index seems much smaller than google's or bing's. I looked for things I know exist on twitter, on an old wordpress blog, on obscure websites I frequent: forcing terms to not be skipped using + I could see that none of my test cases were in the index.
benwills 5y ago
I noticed there were IPs in the source code that seemed to reference yours, and mabye others', home IP addresses. I'm curious if you run any parts of either the crawling, indexing, or searching from home networks? I'm asking since I'm working on similar/different crawling problems that would make some stuff easier to just handle from the hardware I have at home, and have always assumed the provider would shut it down. Have you had any issues with that?

webZero 5y ago

I cant go back to search results from private.sh.

deleted 5y ago

[deleted]

musicale 5y ago

I like the idea of a web search engine that works for searching the web.

SubiculumCode 5y ago

Honestly, I find this search engine pretty dang usable. I've thrown technical to frivolous at it, and i like the mix of results.

maverick74 5y ago

Matt is a great guy and it does not have the credit he should have! Same thing for Gigablast! It is amazing what one person alone can accomplish. Congratulations for that, Matt! You've done impossible things with few resources! It's a shame to have so many money given to so many projects and no one ever remembers Gigablast. (About private.sh: I think it would be nice to have image search on private.sh)

jll29 5y ago
> It is amazing what one person alone can accomplish I was also wondering how Matt did all this mostly alone until I discovered he joined HN only nine months ago. ;)

ronenlh 5y ago

Hi @gbmatt, amazing work! What is the business history of it? Did companies /investors show interest in acquiring it? What do you think needs to be done in terms of business development to extend the index to cover the modern internet, as well as get whitelisted (and shortlisted) properly by cdns?

lkramer 5y ago

Initial searches are very promising. Is there a good way to add this as my default search engine in Firefox?

blobcode 5y ago
You could give https://addons.mozilla.org/en-CA/firefox/addon/gigablast-search/ https://addons.mozilla.org/en-CA/firefox/addon/gigablast-sea... a try.
avery42 5y ago
If you don't want an extension, another option is to find it on Mycroft Project [0], choose Gigablast, and on the "Install plugin" page, right click the address bar and choose "Add Gigablast". Then you can set it as your default from the Firefox search settings. [0]: https://mycroftproject.com/search-engines.html?name=gigablast https://mycroftproject.com/search-engines.html?name=gigablas...

lepouet 5y ago

"Clients" --> Error = Not Found :')

dash2 5y ago

Competition in search would be great. * This needs to be quicker. Nobody wants to wait 3 seconds watching cogs spin. * It needs a UX designer. The search box jumps around the page when you type into it. The left-orientation of the search results is ugly and distracting. If this is a one-person project, then that is really cool, but if it wants to be a serious contender in consumer-facing search, then it is probably time to hire an employee with complementary skills.

fibbberMEN 5y ago
It's been around since early google days, and that seems to be when most of the HTML was written. Every single page has several HTML errors... so definitely needs help there. Basic things like HTML tables are not closed / switching <td> and <tr> around etc.

zandorg 5y ago

I tried to submit my website to Gigablast, but apparently it costs 25 cents. This doesn't make any sense to me for a search engine.