8 ms·

Zinc Search engine. A lightweight alternative to Elasticsearch written in Go

203 points by techplex 5y ago

rzzzt 5y ago
Does it support boolean queries? Bluge, the library backing Zinc appears to have such a searcher, and I can see a few references in Zinc's code, but is it exposed for search expressions?
aliswe 5y ago
I wonder why the choice of technology is even relevant? Because of the hype factor?
- kgersen 5y ago
  A Go project is easy to contribute to. That's a huge factor for an open source project.
- PixyMisa 5y ago
  It matters for performance and reliability. Go is fine, and means you get a nice standalone binary. Java is fine, but you need Java installed. If it were written in Node.js or PHP, on the other hand, you'd know to stay ten miles away.
ochoseis 5y ago
I wonder if this would be a good lightweight alternative to ES for a local development setup. At $work we use ES for deployed environments, but have thus far avoided running it locally because of the resource requirements.
- jillesvangurp 5y ago
  I have Elasticsearch running in docker most of the time for local development. It's fine. You can run it with as little as half a GB of ram. It won't be using much CPU unless you start throwing millions of documents at it for indexing. Our production cluster uses 2 1GB vms. The whole setup costs us about 60$/month. We have a about 7 million documents indexed in there. People here seem to assume Elasticsearch requires lots of memory and CPU. It actually scales down very nicely in addition to its famous ability to scale up very well. Any decent developer laptop should have no issues running this. Running VS Code is more of a burden on my laptop. And intellij is even worse.
maxpert 5y ago
Would be nice to have some benchmarks against ES, Sonic, and Toshi. I am very much interested in stress-testing on large dataset.
02020202 5y ago
there are already Bleve and Riot search engines written in Go. i am always happy to have more options but in this case maybe if the effort would have been put into one of these two would be overall more beneficial for end users.
deknos 5y ago
we have alternatives to elasticsearch as listed here. what we do not have is a alternative to kibana. which works with elasticsearch/opensearch and the elasticsearchalternatives here.
keyle 5y ago
Ha. Lovely, but the missing features is basically what makes Elasticsearch great! Missing features: Clustering and High Availability Nothing that can't be fixed though.
- Andys 5y ago
  I would embed nats, a golang message queue that offers clustering and persistence, a great bulding block for clustered support.
- deleted 5y ago
  [deleted]
- bootcat 5y ago
  yes i agree - its comparatively easy to create software that is optimal as long as it runs in a single machine. The distributed systems and related constraints and guarantees is what takes the resources ! ( I would assume he would use a system based on raft to offer the above, or using something like Infinispan, helix or hazelcast kinda )
  - cyberge99 5y ago
    I wonder if a Nomad/Consul stack could make this clusterable.
pjz 5y ago
>(Kibana is not supported with zinc. Zinc provides its own UI). 1. I was hoping for a drop-in replacement for Elasticsearch. A new/different API means Zinc can't leverage existing tools that use Elasticsearch. 2. I don't like that you're bundling zinc with a UI; that disadvantages anyone else trying to build a better UI and often (usually?) leads to tying the db too closely to the UI (or vice versa)
- jjeaff 5y ago
  If the API really is compatible, then Kibana should work. Just sounds like they don't guarantee it will always work.
- yed 5y ago
  According to the site: > Compatibility with elasticsearch APIs for ingestion of data (single record and bulk API)
nw05678 5y ago
I've always wanted a full text search engine akin to "sqlite".
- nwellnhof 5y ago
  That's what Apache Lucy tried to become, but it's an abandoned project.
- fizx 5y ago
  sqlite is itself a full text search engine. It's gotten pretty good over the years. https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html
- jasonjayr 5y ago
  You are aware sqlite includes a full text search engine too? https://sqlite.org/fts3.html https://sqlite.org/fts3.html https://sqlite.org/fts5.html https://sqlite.org/fts5.html
- simonw 5y ago
  I've been having a lot of fun with the SQLite FTS library. The best public-facing demo I have is probably the search feature on https://datasette.io https://datasette.io - e.g. https://datasette.io/-/beta?q=fts- https://datasette.io/-/beta?q=fts- that's powered by my https://github.com/dogsheep/dogsheep-beta https://github.com/dogsheep/dogsheep-beta tool.
- slig 5y ago
  Typesense and Melisearch are what you're looking for.
- PixyMisa 5y ago
  You could try Xapian. Or there's always SQLite itself.
ordiel 5y ago
When people say "Lightweight alternative" most of the time they are implying less features also, they are not lightweight just because
- jillesvangurp 5y ago
  I would say a fraction of the features. If you don't need those features, that's fine of course. But in my experience, I end up using a lot of those features when dealing with real world customer requirements. I guess if you implement search for some website where search quality just doesn't matter, light weight is fine. For everything else, using proper solutions might be the wiser thing. In terms of performance you see a lot of comments that are stating X is faster than Y. I'd take such comments with a large grain of salt. Unless those lightweight alternatives actually do the same things you can't really compare the performance. It's not that hard to make indexing fast in Elasticsearch if you disable all the features that make it slower. Of course if you don't actually have those features it's going to be fast by default because it simply isn't doing anywhere close to the same things. Elasticsearch is actually pretty damn fast even when you do use its many features. The reason is that it relies on in memory caches, thread pools, etc. and that a lot of very smart people have been implementing and optimizing very efficient algorithms in Lucene for the last 25 years. Elasticsearch actually can run with as little as 256MB but of course you are not going to be able to cache a lot of data with that and performance will suffer accordingly. Mostly large heap sizes with Elasticsearch are all about using larger caches. It also relies on memory mapped files and OS file caches for that. That's what allows it to work with extremely large data sets and still provide query responses in milliseconds. There's no reason you wouldn't be able to do the same with a native implementation of course. But it would be naive to assume you'd end up using a lot less memory or CPU. Using that would be pretty core to matching performance and features. Not using that would make it pretty hard to get even close.
- pjmlp 5y ago
  Except every person has a different set of 10% features.
- vosper 5y ago
  Yeah. There's a lot that Elasticsearch can do. There are libraries and applications that can do bits of what ES can do, but no-one I'm aware of (and I watch this space) is even slightly close to building a genuine replacement (I'm not counting the AWS fork, for obvious reasons)
bootcat 5y ago
I am staunch supporter of software which runs on minimal hardware or resources. Seems like an interesting project, looking forward towards distributed features !
arafalov 5y ago
They had me at: "The only viable solution to search was elasticsearch". /s
xvilka 5y ago
There are also Toshi[1] and Sonic[2] in Rust. And Vector[3] as a Logstash alternative too. There is an issue[4] proposing to integrate Vector with Sonic and Toshi. Maybe Zinc can pursue this goal too. Always good to see people who realize that Java is unwieldy monster that will eat all your memory. Native is a way to go for big systems. [1] https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi [2] https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic [3] https://vector.dev/ https://vector.dev/ [4] https://github.com/vectordotdev/vector/issues/988 https://github.com/vectordotdev/vector/issues/988
- fithisux 5y ago
  You can always write it in Java/Kotlin and make it native with GraalVM. You do not need Rust necessarily. Mixed feelings yet. I am not big fan. Golang is good too
- pkulak 5y ago
  Java will use as much memory as you give it. Because you told it to use that much, and using more generally gives better performance. "Native" or not has nothing to do with the garbage collector. Go is a GCed language too, but I really don't think it's as performant or tuneable as Java's.
  - stiray 5y ago
    Please check this, it explain difference between Java and Go garbage collectors quite nicely: https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26c60 https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26...
    - kaba0 5y ago
      It was criticized quite heavily on the relevant hn thread. But basically, go can get away with a less advanced GC due to having and relying on value types quite often. But Java and certain workloads do require the state-of-the-art GCs that the JVM provides. And a search engine likely constitutes such a one.

203 points by techplex 5y ago

Does it support boolean queries? Bluge, the library backing Zinc appears to have such a searcher, and I can see a few references in Zinc's code, but is it exposed for search expressions?

aliswe 5y ago

I wonder why the choice of technology is even relevant? Because of the hype factor?

kgersen 5y ago
A Go project is easy to contribute to. That's a huge factor for an open source project.
PixyMisa 5y ago
It matters for performance and reliability. Go is fine, and means you get a nice standalone binary. Java is fine, but you need Java installed. If it were written in Node.js or PHP, on the other hand, you'd know to stay ten miles away.

ochoseis 5y ago

I wonder if this would be a good lightweight alternative to ES for a local development setup. At $work we use ES for deployed environments, but have thus far avoided running it locally because of the resource requirements.

jillesvangurp 5y ago
I have Elasticsearch running in docker most of the time for local development. It's fine. You can run it with as little as half a GB of ram. It won't be using much CPU unless you start throwing millions of documents at it for indexing. Our production cluster uses 2 1GB vms. The whole setup costs us about 60$/month. We have a about 7 million documents indexed in there. People here seem to assume Elasticsearch requires lots of memory and CPU. It actually scales down very nicely in addition to its famous ability to scale up very well. Any decent developer laptop should have no issues running this. Running VS Code is more of a burden on my laptop. And intellij is even worse.

maxpert 5y ago

Would be nice to have some benchmarks against ES, Sonic, and Toshi. I am very much interested in stress-testing on large dataset.

02020202 5y ago

there are already Bleve and Riot search engines written in Go. i am always happy to have more options but in this case maybe if the effort would have been put into one of these two would be overall more beneficial for end users.

deknos 5y ago

we have alternatives to elasticsearch as listed here. what we do not have is a alternative to kibana. which works with elasticsearch/opensearch and the elasticsearchalternatives here.

keyle 5y ago

Ha. Lovely, but the missing features is basically what makes Elasticsearch great! Missing features: Clustering and High Availability Nothing that can't be fixed though.

Andys 5y ago
I would embed nats, a golang message queue that offers clustering and persistence, a great bulding block for clustered support.
deleted 5y ago
[deleted]
bootcat 5y ago
yes i agree - its comparatively easy to create software that is optimal as long as it runs in a single machine. The distributed systems and related constraints and guarantees is what takes the resources ! ( I would assume he would use a system based on raft to offer the above, or using something like Infinispan, helix or hazelcast kinda )
- cyberge99 5y ago
  I wonder if a Nomad/Consul stack could make this clusterable.

pjz 5y ago

>(Kibana is not supported with zinc. Zinc provides its own UI). 1. I was hoping for a drop-in replacement for Elasticsearch. A new/different API means Zinc can't leverage existing tools that use Elasticsearch. 2. I don't like that you're bundling zinc with a UI; that disadvantages anyone else trying to build a better UI and often (usually?) leads to tying the db too closely to the UI (or vice versa)

jjeaff 5y ago
If the API really is compatible, then Kibana should work. Just sounds like they don't guarantee it will always work.
yed 5y ago
According to the site: > Compatibility with elasticsearch APIs for ingestion of data (single record and bulk API)

nw05678 5y ago

I've always wanted a full text search engine akin to "sqlite".

nwellnhof 5y ago
That's what Apache Lucy tried to become, but it's an abandoned project.
fizx 5y ago
sqlite is itself a full text search engine. It's gotten pretty good over the years. https://www.sqlite.org/fts5.html https://www.sqlite.org/fts5.html
jasonjayr 5y ago
You are aware sqlite includes a full text search engine too? https://sqlite.org/fts3.html https://sqlite.org/fts3.html https://sqlite.org/fts5.html https://sqlite.org/fts5.html
simonw 5y ago
I've been having a lot of fun with the SQLite FTS library. The best public-facing demo I have is probably the search feature on https://datasette.io https://datasette.io - e.g. https://datasette.io/-/beta?q=fts- https://datasette.io/-/beta?q=fts- that's powered by my https://github.com/dogsheep/dogsheep-beta https://github.com/dogsheep/dogsheep-beta tool.
slig 5y ago
Typesense and Melisearch are what you're looking for.
PixyMisa 5y ago
You could try Xapian. Or there's always SQLite itself.

ordiel 5y ago

When people say "Lightweight alternative" most of the time they are implying less features also, they are not lightweight just because

jillesvangurp 5y ago
I would say a fraction of the features. If you don't need those features, that's fine of course. But in my experience, I end up using a lot of those features when dealing with real world customer requirements. I guess if you implement search for some website where search quality just doesn't matter, light weight is fine. For everything else, using proper solutions might be the wiser thing. In terms of performance you see a lot of comments that are stating X is faster than Y. I'd take such comments with a large grain of salt. Unless those lightweight alternatives actually do the same things you can't really compare the performance. It's not that hard to make indexing fast in Elasticsearch if you disable all the features that make it slower. Of course if you don't actually have those features it's going to be fast by default because it simply isn't doing anywhere close to the same things. Elasticsearch is actually pretty damn fast even when you do use its many features. The reason is that it relies on in memory caches, thread pools, etc. and that a lot of very smart people have been implementing and optimizing very efficient algorithms in Lucene for the last 25 years. Elasticsearch actually can run with as little as 256MB but of course you are not going to be able to cache a lot of data with that and performance will suffer accordingly. Mostly large heap sizes with Elasticsearch are all about using larger caches. It also relies on memory mapped files and OS file caches for that. That's what allows it to work with extremely large data sets and still provide query responses in milliseconds. There's no reason you wouldn't be able to do the same with a native implementation of course. But it would be naive to assume you'd end up using a lot less memory or CPU. Using that would be pretty core to matching performance and features. Not using that would make it pretty hard to get even close.
pjmlp 5y ago
Except every person has a different set of 10% features.
vosper 5y ago
Yeah. There's a lot that Elasticsearch can do. There are libraries and applications that can do bits of what ES can do, but no-one I'm aware of (and I watch this space) is even slightly close to building a genuine replacement (I'm not counting the AWS fork, for obvious reasons)

bootcat 5y ago

I am staunch supporter of software which runs on minimal hardware or resources. Seems like an interesting project, looking forward towards distributed features !

arafalov 5y ago

They had me at: "The only viable solution to search was elasticsearch". /s

xvilka 5y ago

There are also Toshi[1] and Sonic[2] in Rust. And Vector[3] as a Logstash alternative too. There is an issue[4] proposing to integrate Vector with Sonic and Toshi. Maybe Zinc can pursue this goal too. Always good to see people who realize that Java is unwieldy monster that will eat all your memory. Native is a way to go for big systems. [1] https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi [2] https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic [3] https://vector.dev/ https://vector.dev/ [4] https://github.com/vectordotdev/vector/issues/988 https://github.com/vectordotdev/vector/issues/988

fithisux 5y ago
You can always write it in Java/Kotlin and make it native with GraalVM. You do not need Rust necessarily. Mixed feelings yet. I am not big fan. Golang is good too
pkulak 5y ago
Java will use as much memory as you give it. Because you told it to use that much, and using more generally gives better performance. "Native" or not has nothing to do with the garbage collector. Go is a GCed language too, but I really don't think it's as performant or tuneable as Java's.
- stiray 5y ago
  Please check this, it explain difference between Java and Go garbage collectors quite nicely: https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26c60 https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26...
  - kaba0 5y ago
    It was criticized quite heavily on the relevant hn thread. But basically, go can get away with a less advanced GC due to having and relying on value types quite often. But Java and certain workloads do require the state-of-the-art GCs that the JVM provides. And a search engine likely constitutes such a one.