9 ms·

Olric: Distributed, embeddable data structures in Go

113 points by mastabadtomm 4y ago

jensneuse 4y ago
Congrats Burak on making it to the main page!
- mastabadtomm 4y ago
  Thank you, Jens!
kevml 4y ago
> Olric v0.4 and previous versions use Olric Binary Protocol, v0.5 uses Redis serialization protocol for communication and the API was significantly changed. Olric v0.4.x tree is going to receive bug fixes and security updates forever, but I would recommend considering an upgrade to the new version. Seems like a tall order to promise updates “forever”, particularly for a v0 release.
- Philip-J-Fry 4y ago
  But considering there will be no features added to it, it's probably not a lot of effort to fix a few bugs or security issues here and there when people report them.
undefuser 4y ago
While it sounds interesting, I'm not sure in which scenario do I pick this over Redis
- konart 4y ago
  Well, you can use it as a package in your Go service(s) if you don't want to or can't communicate with Redis
abusaidm 4y ago
I was excited to read its embeddable data structure then found it did Redis serialisation. I was hoping for something close to plasma store given that one isn’t getting much love these days. Would be great to get something that can save binary formats like arrow.
- pgwhalen 4y ago
  Ultimately Redis just stores bytes, there's no reason you couldn't put arrow formatted data in it (or in Olric, as I understand it). I do wish plasma wasn't dead/dying though. There is a huge gap in the (extremely crowded) key-value store space for zero-copy shared memory IPC. I assume it doesn't get built because it's not very "cloud native," but such a thing would be undeniably useful.
bpicolo 4y ago
I know Java and Erlang ecosystems have been big on this at times (Hazelcast, mnesia). When are these the appropriate architectural choice, if ever? Definitely fun/cool tech, I just don't understand how pragmatic it is.
- Kinrany 4y ago
  Libraries simplify ops compared to external services, including local development. An embedded database can be used in unit tests with zero effort.
- jlouis 4y ago
  In Erlang, mnesia has a couple of things going for it: One, it stores Erlang terms. The consequence is that you need no data conversion when you want to store data persistently on disk. You just save whatever term you have. Two, it provides Key/Value lookup at memory read speeds. It also does so across Erlang processes, so you don't have to call into another process to grab data in most cases. This was originally a main design constraint. You need it for low latency operation, and the only way to avoid a disk read is to keep things in memory. Nowadays, since we have fast disks with no seek-time, it matters less. But when Mnesia was built, we didn't have that luxury. Note that many other systems run in separate processes (memcached, redis, ...) which incurs a context switch and data transfer across a OS process boundary. When the K/V store is in the same memory space things are much faster. Three, it is an excellent tool for rapid prototyping, and early development work. You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers. Mnesia also allows you to plug in other storage backends, so you can eventually move to a more traditional setup later. Four, mnesia does provide query capability through a query planner. So for the occasional larger query, there are tools built-in which solves your problem in a very small amount of code.
  - dmitriid 4y ago
    > You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers. However, by this time your entire business depends on submillisecond responses from Mnesia in memory and you can't transition to a different database, or even to on-disk Mnesia. This has happened :)
ozirus 4y ago
olric. Great name :)
lokar 4y ago
I don’t understand why anyone would pick the redis wire protocol.
- kgeist 4y ago
  >So Olric has client implementations in all major programming languages.
alexk307 4y ago
I don’t understand why people need another abstraction layer on top of Redis? What can this do that Redis can’t?
- resoluteteeth 4y ago
  I don't think this is an abstraction layer on top of Redis? It appears to be a reimplementation of Redis in go.
- delijati 4y ago
  Two to X programes (any language that has redis client) can access the same memory via redis protocol. Without a running redis server. Or i missed something !?
  - kamikazechaser 4y ago
    Can run as both embedded and server mode. Essentially a re-imagined redis from a general point of view.
didip 4y ago
Relevant discussion when it comes to Redis compatible API: https://news.ycombinator.com/item?id=31796311 https://news.ycombinator.com/item?id=31796311

113 points by mastabadtomm 4y ago

jensneuse 4y ago

Congrats Burak on making it to the main page!

mastabadtomm 4y ago
Thank you, Jens!

kevml 4y ago

> Olric v0.4 and previous versions use Olric Binary Protocol, v0.5 uses Redis serialization protocol for communication and the API was significantly changed. Olric v0.4.x tree is going to receive bug fixes and security updates forever, but I would recommend considering an upgrade to the new version. Seems like a tall order to promise updates “forever”, particularly for a v0 release.

Philip-J-Fry 4y ago
But considering there will be no features added to it, it's probably not a lot of effort to fix a few bugs or security issues here and there when people report them.

undefuser 4y ago

While it sounds interesting, I'm not sure in which scenario do I pick this over Redis

konart 4y ago
Well, you can use it as a package in your Go service(s) if you don't want to or can't communicate with Redis

abusaidm 4y ago

I was excited to read its embeddable data structure then found it did Redis serialisation. I was hoping for something close to plasma store given that one isn’t getting much love these days. Would be great to get something that can save binary formats like arrow.

pgwhalen 4y ago
Ultimately Redis just stores bytes, there's no reason you couldn't put arrow formatted data in it (or in Olric, as I understand it). I do wish plasma wasn't dead/dying though. There is a huge gap in the (extremely crowded) key-value store space for zero-copy shared memory IPC. I assume it doesn't get built because it's not very "cloud native," but such a thing would be undeniably useful.

bpicolo 4y ago

I know Java and Erlang ecosystems have been big on this at times (Hazelcast, mnesia). When are these the appropriate architectural choice, if ever? Definitely fun/cool tech, I just don't understand how pragmatic it is.

Kinrany 4y ago
Libraries simplify ops compared to external services, including local development. An embedded database can be used in unit tests with zero effort.
jlouis 4y ago
In Erlang, mnesia has a couple of things going for it: One, it stores Erlang terms. The consequence is that you need no data conversion when you want to store data persistently on disk. You just save whatever term you have. Two, it provides Key/Value lookup at memory read speeds. It also does so across Erlang processes, so you don't have to call into another process to grab data in most cases. This was originally a main design constraint. You need it for low latency operation, and the only way to avoid a disk read is to keep things in memory. Nowadays, since we have fast disks with no seek-time, it matters less. But when Mnesia was built, we didn't have that luxury. Note that many other systems run in separate processes (memcached, redis, ...) which incurs a context switch and data transfer across a OS process boundary. When the K/V store is in the same memory space things are much faster. Three, it is an excellent tool for rapid prototyping, and early development work. You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers. Mnesia also allows you to plug in other storage backends, so you can eventually move to a more traditional setup later. Four, mnesia does provide query capability through a query planner. So for the occasional larger query, there are tools built-in which solves your problem in a very small amount of code.
- dmitriid 4y ago
  > You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers. However, by this time your entire business depends on submillisecond responses from Mnesia in memory and you can't transition to a different database, or even to on-disk Mnesia. This has happened :)

ozirus 4y ago

olric. Great name :)

lokar 4y ago

I don’t understand why anyone would pick the redis wire protocol.

kgeist 4y ago
>So Olric has client implementations in all major programming languages.

alexk307 4y ago

I don’t understand why people need another abstraction layer on top of Redis? What can this do that Redis can’t?

resoluteteeth 4y ago
I don't think this is an abstraction layer on top of Redis? It appears to be a reimplementation of Redis in go.
delijati 4y ago
Two to X programes (any language that has redis client) can access the same memory via redis protocol. Without a running redis server. Or i missed something !?
- kamikazechaser 4y ago
  Can run as both embedded and server mode. Essentially a re-imagined redis from a general point of view.

didip 4y ago

Relevant discussion when it comes to Redis compatible API: https://news.ycombinator.com/item?id=31796311 https://news.ycombinator.com/item?id=31796311