8 ms·

Show HN: SummitDB – In-Memory NoSQL DB

81 points by tidwall 10y ago

SteveNuts 10y ago
Someone should invent a package manager for databases, the same way we have npm, composer, etc. it's getting difficult to manage all my different databases.
- tshannon 10y ago
  Competitions good, imo. We have an "Embarrassment of riches", as it were, when it comes to databases.
  - m1sta_ 10y ago
    And yet nothing that meets all my requirements. Still so much room to improve!
  - deleted 10y ago
    [deleted]
- arielweisberg 10y ago
  Totally! I can just sudo apt-get install low-latency-strong-consistency-multi-dc-with-declarative-query-language-db as a virtual package and get the right one for my distro.
  - nickpsecurity 10y ago
    Is that the command for installing Google's F1 RDBMS? It has those traits. I really wish they'd spin it off as a product since the only startup offering something similar got acquired and deep sixed by Apple.
    - arielweisberg 10y ago
      I don't think it meets the low latency requirement. It's a CAP theorem joke. Can't have strong consistency across multiple data centers with low latency because you have to read/write from a quorum of DCs in order to maintain availability in case one DC goes down.
      - nickpsecurity 10y ago
        Oh yeah. Low latency eould be saying too much. I think its worst-case for strong-consistency is around 30 seconds or so. Anyway, closest to to low-latency part would be NonStop or OpenVMS clusers across redundant, leased lines. They're fast, scale well at HW level, use ACID databases, and high-availability. Commonly used in transaction processing. One VMS cluster survived 9/11 attack without losing a single transaction. NonStop does up to five 9's. Id be interested in the CAP analysis of these older methods.
        arielweisberg 10y ago
        It also depends on your definition of data center. It's really multi-region that is the problem since then it's the speed of light you have to deal with. Quorum across some data centers is fast enough that it's not hard to manage in new applications.
        nickpsecurity 10y ago
        That makes a lot of since. Many of the NonStop and VMS clusters were in same country but far enough from each other to isolate against many disasters. Could be why they did better on the issue.
      - mirekrusin 10y ago
        You can always use quantum entanglement to help a (qu)bit.
    - lobster_johnson 10y ago
      Take a look at CockroachDB, which had its first public beta earlier this year [1]. It's directly inspired by Google's F1 and Spanner projects. It's similar to FoundationDB (the Apple product you're referring to) in that it's an SQL database layered on top of a K/V store. It's based on some clever technology to accomplish distributed transactions, strong consistency and high availability, and is looking very promising. [1] https://www.cockroachlabs.com/ https://www.cockroachlabs.com/
      - nickpsecurity 10y ago
        Appreciate the tip. Im aware of it. They had a recent post on HN where they were just getting around to dealing with stability issues. I was really excited till I saw that. Im holding off for a while till it matures a bit.
        ddorian43 10y ago
        I think they just went too far with full-sql-joins-indexes-column-families in the first 1.0 version. Better to build it little by little.
      - carterehsmith 10y ago
        "First public beta" of a database product? No.
  - manigandham 10y ago
    Great options already exist: MemSQL for blazing fast distributed full SQL database with cross-datacenter replication and in-memory rowstore + disk-based columnstore. ScyllaDB for Cassandra rewritten in C++ for blazing fast dynamo-style multi-master multi-datacenter disk-based wide-column database. I'll also throw in AMPS (by 60East) as the best messaging platform that supports innovative SQL and real-time state-of-the-world queries on it's message streams (instead of using that rabbitmq or kafka crap).
    - akbar501 10y ago
      https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite Dynomite for production proven, limitless scale for Redis. Provides ability to Redis for both in-memory and on-disk workloads. Dynamo-inspired linearly scalable, shared nothing multi-master architecture. Pairs well with Cassandra. Supports pluggable backends (ex. RocksDB). Been in production 2+ years. One of the larger clusters handles over 3.6 million sustained ops/sec in production, every day.
      - ddorian43 10y ago
        Redis is only the api/driver, not the features, right ? Meaning it's just crud and not the many features/data-types that redis provides, correct ?
        akbar501 10y ago
        It's the full Redis API and protocol. See https://github.com/Netflix/dynomite/blob/dev/notes/redis.md https://github.com/Netflix/dynomite/blob/dev/notes/redis.md for a list of all supported Redis features. The benefit of Dynomite's support for the Redis API + protocol is a.) you can use any Redis client and b.) the same code can be used for standalone Redis on your laptop and on a distributed Dynomite cluster.
      - manigandham 10y ago
        This is neat, but why use this over just using Cassandra-based DBs? Also you were a speaker at Data Layer right?
  - biokoda 10y ago
    http://www.actordb.com/ http://www.actordb.com/
qwertyuiop924 10y ago
Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB? Thus far, I've been pressing SQLite into use for a lot of monotyped data that would have been better served by a NoSQL DB, but I couldn't find one suitable. Apparently, in-memory DBs in now. They're fast and all, that's nice, but I'm still running on a machine with 8 Gigs of ram, and when the dataset starts to exceed that, I want an option other than "grab some more DDR3." I remember the fate of TinyMUD: on-disk operation must at least be an option for any DB I'd use for datasets of a non-fixed size.
- brightball 10y ago
  Mnesia built into Elixir/Erlang potentially?
- misframer 10y ago
  There are several key-value stores you can choose from. RocksDB, LevelDB, lmdb, Berkeley DB, Tokyo Cabinet, sophia, and others. What about those? You can always build abstractions on top.
  - lopatin 10y ago
    BoltDB if you'd like to embed it into a Go program.
    - chaotic-good 10y ago
      BoltDB can be corrupted on crash (kill -9 or hard reset) at least this was the case when I gave it a try. Any database can but in that case that was easy to achieve.
      - mbertschler 10y ago
        did you file a bug for this? I thought it was pretty resilient in such cases
        10y ago
fiatjaf 10y ago
This is exactly like what I wanted Redis to be. Secondary indexes solve all the problems. However, I wanted a feature that allowed custom indexes with Javascript, like CouchDB map functions.
- tidwall 10y ago
  This is available with SummitDB. The SETINDEX command has an EVAL option that allow for custom JavaScript indexes.
- deleted 10y ago
  [deleted]
- dvirsky 10y ago
  I'm working on a secondary index implementation in redis using modules, that should be out in a few days. It indexes redis hashes similar to how you index tables.
  - ddorian43 10y ago
    Check out redis-search module which I think does that (only for keys for now) https://github.com/RedisLabsModules/RediSearch/commit/86eb1c4a76dddd172fc208f3caa162bdc7b0692c https://github.com/RedisLabsModules/RediSearch/commit/86eb1c...
    - dvirsky 10y ago
      I wrote that module as well :) The secondary index module will be more focused on automating more traditional indexing of numbers and simple string values. It has an SQL-ish language for WHERE expressions, and then the result of that is piped to any redis command you want it to. If you're interested, here's a draft of the syntax (it's changed a bit since but you'll get the idea). https://gist.github.com/dvirsky/3ef73143a6d8212f2b50096a8eb68018 https://gist.github.com/dvirsky/3ef73143a6d8212f2b50096a8eb6...
      - ddorian43 10y ago
skrebbel 10y ago
I can't find from the docs at all whether this persists to disk. If not, what is the use case? Why would I need all those ACID-y guarantees if my server can fail at any time and all data is gone?
- pasta 10y ago
  In-Memory could be used for processing data. Or as temp tables. And by the way: a lot of existing databases support this.
- tidwall 10y ago
  SummitDB does persist to disk. Shutting down or kill -9 the server will not lose data.
deforciant 10y ago
looks great, I really like BuntDB, already using it in project :) Glad to see this new SummitDB, will definitely try it out!
ddorian43 10y ago
Looks like it has no sharding unfortunately. Do you have any info/eta/idea on this op ?
- tidwall 10y ago
  Yes it is something that I certainly want soon, but there's no ETA at the moment. I haven't yet fully flushed out the strategy for sharding the key space.

81 points by tidwall 10y ago

Someone should invent a package manager for databases, the same way we have npm, composer, etc. it's getting difficult to manage all my different databases.

tshannon 10y ago
Competitions good, imo. We have an "Embarrassment of riches", as it were, when it comes to databases.
- m1sta_ 10y ago
  And yet nothing that meets all my requirements. Still so much room to improve!
- deleted 10y ago
  [deleted]
arielweisberg 10y ago
Totally! I can just sudo apt-get install low-latency-strong-consistency-multi-dc-with-declarative-query-language-db as a virtual package and get the right one for my distro.
- nickpsecurity 10y ago
  Is that the command for installing Google's F1 RDBMS? It has those traits. I really wish they'd spin it off as a product since the only startup offering something similar got acquired and deep sixed by Apple.
  - arielweisberg 10y ago
    I don't think it meets the low latency requirement. It's a CAP theorem joke. Can't have strong consistency across multiple data centers with low latency because you have to read/write from a quorum of DCs in order to maintain availability in case one DC goes down.
    - nickpsecurity 10y ago
      Oh yeah. Low latency eould be saying too much. I think its worst-case for strong-consistency is around 30 seconds or so. Anyway, closest to to low-latency part would be NonStop or OpenVMS clusers across redundant, leased lines. They're fast, scale well at HW level, use ACID databases, and high-availability. Commonly used in transaction processing. One VMS cluster survived 9/11 attack without losing a single transaction. NonStop does up to five 9's. Id be interested in the CAP analysis of these older methods.
      - arielweisberg 10y ago
        It also depends on your definition of data center. It's really multi-region that is the problem since then it's the speed of light you have to deal with. Quorum across some data centers is fast enough that it's not hard to manage in new applications.
        nickpsecurity 10y ago
        That makes a lot of since. Many of the NonStop and VMS clusters were in same country but far enough from each other to isolate against many disasters. Could be why they did better on the issue.
    - mirekrusin 10y ago
      You can always use quantum entanglement to help a (qu)bit.
  - lobster_johnson 10y ago
    Take a look at CockroachDB, which had its first public beta earlier this year [1]. It's directly inspired by Google's F1 and Spanner projects. It's similar to FoundationDB (the Apple product you're referring to) in that it's an SQL database layered on top of a K/V store. It's based on some clever technology to accomplish distributed transactions, strong consistency and high availability, and is looking very promising. [1] https://www.cockroachlabs.com/ https://www.cockroachlabs.com/
    - nickpsecurity 10y ago
      Appreciate the tip. Im aware of it. They had a recent post on HN where they were just getting around to dealing with stability issues. I was really excited till I saw that. Im holding off for a while till it matures a bit.
      - ddorian43 10y ago
        I think they just went too far with full-sql-joins-indexes-column-families in the first 1.0 version. Better to build it little by little.
    - carterehsmith 10y ago
      "First public beta" of a database product? No.
- manigandham 10y ago
  Great options already exist: MemSQL for blazing fast distributed full SQL database with cross-datacenter replication and in-memory rowstore + disk-based columnstore. ScyllaDB for Cassandra rewritten in C++ for blazing fast dynamo-style multi-master multi-datacenter disk-based wide-column database. I'll also throw in AMPS (by 60East) as the best messaging platform that supports innovative SQL and real-time state-of-the-world queries on it's message streams (instead of using that rabbitmq or kafka crap).
  - akbar501 10y ago
    https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite Dynomite for production proven, limitless scale for Redis. Provides ability to Redis for both in-memory and on-disk workloads. Dynamo-inspired linearly scalable, shared nothing multi-master architecture. Pairs well with Cassandra. Supports pluggable backends (ex. RocksDB). Been in production 2+ years. One of the larger clusters handles over 3.6 million sustained ops/sec in production, every day.
    - ddorian43 10y ago
      Redis is only the api/driver, not the features, right ? Meaning it's just crud and not the many features/data-types that redis provides, correct ?
      - akbar501 10y ago
        It's the full Redis API and protocol. See https://github.com/Netflix/dynomite/blob/dev/notes/redis.md https://github.com/Netflix/dynomite/blob/dev/notes/redis.md for a list of all supported Redis features. The benefit of Dynomite's support for the Redis API + protocol is a.) you can use any Redis client and b.) the same code can be used for standalone Redis on your laptop and on a distributed Dynomite cluster.
    - manigandham 10y ago
      This is neat, but why use this over just using Cassandra-based DBs? Also you were a speaker at Data Layer right?
- biokoda 10y ago
  http://www.actordb.com/ http://www.actordb.com/

qwertyuiop924 10y ago

Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB? Thus far, I've been pressing SQLite into use for a lot of monotyped data that would have been better served by a NoSQL DB, but I couldn't find one suitable. Apparently, in-memory DBs in now. They're fast and all, that's nice, but I'm still running on a machine with 8 Gigs of ram, and when the dataset starts to exceed that, I want an option other than "grab some more DDR3." I remember the fate of TinyMUD: on-disk operation must at least be an option for any DB I'd use for datasets of a non-fixed size.

brightball 10y ago
Mnesia built into Elixir/Erlang potentially?
misframer 10y ago
There are several key-value stores you can choose from. RocksDB, LevelDB, lmdb, Berkeley DB, Tokyo Cabinet, sophia, and others. What about those? You can always build abstractions on top.
- lopatin 10y ago
  BoltDB if you'd like to embed it into a Go program.
  - chaotic-good 10y ago
    BoltDB can be corrupted on crash (kill -9 or hard reset) at least this was the case when I gave it a try. Any database can but in that case that was easy to achieve.
    - mbertschler 10y ago
      did you file a bug for this? I thought it was pretty resilient in such cases
      - 10y ago

fiatjaf 10y ago

This is exactly like what I wanted Redis to be. Secondary indexes solve all the problems. However, I wanted a feature that allowed custom indexes with Javascript, like CouchDB map functions.

tidwall 10y ago
This is available with SummitDB. The SETINDEX command has an EVAL option that allow for custom JavaScript indexes.
deleted 10y ago
[deleted]
dvirsky 10y ago
I'm working on a secondary index implementation in redis using modules, that should be out in a few days. It indexes redis hashes similar to how you index tables.
- ddorian43 10y ago
  Check out redis-search module which I think does that (only for keys for now) https://github.com/RedisLabsModules/RediSearch/commit/86eb1c4a76dddd172fc208f3caa162bdc7b0692c https://github.com/RedisLabsModules/RediSearch/commit/86eb1c...
  - dvirsky 10y ago
    I wrote that module as well :) The secondary index module will be more focused on automating more traditional indexing of numbers and simple string values. It has an SQL-ish language for WHERE expressions, and then the result of that is piped to any redis command you want it to. If you're interested, here's a draft of the syntax (it's changed a bit since but you'll get the idea). https://gist.github.com/dvirsky/3ef73143a6d8212f2b50096a8eb68018 https://gist.github.com/dvirsky/3ef73143a6d8212f2b50096a8eb6...
    - ddorian43 10y ago

skrebbel 10y ago

I can't find from the docs at all whether this persists to disk. If not, what is the use case? Why would I need all those ACID-y guarantees if my server can fail at any time and all data is gone?

pasta 10y ago
In-Memory could be used for processing data. Or as temp tables. And by the way: a lot of existing databases support this.
tidwall 10y ago
SummitDB does persist to disk. Shutting down or kill -9 the server will not lose data.

deforciant 10y ago

looks great, I really like BuntDB, already using it in project :) Glad to see this new SummitDB, will definitely try it out!

ddorian43 10y ago

Looks like it has no sharding unfortunately. Do you have any info/eta/idea on this op ?

tidwall 10y ago
Yes it is something that I certainly want soon, but there's no ETA at the moment. I haven't yet fully flushed out the strategy for sharding the key space.