3 ms·

Thanks for your time. I'm the lead dev on HyperDex, so I'd be happy to address your points. - A SEARCH is simply a mapping from secondary attributes to object

by rescrv 14y ago

Thanks for your time. I'm the lead dev on HyperDex, so I'd be happy to address your points. - A SEARCH is simply a mapping from secondary attributes to objects and it was designed for your use case. - We're still adding server-side atomic operations. In the future we'll definitely have richer types and operations. - Server-side exec of Lua code is definitely an interesting feature. - We've put a lot of effort into making it easy to manage a HyperDex cluster without sacrificing on the scalability or fault tolerance. When you do your assessment comparing the two systems, please let us know the outcome because we'll definitely correct any deficiencies. - The client does indeed maintain a mapping for hashing. Our design is pretty asynchronous already (from the C API), so it'd be a good idea to optimize that further. Of course, we'll eventually document the wire protocol for those who want to go further. Feel free to contact me if you have questions or need a hand.

DEinspanjer 14y ago
@rescrv I remember seeing some of your earlier posts about HyperDex. It continues to be an interesting project to follow. I had several questions that I didn't see covered in the docs. Would love to see them answered somewhere: 1. If you have more than f faults and the cluster shuts down, what are the ramifications? How do you start it back up and recover? What if one or more of the faulted nodes are unrecoverable? Can you easily recover the partial data? 2. Can you perform OR logic in searches? i.e. Where attribute is 'a' OR 'b'? 3. Where is the new count method documented? 4. Is it possible to add new attributes to an existing space? Add new subspaces? Remove subspaces? What is the storage/memory cost associated with subspaces? 5. Are there plans for grouping and aggregation of any sort? I've been searching for a while to find a good key/value or document store system that can provide very expressive searching in combination with grouping and aggregation to provide an analytic platform for storing documents that don't fit very well in a traditional SQL store due to complexity of the structure (nested lists, etc) or dynamic addition of attributes.
- rescrv 14y ago
  1. In the current release it's possible to completely shutdown the cluster. You cannot just cut power, but it's relatively painless to do otherwise. Checkout http://hyperdex.org/doc/06.faults/#shutting-down-and-restoring-a-cluster http://hyperdex.org/doc/06.faults/#shutting-down-and-restori... for more details. 2. Currently you cannot perform OR logic in searches. It'd be relatively easy to add though, so we'll probably do so in the future. It's possible to do two searches concurrently and combine the results. 3. This is an oversight on our part. We have to document it. In the Python bindings, you call it the same way as a regular search, but instead of getting a generator that yields objects, you simply get a number. 4. Currently, things are pretty static. We've got ideas for changing that, but just haven't pursued it yet. Each subspace replicates the data again. The memory cost of a subspace is nearly nothing (just a list of replicas). 5. I think that one-level group-by might be achievable, similar to our sorted_search primitive. There's a lot of research that people have done on this topic, and it's very interesting. That being said, the simple/straightforward approach would probably be what we'd take. Thanks for following the project!
gizzlon 14y ago
@rescrv It's my impression that Redis is well-opitmized C code. How can HyperDex be "2-14 faster" ?
- rescrv 14y ago
  First, HyperDex is well-optimized C++ code. And second, they take issue with the standard benchmark we used to evaluate both systems. While we recognize that Redis may have an impedance mismatch with the benchmark, we were hesitant to create our own lest we inadvertently introduce bias toward our system. Thus the numbers reported are accurate, but the generic nature of the YCSB benchmark may not be exploiting all that Redis offers. We'd love to work with the Redis team on a standardized benchmark we both can get behind but that has not yet happened (and may never happen). In the grander scheme of things, the two systems are very different. In my opinion, Redis is very well suited for inter-process communication where extremely low latency is necessary and data is ephemeral. HyperDex, on the other hand, is great at storing persistent data. The configurable fault tolerance setting makes it very clear how safe your data is, and the strong consistency is very convenient when building applications. Further, HyperDex is built from the ground up to operate in a clustered environment and does so quite well.
  - gizzlon 14y ago
    I see.. I googled it and it seems like the comparison is multi-threaded (multi process? multi machine??) vs single-threaded, single process. It doesn't really seem like a fair comparison, and since you guys are obviously knowledgeable I bet you understand as much. Please don't let what seems like a cool project be tainted by marketing bullshit. (I really, really hate marketing bullshit :)
    - rescrv 14y ago
      I bet you found the thread on their mailing list. I was the one who ran the benchmark and it was one core vs. one core. In the other benchmarks we had multiple machines, but not the Redis benchmark. We clarified after they speculated that a multi/single comparison was performed. The main issue they had was YCSB's method of storing multi-attribute objects as maps (from attribute names to their values) and the fact that Redis cannot support search across keys so YCSB searches within a single set.

by rescrv 14y ago

DEinspanjer 14y ago

@rescrv I remember seeing some of your earlier posts about HyperDex. It continues to be an interesting project to follow. I had several questions that I didn't see covered in the docs. Would love to see them answered somewhere: 1. If you have more than f faults and the cluster shuts down, what are the ramifications? How do you start it back up and recover? What if one or more of the faulted nodes are unrecoverable? Can you easily recover the partial data? 2. Can you perform OR logic in searches? i.e. Where attribute is 'a' OR 'b'? 3. Where is the new count method documented? 4. Is it possible to add new attributes to an existing space? Add new subspaces? Remove subspaces? What is the storage/memory cost associated with subspaces? 5. Are there plans for grouping and aggregation of any sort? I've been searching for a while to find a good key/value or document store system that can provide very expressive searching in combination with grouping and aggregation to provide an analytic platform for storing documents that don't fit very well in a traditional SQL store due to complexity of the structure (nested lists, etc) or dynamic addition of attributes.

rescrv 14y ago
1. In the current release it's possible to completely shutdown the cluster. You cannot just cut power, but it's relatively painless to do otherwise. Checkout http://hyperdex.org/doc/06.faults/#shutting-down-and-restoring-a-cluster http://hyperdex.org/doc/06.faults/#shutting-down-and-restori... for more details. 2. Currently you cannot perform OR logic in searches. It'd be relatively easy to add though, so we'll probably do so in the future. It's possible to do two searches concurrently and combine the results. 3. This is an oversight on our part. We have to document it. In the Python bindings, you call it the same way as a regular search, but instead of getting a generator that yields objects, you simply get a number. 4. Currently, things are pretty static. We've got ideas for changing that, but just haven't pursued it yet. Each subspace replicates the data again. The memory cost of a subspace is nearly nothing (just a list of replicas). 5. I think that one-level group-by might be achievable, similar to our sorted_search primitive. There's a lot of research that people have done on this topic, and it's very interesting. That being said, the simple/straightforward approach would probably be what we'd take. Thanks for following the project!

gizzlon 14y ago

@rescrv It's my impression that Redis is well-opitmized C code. How can HyperDex be "2-14 faster" ?

rescrv 14y ago
First, HyperDex is well-optimized C++ code. And second, they take issue with the standard benchmark we used to evaluate both systems. While we recognize that Redis may have an impedance mismatch with the benchmark, we were hesitant to create our own lest we inadvertently introduce bias toward our system. Thus the numbers reported are accurate, but the generic nature of the YCSB benchmark may not be exploiting all that Redis offers. We'd love to work with the Redis team on a standardized benchmark we both can get behind but that has not yet happened (and may never happen). In the grander scheme of things, the two systems are very different. In my opinion, Redis is very well suited for inter-process communication where extremely low latency is necessary and data is ephemeral. HyperDex, on the other hand, is great at storing persistent data. The configurable fault tolerance setting makes it very clear how safe your data is, and the strong consistency is very convenient when building applications. Further, HyperDex is built from the ground up to operate in a clustered environment and does so quite well.
- gizzlon 14y ago
  I see.. I googled it and it seems like the comparison is multi-threaded (multi process? multi machine??) vs single-threaded, single process. It doesn't really seem like a fair comparison, and since you guys are obviously knowledgeable I bet you understand as much. Please don't let what seems like a cool project be tainted by marketing bullshit. (I really, really hate marketing bullshit :)
  - rescrv 14y ago
    I bet you found the thread on their mailing list. I was the one who ran the benchmark and it was one core vs. one core. In the other benchmarks we had multiple machines, but not the Redis benchmark. We clarified after they speculated that a multi/single comparison was performed. The main issue they had was YCSB's method of storing multi-attribute objects as maps (from attribute names to their values) and the fact that Redis cannot support search across keys so YCSB searches within a single set.