5 ms·
GhostDB – A Fast Distributed Cache
- nodesocket 6y agoGreat project. I did not see an explanation on the architecture of a cluster. Is data replicated to all nodes, or is data only stored on a single node (sharded). What happens when a node in the cluster goes down? Does there have to be a consensus (odd number of nodes) for the cluster to be "healthy"?
- cmckn 6y ago> GhostDB provides a very large hash table that is distributed across multiple machines. Sounds like no replication to me, AKA the memcached model. I can't find how to actually configure a cluster (the `Cluster Configuration` section of the docs doesn't contain anything related to hosts). I also can't find anything client side that would distribute requests to a list of nodes with a consistent hash, for ex. I can't find a client at all, actually. Still, interesting project, kind of aiming for Redis features and a memcached topology.
- GhostDBCache 6y agoThe clients are the SDKs. Our docs are getting an update now to better explain how to configure clusters. The SDKs are in separate repos currently (this is due to how university made us structure the project).
- GhostDBCache 6y agoCurrently we follow the memcached model so there is no replication however, remember this is straight out of university were we had to limit scope considerably and were on a tight time constraint but we aim to add data replication, consensus etc.
- iammru 6y agoDoesn't Redis already address all of this?
- cmckn 6y agoYes, probably with superior performance. The clustering model seems to differ, but as far as I can tell, the 'distributed' part of this project isn't really implemented.
- GhostDBCache 6y agoDepending on the use case. If this is being used to reduce load on your database, then there are no noticeable differences between performance of GhostDB or redis from the clients end.
- totorovirus 6y agoSame question. How is this different from Redis?
- GhostDBCache 6y agoNot too different from Redis at the moment, however this is straight out of a university project were we had a tight deadline and had to limit scope considerably but if you check the roadmap on the repo we want to add a lot to it.
- AlphaSite 6y agoIs this similar to geode/gem fire?
- GhostDBCache 6y agoI'm not aware of these so I can't say
- trishume 6y ago“microsecond performance” irks me because you can’t get latencies of a microsecond without very expensive special network cards and APIs which they don’t seem to mention and I doubt they use. So they either mean 10s to hundreds of microseconds or they’re taking the reciprocal of their throughput which would be weird and misleading. If you’ve made a fast product that’s great! Show us with well-explained benchmarks not the term which sounds best but you can still hand wave as justified, because it’ll make people suspicious.
- dallbee 6y agoOr, perhaps, they're taking the non-networked use case.
- 65536 6y agoI don’t have a stake in this but the page does say > delivers microsecond performance at any scale and then > a very large hash table that is distributed across multiple machines So the way that I read it, the network will be involved when they said “at any scale”.
- Icyphox 6y agoFucking Discord for an open source project.
- matt_f 6y agoCould anyone explain a use case for this? (Distributed KV store) Genuine question, thanks
- mbreese 6y agoI would imagine instances where you'd use memcached. Session storage could be one use-case. Another could be caching DB query results for a specified amount of time (1 min, 5 min, etc).
- GhostDBCache 6y agoIn-memory data lookup Relational and Non-relational database speedup Managing spikes in web/mobile apps Session-store Token caching Gaming - Player profiles & leaderboards Web page caching Global ID or counter generation Fast access to any suitable data
- dvirsky 6y agoWritten in Go? I wonder what the GC pressure is going to look like with many small keys. I've written a redis clone in Go a few years ago and the GC pauses when reaching a few GBs of utilization were awful. Granted the Go GC has improved considerably since but it's still going to be noticeable I bet.
- jldugger 6y ago> I've written a redis clone in Go a few years ago and the GC pauses when reaching a few GBs of utilization were awful. Probably managable with a good slab allocator.
- dvirsky 6y agoYeah, I've seen people do that but I've glanced the code, it's certainly not the case here.
- GhostDBCache 6y agoWe're aware of the issues this could cause and we will be looking into relieving GC pressure
- dvirsky 6y agoGreat. Good luck. As someone who's worked on the redis codebase, I suggest you take a good look at some of the design decisions and optimizations that project took, e.g. wrt persistence and replication.
- boxfire 6y agoHow does this compare to Anna, which makes similar claims (and seems to actually deliver), https://github.com/hydro-project/anna https://github.com/hydro-project/anna (up to date link)
- GhostDBCache 6y agoI've never heard of this so I can't say.
- mesaframe 6y ago> in-memory, general purpose key-value caching > GhostDB can provide you with up to a 25x increase in data retrieval speeds when compared to databases such as MongoDB and MySQL. Isn't it insincere to say your tech is faster than a DBMS which works on a slower hardware.
- WJW 6y agoNot associated with the OP, but I don't think so. It really does return your data 25x faster than a disk-based DBMS. The implied reduction in durability from being in-memory is clearly stated up front.
- Irishsteve 6y agoIt’s listed as a college project that they’d like to turn into something generally available for use. Fair play.
- rawoke083600 6y agoCongrats for getting it out there ! That is more than half the battle. I thing I would like to see also is more Redis-Clones. The redis-parts not the KV parts(lots of options here). I want some alternatives or speed improvements for "Unions" and other set based operations :). PS. GhostDB looks cool !
- GhostDBCache 6y agoAdding support for sets is on our roadmap!
- strogonoff 6y ago> at any scale Would it be feasible to bundle GhostDB with an Electron app, and run it on end-user’s machine?
- GhostDBCache 6y agoI don't know much about electron but if it allows you to bundle binaries and execute them then yes!
- Neoxy 6y agoWill love to dig deeper
- forgotmypw17 6y agofor nojs folk: http://archive.is/U8tTA http://archive.is/U8tTA OP: fyi, your site fails to load or display anything besides a "loading" spinner which never goes away if js is disabled in browser.