6 ms·
Show HN: A bi-directional, persisted KV store that is faster than Redis
we've been working on a KV store for the past year or so which is 2-6x faster than Redis (benchmark link below) yet disk persisted! so you get the speed of in-memory KV stores but with disk persistence.
To achieve this we've created our custom filesystem that is optimized for our special usecase and we're doing smart batching for writes and predictive fetching for reads.
In addition to basic operations, it also provides atomic inc/dec, atomic json patch, range scans and a unique key monitoring mechanism (pub-sub) over WebSockets which essentially allows you to receive notification on registered key changes directly from the KV store. so for example in a realtime web application, you can receive notifications directly in your front-end, with no back-end implementation (no WebSocket server management, no relay etc.) and still be secure and not expose your API keys on front-end.
We have REST, WebSocket and RIOC API and we can't wait to hear your feedback.
We're only providing the free tier for now but let us know and we can increase the limits for you, if have a specific case. please either send us an email to support@hpkv.io or use http://hpkv.io/contact http://hpkv.io/contact if you prefer that way.
sign up: http://hpkv.io/signup http://hpkv.io/signup
documentation: http://hpkv.io/docs http://hpkv.io/docs
realtime pub-sub: http://hpkv.io/blog/2025/03/real-time-pub-sub http://hpkv.io/blog/2025/03/real-time-pub-sub
benchmark vs Redis: http://hpkv.io/blog/2025/02/redis-vs-hpkv-benchmark http://hpkv.io/blog/2025/02/redis-vs-hpkv-benchmark
looking forward to hear your feedback :)
- koushik_indie 2y ago[flagged]
- ranger_danger 2y agoDoes not appear to be open-source.
- mehrant 2y agocorrect. however we're actually planning to make the system open source in future; we can't set an exact date as it depends on various factors, but hopefully not too far out. :)
- huhtenberg 2y agoIn that case you need to provide an SLA for your speed claims. Otherwise the claims are basically moot.
- mehrant 2y agothat's a fair point and you're correct. we will have the SLAs for latency documented and provided soon. in the mean time, please try it out and give us your feedback :)
- huhtenberg 2y agoThe site is very snappy, which matches well your pitch. However your principal selling point - the nanosecond-level speed - falls flat because it's a property important in self-hosted scenarios. Once you put your super speedy stuff behind a web-based API, that selling point becomes completely meaningless. The fact that once our data hits your servers it is handled really quickly doesn't mean much. I am sure you are perfectly aware of that. That is, your pitch is disconnected from your actual offering. If you are selling speed, it needs to be a product, not a service. It doesn't need not be open source though, just looks at something like kdb+.
- mehrant 2y agothanks for the feedback :) our main target for "performance" value proposition are companies and businesses which will setup HPKV either locally (Enterprise plan) for nanosecond performance or in the cloud provider of their choosing, and working via RIOC API (Business Plan), getting ~15 microsecond range over network. however you're totally right, that doesn't really matter much if you're using it REST or WebSocket. for Pro tier, our value proposition is still the fastest managed KV store (you still get <80 ms for writes with a ~30ms ping to our servers) and features such as bi-directional WS, Atomic operations and Range Scans on top basic operations. but given your comment, I think we should perhaps rethink how we're presenting the product. thanks for the feedback again :)
- 2y ago
- deleted 2y ago[deleted]
- Snawoot 2y ago> 2-6x faster than Redis (benchmark link below) yet disk persisted! That's a false contradistinction: Redis is also disk persisted. The benchmark you did mentions Redis benchmarking guide and this guide has following paragraph: > Redis is, mostly, a single-threaded server from the POV of commands execution (actually modern versions of Redis use threads for different things). It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. It is not really fair to compare one single Redis instance to a multi-threaded data store. Did you just benchmarked against only single Redis instance and claimed performance win? Even if so, how do benchmarks compare against source-available competitor DragonflyDB? Finally, documentation doesn't mention how persistence exactly works and what durability guarantees should we expect?
- mehrant 2y agothanks for taking time to write a feedback :) > That's a false contradistinction: Redis is also disk persisted. The performance gain mentioned was vs. Redis in memory. so we weren't claiming that Redis can't be persisted (which of course it can), but we were saying that Redis without persistence (which performs faster that with persistence) was still this much slower than HPKV with persistence. But you're correct that we probably should have been more clear in explaining this :) >Did you just benchmarked against only single Redis instance and claimed performance win? Signle node of Redis vs. Single node of HPKV. so it's an apples to apples comparison >Even if so, how do benchmarks compare against source-available competitor DragonflyDB? Benchmark with DragonFly coming soon :) sorry about lack of that information in documentation, we'll update that. for for now, the durability guarantee on Pro is 30 seconds. on Business with HA is 5 minutes.
- kshmir 2y agoWhy pay what you're asking instead of using dragonfly or something like that and just putting a beefier node?
- ehsanaslani 2y agoWell that's a technical choice depending on the context, but I can list some of the advantages of HPKV: -Persistent by default without any performance penalties -The pub/sub feature which is unique to HPKV and allows for a bi-directional websocket connection from clients to database -Lower cost as we need less expensive infrastructure to provide the same service -Simple API to use
- theonlyvasudev 2y agoAmazing!
- mehrant 2y agothank you :)
- CyberDildonics 2y agoThat person only has one comment but they also made a database 8 months ago. Crazy coincidence, you could get together and compare notes.
- dangoodmanUT 2y agoWhat disks give 600ns persistence _with fsync/fdatasync_? Never heard of anything under 50us p50.
- mehrant 2y agothe 600ns figure represents our optimized write path and not a full fsync operation. we achieve it -among other things- through: 1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers. 2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge. 3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers the transactional guarantees we provide is via: 1- atomic individual operations with retries 2- various conflict resolution strategies (timestamp, etc.) 3- durability through controlled persistence cycles with configurable commit intervals depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes
- dangoodmanUT 2y agoI didn't necessarily mean exactly fsync. I guess I'll ask: Is it actually flushed to persistent disk in 600ns such that if the node crashes, the data can always be read again? Or does that not fully flush?
- mehrant 2y agoyes, in that case data can potentially be lost. 30 sec in a worse case scenario without HA.
- gkbrk 2y agoThere's a tiny 50,000,000x difference between the now admitted 30 seconds and the previously claimed 600 nanoseconds.
- dangoodmanUT 2y agoSo it's not actually persistence then. That's extremely deceptive, and (IANAL) I think false advertisement. I'd clarify it. That's also not HA, that's durability. Concerning.
- quibono 2y agoIs this 2-6x faster because of multi threading/core? Or is this actually 2-6x faster on a single core machine?
- mehrant 2y agothe test was done on a single node and a single thread. on multi thread and batch operations, HPKV was still faster on the same machine
- alex_smart 2y agoI don’t get it. How could you be fsyncing the WAL in 600ns? What are the transactional guarantees that you are offering?
- mehrant 2y agothat's a great question. the 600ns figure represents our optimized write path and not a full fsync operation. we achieve it -among other things- through: 1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers. 2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge. 3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers the transactional guarantees we provide is via: 1- atomic individual operations with retries 2- various conflict resolution strategies (timestamp, etc.) 3- durability through controlled persistence cycles with configurable commit intervals depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes
- buenzlikoder 2y agoWhat storage backend are you using? A write operation on a SSD takes 10s of uS - without any VFS layers
- mehrant 2y agosorry for not being clear again. by saying this number does not represent full fsync operation, I meant it doesn't include the SSD write time. this is the time to update KVs internal memory structure + adding to write buffers. this is fair because we provide transactional guarantee and immediate consistency, regardless of the state of the append-only write buffer entry. during that speed, for a given key, the value might change and a new write buffer entry might be added for the said key before the write buffer had the chance to complete (as you mentioned the actual write on disk is slower) but the conflict resolution still ensures the write of the last valid entry and skips the rest. before this operation HPKV is acting like an in-memory KV store.
- 2y ago
- tobyhinloopen 2y agoNot open source, not interested. It looks neat though, but I won't burn myself on anything closed source if there's open source and/or self-hosted alternatives.
- mehrant 2y agothanks for taking time and commenting :) we'd still be happy if you decided to use it and give us your thoughts. as I mentioned in one of the comments below, we're hoping to go open source in future :)
- verdverm 2y ago> we're hoping to go open source in future :) That's just lip service, either you intend to or you don't, it's not up to hope We hope you will see the dev tooling space is based around open source and you will alienate many potential users by not being open source
- tobyhinloopen 2y agoLet me know when it's open source and I'd be happy to give it a try! It doesn't even have to be free - I'm fine with Unreal's model where you get access to the source even though commercial use requires a paid license. I want something running locally on my machine that doesn't rely on calling home.
- mehrant 2y agoyour concern is understandable. we'll be in touch :)
- hdjjhhvvhga 2y agoGive me one reason to use a closed-source Redis alternative rather than one of many open ones, starting with KeyDB. If I wanted a closed clone, I'd probably go with DragonflyDB (whose license is "feel free to run it in production unless you offer it as a managed redis service").
- tschellenbach 2y ago
- cess11 2y agoDoes it have ACID guarantees?
- mehrant 2y agoWe provide some elements of ACID guarantees, but not full ACID compliance as traditionally defined in database systems: Atomicity: Yes, for individual operations. Each key-value operation is atomic (it either completes fully or not at all). Consistency: Partial. We ensure data validity through our conflict resolution strategies, but we don't support multi-key constraints or referential integrity. Isolation: Limited. Operations on individual keys are isolated, but we don't provide transaction isolation levels across multiple keys. Durability: Yes. Our persistence model allows for tunable durability guarantees with corresponding performance trade-offs. So while we provide strong guarantees for individual operations, HPKV is not a full ACID-compliant database system. We've optimized for high-performance key-value operations with practical durability assurances rather than complete ACID semantics.
- gcbirzan 2y ago> Consistency: Partial. We ensure data validity through our conflict resolution strategies, but we don't support multi-key constraints or referential integrity. That's not what consistency means in ACID. > Durability: Yes. Our persistence model allows for tunable durability guarantees with corresponding performance trade-offs. > ~600ns p50 for writes with disk persistence I'm pretty sure there's no durability there. That statement is pretty disingenuous in itself, but it'd be nice to see a number for durability (which, granted, is not something you advertise the product for). My main concern is that all these speed benefits are going to be eclipsed by the 0.5ms of network latency.
- cess11 2y agoOK, thanks. Those tradeoffs aren't suitable for my purposes.
- edoceo 2y agoHow will it be faster than my Redis or KeyVal which is very close if your servers are far away? Network time matters here, right?
- mehrant 2y agoof course. the speeds down to 15us can be achieved over network over our custom protocol on the same region. for sub-microsecond latency, you need to have HPKV running on the same machine as yours :)
- bjornsing 2y agoInteresting. I did some work on a related but different product idea (https://www.haystackdb.dev/ https://www.haystackdb.dev/) a few years back. Gave up though as it seemed hard to get traction / find customers. What’s your thinking on that? How are you going to reach your initial customers? Would love to have a chat about possible collaboration or if I could help out in some way. Nice to see foundational tech coming out of the EU!
- mehrant 2y agothank you :) it would be interesting to have a chat for sure. would you mind dropping an email on the email I mentioned in OP and I'll reach out to you.
- mrbluecoat 2y agoSomething must be in the water.. this is the third similar tool in three days on HN https://news.ycombinator.com/item?id=43379262 https://news.ycombinator.com/item?id=43379262 https://news.ycombinator.com/item?id=43371097 https://news.ycombinator.com/item?id=43371097
- linotype 2y agoYeah, Redis fucked around and found out. https://redis.io/blog/redis-adopts-dual-source-available-licensing/ https://redis.io/blog/redis-adopts-dual-source-available-lic...
- rendaw 2y agoThat was a year ago.
- conception 2y agoI guess we know how long it takes to make a redis clone.
- avinassh 2y agoIf it based on some research papers, could you link them please
- mehrant 2y agoOne thing we'd like to know your opinion on, is our key monitoring via WebSocket (pub-sub) feature. You can read more about it in our documentation under WebSocket. Is it something that you think it's useful and you might have use case for or you can't see any value in it? In other words, is it something that you might consider using HPKV because of it?
- alexpadula 2y agoWhy no open source :<