4 ms·

Show HN: Skytable's new NoSQL engine with BlueQL, injection safety, better perf

17 points by sayann 3y ago

Hey HN! I've been working on Skytable since 2020 and after several iterations from a simple K/V store, we've walked the path to this release. The goal of Skytable is to deliver a solid foundation for building data intensive applications. Skytable's primary goal is performance and scale. Even with a query language it can outperform K/V stores which use simple commands (benchmarks will be shared in another post). Several implementations in Skytable (especially around query evaluation and execution) are fundamentally different from SQL and even NoSQL counterparts and there are some entirely new concepts which might make it a little hard to grasp. BlueQL is a very important part of Skytable and it employs some interesting concepts to try and reduce the surface for injection attacks and tries to be a modern and secure alternative to SQL. - Source code: https://github.com/skytable/skytable https://github.com/skytable/skytable - Documentation: https://docs.skytable.io https://docs.skytable.io - Rust client driver (more client drivers are underway): https://github.com/skytable/client-rust https://github.com/skytable/client-rust I'm here for any questions! -- Sayan NB: Clustering and Replication will be released around Q1'24. Choice: AP in CAP. The original post has been updated for clarity.

johnwoods 3y ago
Wow, this looks incredible! It looks like the release post has changed significantly since I saw and shared it. Looking forward to the benchmarks and 1.0!
- sayann 3y ago
  Thank you. Yes, the early benchmarks definitely seem exciting! I'm working on a comprehensive benchmark with resource usage, latency reports, throughput and more (and benchmarking against more systems now).
zackify 3y ago
Congrats on pursuing this for almost 4 years now. Exciting to see the progress and your continued dedication.
- sayann 3y ago
  Thank you for following along with us for so long!
kadomony 3y ago
How does it measure up against something like MongoDB?
- sayann 3y ago
  Perf tests will be done soon. The data model is different from mongo and the query language differs as well. Otherwise you should be able to store what you can in Mongo on Skytable as well (excluding full text search, a feature that we're working on).
  - mdaniel 3y ago
    Any plans on using Jepsen? I searched the issue tracker and didn't see it, but maybe that's not where such a thing would appear in your roadmap
    - sayann 3y ago
      Yes. Jepsen is very much planned once C&R is shipped (ETA Q1'24)
avinassh 3y ago
> Skytable's primary goal is performance and scale. Even with a query language it can outperform K/V stores which use simple commands (benchmarks will be shared in another post). How does Skytable achieves this? In other words, what makes it different than others?
- sayann 3y ago
  The linked blog post has some notes on the architectural differences. Overall, we use fundamentally different underlying implementations for various components, from the network layer to the storage engine. If you want a more in-depth review you can read https://docs.skytable.io/architecture https://docs.skytable.io/architecture That document will be updated frequently!
  - avinassh 3y ago
    I skimmed through the article, still I don’t get it what makes it so fast that it can beat other KV Stores. Does it use a B Tree / LSM / Bitcask? Or something else entirely? How does your storage engine compare with sled? The storage section also mentioned that most data stays in RAM and it is planned to change. Is it still the same? The benchmarks were ran on the in-memory version or disk one (if that is changed) Also, can you share against which all KV Stores the benchmark was done? The project looks very exciting and I’d love to know more details
    - sayann 3y ago
      Skytable is currently primarily in-memory (which is to change soon; approximately Q1 next year). The storage engine actually uses various drivers and most of them are AOF-style custom implementations. Real-time/background compaction and more will be implemented shortly before GA is shipped (this is currently a beta).
mdaniel 3y ago
https://docs.skytable.io/blueql/overview#parameters https://docs.skytable.io/blueql/overview#parameters Wow, that's very aggressive. It smells to the outside observer like overcorrecting for a real but blown out of proportion problem
- sayann 3y ago
  I know it might look that way. But we're just trying to reduce the surface area for mistakes. In the end, one can rest knowing that they didn't at least write vulnerable code, the most common one!

17 points by sayann 3y ago

johnwoods 3y ago

Wow, this looks incredible! It looks like the release post has changed significantly since I saw and shared it. Looking forward to the benchmarks and 1.0!

sayann 3y ago
Thank you. Yes, the early benchmarks definitely seem exciting! I'm working on a comprehensive benchmark with resource usage, latency reports, throughput and more (and benchmarking against more systems now).

zackify 3y ago

Congrats on pursuing this for almost 4 years now. Exciting to see the progress and your continued dedication.

sayann 3y ago
Thank you for following along with us for so long!

kadomony 3y ago

How does it measure up against something like MongoDB?

sayann 3y ago
Perf tests will be done soon. The data model is different from mongo and the query language differs as well. Otherwise you should be able to store what you can in Mongo on Skytable as well (excluding full text search, a feature that we're working on).
- mdaniel 3y ago
  Any plans on using Jepsen? I searched the issue tracker and didn't see it, but maybe that's not where such a thing would appear in your roadmap
  - sayann 3y ago
    Yes. Jepsen is very much planned once C&R is shipped (ETA Q1'24)

avinassh 3y ago

> Skytable's primary goal is performance and scale. Even with a query language it can outperform K/V stores which use simple commands (benchmarks will be shared in another post). How does Skytable achieves this? In other words, what makes it different than others?

sayann 3y ago
The linked blog post has some notes on the architectural differences. Overall, we use fundamentally different underlying implementations for various components, from the network layer to the storage engine. If you want a more in-depth review you can read https://docs.skytable.io/architecture https://docs.skytable.io/architecture That document will be updated frequently!
- avinassh 3y ago
  I skimmed through the article, still I don’t get it what makes it so fast that it can beat other KV Stores. Does it use a B Tree / LSM / Bitcask? Or something else entirely? How does your storage engine compare with sled? The storage section also mentioned that most data stays in RAM and it is planned to change. Is it still the same? The benchmarks were ran on the in-memory version or disk one (if that is changed) Also, can you share against which all KV Stores the benchmark was done? The project looks very exciting and I’d love to know more details
  - sayann 3y ago
    Skytable is currently primarily in-memory (which is to change soon; approximately Q1 next year). The storage engine actually uses various drivers and most of them are AOF-style custom implementations. Real-time/background compaction and more will be implemented shortly before GA is shipped (this is currently a beta).

mdaniel 3y ago

https://docs.skytable.io/blueql/overview#parameters https://docs.skytable.io/blueql/overview#parameters Wow, that's very aggressive. It smells to the outside observer like overcorrecting for a real but blown out of proportion problem

sayann 3y ago
I know it might look that way. But we're just trying to reduce the surface area for mistakes. In the end, one can rest knowing that they didn't at least write vulnerable code, the most common one!