4 ms·

Can someone tell me what does this do? And why its better.

by ms7892 2y ago

moralestapia 2y ago
It's a message queue on the cloud. https://chatgpt.com/c/676703d4-7bc8-8003-9e5d-d6a402050439 https://chatgpt.com/c/676703d4-7bc8-8003-9e5d-d6a402050439 Edit: Keep downvoting, only 5.6k to go!
- ms7892 2y ago
  Thank you
alanfranz 2y ago
Sort of serverless Kafka, which natively uses object storage and promises better latencies than things like warpstream.
- ivankelly 2y ago
  A interesting difference is the ability to have exclusive access to writes on the log (the fencing token). This allows you to use the logs as write ahead logs.
shikhar 2y ago
(Founder) There is a table on the landing page https://s2.dev/ https://s2.dev/ which hopefully gives a nice overview :) It's like S3, but for streams. Cheap appends, and instead of dealing with blocks of data and byte ranges, you work with records. S2 takes care of ordering records, and letting you read from anywhere in the stream. This is an alternative to systems like Kafka which don't do great at giving a serverless experience.
- stogot 2y ago
  Could you clarify the Kafka difference further? Or more generally, when is it better to choose S2 vs services like SQS or Kinesis? S2 sounds like an ordered queue to me, but those exist?
  - shikhar 2y ago
    (Founder here) Managed cloud offerings for streaming limit ordered throughput pretty low, e.g. Kinesis at 1 MiBps, Redpanda serverless at 1 MiBps, Confluent's even higher-end clusters at 10-20 MiBps IIRC. If you really need ordering, this can indeed be a limit. S2 lets you push 125 MiBps currently, and we may grow that. Another factor is how many ordered streams you can have. Typically a few thousand at most with those systems. We take the serverless spirit of S3 here, when did you have to worry about the number of objects in a bucket? We are also able to offer latency comparable to disk-based streaming like Confluent's Kora and Kinesis, with our Express storage class (under 50 milliseconds end-to-end latency for client in the same cloud region) - while being backed by S3 with regional durability! Not a disk in the system. We want people to be able to build safe distributed data systems on top of S2, so we also allow concurrency control mechanisms on the stream like fencing. Kafka or Kinesis won't let you do that. This is the approach AWS takes internally (https://brooker.co.za/blog/2024/04/25/memorydb.html https://brooker.co.za/blog/2024/04/25/memorydb.html), but they don't have that as a service. We want to democratize the pattern. ED: on throughtputs, to clarify, I am talking about ordered throughput, i.e. per Kafka partition or Kinesis shard. WarpStream also does well here because of their architectural approach to separate ordering, but at a latency cost.
    - alwa 2y ago
      Between your site copy and your early comments on this thread, it was this rundown that made the product click in my mind. I’m sure that in this early preview you’re trying to reach mainly devs with existing domain expertise, but the way that, in this comment, you laid out existing constraints and what possibilities might lie beyond them—it really helped me situate your S2 product in the constellation of cloud primitives. Just wanted to offer that feedback in the hope that the spirit of your comment here doesn’t get buried down-thread!
      - shikhar 2y ago
        thank you for the feedback!
    - agallego 2y ago
deleted 2y ago
[deleted]

Can someone tell me what does this do? And why its better.

by ms7892 2y ago

Can someone tell me what does this do? And why its better.

It's a message queue on the cloud. https://chatgpt.com/c/676703d4-7bc8-8003-9e5d-d6a402050439 https://chatgpt.com/c/676703d4-7bc8-8003-9e5d-d6a402050439 Edit: Keep downvoting, only 5.6k to go!

ms7892 2y ago
Thank you

alanfranz 2y ago

Sort of serverless Kafka, which natively uses object storage and promises better latencies than things like warpstream.

ivankelly 2y ago
A interesting difference is the ability to have exclusive access to writes on the log (the fencing token). This allows you to use the logs as write ahead logs.

shikhar 2y ago

(Founder) There is a table on the landing page https://s2.dev/ https://s2.dev/ which hopefully gives a nice overview :) It's like S3, but for streams. Cheap appends, and instead of dealing with blocks of data and byte ranges, you work with records. S2 takes care of ordering records, and letting you read from anywhere in the stream. This is an alternative to systems like Kafka which don't do great at giving a serverless experience.

stogot 2y ago
Could you clarify the Kafka difference further? Or more generally, when is it better to choose S2 vs services like SQS or Kinesis? S2 sounds like an ordered queue to me, but those exist?
- shikhar 2y ago
  (Founder here) Managed cloud offerings for streaming limit ordered throughput pretty low, e.g. Kinesis at 1 MiBps, Redpanda serverless at 1 MiBps, Confluent's even higher-end clusters at 10-20 MiBps IIRC. If you really need ordering, this can indeed be a limit. S2 lets you push 125 MiBps currently, and we may grow that. Another factor is how many ordered streams you can have. Typically a few thousand at most with those systems. We take the serverless spirit of S3 here, when did you have to worry about the number of objects in a bucket? We are also able to offer latency comparable to disk-based streaming like Confluent's Kora and Kinesis, with our Express storage class (under 50 milliseconds end-to-end latency for client in the same cloud region) - while being backed by S3 with regional durability! Not a disk in the system. We want people to be able to build safe distributed data systems on top of S2, so we also allow concurrency control mechanisms on the stream like fencing. Kafka or Kinesis won't let you do that. This is the approach AWS takes internally (https://brooker.co.za/blog/2024/04/25/memorydb.html https://brooker.co.za/blog/2024/04/25/memorydb.html), but they don't have that as a service. We want to democratize the pattern. ED: on throughtputs, to clarify, I am talking about ordered throughput, i.e. per Kafka partition or Kinesis shard. WarpStream also does well here because of their architectural approach to separate ordering, but at a latency cost.
  - alwa 2y ago
    Between your site copy and your early comments on this thread, it was this rundown that made the product click in my mind. I’m sure that in this early preview you’re trying to reach mainly devs with existing domain expertise, but the way that, in this comment, you laid out existing constraints and what possibilities might lie beyond them—it really helped me situate your S2 product in the constellation of cloud primitives. Just wanted to offer that feedback in the hope that the spirit of your comment here doesn’t get buried down-thread!
    - shikhar 2y ago
      thank you for the feedback!
  - agallego 2y ago

deleted 2y ago

[deleted]