shikhar

6 ms·

61.
Fast IoT data pipeline for infrared monitoring(s2.dev)
4 points by shikhar 2y ago|0 comments
62.
by shikhar 2y ago
Indeed. We are trying to democratize that secret sauce. Since it is backed by object storage, the latencies are not what AWS enjoys with its internal Journal service, but we intend to get there with a NVMe-based tier later. In the meantime
63.
by shikhar 2y ago
This is why we didn't actually call it logs as a service, but streams :P I meant to refer to the log abstraction this post talks about, see links therein. Observability events are but one kind of data you may want as a stream of durabl
64.
by shikhar 2y ago
This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional ap
65.
One weird trick to durably replicate your KV store(s2.dev)
3 points by shikhar 2y ago|0 comments
66.
by shikhar 2y ago
I failed to mention that we do want to support single-tenant cells for customers that need isolation.
67.
by shikhar 2y ago
Fair :) Yes, we are pretty hyped about the possibilities with Iceberg, especially now with S3 Table buckets.
68.
by shikhar 2y ago
(S2 Founder) Congrats on the success with Estuary! You are not the first person to tell me there is no/tiny market for this. Clearly _you_ thought there was something to it, when you looked to HN for validation. We may do a lot more on
69.
by shikhar 2y ago
(Founder) Thanks for the deep questions! 1) Storage is priced on uncompressed data. We don't currently compress segments. 2) It only affects chunk storage. We do have a 'Native' chunk store in mind, the sketch involves introd
70.
by shikhar 2y ago
(Founder) Unfortunately, if you access over a public IP, it is internet egress. Even if the client is in the same AWS cloud region. PrivateLink is the only option.
71.
by shikhar 2y ago
(Founder) It's a number of dimensions. I get excited about the ordered throughput angle because I have personally cared about this in the past, and yeah a lot of folks may not need that :) Simple API, reasonable pricing, latency flexib
72.
by shikhar 2y ago
(Founder) Thanks for sharing your thoughts. We are early and figuring things out. I agree egress cost is going to be a big concern. We want to do the best we can for users as we unlock some scale. During preview, we are focused on getting f
73.
by shikhar 2y ago
(Founder) An S2 stream is a durable log and can be replayed! We do want to add compaction support. Event sourcing is a great use case for S2.
74.
by shikhar 2y ago
(Founder) Both S3 Express _One Zone_ appends and Azure's append blobs charge the regular PUT price for appends. It may work for you, but probably not if you want to do smaller writes. Blob stores will also not let you do tailing reads,
75.
by shikhar 2y ago
(Founder) Yes we should create a video, thanks for the feedback. In the meantime, checkout this quickstart which will have to streaming Star Wars with the S2 CLI and give you a pretty good sense of things https://s2.dev/docs
76.
by shikhar 2y ago
(Founder) We are not charging during preview. If anything, I wanted to be transparent about our planned pricing. Our mission is to make streams a cloud storage primitive, and I worked backwards from there in terms of our costs and expected
77.
by shikhar 2y ago
(Founder) I thought you were joking but coming back it could well be serious :) When we say stream, we really mean The Log that Jay Kreps has a famous blog about https://engineering.linkedin.com/distributed-systems/log-
78.
by shikhar 2y ago
(Founder) Yes, we will charge less for private connectivity. Pricing is transparent https://s2.dev/pricing - free during preview.
79.
by shikhar 2y ago
(Founder) We plan to be multi-cloud over time. Kinesis has pretty low ordered throughput limit (i.e. at the level of a stream shard) of 1 MBps, if you need higher. S2 will be cheaper and faster than Kinesis with the Express storage class. S
80.
by shikhar 2y ago
(Founder) That makes sense. We would eventually host the Kafka layer too - and will be able to avoid a hop by inlining our edge service logic in there.
81.
by shikhar 2y ago
(Founder) No, we want to be in the same cloud regions as customers.
82.
by shikhar 2y ago
(Founder) Thank you for the advice. I hope we can offer better when the deals come into play, but for now setting our planned internet egress price to $0.08/GiB.
83.
by shikhar 2y ago
(Founder) Nope! We have a FAQ for this ;)
84.
by shikhar 2y ago
(Founder) We are happy for S2 API to have alternate implementations, we are considering an in-memory emulator to open source ourselves. It is not a very complicated API. If you would prefer to stick with the Kafka API but benefit from featu
85.
by shikhar 2y ago
(Founder) Yes I understand, and this could definitely do with work. I struggle with it personally because it is so obvious to me. I don't even know where to start? How do you pitch use cases for object storage? Stream storage feels jus
86.
by shikhar 2y ago
(Founder) There are definitely some interesting possibilities. Pretty hyped about S3 Table (Iceberg) buckets. S2 stream to buffer small writes so you can flush decent size Parquet into the table, and avoid compaction costs.
87.
by shikhar 2y ago
Forgot to mention storage classes to tune your latency vs cost tradeoff. That you can even reconfigure - soon we will make that a live migration.
88.
by shikhar 2y ago
(Founder) Thank you for the kind comment! Yes we are not trying to confuse S2 with S3, we just think S3 is the best damn serverless experience out there, and we aspire to that greatness. We borrowed the structure of that name to reflect tha
89.
by shikhar 2y ago
(Founder) We will be using authenticated encryption with per-basin (our term for bucket) or per-stream keys, but we don't have this yet. This is noted on https://s2.dev/docs/security#encryption
90.
by shikhar 2y ago
(Founder) Besides simple API, - Unlimited streams. Current cloud systems limit to a few thousand. With dedicated clusters, few hundred K? If you want a stream per user, you are now dealing with multiple clusters. - Elastic throughput per st

More ›