4 ms·

Show HN: SwimOS Rust – A framework for real-time streaming data applications

71 points by swimos 2y ago

swimos 2y ago
The Swim Rust SDK contains software framework for building stateful applications that can be interacted with via multiplexed streaming APIs. It is built on top of the Tokio asynchronous runtime and a Tokio runtime is required for any Swim application. Each application consists of some number of stateful agents, each of which runs as a separate Tokio task and can be individually addressed by a URI. An agent may have both public and private state which can either be held solely in memory or, optionally, in persistent storage. The public state of the agent consists of a number of lanes, analogous to a field in a record. There are multiple kinds of lanes that, for example, lanes containing single values and those containing a map of key-value pairs. The state of any lane can be observed by establishing a link to it (either from another agent instance or a dedicated client). A established link will push all updates to the state of that lane to the subscriber and will also allow the subscriber to request changes to the state (for lane kinds that support this). Links operate over a web-socket connection and are multiplexed, meaning that links to multiple lanes on the same host can share a single web-socket connection. There's a number of example applications available here: https://github.com/swimos/swim-rust/tree/main/example_apps https://github.com/swimos/swim-rust/tree/main/example_apps If you're interested in getting started with it a developer guide is available here https://www.swimos.org/server/rust/developer-guide/ https://www.swimos.org/server/rust/developer-guide/ as well as reference documentation here https://www.swimos.org/server/rust/ https://www.swimos.org/server/rust/ Previously on HN: - SwimOS: Distributed platform for building stateful, real-time streaming apps https://news.ycombinator.com/item?id=22920764 https://news.ycombinator.com/item?id=22920764 - Real-time traffic light status in Palo Alto powered by swim.ai https://news.ycombinator.com/item?id=19234286 https://news.ycombinator.com/item?id=19234286
- 01HNNWZ0MV43FF 2y ago
  Doesn't run on bare metal though?
  - swimos 2y ago
    It doesn't unfortunately but this is something that we have discussed internally about targeting. We would be interested to hear about your use-case for it!
    - 01HNNWZ0MV43FF 2y ago
      I'm just picky about maintaining purity of the term "OS". If "OS" can include WebKit and Lua and anything with a file API and an interpreter then "OS" just means "an abstraction" and I don't need two words for the same thing Like, Chrome is not an OS, no matter how big a bonus someone at Google got for that clever bit of marketing.
- duped 2y ago
  How is it real time if you are using tokio? What latency and deadline guarantees are you able to provide?
  - filleokus 2y ago
    "real time" as in figma multiplayer not as in control system for a car (I'm assuming, never used SwimOS)
jitl 2y ago
The README tells me mechanically what swim does but even with the example it’s quite abstract and I don’t know why I would choose to build with swim, what it’s similar to, or what it’s especially good at. I think with frameworks you should make the value proposition very clear otherwise it’s a tall ask for someone to adopt a way of building. Versus a library which is easy to drop into existing software to solve a specific, scoped task. I guess a bunch of what I’m looking for is discussed in the main repo of your project, but even there it’s a bit heavy on the “what” and too light on the “why” for me. This is a deep framework. I work on a realtime reactive app and am currently building stateful services on Kafka + Typescript + SQLite, so I think I’m squarely in your target customer demographic. It’s not until I found the website that the value proposition is clearly described. I encourage you to copy-paste the headline of your website to all the components of your framework, so it’s easier for a casually interested person on HN to grok what it’s all about. EDIT: there’s like a 1% chance I actually do convert, since I realize I could run my existing Typescript business logic in a Rust agent via Deno… perhaps the scattered documentation hunt on a Sunday is good marketing strategy after all.
- the__alchemist 2y ago
  I had a similar reaction, and assumed I'm not the target audience. (Including skimming the examples folder and not being sure where I would use one of them) I have a hard time understanding the purpose of these abstract libraries; I assume people who have had the problems they solve recognize their utility. As a reference point for rust libs I've worked with recently: I see things like EGUI: This clearly is a library that allows you to add a GUI to programs. Or Bio: This lets you read and write FASTA format DNA etc sequences, find matches in sequences etc. Bincode: Provides a "derive" that allows you to easily serialize data to and from binary formats. Or the various embedded infrastructure libs that provide high-level APIs for performing hardware options. (I/O, ADCs, send a packet over USB or a radio etc) By contrast to these and every other lib I've worked with, Swim is abstract. I start wondering if I'm too dumb or otherwise incapable of reasoning abstracting to understand the Async part of Rust's ecosystem.
- swimos 2y ago
  Thank you for the feedback! We could definitely improve the documentation and we will expand on the "why?"; we will get an updated readme up that expands on this in more detail. Could you please tell me which part of the website clearly explained it to you such that you understood the value proposition?
worldmerge 2y ago
This is super cool!! Back many years ago now I had a project that dealt with realtime data + geofences so I used tile38. I'm super curious to check this out!
- swimos 2y ago
  We hope that it goes well and we would love to hear your feedback!
alexmorley 2y ago
Maybe stating the obvious but took me an embarrassingly long time to clock this: SwimOS is an existing streaming framework (Java-based) targeted at full stack / app development. This announcement is a new Rust SDK for the above - not a new framework.
- swimos 2y ago
  That is correct. We're working towards replacing our Java implementation with the Rust version and then provide bindings to Java
qwerty456127 2y ago
How does this compare to Erlang?
- thoughtpoet 2y ago
  I've worked with Erlang/OTP in the past, although it's been a while since I last used it. I have fond memories of working with it, and I appreciate its unique strengths. One notable similarity between Erlang and SwimOS is the actor model, but their use cases are quite different. Erlang excels at fault tolerance, robust supervision, and lightweight isolated processes, with a focus on message passing (which, if I recall correctly, involves significant data copying underneath). In contrast, SwimOS is designed for high-performance streaming, boasting a heavily optimized WebSocket implementation that leverages multicast, multiplexing, and differential dataflow (delta encoding) to efficiently handle massive data volumes. SwimOS utilizes zero-cost abstractions and lock-free mechanisms to achieve its goals. Its primary use cases are data observability and real-time collaboration applications, with an emphasis on in-memory processing and recent data. This processing window is similar to a retention window in distributed log technology like Kafka, where the focus is on handling large volumes of data in real-time to enable timely decision-making, without necessarily storing all the data long-term. While both technologies share some similarities, their design and use cases make them suited for very different problem domains.
VyseofArcadia 2y ago
Why do so many products that aren't an operating system calling themselves WhateverOS?
Sunscratch 2y ago
Looks very similar to Akka, where agent is actor and Lifecycle is Behaviors. An interesting project for a complex task - streaming stateful processing.
shae 2y ago
What about adding differential data flow?

71 points by swimos 2y ago

The Swim Rust SDK contains software framework for building stateful applications that can be interacted with via multiplexed streaming APIs. It is built on top of the Tokio asynchronous runtime and a Tokio runtime is required for any Swim application. Each application consists of some number of stateful agents, each of which runs as a separate Tokio task and can be individually addressed by a URI. An agent may have both public and private state which can either be held solely in memory or, optionally, in persistent storage. The public state of the agent consists of a number of lanes, analogous to a field in a record. There are multiple kinds of lanes that, for example, lanes containing single values and those containing a map of key-value pairs. The state of any lane can be observed by establishing a link to it (either from another agent instance or a dedicated client). A established link will push all updates to the state of that lane to the subscriber and will also allow the subscriber to request changes to the state (for lane kinds that support this). Links operate over a web-socket connection and are multiplexed, meaning that links to multiple lanes on the same host can share a single web-socket connection. There's a number of example applications available here: https://github.com/swimos/swim-rust/tree/main/example_apps https://github.com/swimos/swim-rust/tree/main/example_apps If you're interested in getting started with it a developer guide is available here https://www.swimos.org/server/rust/developer-guide/ https://www.swimos.org/server/rust/developer-guide/ as well as reference documentation here https://www.swimos.org/server/rust/ https://www.swimos.org/server/rust/ Previously on HN: - SwimOS: Distributed platform for building stateful, real-time streaming apps https://news.ycombinator.com/item?id=22920764 https://news.ycombinator.com/item?id=22920764 - Real-time traffic light status in Palo Alto powered by swim.ai https://news.ycombinator.com/item?id=19234286 https://news.ycombinator.com/item?id=19234286

01HNNWZ0MV43FF 2y ago
Doesn't run on bare metal though?
- swimos 2y ago
  It doesn't unfortunately but this is something that we have discussed internally about targeting. We would be interested to hear about your use-case for it!
  - 01HNNWZ0MV43FF 2y ago
    I'm just picky about maintaining purity of the term "OS". If "OS" can include WebKit and Lua and anything with a file API and an interpreter then "OS" just means "an abstraction" and I don't need two words for the same thing Like, Chrome is not an OS, no matter how big a bonus someone at Google got for that clever bit of marketing.
duped 2y ago
How is it real time if you are using tokio? What latency and deadline guarantees are you able to provide?
- filleokus 2y ago
  "real time" as in figma multiplayer not as in control system for a car (I'm assuming, never used SwimOS)

jitl 2y ago

The README tells me mechanically what swim does but even with the example it’s quite abstract and I don’t know why I would choose to build with swim, what it’s similar to, or what it’s especially good at. I think with frameworks you should make the value proposition very clear otherwise it’s a tall ask for someone to adopt a way of building. Versus a library which is easy to drop into existing software to solve a specific, scoped task. I guess a bunch of what I’m looking for is discussed in the main repo of your project, but even there it’s a bit heavy on the “what” and too light on the “why” for me. This is a deep framework. I work on a realtime reactive app and am currently building stateful services on Kafka + Typescript + SQLite, so I think I’m squarely in your target customer demographic. It’s not until I found the website that the value proposition is clearly described. I encourage you to copy-paste the headline of your website to all the components of your framework, so it’s easier for a casually interested person on HN to grok what it’s all about. EDIT: there’s like a 1% chance I actually do convert, since I realize I could run my existing Typescript business logic in a Rust agent via Deno… perhaps the scattered documentation hunt on a Sunday is good marketing strategy after all.

the__alchemist 2y ago
I had a similar reaction, and assumed I'm not the target audience. (Including skimming the examples folder and not being sure where I would use one of them) I have a hard time understanding the purpose of these abstract libraries; I assume people who have had the problems they solve recognize their utility. As a reference point for rust libs I've worked with recently: I see things like EGUI: This clearly is a library that allows you to add a GUI to programs. Or Bio: This lets you read and write FASTA format DNA etc sequences, find matches in sequences etc. Bincode: Provides a "derive" that allows you to easily serialize data to and from binary formats. Or the various embedded infrastructure libs that provide high-level APIs for performing hardware options. (I/O, ADCs, send a packet over USB or a radio etc) By contrast to these and every other lib I've worked with, Swim is abstract. I start wondering if I'm too dumb or otherwise incapable of reasoning abstracting to understand the Async part of Rust's ecosystem.
swimos 2y ago
Thank you for the feedback! We could definitely improve the documentation and we will expand on the "why?"; we will get an updated readme up that expands on this in more detail. Could you please tell me which part of the website clearly explained it to you such that you understood the value proposition?

worldmerge 2y ago

This is super cool!! Back many years ago now I had a project that dealt with realtime data + geofences so I used tile38. I'm super curious to check this out!

swimos 2y ago
We hope that it goes well and we would love to hear your feedback!

alexmorley 2y ago

Maybe stating the obvious but took me an embarrassingly long time to clock this: SwimOS is an existing streaming framework (Java-based) targeted at full stack / app development. This announcement is a new Rust SDK for the above - not a new framework.

swimos 2y ago
That is correct. We're working towards replacing our Java implementation with the Rust version and then provide bindings to Java

qwerty456127 2y ago

How does this compare to Erlang?

thoughtpoet 2y ago
I've worked with Erlang/OTP in the past, although it's been a while since I last used it. I have fond memories of working with it, and I appreciate its unique strengths. One notable similarity between Erlang and SwimOS is the actor model, but their use cases are quite different. Erlang excels at fault tolerance, robust supervision, and lightweight isolated processes, with a focus on message passing (which, if I recall correctly, involves significant data copying underneath). In contrast, SwimOS is designed for high-performance streaming, boasting a heavily optimized WebSocket implementation that leverages multicast, multiplexing, and differential dataflow (delta encoding) to efficiently handle massive data volumes. SwimOS utilizes zero-cost abstractions and lock-free mechanisms to achieve its goals. Its primary use cases are data observability and real-time collaboration applications, with an emphasis on in-memory processing and recent data. This processing window is similar to a retention window in distributed log technology like Kafka, where the focus is on handling large volumes of data in real-time to enable timely decision-making, without necessarily storing all the data long-term. While both technologies share some similarities, their design and use cases make them suited for very different problem domains.

VyseofArcadia 2y ago

Why do so many products that aren't an operating system calling themselves WhateverOS?

Sunscratch 2y ago

Looks very similar to Akka, where agent is actor and Lifecycle is Behaviors. An interesting project for a complex task - streaming stateful processing.

shae 2y ago

What about adding differential data flow?