16 ms·

We built our data transformation engine with the Wasm runtime

34 points by sorenbs 5y ago

followtherhythm 5y ago
Hi I'm one of the authors of the blog post, happy to answer any questions here !
deleted 5y ago
[deleted]
- sorenbs 5y ago
  Hi Alex! Do you already have companies using this in production or maybe evaluating it? What language do you see most demand for?
  - followtherhythm 5y ago
    We see a lot of demand for onboard transformations, and JS is our first trial, since most devs are familiar with it. We plan to look into Rust and all of the Wasm supported languages, based on user demand
  - agallego 5y ago
    Yeah, we went from low throughput to multi-gigabyte/second workloads, for example consuming the NYSE entire firehose+downstream analytics.
    - agallego 5y ago
      I accidentally deleted the parent thread, when I meant to hit edit. but, my name is alex & I am the founder. happy to answer questions.
northstar702 5y ago
Are there other wasm runtimes to consider? Why v8?
- followtherhythm 5y ago
  Yup, as of now we are leveraging Nodejs for the technical preview. We also have another preview using embedded v8 for inline transforms, that is your transform runs in-process and apply() boils down to a function call. We are still evaluating other runtimes like wasmer as well, v8 was initially chosen for its maturity and community support.
pandemiclatte 5y ago
can materialized topics have custom properties? what if I want a longer retention in a materialized topic, perhaps I'm just counting clicks and I don't want the whole header data from that topic.
- followtherhythm 5y ago
  Currently they cannot, as they share all of the same properties as their respective source topics, but this idea is good! I'm adding it to our list of design enhancements.
  - pandemiclatte 5y ago
    That's awesome! I would love to try that option one day! Thanks!
abd12 5y ago
Nice post! Question for you -- what are the performance implications of "re-keying" my records during the transform? Should I try to keep the same key for my transformed record as from my original record so that they align with the same partitions, or are they likely going to get sent to different partitions across different brokers anyway?
- followtherhythm 5y ago
  Good question! You can't re-key your records but this done is by design. We want the materialized partitions to reside on the same brokers as their sources so there is no network overhead for performing the transform, and it occurs on all nodes. So by choosing this design we traded off more CPU for lower latency and less network bandwidth.
PeterCorless 5y ago
Great minds think alike! We're adding Wasm support to ScyllaDB, too. https://github.com/scylladb/scylla/pull/9108 https://github.com/scylladb/scylla/pull/9108 Folks can hear a talk on the implementation at Scylla Summit next week [free to register and yes, will be available free on demand after]. We'll also have a talk by Redpanda's Denis Rystsov. https://www.scylladb.com/summit/ https://www.scylladb.com/summit/
- agallego 5y ago
  I'm pretty excited to see the wasm chat too. very cool. I think that wasm is going to do to server side what javascript did for the web in the 90's
dang 5y ago
This is a good post but besides the non-organic upvotes, it looks like your friends/fans/teammates have seeded the thread with comments and questions. This is a super bad idea. HN readers can smell this kind of thing in the parts per million and will flame you for it. It's also against the rules—this is in both the guidelines and the FAQ: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html
- agallego 5y ago
  i got a notification in slack. there is less organization than you think :D. if we were, we'd have done this w/ the last 15 posts.
  - dang 5y ago
    Slack messages are how most of this works. It doesn't change the point.
    - agallego 5y ago
      dunno what to say. the answer doesn't change.
hammersmith 5y ago
Hiya! Excited to try this, The devil is of-course in the details :) What are the throughput implications for the rest of the cluster? IS there any way to profile the transforms usage as a percentage of the whole? How much of the API is exposed? can it create topics?
- followtherhythm 5y ago
  Hi, of course there are :) So the transformations are deployed in a separate NodeJs process per node. You can use familiar tools to take a look at its cpu + memory usage. As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.
markpapadakis 5y ago
We have been running hundreds of services that effectively consume from various topics, perform mostly simple transformations and produce back to other topics. Providing this kind of functionality makes a lot of sense. We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.

34 points by sorenbs 5y ago

followtherhythm 5y ago

Hi I'm one of the authors of the blog post, happy to answer any questions here !

deleted 5y ago

[deleted]

sorenbs 5y ago
Hi Alex! Do you already have companies using this in production or maybe evaluating it? What language do you see most demand for?
- followtherhythm 5y ago
  We see a lot of demand for onboard transformations, and JS is our first trial, since most devs are familiar with it. We plan to look into Rust and all of the Wasm supported languages, based on user demand
- agallego 5y ago
  Yeah, we went from low throughput to multi-gigabyte/second workloads, for example consuming the NYSE entire firehose+downstream analytics.
  - agallego 5y ago
    I accidentally deleted the parent thread, when I meant to hit edit. but, my name is alex & I am the founder. happy to answer questions.

northstar702 5y ago

Are there other wasm runtimes to consider? Why v8?

followtherhythm 5y ago
Yup, as of now we are leveraging Nodejs for the technical preview. We also have another preview using embedded v8 for inline transforms, that is your transform runs in-process and apply() boils down to a function call. We are still evaluating other runtimes like wasmer as well, v8 was initially chosen for its maturity and community support.

pandemiclatte 5y ago

can materialized topics have custom properties? what if I want a longer retention in a materialized topic, perhaps I'm just counting clicks and I don't want the whole header data from that topic.

followtherhythm 5y ago
Currently they cannot, as they share all of the same properties as their respective source topics, but this idea is good! I'm adding it to our list of design enhancements.
- pandemiclatte 5y ago
  That's awesome! I would love to try that option one day! Thanks!

abd12 5y ago

Nice post! Question for you -- what are the performance implications of "re-keying" my records during the transform? Should I try to keep the same key for my transformed record as from my original record so that they align with the same partitions, or are they likely going to get sent to different partitions across different brokers anyway?

followtherhythm 5y ago
Good question! You can't re-key your records but this done is by design. We want the materialized partitions to reside on the same brokers as their sources so there is no network overhead for performing the transform, and it occurs on all nodes. So by choosing this design we traded off more CPU for lower latency and less network bandwidth.

PeterCorless 5y ago

Great minds think alike! We're adding Wasm support to ScyllaDB, too. https://github.com/scylladb/scylla/pull/9108 https://github.com/scylladb/scylla/pull/9108 Folks can hear a talk on the implementation at Scylla Summit next week [free to register and yes, will be available free on demand after]. We'll also have a talk by Redpanda's Denis Rystsov. https://www.scylladb.com/summit/ https://www.scylladb.com/summit/

agallego 5y ago
I'm pretty excited to see the wasm chat too. very cool. I think that wasm is going to do to server side what javascript did for the web in the 90's

dang 5y ago

This is a good post but besides the non-organic upvotes, it looks like your friends/fans/teammates have seeded the thread with comments and questions. This is a super bad idea. HN readers can smell this kind of thing in the parts per million and will flame you for it. It's also against the rules—this is in both the guidelines and the FAQ: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html

agallego 5y ago
i got a notification in slack. there is less organization than you think :D. if we were, we'd have done this w/ the last 15 posts.
- dang 5y ago
  Slack messages are how most of this works. It doesn't change the point.
  - agallego 5y ago
    dunno what to say. the answer doesn't change.

hammersmith 5y ago

Hiya! Excited to try this, The devil is of-course in the details :) What are the throughput implications for the rest of the cluster? IS there any way to profile the transforms usage as a percentage of the whole? How much of the API is exposed? can it create topics?

followtherhythm 5y ago
Hi, of course there are :) So the transformations are deployed in a separate NodeJs process per node. You can use familiar tools to take a look at its cpu + memory usage. As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.

markpapadakis 5y ago

We have been running hundreds of services that effectively consume from various topics, perform mostly simple transformations and produce back to other topics. Providing this kind of functionality makes a lot of sense. We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.