3 ms·

Blink and you missed the interesting bit, which is how they hooked them together. Looks like it's with this project: https://napi.rs/ https://napi.rs/ That pro

Blink and you missed the interesting bit, which is how they hooked them together. Looks like it's with this project: https://napi.rs/ https://napi.rs/ That project seems to be creating platform-specific DLL's, which I guess doesn't matter too much if it's wrapped up nicely in an npm package and you're deploying Docker files on Linux anyway? But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. Alternatively, writing very C-like code using a Uint8Array and not allocating in the loop might have fixed it too? But if you don't use JavaScript string libraries, it's not much like writing JavaScript anymore.

leononame 3y ago
Yeah, I think Uint8Array might've been a strong solution. But I've got the feeling this isn't the whole story. Unless the TSV is extremely wide, this alone feels a bit insignificant. I don't know the details so this is pure speculation, but using a transform stream on each row to split it up might've been enough to reduce memory pressure. I've done a bit of data manipulation in node and while imo js is not the ideal language for the job, it can be very fast. Strings are hard though, I'll admit that
eyelidlessness 3y ago
> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. This is true, but N-API will generally perform better (sometimes significantly, usually in exactly the same CPU-bound scenarios where WASM will perform better than V8). And NAPI-RS is quite good at paving the happy path for multi-target packages.
pimeys 3y ago
> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. If the database IO happens in Rust, the option with WebAssembly is fetch, meaning HTTP and the Atlas-only Data API. If the IO is done in JS, you pay some extra CPU time transferring UTF-16 strings over the border and copy them as UTF-8 in Rust.

leononame 3y ago

Yeah, I think Uint8Array might've been a strong solution. But I've got the feeling this isn't the whole story. Unless the TSV is extremely wide, this alone feels a bit insignificant. I don't know the details so this is pure speculation, but using a transform stream on each row to split it up might've been enough to reduce memory pressure. I've done a bit of data manipulation in node and while imo js is not the ideal language for the job, it can be very fast. Strings are hard though, I'll admit that

eyelidlessness 3y ago

> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. This is true, but N-API will generally perform better (sometimes significantly, usually in exactly the same CPU-bound scenarios where WASM will perform better than V8). And NAPI-RS is quite good at paving the happy path for multi-target packages.

pimeys 3y ago

> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. If the database IO happens in Rust, the option with WebAssembly is fetch, meaning HTTP and the Atlas-only Data API. If the IO is done in JS, you pay some extra CPU time transferring UTF-16 strings over the border and copy them as UTF-8 in Rust.