4 ms·

It makes sense if you have multiple machines to actually distribute the load (files could be on a NFS).

• emj 7y ago
  You can send the files through the pubsub directly and just convert a stream of bytes. It is a beautiful pattern, but needs some tools to make it really easy to use (e.g. how to I metadata and bytestream to a pipe)
  • anderspitman 7y ago
    Sending the actual files would probably be really slow on the free server, due to bandwidth limiting. As for metadata, you might consider multipart/form-data, or just packing it into the query string (obviously depends on how much data you're talking about). Of course you'd probably need to write that in something other than bash. EDIT: It's pretty easy to send form-data with curl: https://ec.haxx.se/http-multipart.html https://ec.haxx.se/http-multipart.html
    • emj 7y ago
      Yes but it is the receiving that mess you up, if you want to use standard shell tools and just stream it. Everything can be solved by using other tools. # preserve filename for i in a b; do gzip -9 $i; done # does not preserve filenames while true; done curl $patchbay| mangle | gzip -9 | curl -d@- $patchbay; done