4 ms·

There is a huge problem with this sadly, the startup time of the BEAM is too much! I was thinking about having a running VM process and injecting code directly,

There is a huge problem with this sadly, the startup time of the BEAM is too much! I was thinking about having a running VM process and injecting code directly, but I think this could be quite insecure. Love the idea though.

doawoo 6y ago
Totally understandable concern, in my quick experiments I haven't been bothered too much by the bootup time. But there's definitely better performance aware solutions for scripting
- andy_ppp 6y ago
  What is the startup time compared to bash?
  - sususu 6y ago
    I haven't test, but IIRC, Jose Valim in an interview commented that the BEAM startup time is about 300 to 500 milliseconds
- GlennS 6y ago
  Powershell suffers from this same problem, and I find it very annoying. Really recommend you speed this up if it's possible.
bostonvaulter2 6y ago
But would that be more insecure than running arbitrary scripts on your machine? I imagine you could build something akin to Ruby's spring for elixir, might be interesting.
elcritch 6y ago
You could spawn independent actors for each invocation. That provide a degree of separation and be fine for secuirty, if the beam process was run as your user pid. Handling multiple stdin/stdouts would take some work, but doable with standard erlang libraries.
toast0 6y ago
ok on my pentium G3470 running FreeBSD 12.2: time /bin/sh -c "echo hi" hi real 0m0.003s user 0m0.000s sys 0m0.003s time /usr/local/bin/bash -c "echo hi" hi real 0m0.006s user 0m0.000s sys 0m0.006s time /usr/local/lib/erlang23/bin/erl -eval 'io:format("hi~n"), exit(0)' Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [dtrace] hi {"init terminating in do_boot",0} init terminating in do_boot (0) Crash dump is being written to: erl_crash.dump...done real 0m0.304s user 0m0.117s sys 0m0.010s Ok, so 300 ms is clearly too much, but that's running a bunch of stuff. Turning a bunch of stuff off, I get time /usr/local/lib/erlang23/bin/erl -eval 'io:format("hi~n"), halt(0, [{flush, false}])' -noshell -start_epmd false -boot no_dot_erlang hi real 0m0.077s user 0m0.094s sys 0m0.008s (I tried tuning down the number of schedulers, but couldn't get the real time lower) This is still a lot worse than bash, but about 75 ms is sort of okish. Maybe a non-stock boot file could help. I usually plan for my BEAMs to run for months or years, so I don't worry too much about startup time :)
- andy_ppp 6y ago
  Nice that you’ve managed to speed it up quite a lot! Still it’s fairly slow, that’s why I was thinking about streaming stuff to a running BEAM from a Golang executable, say. Presumably then this problem goes away (compilation may still take a while). I guess if the #! just streams the file content over we are still protected by the execution bit, but what about malicious programs streaming their own content?
  - toast0 6y ago
    You could do something like that, but it adds a lot of complexity. I suspect if I can get the time down to 75 ms with a frw minutes of fiddling, someone who actually cares to run this could spend a couple hours or days on it and get it down to 30ms, which should be tollerable. Reduce as much runtime startup as possible (probably allocator tuning), and then reduce or postpone as much OTP startup as possible. Otherwise you have to deal with things like how do you kill one script without killing the others, version fun including what happens if you are running scripts and update the beam, how to keep scripts separated for reals, etc.

doawoo 6y ago

Totally understandable concern, in my quick experiments I haven't been bothered too much by the bootup time. But there's definitely better performance aware solutions for scripting

andy_ppp 6y ago
What is the startup time compared to bash?
- sususu 6y ago
  I haven't test, but IIRC, Jose Valim in an interview commented that the BEAM startup time is about 300 to 500 milliseconds
GlennS 6y ago
Powershell suffers from this same problem, and I find it very annoying. Really recommend you speed this up if it's possible.

bostonvaulter2 6y ago

But would that be more insecure than running arbitrary scripts on your machine? I imagine you could build something akin to Ruby's spring for elixir, might be interesting.

elcritch 6y ago

You could spawn independent actors for each invocation. That provide a degree of separation and be fine for secuirty, if the beam process was run as your user pid. Handling multiple stdin/stdouts would take some work, but doable with standard erlang libraries.

toast0 6y ago

ok on my pentium G3470 running FreeBSD 12.2: time /bin/sh -c "echo hi" hi real 0m0.003s user 0m0.000s sys 0m0.003s time /usr/local/bin/bash -c "echo hi" hi real 0m0.006s user 0m0.000s sys 0m0.006s time /usr/local/lib/erlang23/bin/erl -eval 'io:format("hi~n"), exit(0)' Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [dtrace] hi {"init terminating in do_boot",0} init terminating in do_boot (0) Crash dump is being written to: erl_crash.dump...done real 0m0.304s user 0m0.117s sys 0m0.010s Ok, so 300 ms is clearly too much, but that's running a bunch of stuff. Turning a bunch of stuff off, I get time /usr/local/lib/erlang23/bin/erl -eval 'io:format("hi~n"), halt(0, [{flush, false}])' -noshell -start_epmd false -boot no_dot_erlang hi real 0m0.077s user 0m0.094s sys 0m0.008s (I tried tuning down the number of schedulers, but couldn't get the real time lower) This is still a lot worse than bash, but about 75 ms is sort of okish. Maybe a non-stock boot file could help. I usually plan for my BEAMs to run for months or years, so I don't worry too much about startup time :)

andy_ppp 6y ago
Nice that you’ve managed to speed it up quite a lot! Still it’s fairly slow, that’s why I was thinking about streaming stuff to a running BEAM from a Golang executable, say. Presumably then this problem goes away (compilation may still take a while). I guess if the #! just streams the file content over we are still protected by the execution bit, but what about malicious programs streaming their own content?
- toast0 6y ago
  You could do something like that, but it adds a lot of complexity. I suspect if I can get the time down to 75 ms with a frw minutes of fiddling, someone who actually cares to run this could spend a couple hours or days on it and get it down to 30ms, which should be tollerable. Reduce as much runtime startup as possible (probably allocator tuning), and then reduce or postpone as much OTP startup as possible. Otherwise you have to deal with things like how do you kill one script without killing the others, version fun including what happens if you are running scripts and update the beam, how to keep scripts separated for reals, etc.