11 ms·

Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together,

Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. Moreover, even in some server-side applications it's super neat to have the luxury to spawn a new process to service certain requests, not having to worry about memory leaks. It's a perfect "API" which allows multiple languages to interact together. It always bothered me when people dismiss the start-up time by adding "just" in front of it. We're not talking about web frameworks, these are _general purpose_ programming languages, and horrendous start-up time automatically disqualifies them from being general purpose and places them into a niche category, in my humble opinion.

8ytecoder 7y ago
Not necessarily. It's entirely possible to keep a daemon running with the loaded runtime to speed up start time. I'm not arguing if and whether these languages are suitable for scripting rather startup time alone need not be a disqualifying factor. If you evaluate holistically and arrive at a language you'll always have one thing or the other that is not as what you want.
- skissane 7y ago
  A few years ago, as a hobby project, I designed a programming language and wrote an interpreter for it in Java. (I've never released it or shared it with anyone; I was never quite happy with it and eventually moved on to other things.) As you can imagine, an interpreted language with the interpreter written in Java is going to be rather slow to start. So, I implemented exactly the solution you describe here. I made my interpreter run as a daemon and listen for requests on a Unix domain socket (with a custom binary protocol). I then wrote a client in C, which opened that Unix domain socket and sent it a script to run. The C program also redirected stdin/stdout/stderr to/from the interpreter through the Unix domain socket, if connected to a tty it could switch to/from raw mode, it notified the server if certain signals occurred, etc. Using the interpreter this way, startup time was a lot faster. This approach can be applied to any language which supports runtime evaluation of code.
  - ghusbands 7y ago
    A long-running background interpreter carries significant security risks, though. A process that lacks most linux capabilities could use it to regain access to things it shouldn't have access to. A process could interpose itself and get access to data and code it shouldn't. If there are multiple users involved, one could bypass ACLs and similar. Essentially, you're removing a significant proportion of the security guarantees the OS provides around isolated processes.
    - skissane 7y ago
      Some of the concerns you raise can be addressed. For example, make the daemon and socket per-user and give the socket 600 permissions. Also, the SCM_CREDENTIALS message (on Linux) or LOCAL_PEERCRED/getpeereid (macOS/*BSD) can be used to validate the caller has expected UID. The issue with missing-normal-capability processes is more difficult to address. One possibility, at least on Linux, would be to get the pid from SCM_CREDENTIALS message, and then read /proc/$PID/status to check capability bits. It could default to denying access to processes with less capabilities than it itself has. (SCM_SECURITY can be used to pass the SELinux security label from client to server, which could also be used as a security measure; maybe the server could refuse access to processes running with a different label, or have a whitelist of allowed labels; if SELinux is being used to sandbox a process, that would prevent it from accessing the background interpreter unless that was explicitly allowed by whitelisting.)
StreamBright 7y ago
Totally, this is why have the right tool for the job approach. I would not start up a Spark cluster for processing 100MB of data I also would not start to write a CLI tool in Clojure (even though it is one of my favorite languages).
- peatmoss 7y ago
  I would write a CLI in Clojure if it were a reasonable option. Something like Gambit or Chicken but for Clojure would be a dream. I guess there’s Ferret, so maybe I should give that a try.
  - smnplk 7y ago
    Also checkout Joker language. It's an interpreter that implements a subset of Clojure.
- owl57 7y ago
  You got me thinking about that word, CLI. I know that usually means a Unix shell, but when I just run my bespoke file conversion tool from CIDER REPL, that interface is also a command line one, what else is it?
  - pjmlp 7y ago
    What a CLI wants to be when it grows up. Check the REPL experience from Lisp Machines and Xerox PARC Lisp/Mesa/Smalltalk workstations. Think of Swift Playgrounds or Jupiter notebooks, done in late 70's hardware.
ken 7y ago
> it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter Or that different languages require different approaches, and you can't translate 1:1 between technologies. My Mac starts up 27x slower than my C=64, but that doesn't make it useless for any task that requires turning it on. > We're not talking about web frameworks, these are _general purpose_ programming languages Are we? I've never written a line of Julia in my life but the impression that I get is that this isn't intended to be a general-purpose language: > Of course, one might argue that Julia is not intended to be a general-purpose programming language, but a language for numerical computing. > As has been pointed out elsewhere “Base APIs outside of the niche Julia targets often don’t make sense” and the general-purpose APIs are somewhat limited. The Julia webpage has 6 tabs listing features, and 5 of them are about numerics. The 6th says "General Purpose", but it's mostly about FFI.
- ddragon 7y ago
  Julia is intended to be (and already is) a general purpose programming language. What it does not try to be is a language that tries to be good for every purpose. Python for example is a general purpose programming language but you wouldn't use it to write kernel modules, and likewise you wouldn't use C to write your simple website (even though you obviously can). A language that tries to be everything to everyone will either be too bloated and complex, or lack the included batteries for pretty much anything. Even if we don't use technicalities, Julia is a very powerful language that allows people to extend it for purposes it was not built for. If it didn't support JSON, you could write a JSON type as integrated and fast as what the stdlib offers, it's metaprogramming/multiple dispatch paradigm allows for easily creating frameworks for many purposes and while numerical processing gets special attention the language will still outperform in terms of speed most dynamic languages in any domain. And I do think short scripts are within Julia's main targets and are merely victims of the fact that the language is still too young and the battles the devs chose to fight within that limited time (and either AoT options or an interpreter that runs code while it's compiling could solve it really well for example, but both would require a lot of time and work which could be used for other features).
  - 7y ago
phonebucket 7y ago
> like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. If you want to build CLIs without the JIT overhead, you can compile your Julia code: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl
- peatmoss 7y ago
  How big are the resulting binaries these days? It’s been some time since I tried this, but I decided that the size of the compiled code didn’t fit my use case (a serverless function IIRC)
  - myrryr 7y ago
    Horrendous is the answer :) It is still crazy big, because it still has to include the LLVM compiler in it.
- norcalli 7y ago
  It doesn't work very well. I got SEGFAULTs every time I tried; filed a bug report; no resolution. So, while it does exist, it's not a high priority for the Julia team, so I wouldn't rely on it.
  - jruth 7y ago
    I had the exact same experience when I tried over a year ago. The package was eventually closed IIRC, but a recent news letter claimed to have fixed it.
  - xiaodai 7y ago
    It's getting better. See how I managed to compile a bunch of data science packages https://discourse.julialang.org/t/201908-compiling-a-data-science-image-for-julia-on-windows-to-combat-compilation-latency/27640 https://discourse.julialang.org/t/201908-compiling-a-data-sc...
smallnamespace 7y ago
> it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter This is not true for Julia, you can simply precompile the code: https://medium.com/@sdanisch/compiling-julia-binaries-ddd6d4e0caf4 https://medium.com/@sdanisch/compiling-julia-binaries-ddd6d4... Julia's default model is that script functions are only JIT compiled when run (just like the JVM, etc.), but you can just as easily force the compile and stick it in an image.
- dnautics 7y ago
  > you can just as easily force the compiler Have you actually tried the instructions in there? I did, to get a better time for julia in the n-body programming languages shootout https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... (I am the current leader among julia implementations). Suffice it to say, it did not work. I mean i'm not a brilliant programmer or anything, but I would hardly say it's "just as easily".
  - bocklund 7y ago
    I am a Julia noob and managed to use PackageCompiler.jl for a package I wrote and it just worked out of the box. I think the only non-stdlib package I used was StaticArrays, but I was able to use the happy path described in the PackageCompiler docs and got it working in an hour or so max. This was probably in March or April, so relatively recent.
    - dnautics 7y ago
      As you might be able to see I ran it on 25 Jun 2019 I didn't have any non-stdlib packages at all!! I'll give it another shot.
cortesoft 7y ago
Depends on how long the piped commands are going to run for... if they run for minutes doing stuff, who cares about the extra startup time?
paultopia 7y ago
Eh? Swift surely doesn't fall in this category, as it's meant to be used compiled, not from a REPL, and doesn't have a massive heavy runtime like Clojure. I refuse to believe that a Swift CLI compiled to native code as intended has a startup time problem.
deleted 7y ago
[deleted]
slavapestov 7y ago
> Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. The REPL is a tad slow but compiled Swift binaries have virtually no startup time overhead.
dominotw 7y ago
>Clojure are just as useless as Julia for numerous use cases where start-up times matter, Clojure can be targeted for graal afaik. Not sure how well it works though.
- FPGAhacker 7y ago
  I’ve used it a few times in the past experimentally. It definitely improved startup time. It was hard to set up though, so I generally didn't find it useful. I’ve become disenchanted with clojure over the last couple years, so I haven’t tried anything recently.
  - dominotw 7y ago
    > I’ve become disenchanted with clojure over the last couple years I've been trying to get into it last 6 months :p. Now I am Wondering why you say this.
wallstquant 7y ago
try the following `date 1>&2 | sleep 1 | date 1>&2 | sleep 1 | date 1>&2` you'll actually see the three date commands output immediately so the three julia processes would actually launch in parallel if they were in a pipe

8ytecoder 7y ago

Not necessarily. It's entirely possible to keep a daemon running with the loaded runtime to speed up start time. I'm not arguing if and whether these languages are suitable for scripting rather startup time alone need not be a disqualifying factor. If you evaluate holistically and arrive at a language you'll always have one thing or the other that is not as what you want.

skissane 7y ago
A few years ago, as a hobby project, I designed a programming language and wrote an interpreter for it in Java. (I've never released it or shared it with anyone; I was never quite happy with it and eventually moved on to other things.) As you can imagine, an interpreted language with the interpreter written in Java is going to be rather slow to start. So, I implemented exactly the solution you describe here. I made my interpreter run as a daemon and listen for requests on a Unix domain socket (with a custom binary protocol). I then wrote a client in C, which opened that Unix domain socket and sent it a script to run. The C program also redirected stdin/stdout/stderr to/from the interpreter through the Unix domain socket, if connected to a tty it could switch to/from raw mode, it notified the server if certain signals occurred, etc. Using the interpreter this way, startup time was a lot faster. This approach can be applied to any language which supports runtime evaluation of code.
- ghusbands 7y ago
  A long-running background interpreter carries significant security risks, though. A process that lacks most linux capabilities could use it to regain access to things it shouldn't have access to. A process could interpose itself and get access to data and code it shouldn't. If there are multiple users involved, one could bypass ACLs and similar. Essentially, you're removing a significant proportion of the security guarantees the OS provides around isolated processes.
  - skissane 7y ago
    Some of the concerns you raise can be addressed. For example, make the daemon and socket per-user and give the socket 600 permissions. Also, the SCM_CREDENTIALS message (on Linux) or LOCAL_PEERCRED/getpeereid (macOS/*BSD) can be used to validate the caller has expected UID. The issue with missing-normal-capability processes is more difficult to address. One possibility, at least on Linux, would be to get the pid from SCM_CREDENTIALS message, and then read /proc/$PID/status to check capability bits. It could default to denying access to processes with less capabilities than it itself has. (SCM_SECURITY can be used to pass the SELinux security label from client to server, which could also be used as a security measure; maybe the server could refuse access to processes running with a different label, or have a whitelist of allowed labels; if SELinux is being used to sandbox a process, that would prevent it from accessing the background interpreter unless that was explicitly allowed by whitelisting.)

StreamBright 7y ago

Totally, this is why have the right tool for the job approach. I would not start up a Spark cluster for processing 100MB of data I also would not start to write a CLI tool in Clojure (even though it is one of my favorite languages).

peatmoss 7y ago
I would write a CLI in Clojure if it were a reasonable option. Something like Gambit or Chicken but for Clojure would be a dream. I guess there’s Ferret, so maybe I should give that a try.
- smnplk 7y ago
  Also checkout Joker language. It's an interpreter that implements a subset of Clojure.
owl57 7y ago
You got me thinking about that word, CLI. I know that usually means a Unix shell, but when I just run my bespoke file conversion tool from CIDER REPL, that interface is also a command line one, what else is it?
- pjmlp 7y ago
  What a CLI wants to be when it grows up. Check the REPL experience from Lisp Machines and Xerox PARC Lisp/Mesa/Smalltalk workstations. Think of Swift Playgrounds or Jupiter notebooks, done in late 70's hardware.

ken 7y ago

> it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter Or that different languages require different approaches, and you can't translate 1:1 between technologies. My Mac starts up 27x slower than my C=64, but that doesn't make it useless for any task that requires turning it on. > We're not talking about web frameworks, these are _general purpose_ programming languages Are we? I've never written a line of Julia in my life but the impression that I get is that this isn't intended to be a general-purpose language: > Of course, one might argue that Julia is not intended to be a general-purpose programming language, but a language for numerical computing. > As has been pointed out elsewhere “Base APIs outside of the niche Julia targets often don’t make sense” and the general-purpose APIs are somewhat limited. The Julia webpage has 6 tabs listing features, and 5 of them are about numerics. The 6th says "General Purpose", but it's mostly about FFI.

ddragon 7y ago
Julia is intended to be (and already is) a general purpose programming language. What it does not try to be is a language that tries to be good for every purpose. Python for example is a general purpose programming language but you wouldn't use it to write kernel modules, and likewise you wouldn't use C to write your simple website (even though you obviously can). A language that tries to be everything to everyone will either be too bloated and complex, or lack the included batteries for pretty much anything. Even if we don't use technicalities, Julia is a very powerful language that allows people to extend it for purposes it was not built for. If it didn't support JSON, you could write a JSON type as integrated and fast as what the stdlib offers, it's metaprogramming/multiple dispatch paradigm allows for easily creating frameworks for many purposes and while numerical processing gets special attention the language will still outperform in terms of speed most dynamic languages in any domain. And I do think short scripts are within Julia's main targets and are merely victims of the fact that the language is still too young and the battles the devs chose to fight within that limited time (and either AoT options or an interpreter that runs code while it's compiling could solve it really well for example, but both would require a lot of time and work which could be used for other features).
- 7y ago

phonebucket 7y ago

> like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. If you want to build CLIs without the JIT overhead, you can compile your Julia code: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl

peatmoss 7y ago
How big are the resulting binaries these days? It’s been some time since I tried this, but I decided that the size of the compiled code didn’t fit my use case (a serverless function IIRC)
- myrryr 7y ago
  Horrendous is the answer :) It is still crazy big, because it still has to include the LLVM compiler in it.
norcalli 7y ago
It doesn't work very well. I got SEGFAULTs every time I tried; filed a bug report; no resolution. So, while it does exist, it's not a high priority for the Julia team, so I wouldn't rely on it.
- jruth 7y ago
  I had the exact same experience when I tried over a year ago. The package was eventually closed IIRC, but a recent news letter claimed to have fixed it.
- xiaodai 7y ago
  It's getting better. See how I managed to compile a bunch of data science packages https://discourse.julialang.org/t/201908-compiling-a-data-science-image-for-julia-on-windows-to-combat-compilation-latency/27640 https://discourse.julialang.org/t/201908-compiling-a-data-sc...

smallnamespace 7y ago

> it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter This is not true for Julia, you can simply precompile the code: https://medium.com/@sdanisch/compiling-julia-binaries-ddd6d4e0caf4 https://medium.com/@sdanisch/compiling-julia-binaries-ddd6d4... Julia's default model is that script functions are only JIT compiled when run (just like the JVM, etc.), but you can just as easily force the compile and stick it in an image.

dnautics 7y ago
> you can just as easily force the compiler Have you actually tried the instructions in there? I did, to get a better time for julia in the n-body programming languages shootout https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... (I am the current leader among julia implementations). Suffice it to say, it did not work. I mean i'm not a brilliant programmer or anything, but I would hardly say it's "just as easily".
- bocklund 7y ago
  I am a Julia noob and managed to use PackageCompiler.jl for a package I wrote and it just worked out of the box. I think the only non-stdlib package I used was StaticArrays, but I was able to use the happy path described in the PackageCompiler docs and got it working in an hour or so max. This was probably in March or April, so relatively recent.
  - dnautics 7y ago
    As you might be able to see I ran it on 25 Jun 2019 I didn't have any non-stdlib packages at all!! I'll give it another shot.

cortesoft 7y ago

Depends on how long the piped commands are going to run for... if they run for minutes doing stuff, who cares about the extra startup time?

paultopia 7y ago

Eh? Swift surely doesn't fall in this category, as it's meant to be used compiled, not from a REPL, and doesn't have a massive heavy runtime like Clojure. I refuse to believe that a Swift CLI compiled to native code as intended has a startup time problem.

deleted 7y ago

[deleted]

slavapestov 7y ago

> Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. The REPL is a tad slow but compiled Swift binaries have virtually no startup time overhead.

dominotw 7y ago

>Clojure are just as useless as Julia for numerous use cases where start-up times matter, Clojure can be targeted for graal afaik. Not sure how well it works though.

FPGAhacker 7y ago
I’ve used it a few times in the past experimentally. It definitely improved startup time. It was hard to set up though, so I generally didn't find it useful. I’ve become disenchanted with clojure over the last couple years, so I haven’t tried anything recently.
- dominotw 7y ago
  > I’ve become disenchanted with clojure over the last couple years I've been trying to get into it last 6 months :p. Now I am Wondering why you say this.

wallstquant 7y ago

try the following `date 1>&2 | sleep 1 | date 1>&2 | sleep 1 | date 1>&2` you'll actually see the three date commands output immediately so the three julia processes would actually launch in parallel if they were in a pipe