13 ms·

Nim – Natively compiled language with hot code-reloading at runtime [video]

186 points by Clyybber 7y ago

dunefox 7y ago
Afaik Common Lisp would be the first, wouldn't it?
- vthriller 7y ago
  The answer to that would depend on the CL implementation, since not all of them can do AOT compilation.
  - coliveira 7y ago
    But the same can be said about Nim, this ability depends on the implementation.
  - jlarocco 7y ago
    Not really... It's in the CL standard, and compliant implementations support it. SBCL, LispWorks, Allegro, CCL, and others all do AOT native compilation, from the REPL and via eval (which should be avoided, in any case). ABCL does AOT compilation to JVM bytecode, and Clisp to its own bytecode. It's a cool feature for Nim to have, but claiming to be the first makes them look bad.
    - lispm 7y ago
      Several Lisp implementations even do similar things which were shown in the video: compiling Lisp to C/C++, then calling the C or C++ compiler, then loading the compiled code into a running Lisp, replacing/extending code. For example ECL. The maintainer of ECL also showed how to write inline C in a Lisp function and compile/load that in a REPL - using only a couple of lines of code - basically a C REPL inside Lisp. https://www.reddit.com/r/lisp/comments/bgwtsh/fun_ecl_hack/ https://www.reddit.com/r/lisp/comments/bgwtsh/fun_ecl_hack/
- lispm 7y ago
  Lisp in 1962 did it, it had a compiler and the implementation could mix interpreted and compiled code. One takes an interpreted function, which can be defined at runtime, compile it to assembler and have the assembler generate machine code in binary program space. The Lisp system then notes that this is now a compiled function. http://www.softwarepreservation.org/projects/LISP/book/LISP%201.5%20Programmers%20Manual.pdf http://www.softwarepreservation.org/projects/LISP/book/LISP%... > The LISP Compiler is a program written in LISP that translates S-expression definitions of functions into machine language subroutines. It is an optional feature that makes programs run many times faster than they would if they were to be interpreted at run time by the interpreter. > When the compiler is called upon to compile a function, it looks for an EXPR or FEXPR on the property list of the function name. The compiler then translates this S-expression into an S-expression that represents a subroutine in the LISP Assembly Language (LAP). LAP then proceeds to assemble this program into binary program space. Thus an EXPR, or an FEXPR, has been changed to a SUBR or an FSUBR, respectively. ... > 1. It is not necessary to compile all of the functions that are used in a particular run. The interpreter is designed to link with compiled functions. Compiled functions that use interpreted functions will call the interpreter to evaluate these at run time. > 2. The order in which functions are compiled is of no significance. It is not even necessary to have all of the functions defined until they are actually used at run time. (Specialforms are an exception to this rule. They must be defined before any function that calls them is compiled. )
lelf 7y ago
(Natively compiled) Erlang might want to have a word with you.
- teilo 7y ago
  Educate me if I am wrong, but there are natively compiled Erlang modules, but no longer any natively compiled Erlang. It only runs on the BEAM, and ever since BEAM/C fell out of use in OTP R5 no native code is generated.
  - lelf 7y ago
    erlc +native You will get beam file containing both the byte-code and native code.
  - darkmarmot 7y ago
    It even runs bare metal IoT https://www.grisp.org/ https://www.grisp.org/ (and https://github.com/nerves-project https://github.com/nerves-project is cool too for other embedded work)
  - strmpnk 7y ago
    BEAM is still present but HiPE interfaces with it using some specific ABI that allows mixing AOT compiled native modules (just machine code) and the BEAM register machine. I'm not sure if there are new caveats as I regard HiPE as a compatibility feature rather than something under active development (I may be wrong but it seems to fail to compile all of OTP's Erlang code now). The overhead of that context switch is a bit high but it allows the code loading facilities of the BEAM to reload HiPE compiled modules. This works because all processes yield to the scheduler, which acts as a kind of code-swap safe-point. The usual module-local vs module-remote call rules apply here when old versions are purged.
    - anthony_doan 7y ago
      > http://erlang.org/download/otp_src_22.0-rc3.readme http://erlang.org/download/otp_src_22.0-rc3.readme It's in active dev. ctrl+f "hipe-3.19" > I may be wrong but it seems to fail to compile all of OTP's Erlang code now See OTP-15596.
jamieson-becker 7y ago
Nim is an amazing language. The syntax is cleaner (IMO) and easier-to-read than Go and approaches Python in its readability, which is impressive for a statically typed, compiled language[0]. The design is focused on performance above all else, but it still has metaprogramming[1] and functional features. However, Go and other languages have a huge ecosystem and many more libraries. Nim only has a few web servers/frameworks, for example. Even if Nim's web frameworks/servers (like httpbeast) are quite fast[2], they lack the completeness that exist for other languages. Until then, if you are looking for a systems programming language, you owe it to yourself to investigate Nim[3], alongside Go, Crystal, Julia, D, Rust, Haskell, etc. The tooling is fantastic and the Nim compiler produces clean, cross-platform C code (or JS[4]!) that can be automatically fed into gcc, clang, mingw-w64, etc. It's a language that's undergoing rapid changes, but almost all of the changes are in the libraries and it's exciting to see all of the innovation there -- and as libraries increase and mature, it will become a really compelling application language as well. The community is extremely active, and issues are promptly dealt with. For HNers, it's an opportunity to still make a huge difference by contributing to a relatively young language, compared to getting drowned out by all the noise in a more mature language community. 0. https://nim-lang.org/docs/tut1.html https://nim-lang.org/docs/tut1.html 1. https://en.wikipedia.org/wiki/Metaprogramming https://en.wikipedia.org/wiki/Metaprogramming 2. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ 3. https://nim-lang.org/features.html https://nim-lang.org/features.html 4. https://picheta.me/snake/ https://picheta.me/snake/
- leshow 7y ago
  > Until then, if you are looking for a systems programming language, you owe it to yourself to investigate Nim[3], alongside Go, Crystal, Julia, D, Rust, Haskell, etc. Go, Crystal, Julia, Haskell and Nim all have a runtime that sort of precludes their use as a true systems lang. I agree with your other points, and I like Nim, I wish it was more popular, but I can't convince myself it's worth the time investment to learn it. I know Rust and Haskell already, between those two there isn't much space where Nim would be a good fit that the others aren't.
  - pjmlp 7y ago
nonsince 7y ago
To add another existing implementation of this, Naughty Dog's Lisp-based assembler could do hot code reloading, and that workflow was extensively used for developing Jak & Daxter. Plus, obviously any JIT'd language can do this but it's fair not to count that. Still, that's just splitting hairs and it's really exciting to see this in Nim. It's a great language and it's a shame that it doesn't seem to have found a real niche yet. http://www.codersnotes.com/notes/disassembling-jak/ http://www.codersnotes.com/notes/disassembling-jak/
- OskarS 7y ago
  There are many game engines do some form of hot code reloading of the game scripting code (as opposed to engine code). Unity supports it for C#, for instance (though it's finicky, you have to program with hot reloading in mind).
  - pjmlp 7y ago
    And Unreal does it for C++ as well now.
cmm 7y ago
I see that at this point in time Smalltalk and Lisp machines have become completely memory-holed (oh wait, Smalltalk & Lisp are interactive, thus "interpreted", thus obviously not compiled! I'll show myself out)
- IronBacon 7y ago
  Don't know about Smalltalk but there are numerous CL compilers...
  - bitwize 7y ago
    Smalltalk is compiled to bytecode.
    - pjmlp 7y ago
      Which were originally the CPU instructions of Dorado, given its microcoded CPU. And gave birth to all modern JITs.
    - gokr 7y ago
      Smalltalk is a family of dialects and implementations. Most of them are indeed compiled to bytecode which is then executed by a JIT VM. Most of them use the "image" concept where you are always in runtime like in a classic Lisp, and yes, the VM handles shape changes etc etc. But some Smalltalks actually have other characteristics, like non JIT or even compiling via C (Smalltalk/X does that I believe).
  - hencq 7y ago
    I'm pretty sure that's exactly their point. People assume that interactive means interpreted, but that doesn't need to be the case.
    - IronBacon 7y ago
      Ah, my bad, I didn't caught the sarcasm in that post. Yeah, it makes sense now... ^__^;
naasking 7y ago
Could be a cool feature, but I wouldn't say "first". Been done in C for years: * Safe and Automatic Live Update for Operating Systems, https://www.cs.vu.nl/~giuffrida/papers/asplos-2013.pdf https://www.cs.vu.nl/~giuffrida/papers/asplos-2013.pdf * Automating Live Update for Generic Server Programs, https://www.cs.vu.nl/~giuffrida/papers/lu_tse16.pdf https://www.cs.vu.nl/~giuffrida/papers/lu_tse16.pdf And I'm sure hot code reloading has been done in Forth for decades.
bjourne 7y ago
I only skimmed the video, but I couldn't see him describing it doing type reloading. Perhaps that is what he is planning on implementing. It is very hard to implement (especially if you can't control the compiler) but very useful for general code reloading. Of course everything is relative, and even if I think it is "very hard" it might be a weekend project for someone else. Generally though, code reloading is one of the hardest things to implement in a vm. The holy grail of code reloading is to upgrade the code of a HTTP server while it is running and without disturbing any requests being processed. Very few languages except for Erlang are able to do that correctly. Some languages claim to support that, but when you experiment with them you discover "quirks" making it impossible in practice.
- MuffinFlavored 7y ago
  > The holy grail of code reloading is to upgrade the code of a HTTP server while it is running and without disturbing any requests being processed. Very few languages except for Erlang are able to do that correctly. Some languages claim to support that, but when you experiment with them you discover "quirks" making it impossible in practice. You can kind of do it with node.js, but man do things get ugly fast when you manage state/connections in modules on reload.
  - muddybytes 7y ago
    in the olden days there was a “copyover” patch for a MUD codebase that allowed hot reloading with hundreds of active sockets in a single threaded app.
- jerf 7y ago
  If you put your mind to it just a bit (i.e., I wouldn't call this "automatic" exactly but it's not too hard to use the support the language, runtime, and libraries have), Erlang can even do a harder thing, which is update the code handling a given socket connection or something live, in a principled manner, while never dropping the connection. Since HTTP is transient, it's actually a bit easier than a raw socket since you can expect it to go away soon, or even in the case of HTTP2, you can often expect to just close a socket as long as it's not currently active and get away with it. Many languages can smoothly upgrade an HTTP server by handing off a listening socket to a new process with new code. But even that won't save you for live sockets, because even if you hand off the socket, you haven't got a clean mechanism for handing off its accompanying state. Several interpreted languages can sort of do this, but I'd call it in an "unprincipled" manner by just slamming new code in place and hoping for the best. Erlang explicitly upgrades the gen_* instances and you can provide a function for converting the old state to the new state cleanly.
wdroz 7y ago
You can hotpatch C function too : https://news.ycombinator.com/item?id=11403263 https://news.ycombinator.com/item?id=11403263
- kazinator 7y ago
  Loading and unloading dynamic libraries (dlopen in Unix, LoadLibrary in Windows ...) and operating system kernel modules are also examples of hot code reloading in C.
quangio 7y ago
+1 Nim is a really nice and simple language. Extremely easy FFI and Python-like. For me, using Nim and Julia to write small "scripts" for my data science (student) works (processing 10Gb-20Gb of data) makes me feel quite productive compared to something like Go (too `ugly` for me), Python (too slow)
- everdev 7y ago
  Are there any Nim benchmarks vs. Go, Crystal, Python, Rust, etc.?
  - maow83jejj 7y ago
    You can find them although they tend to be out of date and/or piecemeal. I think they're still useful. For example: https://github.com/drujensen/fib https://github.com/drujensen/fib https://gist.github.com/sdwfrost/7c660322c6c33961297a826df4cbc30d https://gist.github.com/sdwfrost/7c660322c6c33961297a826df4c... https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks My general sense is that Nim is most competitive as an alternative to Python and/or things like Julia, in that it's as expressive and easy to understand as those, but has performance closer to C or Rust than something like Python. Julia is similar to Nim but I think Nim seems cleaner in its implementation overall, and targets much more general use scenarios. I've been really impressed by Nim. There are some little things that have irritated me, like case insensitivity, but I wish it got more traction in the community. Right now the only thing that it doesn't seem to have going for it is library support. For the numerical applications I do I'd prefer it over everything else, except everything else has huge resource bases, so something that's already packaged in those languages would have to be done from scratch in Nim, which isn't feasible.
    - 0xdeadbeefbabe 7y ago
      Isn't the stdlib problem made less of a problem by the easy FFI to C? (it wasn't easy last time I tried btw)
    - jp_sc 7y ago
      As far as I understanf, you can call numpy from nim using a lightweight wrapper.
benj111 7y ago
Can someone give me some pointers to reading material on this. I'm aware of hot code reloading for interpreted languages, google just gives links to web page loading. I'm also aware of it at the OS level. I don't really understand why you'd want this in a compiled language.
- scriptkiddy 7y ago
  > Can someone give me some pointers to reading material on this. Wish I could help you with this, but I have been out of the loop with Nim recently. > I don't really understand why you'd want this in a compiled language. Quick iteration in development. Not needing to manually recompile every time you make a small change.
  - lloda 7y ago
    or recompute the data you're working with.
- lispm 7y ago
  > I don't really understand why you'd want this in a compiled language. Why not?
- kazinator 7y ago
  > why you'd want this in a compiled language. Shared-library-based plugins, kernel modules, ...
- detaro 7y ago
  If you understand why people have it in interpreted languages, why do those reasons not apply to compiled languages?
  - benj111 7y ago
    Because for an interpreted language you can use it for interactive development. I had thought different reasons would apply to a compiled language but judging by the other replies obviously not!
ungzd 7y ago
I remember using "edit and continue" in Visual C++ 6, seems that this feature still exists in newer versions[1], but I don't know what limitations it has. [1] https://docs.microsoft.com/en-us/visualstudio/debugger/edit-and-continue?view=vs-2019 https://docs.microsoft.com/en-us/visualstudio/debugger/edit-...
- Jach 7y ago
  That's a great doc page. In the "Supported Code Changes" for C++ and C# it has a helpful list of what isn't supported too. Java also has the feature in the JVM (IDEs / mvn-hotswap plugin can hook into it), but it's similarly limited. There's a proprietary java agent called JRebel that significantly expands the support (https://jrebel.com/software/jrebel/features/comparison-matrix/ https://jrebel.com/software/jrebel/features/comparison-matri...), and a couple open source versions I haven't tried, but it's still got some limitations. It's worth a comparison to Common Lisp which thought about this feature a lot more and built it into the language rather than having it hacked in by external tools... e.g. if you redefine a class definition in Java say to add a new storage field, well you can't using default hotswap but with JRebel you can, but still any existing objects will continue to point to the "old class"'s code and the new field won't be available... Common Lisp defines a generic function 'update-instance-for-redefined-class that you extend before you do the class edit, and now your existing objects will work with the new code.
- vram22 7y ago
  I had been exploring Lisp (in particular, Franz's Allegro Lisp and LispWorks Lisp) some years ago, and around the same time, came across the "edit and continue" feature in Visual C++ / Visual Studio (may have been 6 or other version). IIRC I blogged or tweeted about both, and that maybe that VC++ feature was inspired by something similar in Lisp.
azhenley 7y ago
Nim enticed me to create my own systems-ish language: https://github.com/AZHenley/knox https://github.com/AZHenley/knox It is still just a toy and in the early stages. It’s been fun. Thanks for that, Nim.
zevv 7y ago
Can't recommend this enough: Nim is a very low-friction language with performance in the C/C++ range, also offers introspection and metaprogramming with a very powerful macro system.
jdc 7y ago
Looks like it's still a pre-release feature at this point. Hot code-reloading part of the talk: https://youtu.be/7WgCt0Wooeo?t=1519 https://youtu.be/7WgCt0Wooeo?t=1519 RFC: https://github.com/nim-lang/Nim/issues/8927 https://github.com/nim-lang/Nim/issues/8927 Code: https://github.com/onqtam/Nim/tree/hcr-1 https://github.com/onqtam/Nim/tree/hcr-1
- narimiran 7y ago
  > Looks like it's still a pre-release feature at this point. It has been merged to devel branch: https://github.com/nim-lang/Nim/pull/10729 https://github.com/nim-lang/Nim/pull/10729
  - akavel 7y ago
    What's the situation with the TODOs mentioned in the presentation? Primarily, the missing standard library support cleanup ("Currently no real-world project can be built with HCR")?
    - onqtam 7y ago
      those issues will hopefully be addressed in the coming months - when I get the time
      - akavel 7y ago
        Cool, and thanks for all the work! Keeping fingers crossed that you'll manage to make the final push!
squarefoot 7y ago
+1 for Nim. I'm dreaming some day to see a RAD where Lazarus generates Nim code with full LCL support; that would be a truly killer dev system for desktop apps. Lazarus is already a wonderful RAD, one can build native apps on a *PI board and run them directly on that target; Nim support would make it even greater.
- pjmlp 7y ago
  Not sure what that would buy over Object Pascal and the plethora of libraries develop since Turbo Pascal/Apple Pascal days.
  - squarefoot 7y ago
    Admittedly not much on the technical side beyond new languages constructs etc, but the main benefit would be ending for good the wrong argument "Pascal is old therefore Lazarus sucks". Like 20 years ago I've written network code in Delphi that would send and receive binary data between different endianess and word size machines, so I had to align them and manage data down to bitfields in the Pascal equivalent of C unions, that is, the language offers enough bullets and rope to hang then shoot oneself in the foot just like C, but every time I talked with someone about Delphi and later Lazarus, it was just about a matter of minutes before the inevitable "yeah, but Pascal is old".
    - pjmlp 7y ago
      Here is an answer for them, all mainstream languages in use today are at least 20 years old. And their beloved C is reaching 50 years old.
- ScottFree 7y ago
  > I'm dreaming some day to see a RAD where Lazarus generates Nim code with full LCL support What does LCL stand for in this context?
peheje 7y ago
I really like Nim. I hope the threading interface improves though, I found it lacking last time I checked about half a year ago.
- hathawsh 7y ago
  I'm curious, what more are you looking for? From the nim manual: "Each thread has its own (garbage collected) heap and sharing of memory is restricted to global variables. This helps to prevent race conditions. GC efficiency is improved quite a lot, because the GC never has to stop other threads and see what they reference. Memory allocation requires no lock at all! This design easily scales to massive multicore processors that are becoming the norm." To me, that sounds perfect for writing typical apps.
  - KallDrexx 7y ago
    How does Nim do thread-based GC if an object is shared across threads? Does it not allow that sharing references cross-thread (not sure how it would without lifetimes though)?
    - hathawsh 7y ago
      If I understand the Nim manual correctly, there is no sharing of objects across threads. Instead, Nim assumes message passing. When you communicate between threads, Nim makes a deep copy of the message. In theory, that strategy could be very inefficient if the messages are large, but it also means the garbage collector is unaware of threads and therefore simpler.
      - btym 7y ago
        >there is no sharing of objects across threads You can message-pass a pointer if you are sure the referenced object won't be GCed.
    - btym 7y ago
      You communicate via channels, like in Go. Objects passed through a channel are deep copied. To share an object, you can create an actor thread that owns it, or if you're sure the object won't be GCed (i.e. by exempting it with the `GC_ref` function), you can pass a pointer through the channel.
pjmlp 7y ago
In addition to several other examples. C++ did it in the 80's with Lucid C++ and the ill fated Visual Age for C++ v4.0. Eiffel also thanks to its MELT VM.
KallDrexx 7y ago
I can't tell from the video if the hot code reloading is meant for production workflows or dev only workflows. For example being able to use HCR to update a long-lived TCP server (like a video streaming server) without disconnecting clients. It seems like it forces pointers for everything which, which seems to me will break a lot of optimizations (like inlining) Edit: I guess production use is limited anyway since you can't add, modify, or remove types with HCR yet. Edit2: I just saw in the video they said it's 2x times slower, so definitely can't be used for a lot of workflows in production. Still useful though.
- onqtam 7y ago
  that is indeed the case - HCR is meant mainly for development. But if you can stomach the x2-x3-x4 slowdown for production nothing would stop you
drmeister 7y ago
IMHO - you should learn Common Lisp - it solved the problems described in the talk (Nim can't redefine types on the fly) decades ago. In Common Lisp you can redefine _everything_ on the fly. You can change types (classes) in a running system and objects in memory that have the layout of the old class get upgraded automatically when they are used. Programming with Common Lisp in Slime/emacs is an amazing experience that will spoil you for other languages.
revskill 7y ago
Nim is an interesting language. Main features: - Type system: Strong and static typings. - Support Generics (Unlike Go) - Support Modules (Unlink C++) - Multiple and optional GC. - Metaprogramming support. - Executed binary. What excites me most is the efficiency as C. I hope it'll get mature more this year. At least for the Javascript backend, so that i could write Nim for frontend (with bindings for popular JS frameworks like React, Vue,..) , too.
- pjmlp 7y ago
  C++20 does support modules.
gregwebs 7y ago
Haskell (a compiled lanaguage) has done hot code reloading with ghci for a long time. Facebook actually deploys in production by reloading. https://simonmar.github.io/posts/2017-10-17-hotswapping-haskell.html https://simonmar.github.io/posts/2017-10-17-hotswapping-hask...
interfixus 7y ago
Whenever I've been working in Nim, going back to other languages is always - codewise - a letdown. Alas, I often must go back, almost always for the same reason: The lack of libraries (and || or) attendant documentation. One doesn't always have the time, inclination, or ability to work everything up from scratch. Otherwise, it's such an overwhelmingly pleasing toolset to work with. Clean code, ultrafast compilation producing rocksolid executables as small and tight as you care to make them. Joy all around.
- canada_dry 7y ago
  > The lack of libraries Yup. It's too bad there has been no en-mass conversion/import of popular libraries. The thing that makes python such a terrific swiss-army-knife is that there are libraries available for just about every imaginable use case.

186 points by Clyybber 7y ago

dunefox 7y ago

Afaik Common Lisp would be the first, wouldn't it?

vthriller 7y ago
The answer to that would depend on the CL implementation, since not all of them can do AOT compilation.
- coliveira 7y ago
  But the same can be said about Nim, this ability depends on the implementation.
- jlarocco 7y ago
  Not really... It's in the CL standard, and compliant implementations support it. SBCL, LispWorks, Allegro, CCL, and others all do AOT native compilation, from the REPL and via eval (which should be avoided, in any case). ABCL does AOT compilation to JVM bytecode, and Clisp to its own bytecode. It's a cool feature for Nim to have, but claiming to be the first makes them look bad.
  - lispm 7y ago
    Several Lisp implementations even do similar things which were shown in the video: compiling Lisp to C/C++, then calling the C or C++ compiler, then loading the compiled code into a running Lisp, replacing/extending code. For example ECL. The maintainer of ECL also showed how to write inline C in a Lisp function and compile/load that in a REPL - using only a couple of lines of code - basically a C REPL inside Lisp. https://www.reddit.com/r/lisp/comments/bgwtsh/fun_ecl_hack/ https://www.reddit.com/r/lisp/comments/bgwtsh/fun_ecl_hack/
lispm 7y ago
Lisp in 1962 did it, it had a compiler and the implementation could mix interpreted and compiled code. One takes an interpreted function, which can be defined at runtime, compile it to assembler and have the assembler generate machine code in binary program space. The Lisp system then notes that this is now a compiled function. http://www.softwarepreservation.org/projects/LISP/book/LISP%201.5%20Programmers%20Manual.pdf http://www.softwarepreservation.org/projects/LISP/book/LISP%... > The LISP Compiler is a program written in LISP that translates S-expression definitions of functions into machine language subroutines. It is an optional feature that makes programs run many times faster than they would if they were to be interpreted at run time by the interpreter. > When the compiler is called upon to compile a function, it looks for an EXPR or FEXPR on the property list of the function name. The compiler then translates this S-expression into an S-expression that represents a subroutine in the LISP Assembly Language (LAP). LAP then proceeds to assemble this program into binary program space. Thus an EXPR, or an FEXPR, has been changed to a SUBR or an FSUBR, respectively. ... > 1. It is not necessary to compile all of the functions that are used in a particular run. The interpreter is designed to link with compiled functions. Compiled functions that use interpreted functions will call the interpreter to evaluate these at run time. > 2. The order in which functions are compiled is of no significance. It is not even necessary to have all of the functions defined until they are actually used at run time. (Specialforms are an exception to this rule. They must be defined before any function that calls them is compiled. )

lelf 7y ago

(Natively compiled) Erlang might want to have a word with you.

teilo 7y ago
Educate me if I am wrong, but there are natively compiled Erlang modules, but no longer any natively compiled Erlang. It only runs on the BEAM, and ever since BEAM/C fell out of use in OTP R5 no native code is generated.
- lelf 7y ago
  erlc +native You will get beam file containing both the byte-code and native code.
- darkmarmot 7y ago
  It even runs bare metal IoT https://www.grisp.org/ https://www.grisp.org/ (and https://github.com/nerves-project https://github.com/nerves-project is cool too for other embedded work)
- strmpnk 7y ago
  BEAM is still present but HiPE interfaces with it using some specific ABI that allows mixing AOT compiled native modules (just machine code) and the BEAM register machine. I'm not sure if there are new caveats as I regard HiPE as a compatibility feature rather than something under active development (I may be wrong but it seems to fail to compile all of OTP's Erlang code now). The overhead of that context switch is a bit high but it allows the code loading facilities of the BEAM to reload HiPE compiled modules. This works because all processes yield to the scheduler, which acts as a kind of code-swap safe-point. The usual module-local vs module-remote call rules apply here when old versions are purged.
  - anthony_doan 7y ago
    > http://erlang.org/download/otp_src_22.0-rc3.readme http://erlang.org/download/otp_src_22.0-rc3.readme It's in active dev. ctrl+f "hipe-3.19" > I may be wrong but it seems to fail to compile all of OTP's Erlang code now See OTP-15596.

jamieson-becker 7y ago

Nim is an amazing language. The syntax is cleaner (IMO) and easier-to-read than Go and approaches Python in its readability, which is impressive for a statically typed, compiled language[0]. The design is focused on performance above all else, but it still has metaprogramming[1] and functional features. However, Go and other languages have a huge ecosystem and many more libraries. Nim only has a few web servers/frameworks, for example. Even if Nim's web frameworks/servers (like httpbeast) are quite fast[2], they lack the completeness that exist for other languages. Until then, if you are looking for a systems programming language, you owe it to yourself to investigate Nim[3], alongside Go, Crystal, Julia, D, Rust, Haskell, etc. The tooling is fantastic and the Nim compiler produces clean, cross-platform C code (or JS[4]!) that can be automatically fed into gcc, clang, mingw-w64, etc. It's a language that's undergoing rapid changes, but almost all of the changes are in the libraries and it's exciting to see all of the innovation there -- and as libraries increase and mature, it will become a really compelling application language as well. The community is extremely active, and issues are promptly dealt with. For HNers, it's an opportunity to still make a huge difference by contributing to a relatively young language, compared to getting drowned out by all the noise in a more mature language community. 0. https://nim-lang.org/docs/tut1.html https://nim-lang.org/docs/tut1.html 1. https://en.wikipedia.org/wiki/Metaprogramming https://en.wikipedia.org/wiki/Metaprogramming 2. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ 3. https://nim-lang.org/features.html https://nim-lang.org/features.html 4. https://picheta.me/snake/ https://picheta.me/snake/

leshow 7y ago
> Until then, if you are looking for a systems programming language, you owe it to yourself to investigate Nim[3], alongside Go, Crystal, Julia, D, Rust, Haskell, etc. Go, Crystal, Julia, Haskell and Nim all have a runtime that sort of precludes their use as a true systems lang. I agree with your other points, and I like Nim, I wish it was more popular, but I can't convince myself it's worth the time investment to learn it. I know Rust and Haskell already, between those two there isn't much space where Nim would be a good fit that the others aren't.
- pjmlp 7y ago

nonsince 7y ago

To add another existing implementation of this, Naughty Dog's Lisp-based assembler could do hot code reloading, and that workflow was extensively used for developing Jak & Daxter. Plus, obviously any JIT'd language can do this but it's fair not to count that. Still, that's just splitting hairs and it's really exciting to see this in Nim. It's a great language and it's a shame that it doesn't seem to have found a real niche yet. http://www.codersnotes.com/notes/disassembling-jak/ http://www.codersnotes.com/notes/disassembling-jak/

OskarS 7y ago
There are many game engines do some form of hot code reloading of the game scripting code (as opposed to engine code). Unity supports it for C#, for instance (though it's finicky, you have to program with hot reloading in mind).
- pjmlp 7y ago
  And Unreal does it for C++ as well now.

cmm 7y ago

I see that at this point in time Smalltalk and Lisp machines have become completely memory-holed (oh wait, Smalltalk & Lisp are interactive, thus "interpreted", thus obviously not compiled! I'll show myself out)

IronBacon 7y ago
Don't know about Smalltalk but there are numerous CL compilers...
- bitwize 7y ago
  Smalltalk is compiled to bytecode.
  - pjmlp 7y ago
    Which were originally the CPU instructions of Dorado, given its microcoded CPU. And gave birth to all modern JITs.
  - gokr 7y ago
    Smalltalk is a family of dialects and implementations. Most of them are indeed compiled to bytecode which is then executed by a JIT VM. Most of them use the "image" concept where you are always in runtime like in a classic Lisp, and yes, the VM handles shape changes etc etc. But some Smalltalks actually have other characteristics, like non JIT or even compiling via C (Smalltalk/X does that I believe).
- hencq 7y ago
  I'm pretty sure that's exactly their point. People assume that interactive means interpreted, but that doesn't need to be the case.
  - IronBacon 7y ago
    Ah, my bad, I didn't caught the sarcasm in that post. Yeah, it makes sense now... ^__^;

naasking 7y ago

Could be a cool feature, but I wouldn't say "first". Been done in C for years: * Safe and Automatic Live Update for Operating Systems, https://www.cs.vu.nl/~giuffrida/papers/asplos-2013.pdf https://www.cs.vu.nl/~giuffrida/papers/asplos-2013.pdf * Automating Live Update for Generic Server Programs, https://www.cs.vu.nl/~giuffrida/papers/lu_tse16.pdf https://www.cs.vu.nl/~giuffrida/papers/lu_tse16.pdf And I'm sure hot code reloading has been done in Forth for decades.

bjourne 7y ago

I only skimmed the video, but I couldn't see him describing it doing type reloading. Perhaps that is what he is planning on implementing. It is very hard to implement (especially if you can't control the compiler) but very useful for general code reloading. Of course everything is relative, and even if I think it is "very hard" it might be a weekend project for someone else. Generally though, code reloading is one of the hardest things to implement in a vm. The holy grail of code reloading is to upgrade the code of a HTTP server while it is running and without disturbing any requests being processed. Very few languages except for Erlang are able to do that correctly. Some languages claim to support that, but when you experiment with them you discover "quirks" making it impossible in practice.

MuffinFlavored 7y ago
> The holy grail of code reloading is to upgrade the code of a HTTP server while it is running and without disturbing any requests being processed. Very few languages except for Erlang are able to do that correctly. Some languages claim to support that, but when you experiment with them you discover "quirks" making it impossible in practice. You can kind of do it with node.js, but man do things get ugly fast when you manage state/connections in modules on reload.
- muddybytes 7y ago
  in the olden days there was a “copyover” patch for a MUD codebase that allowed hot reloading with hundreds of active sockets in a single threaded app.
jerf 7y ago
If you put your mind to it just a bit (i.e., I wouldn't call this "automatic" exactly but it's not too hard to use the support the language, runtime, and libraries have), Erlang can even do a harder thing, which is update the code handling a given socket connection or something live, in a principled manner, while never dropping the connection. Since HTTP is transient, it's actually a bit easier than a raw socket since you can expect it to go away soon, or even in the case of HTTP2, you can often expect to just close a socket as long as it's not currently active and get away with it. Many languages can smoothly upgrade an HTTP server by handing off a listening socket to a new process with new code. But even that won't save you for live sockets, because even if you hand off the socket, you haven't got a clean mechanism for handing off its accompanying state. Several interpreted languages can sort of do this, but I'd call it in an "unprincipled" manner by just slamming new code in place and hoping for the best. Erlang explicitly upgrades the gen_* instances and you can provide a function for converting the old state to the new state cleanly.

wdroz 7y ago

You can hotpatch C function too : https://news.ycombinator.com/item?id=11403263 https://news.ycombinator.com/item?id=11403263

kazinator 7y ago
Loading and unloading dynamic libraries (dlopen in Unix, LoadLibrary in Windows ...) and operating system kernel modules are also examples of hot code reloading in C.

quangio 7y ago

+1 Nim is a really nice and simple language. Extremely easy FFI and Python-like. For me, using Nim and Julia to write small "scripts" for my data science (student) works (processing 10Gb-20Gb of data) makes me feel quite productive compared to something like Go (too `ugly` for me), Python (too slow)

everdev 7y ago
Are there any Nim benchmarks vs. Go, Crystal, Python, Rust, etc.?
- maow83jejj 7y ago
  You can find them although they tend to be out of date and/or piecemeal. I think they're still useful. For example: https://github.com/drujensen/fib https://github.com/drujensen/fib https://gist.github.com/sdwfrost/7c660322c6c33961297a826df4cbc30d https://gist.github.com/sdwfrost/7c660322c6c33961297a826df4c... https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks My general sense is that Nim is most competitive as an alternative to Python and/or things like Julia, in that it's as expressive and easy to understand as those, but has performance closer to C or Rust than something like Python. Julia is similar to Nim but I think Nim seems cleaner in its implementation overall, and targets much more general use scenarios. I've been really impressed by Nim. There are some little things that have irritated me, like case insensitivity, but I wish it got more traction in the community. Right now the only thing that it doesn't seem to have going for it is library support. For the numerical applications I do I'd prefer it over everything else, except everything else has huge resource bases, so something that's already packaged in those languages would have to be done from scratch in Nim, which isn't feasible.
  - 0xdeadbeefbabe 7y ago
    Isn't the stdlib problem made less of a problem by the easy FFI to C? (it wasn't easy last time I tried btw)
  - jp_sc 7y ago
    As far as I understanf, you can call numpy from nim using a lightweight wrapper.

benj111 7y ago

Can someone give me some pointers to reading material on this. I'm aware of hot code reloading for interpreted languages, google just gives links to web page loading. I'm also aware of it at the OS level. I don't really understand why you'd want this in a compiled language.

scriptkiddy 7y ago
> Can someone give me some pointers to reading material on this. Wish I could help you with this, but I have been out of the loop with Nim recently. > I don't really understand why you'd want this in a compiled language. Quick iteration in development. Not needing to manually recompile every time you make a small change.
- lloda 7y ago
  or recompute the data you're working with.
lispm 7y ago
> I don't really understand why you'd want this in a compiled language. Why not?
kazinator 7y ago
> why you'd want this in a compiled language. Shared-library-based plugins, kernel modules, ...
detaro 7y ago
If you understand why people have it in interpreted languages, why do those reasons not apply to compiled languages?
- benj111 7y ago
  Because for an interpreted language you can use it for interactive development. I had thought different reasons would apply to a compiled language but judging by the other replies obviously not!

ungzd 7y ago

I remember using "edit and continue" in Visual C++ 6, seems that this feature still exists in newer versions[1], but I don't know what limitations it has. [1] https://docs.microsoft.com/en-us/visualstudio/debugger/edit-and-continue?view=vs-2019 https://docs.microsoft.com/en-us/visualstudio/debugger/edit-...

Jach 7y ago
That's a great doc page. In the "Supported Code Changes" for C++ and C# it has a helpful list of what isn't supported too. Java also has the feature in the JVM (IDEs / mvn-hotswap plugin can hook into it), but it's similarly limited. There's a proprietary java agent called JRebel that significantly expands the support (https://jrebel.com/software/jrebel/features/comparison-matrix/ https://jrebel.com/software/jrebel/features/comparison-matri...), and a couple open source versions I haven't tried, but it's still got some limitations. It's worth a comparison to Common Lisp which thought about this feature a lot more and built it into the language rather than having it hacked in by external tools... e.g. if you redefine a class definition in Java say to add a new storage field, well you can't using default hotswap but with JRebel you can, but still any existing objects will continue to point to the "old class"'s code and the new field won't be available... Common Lisp defines a generic function 'update-instance-for-redefined-class that you extend before you do the class edit, and now your existing objects will work with the new code.
vram22 7y ago
I had been exploring Lisp (in particular, Franz's Allegro Lisp and LispWorks Lisp) some years ago, and around the same time, came across the "edit and continue" feature in Visual C++ / Visual Studio (may have been 6 or other version). IIRC I blogged or tweeted about both, and that maybe that VC++ feature was inspired by something similar in Lisp.

azhenley 7y ago

Nim enticed me to create my own systems-ish language: https://github.com/AZHenley/knox https://github.com/AZHenley/knox It is still just a toy and in the early stages. It’s been fun. Thanks for that, Nim.

zevv 7y ago

Can't recommend this enough: Nim is a very low-friction language with performance in the C/C++ range, also offers introspection and metaprogramming with a very powerful macro system.

jdc 7y ago

Looks like it's still a pre-release feature at this point. Hot code-reloading part of the talk: https://youtu.be/7WgCt0Wooeo?t=1519 https://youtu.be/7WgCt0Wooeo?t=1519 RFC: https://github.com/nim-lang/Nim/issues/8927 https://github.com/nim-lang/Nim/issues/8927 Code: https://github.com/onqtam/Nim/tree/hcr-1 https://github.com/onqtam/Nim/tree/hcr-1

narimiran 7y ago
> Looks like it's still a pre-release feature at this point. It has been merged to devel branch: https://github.com/nim-lang/Nim/pull/10729 https://github.com/nim-lang/Nim/pull/10729
- akavel 7y ago
  What's the situation with the TODOs mentioned in the presentation? Primarily, the missing standard library support cleanup ("Currently no real-world project can be built with HCR")?
  - onqtam 7y ago
    those issues will hopefully be addressed in the coming months - when I get the time
    - akavel 7y ago
      Cool, and thanks for all the work! Keeping fingers crossed that you'll manage to make the final push!

squarefoot 7y ago

+1 for Nim. I'm dreaming some day to see a RAD where Lazarus generates Nim code with full LCL support; that would be a truly killer dev system for desktop apps. Lazarus is already a wonderful RAD, one can build native apps on a *PI board and run them directly on that target; Nim support would make it even greater.

pjmlp 7y ago
Not sure what that would buy over Object Pascal and the plethora of libraries develop since Turbo Pascal/Apple Pascal days.
- squarefoot 7y ago
  Admittedly not much on the technical side beyond new languages constructs etc, but the main benefit would be ending for good the wrong argument "Pascal is old therefore Lazarus sucks". Like 20 years ago I've written network code in Delphi that would send and receive binary data between different endianess and word size machines, so I had to align them and manage data down to bitfields in the Pascal equivalent of C unions, that is, the language offers enough bullets and rope to hang then shoot oneself in the foot just like C, but every time I talked with someone about Delphi and later Lazarus, it was just about a matter of minutes before the inevitable "yeah, but Pascal is old".
  - pjmlp 7y ago
    Here is an answer for them, all mainstream languages in use today are at least 20 years old. And their beloved C is reaching 50 years old.
ScottFree 7y ago
> I'm dreaming some day to see a RAD where Lazarus generates Nim code with full LCL support What does LCL stand for in this context?

peheje 7y ago

I really like Nim. I hope the threading interface improves though, I found it lacking last time I checked about half a year ago.

hathawsh 7y ago
I'm curious, what more are you looking for? From the nim manual: "Each thread has its own (garbage collected) heap and sharing of memory is restricted to global variables. This helps to prevent race conditions. GC efficiency is improved quite a lot, because the GC never has to stop other threads and see what they reference. Memory allocation requires no lock at all! This design easily scales to massive multicore processors that are becoming the norm." To me, that sounds perfect for writing typical apps.
- KallDrexx 7y ago
  How does Nim do thread-based GC if an object is shared across threads? Does it not allow that sharing references cross-thread (not sure how it would without lifetimes though)?
  - hathawsh 7y ago
    If I understand the Nim manual correctly, there is no sharing of objects across threads. Instead, Nim assumes message passing. When you communicate between threads, Nim makes a deep copy of the message. In theory, that strategy could be very inefficient if the messages are large, but it also means the garbage collector is unaware of threads and therefore simpler.
    - btym 7y ago
      >there is no sharing of objects across threads You can message-pass a pointer if you are sure the referenced object won't be GCed.
  - btym 7y ago
    You communicate via channels, like in Go. Objects passed through a channel are deep copied. To share an object, you can create an actor thread that owns it, or if you're sure the object won't be GCed (i.e. by exempting it with the `GC_ref` function), you can pass a pointer through the channel.

pjmlp 7y ago

In addition to several other examples. C++ did it in the 80's with Lucid C++ and the ill fated Visual Age for C++ v4.0. Eiffel also thanks to its MELT VM.

KallDrexx 7y ago

I can't tell from the video if the hot code reloading is meant for production workflows or dev only workflows. For example being able to use HCR to update a long-lived TCP server (like a video streaming server) without disconnecting clients. It seems like it forces pointers for everything which, which seems to me will break a lot of optimizations (like inlining) Edit: I guess production use is limited anyway since you can't add, modify, or remove types with HCR yet. Edit2: I just saw in the video they said it's 2x times slower, so definitely can't be used for a lot of workflows in production. Still useful though.

onqtam 7y ago
that is indeed the case - HCR is meant mainly for development. But if you can stomach the x2-x3-x4 slowdown for production nothing would stop you

drmeister 7y ago

IMHO - you should learn Common Lisp - it solved the problems described in the talk (Nim can't redefine types on the fly) decades ago. In Common Lisp you can redefine _everything_ on the fly. You can change types (classes) in a running system and objects in memory that have the layout of the old class get upgraded automatically when they are used. Programming with Common Lisp in Slime/emacs is an amazing experience that will spoil you for other languages.

revskill 7y ago

Nim is an interesting language. Main features: - Type system: Strong and static typings. - Support Generics (Unlike Go) - Support Modules (Unlink C++) - Multiple and optional GC. - Metaprogramming support. - Executed binary. What excites me most is the efficiency as C. I hope it'll get mature more this year. At least for the Javascript backend, so that i could write Nim for frontend (with bindings for popular JS frameworks like React, Vue,..) , too.

pjmlp 7y ago
C++20 does support modules.

gregwebs 7y ago

Haskell (a compiled lanaguage) has done hot code reloading with ghci for a long time. Facebook actually deploys in production by reloading. https://simonmar.github.io/posts/2017-10-17-hotswapping-haskell.html https://simonmar.github.io/posts/2017-10-17-hotswapping-hask...

interfixus 7y ago

Whenever I've been working in Nim, going back to other languages is always - codewise - a letdown. Alas, I often must go back, almost always for the same reason: The lack of libraries (and || or) attendant documentation. One doesn't always have the time, inclination, or ability to work everything up from scratch. Otherwise, it's such an overwhelmingly pleasing toolset to work with. Clean code, ultrafast compilation producing rocksolid executables as small and tight as you care to make them. Joy all around.

canada_dry 7y ago
> The lack of libraries Yup. It's too bad there has been no en-mass conversion/import of popular libraries. The thing that makes python such a terrific swiss-army-knife is that there are libraries available for just about every imaginable use case.