6 ms·

GenieFramework – Build web applications with Julia

207 points by msravi 5y ago

transfire 5y ago
I started a new web project recently and I decided to either use Elixir/Phoenix or Julia/Genie. I wanted to use the later but two main issues led me ultimately to the former. 1) JIT compile times to first view were very slow — like 10 seconds, 2) Macro usage in Julia seems too common making the code ugly and difficult to read. e.g. @with_kw. What’s the point of an elegant language if you have to annotate it with obtuse macros at every turn?
melkael 5y ago
To Genie users, what is your workflow ? I love Julia for scientific computing but the compilation time is a bit of an issue. I would imagine I would need to compile a lot more when doing web dev. How do you work around this ? If there is a solution to this, Genie would be awesome as I could directly put into production my Julia models without using a third party language.
- huijzer 5y ago
  Have you tried PackageCompiler.jl?
  - abstracto 5y ago
    Honestly, this is an area where the op would be far better served by using go/rust/c++ or another backend language. Can launch a server with microsecond latency, and if it grows get good elastic behavior. No awkward route precompilation or thirty minute build times... Julia sure is neat, but this isn't an area where it's strong.
    - 0des 5y ago
      I do webdev in Go, and freely recompile as it takes a second or less on my hardware which consists of a rock, a rubber band and two sesame seeds on 20x2 liquid crystal. Its no struggle to recompile 100x in a day just iterating.
    - jjtheblunt 5y ago
      microsecond latency? on what exact hardware are you seeing that?
      - abstracto 5y ago
        That was a somewhat hyperbolic statement. Make an application in node, genie, and rusts actix. Run them and see how fast it takes for the cli to load each of them. Feel free to package compiler the Julia one if you want. It's a good experiment with visual results.
      - mirekrusin 5y ago
        Must be new M1 Ultra or something.
        abstracto 5y ago
        P50 for basic routes using other web frameworks can be sub 0.5 ms. See the techempower benchmarks. These benchmarks are somewhat flawed, but I'd be surprised if genie was faster then flask all around, and flask is widely considered expensively slow.
        jjtheblunt 5y ago
        0.5 ms is 500 times longer than the comment above, though
- andyferris 5y ago
  I haven’t used Genie (though after this post I am tempted!) but I imagine compilation overhead would only occur once (e.g. the first time each route is used) and that you could pretty easily use precompile statements and/or PackageCompiler.jl to do the compilation before the service goes live. After that point, Julia should have produced fast code and provides a good runtime for these kinds of async/multithreaded applications with built-in M:N scheduling (perhaps a concurrent/multithreaded GC might help reduce tail latency but I imagine it should already be competitive with everything but well-written Rust/C++/C).
- alhirzel 5y ago
  Revise.jl[1] is very helpful for development, as it allows for seamless recompilation without restarts. Cold start can also take a long time, so sometimes it helps to add options[2] such as -O0 when developing. Two paths forward for AOT compilation are PackageCompiler.jl[3] mentioned already in this thread (things like RelocatableFolders.jl[4] come in handy) and the possibility of GPUCompiler.jl[5] some day. [1]: https://github.com/timholy/Revise.jl https://github.com/timholy/Revise.jl [2]: https://docs.julialang.org/en/v1/manual/command-line-options/ https://docs.julialang.org/en/v1/manual/command-line-options... [3]: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl [4]: https://github.com/JuliaPackaging/RelocatableFolders.jl https://github.com/JuliaPackaging/RelocatableFolders.jl [5]: https://github.com/JuliaGPU/GPUCompiler.jl/issues/3 https://github.com/JuliaGPU/GPUCompiler.jl/issues/3
- deleted 5y ago
  [deleted]
- zorked 5y ago
  I only use Genie slightly to add a simple UI to my projects - I'm no web dev. Time to first request is as terrible as you would expect, but one can then dynamically iterate on a running interpreter by changing and adding pieces, and then it's not that bad as only the new bits need recompilation. I never tried the fancy to speed up compilation.
epolanski 5y ago
I'm curious about this project. Web development benefits a lot from near-instant recompilations, which I heard aren't possible with the Julia programming language as codebases grow. Is that still the case?
- clarkevans 5y ago
  Revise.jl is used to provide incremental runtime recompilation; and Genie now has Revise built-in.
  - abstracto 5y ago
    Probably worth staying revises limitations. Ie can't change structs, etc.
  - NeutralForest 5y ago
    Good to know, thanks for pointing it out.
clarkevans 5y ago
I'm curious if Julia's multiple dispatch could work naturally for server-side web development. In many content management problems, you often run into pages that are parameterized in multiple ways: the kind of data object being viewed, role based permission, and display capabilities, among others. In object oriented systems this often leads to double dispatch in one way or another. With Julia, it may instead result with an "just ahead of time" compile for each permutation; once that's done, page rendering should be snappy. One could use PackageCompiler to build an image with commonly needed permutations so that each computational unit's startup time is reasonable. During development, Revise.jl provides incremental updates to pages affected via code dependency graph.
- pjmlp 5y ago
  If it works for Common Lisp and C++ [0], it certainly does for Julia as well. [0] - In C++ you can emulate it via a mix of function overloading and template metaprogramming.
- freemint 5y ago
  > With Julia, it may instead result with an "ahead of time" compile for each permutation; This would be an odd approach, just have the first person who hits a new combination of function and type pay a few ms of compile time. That should work fine.
  - clarkevans 5y ago
    Oh dear. I meant to write "just ahead of time" (I corrected the post above). This is the Julia community's take on "just in time", but taking into account that there isn't an interpreted fallback. Exactly when a new type specialization is used, it is compiled then and there.
    - dTal 5y ago
      >there isn't an interpreted fallback julia --compile=no
NeutralForest 5y ago
Good stuff! I think it's important that analyses done in Julia can be presented in dashboard form with Julia tools as well. Having a one language stack is just that much less work.
white_dragon88 5y ago
This looks utterly bananas. I mean… why. Because variety is the spice of life? How are also those companies using this framework?
- FieryTransition 5y ago
  Imagine spending a lot of time on learning Julia, then imagine you want to do web dev, then imagine your favorite language is Julia, and you don't want to use time on learning something else? So why not I guess. From reading the landing page, I don't see a compelling reason to use it unless you are already in the ecosystem, which is fair.
- clarkevans 5y ago
  Frameworks are helpful when a community would benefit by having a bit less variety in how things are done. Even if it's sub-optimal for most individual use cases, having a standard way of building applications in a community makes it easier to construct and maintain projects.
- lf-non 5y ago
  why not ? People are free to use their time & energy to develop interesting things in whatever language/toolchain/techstack they like and share with the community. That is how ecosystems evolve, expand, and new ecosystems emerge. We don't want to be locked into incumbent solutions forever. Not everything needs to have corporate support or large scale use from day one. It doesn't even have to be useful at all on day one, or ever for that matter.
- qualudeheart 5y ago
  Single language stacks are simpler. For heavy duty webdev tasks you should use a real web language. For simple sites this looks fine.
- cmrdporcupine 5y ago
  Julia is a fairly elegant nice language and multiple dispatch makes a lot of sense in all sorts of places. I don't really get why Julia is pigeon-holed for numerics. I prefer its syntax to Python and would consider it for scripting applications where I use Julia now.
logankilpatrick 5y ago
Folks might be interested in: https://www.freecodecamp.org/news/how-to-build-web-apps-in-julia/ https://www.freecodecamp.org/news/how-to-build-web-apps-in-j...
svnpenn 5y ago
In regards to startup time, isn't Julia an order of magnitude slower than everything, even Ruby? Why would anyone want this?
- adgjlsfhk1 5y ago
  For this use-case, PackageCompiler (for dependencies) + Revise (for your code) fixes that.
  - blindseer 5y ago
    This is just not true. Even with PackageCompiler I was getting 30 seconds or more startup time for a medium size codebase with Genie.
    - abstracto 5y ago
      Also package compiler takes a long time to run...
- freemint 5y ago
  A webserver spends a minuscule amount of its time in startup.
  - freedomben 5y ago
    Startup time in a big Spring Boot or Rails project can easily be two minutes or more. Sadly I speak from experience.
nancarrow 5y ago
why? i like Julia but it's not a great general purpose language IMO.
blindseer 5y ago
I would caution someone interested in this to consider some of the ramifications of using Julia / Genie. When I last used it, it was a extremely painful development experience because of the time it took to for Julia to reload when "live reload" was enabled. Further more, many features like authetication were nowhere to be found. Also, startup time for the server was SO large that it just does not work on heroku, even with PackageCompiler. And forget about using AWS Lambdas or Cloudflare pages or something like that. IMO, Rust, Go, Javascript or even Python is a waaayyy better option for this. Kudos to the devs for continuing this work though. I'm sure it'll get there some day. If you absolutely want to use Julia because you have an existing Julia code, you can use HTTP.jl instead. It's more lightweight and you can build routes using that into your application.
- sundarurfriend 5y ago
  Adding a couple of responses from Genie's creator [1], when parent commenter mentioned these issues a few months ago, for visibility in case it's helpful to someone here: > startup time for the server was SO large that it just does not work on heroku Response: The solution (which granted should be maybe better documented) is to set the `EARLYBIND` env variable to `true` - this will allow Genie to bind to the port very early so Heroku will love it. > many features like authetication were nowhere to be found. Response: Genie has built in support for handling (encrypted) cookies and sessions and plugins like GenieAuthentication (or GenieAuthorisation) and SearchLight provide straightforward features. Have you used these? ---- With that said, I myself do think of Genie as (currently) something to be used when (a) you already have Julia code and need a web front-end for it, or (b) you want to experiment and possibly break things for the potential of expanding horizons. [1] https://news.ycombinator.com/item?id=29830003 https://news.ycombinator.com/item?id=29830003
  - blindseer 5y ago
    Thanks for sharing the information about EARLYBIND and GenieAuthentication. I will say that when I used EARLYBIND it still took too long for Heroku. That's not to mention that didn't solve any of the live development workflow issues. I looked into GenieAuthetication but it doesn't support SSO and a bunch of other features that other modern authetications libraries support.

207 points by msravi 5y ago

I started a new web project recently and I decided to either use Elixir/Phoenix or Julia/Genie. I wanted to use the later but two main issues led me ultimately to the former. 1) JIT compile times to first view were very slow — like 10 seconds, 2) Macro usage in Julia seems too common making the code ugly and difficult to read. e.g. @with_kw. What’s the point of an elegant language if you have to annotate it with obtuse macros at every turn?

melkael 5y ago

To Genie users, what is your workflow ? I love Julia for scientific computing but the compilation time is a bit of an issue. I would imagine I would need to compile a lot more when doing web dev. How do you work around this ? If there is a solution to this, Genie would be awesome as I could directly put into production my Julia models without using a third party language.

huijzer 5y ago
Have you tried PackageCompiler.jl?
- abstracto 5y ago
  Honestly, this is an area where the op would be far better served by using go/rust/c++ or another backend language. Can launch a server with microsecond latency, and if it grows get good elastic behavior. No awkward route precompilation or thirty minute build times... Julia sure is neat, but this isn't an area where it's strong.
  - 0des 5y ago
    I do webdev in Go, and freely recompile as it takes a second or less on my hardware which consists of a rock, a rubber band and two sesame seeds on 20x2 liquid crystal. Its no struggle to recompile 100x in a day just iterating.
  - jjtheblunt 5y ago
    microsecond latency? on what exact hardware are you seeing that?
    - abstracto 5y ago
      That was a somewhat hyperbolic statement. Make an application in node, genie, and rusts actix. Run them and see how fast it takes for the cli to load each of them. Feel free to package compiler the Julia one if you want. It's a good experiment with visual results.
    - mirekrusin 5y ago
      Must be new M1 Ultra or something.
      - abstracto 5y ago
        P50 for basic routes using other web frameworks can be sub 0.5 ms. See the techempower benchmarks. These benchmarks are somewhat flawed, but I'd be surprised if genie was faster then flask all around, and flask is widely considered expensively slow.
        jjtheblunt 5y ago
        0.5 ms is 500 times longer than the comment above, though
andyferris 5y ago
I haven’t used Genie (though after this post I am tempted!) but I imagine compilation overhead would only occur once (e.g. the first time each route is used) and that you could pretty easily use precompile statements and/or PackageCompiler.jl to do the compilation before the service goes live. After that point, Julia should have produced fast code and provides a good runtime for these kinds of async/multithreaded applications with built-in M:N scheduling (perhaps a concurrent/multithreaded GC might help reduce tail latency but I imagine it should already be competitive with everything but well-written Rust/C++/C).
alhirzel 5y ago
Revise.jl[1] is very helpful for development, as it allows for seamless recompilation without restarts. Cold start can also take a long time, so sometimes it helps to add options[2] such as -O0 when developing. Two paths forward for AOT compilation are PackageCompiler.jl[3] mentioned already in this thread (things like RelocatableFolders.jl[4] come in handy) and the possibility of GPUCompiler.jl[5] some day. [1]: https://github.com/timholy/Revise.jl https://github.com/timholy/Revise.jl [2]: https://docs.julialang.org/en/v1/manual/command-line-options/ https://docs.julialang.org/en/v1/manual/command-line-options... [3]: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl [4]: https://github.com/JuliaPackaging/RelocatableFolders.jl https://github.com/JuliaPackaging/RelocatableFolders.jl [5]: https://github.com/JuliaGPU/GPUCompiler.jl/issues/3 https://github.com/JuliaGPU/GPUCompiler.jl/issues/3
deleted 5y ago
[deleted]
zorked 5y ago
I only use Genie slightly to add a simple UI to my projects - I'm no web dev. Time to first request is as terrible as you would expect, but one can then dynamically iterate on a running interpreter by changing and adding pieces, and then it's not that bad as only the new bits need recompilation. I never tried the fancy to speed up compilation.

epolanski 5y ago

I'm curious about this project. Web development benefits a lot from near-instant recompilations, which I heard aren't possible with the Julia programming language as codebases grow. Is that still the case?

clarkevans 5y ago
Revise.jl is used to provide incremental runtime recompilation; and Genie now has Revise built-in.
- abstracto 5y ago
  Probably worth staying revises limitations. Ie can't change structs, etc.
- NeutralForest 5y ago
  Good to know, thanks for pointing it out.

clarkevans 5y ago

I'm curious if Julia's multiple dispatch could work naturally for server-side web development. In many content management problems, you often run into pages that are parameterized in multiple ways: the kind of data object being viewed, role based permission, and display capabilities, among others. In object oriented systems this often leads to double dispatch in one way or another. With Julia, it may instead result with an "just ahead of time" compile for each permutation; once that's done, page rendering should be snappy. One could use PackageCompiler to build an image with commonly needed permutations so that each computational unit's startup time is reasonable. During development, Revise.jl provides incremental updates to pages affected via code dependency graph.

pjmlp 5y ago
If it works for Common Lisp and C++ [0], it certainly does for Julia as well. [0] - In C++ you can emulate it via a mix of function overloading and template metaprogramming.
freemint 5y ago
> With Julia, it may instead result with an "ahead of time" compile for each permutation; This would be an odd approach, just have the first person who hits a new combination of function and type pay a few ms of compile time. That should work fine.
- clarkevans 5y ago
  Oh dear. I meant to write "just ahead of time" (I corrected the post above). This is the Julia community's take on "just in time", but taking into account that there isn't an interpreted fallback. Exactly when a new type specialization is used, it is compiled then and there.
  - dTal 5y ago
    >there isn't an interpreted fallback julia --compile=no

NeutralForest 5y ago

Good stuff! I think it's important that analyses done in Julia can be presented in dashboard form with Julia tools as well. Having a one language stack is just that much less work.

white_dragon88 5y ago

This looks utterly bananas. I mean… why. Because variety is the spice of life? How are also those companies using this framework?

FieryTransition 5y ago
Imagine spending a lot of time on learning Julia, then imagine you want to do web dev, then imagine your favorite language is Julia, and you don't want to use time on learning something else? So why not I guess. From reading the landing page, I don't see a compelling reason to use it unless you are already in the ecosystem, which is fair.
clarkevans 5y ago
Frameworks are helpful when a community would benefit by having a bit less variety in how things are done. Even if it's sub-optimal for most individual use cases, having a standard way of building applications in a community makes it easier to construct and maintain projects.
lf-non 5y ago
why not ? People are free to use their time & energy to develop interesting things in whatever language/toolchain/techstack they like and share with the community. That is how ecosystems evolve, expand, and new ecosystems emerge. We don't want to be locked into incumbent solutions forever. Not everything needs to have corporate support or large scale use from day one. It doesn't even have to be useful at all on day one, or ever for that matter.
qualudeheart 5y ago
Single language stacks are simpler. For heavy duty webdev tasks you should use a real web language. For simple sites this looks fine.
cmrdporcupine 5y ago
Julia is a fairly elegant nice language and multiple dispatch makes a lot of sense in all sorts of places. I don't really get why Julia is pigeon-holed for numerics. I prefer its syntax to Python and would consider it for scripting applications where I use Julia now.

logankilpatrick 5y ago

Folks might be interested in: https://www.freecodecamp.org/news/how-to-build-web-apps-in-julia/ https://www.freecodecamp.org/news/how-to-build-web-apps-in-j...

svnpenn 5y ago

In regards to startup time, isn't Julia an order of magnitude slower than everything, even Ruby? Why would anyone want this?

adgjlsfhk1 5y ago
For this use-case, PackageCompiler (for dependencies) + Revise (for your code) fixes that.
- blindseer 5y ago
  This is just not true. Even with PackageCompiler I was getting 30 seconds or more startup time for a medium size codebase with Genie.
  - abstracto 5y ago
    Also package compiler takes a long time to run...
freemint 5y ago
A webserver spends a minuscule amount of its time in startup.
- freedomben 5y ago
  Startup time in a big Spring Boot or Rails project can easily be two minutes or more. Sadly I speak from experience.

nancarrow 5y ago

why? i like Julia but it's not a great general purpose language IMO.

blindseer 5y ago

I would caution someone interested in this to consider some of the ramifications of using Julia / Genie. When I last used it, it was a extremely painful development experience because of the time it took to for Julia to reload when "live reload" was enabled. Further more, many features like authetication were nowhere to be found. Also, startup time for the server was SO large that it just does not work on heroku, even with PackageCompiler. And forget about using AWS Lambdas or Cloudflare pages or something like that. IMO, Rust, Go, Javascript or even Python is a waaayyy better option for this. Kudos to the devs for continuing this work though. I'm sure it'll get there some day. If you absolutely want to use Julia because you have an existing Julia code, you can use HTTP.jl instead. It's more lightweight and you can build routes using that into your application.

sundarurfriend 5y ago
Adding a couple of responses from Genie's creator [1], when parent commenter mentioned these issues a few months ago, for visibility in case it's helpful to someone here: > startup time for the server was SO large that it just does not work on heroku Response: The solution (which granted should be maybe better documented) is to set the `EARLYBIND` env variable to `true` - this will allow Genie to bind to the port very early so Heroku will love it. > many features like authetication were nowhere to be found. Response: Genie has built in support for handling (encrypted) cookies and sessions and plugins like GenieAuthentication (or GenieAuthorisation) and SearchLight provide straightforward features. Have you used these? ---- With that said, I myself do think of Genie as (currently) something to be used when (a) you already have Julia code and need a web front-end for it, or (b) you want to experiment and possibly break things for the potential of expanding horizons. [1] https://news.ycombinator.com/item?id=29830003 https://news.ycombinator.com/item?id=29830003
- blindseer 5y ago
  Thanks for sharing the information about EARLYBIND and GenieAuthentication. I will say that when I used EARLYBIND it still took too long for Heroku. That's not to mention that didn't solve any of the live development workflow issues. I looked into GenieAuthetication but it doesn't support SSO and a bunch of other features that other modern authetications libraries support.