6 ms·

Lapis – A Lua, Moonscript Framework built on OpenResty

164 points by statenjason 12y ago

pspeter3 12y ago
I was just looking at this yesterday. It seems awesome. My one question is how you do forking dependency graphs since everything is non blocking but looks blocking. Eg. I have takes A, B and C. C depends on A and B so I want to do A and B concurrently but then wait on them for C.
- fab13n 12y ago
  I don't know how Lapis does it, but it's not difficult to write a coroutine-cloning function in C for Lua: http://lua-users.org/lists/lua-l/2006-01/threads.html#00650 http://lua-users.org/lists/lua-l/2006-01/threads.html#00650, cf. thread "coroutine.clone and stateful web applications". (Coroutines are essentially one-shot continuations; if you can clone them, you can bookmark execution states at will. Look at Lua's upvalue semantics to anticipate what will happen to variables shared by multiple coroutine clones).
- justincormack 12y ago
  You can use https://github.com/openresty/lua-nginx-module#ngxlocationcapture_multi https://github.com/openresty/lua-nginx-module#ngxlocationcap... to do multiple parallel requests
  - pspeter3 12y ago
    Interesting. Thank you!
xfalcox 12y ago
I just need to say that OpenResty is great, and the mailing list is awesome. After a topic here in HN saying about doing auth in lua+nginx, we successfully migrated our auth from inside the applications (all java) and now it's a 100 LOC of Lua, easy and decoupled from the application.
- justincormack 12y ago
  Doing auth in OpenResty is a good gateway into using it for more stuff, it is really easy to do whatever auth you need without your application having to know.
  - srean 12y ago
    Are there any Lua libraries targeting openresty that makes it easy to use integrate with oauth2 services of Google, facebook, 4square etc ? Since everyone does oauth differently it would be nice to have a go to library for this.
  - xfalcox 12y ago
    Yeah, a coworker got excited and did the identicons in Lua too, with openresty generating the png on the first visit of the user. We're a intranet app, in a company with 120k workers, some with intranet only access.
- Keats 12y ago
  Do you have the link for that topic ?
izietto 12y ago
This is great! I'm curious about CPU/RAM usage too
- fasteo 12y ago
  An order of magnitude less than any framework you have in your mind
- jbeja 12y ago
  Is so minimal that is not interesting.
networked 12y ago
Is Unicode ever a problem when you do real-life web development in Lua? Lapis looks appealing otherwise but the lack of a built-in Unicode string type in Lua makes me somewhat wary.
- justincormack 12y ago
  There are unicode libraries for Lua should you need to do unicode manipulation. In real life though I rarely have needed much. What functions do you need?
  - networked 12y ago
    What I had in mind was Unicode string normalization, comparison and sorting as well as finding the length of strings correctly when they contain combining characters, ligatures and such. What is the library of choice for this? For anyone curious about the potential issues there is a good article about the flaws in JavaScript's handling of Unicode at http://mathiasbynens.be/notes/javascript-unicode#accounting-for-lookalikes http://mathiasbynens.be/notes/javascript-unicode#accounting-.... HN discussion: https://news.ycombinator.com/item?id=7221362 https://news.ycombinator.com/item?id=7221362.
    - meric 12y ago
      Yes, you'll need a unicode library for that. Lua's length operator count bytes not characters.
- plorkyeran 12y ago
  Not particularly. Lua doesn't do anything to make storing utf-8 in strings difficult. You have to bring your own Unicode library for doing anything nontrivial, but that's true of everything in Lua. Using ICU from Lua isn't substantially different from using it from any other lanugage.
  - shanth 12y ago
    And they said, "Lua doesn't come with Batteries included"
erikcw 12y ago
We've been using it in production for about 6 months as the endpoint for an analytics system. We went from 5 medium EC2 app servers down to 1 micro instance in testing for our load at the time (we have an HA setup in production). Blazingly fast. Less than 100 LOC for the service. The author of Lapis is also very responsive to PRs and bug reports.
- mojoe 12y ago
  What framework were you using before that required 5 medium ec2 instances? I'd really appreciate knowing what you were comparing Lapis to.
  - erikcw 12y ago
    We use Django behind gunicorn for most of the stack. Not that Django is slow (we still happily use it for 99% of our public facing system) -- it's just that OpenResty is so fast[1]. Besides the fact that OpenResty is built on C/Lua, Nginx's evented architecture makes it scale concurrently really well. We use Gunicorn in a multiprocessing configuration for the rest of the stack -- so each worker consumes much more RAM than the equivalent evented configuration. We were toying with the idea of implementing the service with Tornado (also evented, but in Python) -- but OpenResty/Lapis has been so great for our use case that it's tough to justify the test at this point. [1] http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/
fasteo 12y ago
As a sidenote, openresty (nginx+lua) is probably one of the most underrated technologies we have available nowadays. In my experience, every webapp can offload a ton of work(1) to openresty. The lua code is usually much simpler that the equivalent code in the app. (1) think caching, auth/acl, security filters, some of the stats, some of the logs, etc
- vanessarp 12y ago
  +1 At apitools.com we use openresty as well. We chose lua (over javascript) because it's super fast, and because it has good sandboxing/ isolation properties which make it safe to handle arbitrary middleware.
leafo 12y ago
Hello, I create lapis. I'm actually on the verge of releasing a new huge update. New features include: * first class lua support * built in support for a lua templating language: https://github.com/leafo/etlua https://github.com/leafo/etlua * a new postgres driver written in pure lua, should play nicer with luajit and and be available in more stages of nginx request cycle: https://github.com/leafo/pgmoon https://github.com/leafo/pgmoon * lots of other smaller changes All of this code is in master I just need to write all the documentation for it, sigh. The biggest site I've created with lapis is itch.io: http://itch.io http://itch.io it's currently 43,209 lines of code (excluding libraries). I run it on a really crappy 10 dollar a month vps, memory and cpu usage is tiny. I've handled well over a million requests per day in the past with no issues. The app communicates with a lot of third party payment services over http in the server, because of the non-blocking nature of openresty throughput is not affected when these services are slow (eg. paypal). Another interesting lapis project is MoonRocks: http://rocks.moonscript.org/ http://rocks.moonscript.org/ it's a public lua module hosting site. It's opensource so you can check out what a developed lapis application looks like: https://github.com/leafo/moonrocks-site https://github.com/leafo/moonrocks-site Thanks for checking out lapis!
- statenjason 12y ago
  Thanks for the great documentation. It makes lapis stand out from just some weekend project.
- egeozcan 12y ago
  Awesome work! By the way, does pgmoon support arrays in postgres?
  - leafo 12y ago
    I was just experimenting this today, select queries will be able to deserialize array results into lua tables automatically. Same for json values as well.
- pspeter3 12y ago
  With all of these projects, I'm unsure of the relationship between Lua and MoonScript. Is it similar to the relationship between JavaScript and CoffeeScript? Also, does it matter which one is used ultimately?
reinhardt 12y ago
Performance aside, does anyone have experience with how Lapis/OpenResty compare to, say, Django or Rails in terms of stability, included batteries, 3rd party libraries and overall developer productivity and production readiness? I guess this overlaps a lot with the respective question for lua vs python/ruby ecosystems.
- xt 12y ago
  The community for Lua in general, and specifically for Lapis is much smaller.
- shuzchen 12y ago
  As a person that uses Django to make a living (at my full-time job, and for my freelance work) and who has tinkered with a lapis/openresty site on my spare time (http://graphcake.com http://graphcake.com), I'd say productivity is on the side of Django. There are too many included batteries (form validation, django ORM more powerful than lapis ORM, staticfiles) and high quality 3rd party libraries (task queue (celery, python-rq), image processing, caching) that I'd hate to reimplement. That said, performance is totally on the side of lapis/openresty. The performance of my side project totally depends on how fast postgres responds to the needed queries.
avargas 12y ago
Lua within Ngnix is amazingly fast. I developed a custom solution for a client last year that basically did bot detection (similar to CloudFlare). It's amazing what you can do within nginx without touching an external fastcgi server or proxy.
dubcanada 12y ago
I've been experimenting with this with a fairly large website. The lack of MySQL support is the only thing keeping me away.
- taf2 12y ago
  You migh look at this nginx module wiki.nginx.org/HttpDrizzleModule. It should be possible to access mysql this way with lua as well
xt 12y ago
If someone is eager to try Lapis I have a Dockerfile available here: https://github.com/torhve/lapis-docker https://github.com/torhve/lapis-docker it includes a readme for getting started with Openresty in general.
- _mikz 12y ago
  For plain openresty you can try https://github.com/3scale/docker-openresty https://github.com/3scale/docker-openresty It does not have fancy readme, but has batteries included. Luarocks works, supervisor is set up, cron is there...
frik 12y ago
Interesting, and I found it in the techempower benchmark too: http://www.techempower.com/benchmarks/#section=data-r9&hw=i7&test=fortune http://www.techempower.com/benchmarks/#section=data-r9&hw=i7...
- pspeter3 12y ago
  I don't get how it does so badly in the queries and writes section when compared to node.js while the serving plaintext and json is so high.
deleted 12y ago
[deleted]
thinxer 12y ago
I don't understand. How is Lua's performance compared to NodeJS, or compiled languages such as Java or Go ? Does it make sense to stuck everything into a (used to be ) reverse proxy server ?
- otikik 12y ago
  > How is Lua's performance compared to NodeJS, or compiled languages such as Java or Go This might help: http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ > Does it make sense to stuck everything into a (used to be ) reverse proxy server ? Just like putting stuff into Apache or whatever other webserver you like.
martijn_himself 12y ago
Absolutely amazing work. I have been meaning to give this a go for ages.

164 points by statenjason 12y ago

I was just looking at this yesterday. It seems awesome. My one question is how you do forking dependency graphs since everything is non blocking but looks blocking. Eg. I have takes A, B and C. C depends on A and B so I want to do A and B concurrently but then wait on them for C.

fab13n 12y ago
I don't know how Lapis does it, but it's not difficult to write a coroutine-cloning function in C for Lua: http://lua-users.org/lists/lua-l/2006-01/threads.html#00650 http://lua-users.org/lists/lua-l/2006-01/threads.html#00650, cf. thread "coroutine.clone and stateful web applications". (Coroutines are essentially one-shot continuations; if you can clone them, you can bookmark execution states at will. Look at Lua's upvalue semantics to anticipate what will happen to variables shared by multiple coroutine clones).
justincormack 12y ago
You can use https://github.com/openresty/lua-nginx-module#ngxlocationcapture_multi https://github.com/openresty/lua-nginx-module#ngxlocationcap... to do multiple parallel requests
- pspeter3 12y ago
  Interesting. Thank you!

xfalcox 12y ago

I just need to say that OpenResty is great, and the mailing list is awesome. After a topic here in HN saying about doing auth in lua+nginx, we successfully migrated our auth from inside the applications (all java) and now it's a 100 LOC of Lua, easy and decoupled from the application.

justincormack 12y ago
Doing auth in OpenResty is a good gateway into using it for more stuff, it is really easy to do whatever auth you need without your application having to know.
- srean 12y ago
  Are there any Lua libraries targeting openresty that makes it easy to use integrate with oauth2 services of Google, facebook, 4square etc ? Since everyone does oauth differently it would be nice to have a go to library for this.
- xfalcox 12y ago
  Yeah, a coworker got excited and did the identicons in Lua too, with openresty generating the png on the first visit of the user. We're a intranet app, in a company with 120k workers, some with intranet only access.
Keats 12y ago
Do you have the link for that topic ?

izietto 12y ago

This is great! I'm curious about CPU/RAM usage too

fasteo 12y ago
An order of magnitude less than any framework you have in your mind
jbeja 12y ago
Is so minimal that is not interesting.

networked 12y ago

Is Unicode ever a problem when you do real-life web development in Lua? Lapis looks appealing otherwise but the lack of a built-in Unicode string type in Lua makes me somewhat wary.

justincormack 12y ago
There are unicode libraries for Lua should you need to do unicode manipulation. In real life though I rarely have needed much. What functions do you need?
- networked 12y ago
  What I had in mind was Unicode string normalization, comparison and sorting as well as finding the length of strings correctly when they contain combining characters, ligatures and such. What is the library of choice for this? For anyone curious about the potential issues there is a good article about the flaws in JavaScript's handling of Unicode at http://mathiasbynens.be/notes/javascript-unicode#accounting-for-lookalikes http://mathiasbynens.be/notes/javascript-unicode#accounting-.... HN discussion: https://news.ycombinator.com/item?id=7221362 https://news.ycombinator.com/item?id=7221362.
  - meric 12y ago
    Yes, you'll need a unicode library for that. Lua's length operator count bytes not characters.
plorkyeran 12y ago
Not particularly. Lua doesn't do anything to make storing utf-8 in strings difficult. You have to bring your own Unicode library for doing anything nontrivial, but that's true of everything in Lua. Using ICU from Lua isn't substantially different from using it from any other lanugage.
- shanth 12y ago
  And they said, "Lua doesn't come with Batteries included"

erikcw 12y ago

We've been using it in production for about 6 months as the endpoint for an analytics system. We went from 5 medium EC2 app servers down to 1 micro instance in testing for our load at the time (we have an HA setup in production). Blazingly fast. Less than 100 LOC for the service. The author of Lapis is also very responsive to PRs and bug reports.

mojoe 12y ago
What framework were you using before that required 5 medium ec2 instances? I'd really appreciate knowing what you were comparing Lapis to.
- erikcw 12y ago
  We use Django behind gunicorn for most of the stack. Not that Django is slow (we still happily use it for 99% of our public facing system) -- it's just that OpenResty is so fast[1]. Besides the fact that OpenResty is built on C/Lua, Nginx's evented architecture makes it scale concurrently really well. We use Gunicorn in a multiprocessing configuration for the rest of the stack -- so each worker consumes much more RAM than the equivalent evented configuration. We were toying with the idea of implementing the service with Tornado (also evented, but in Python) -- but OpenResty/Lapis has been so great for our use case that it's tough to justify the test at this point. [1] http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/

fasteo 12y ago

As a sidenote, openresty (nginx+lua) is probably one of the most underrated technologies we have available nowadays. In my experience, every webapp can offload a ton of work(1) to openresty. The lua code is usually much simpler that the equivalent code in the app. (1) think caching, auth/acl, security filters, some of the stats, some of the logs, etc

vanessarp 12y ago
+1 At apitools.com we use openresty as well. We chose lua (over javascript) because it's super fast, and because it has good sandboxing/ isolation properties which make it safe to handle arbitrary middleware.

leafo 12y ago

Hello, I create lapis. I'm actually on the verge of releasing a new huge update. New features include: * first class lua support * built in support for a lua templating language: https://github.com/leafo/etlua https://github.com/leafo/etlua * a new postgres driver written in pure lua, should play nicer with luajit and and be available in more stages of nginx request cycle: https://github.com/leafo/pgmoon https://github.com/leafo/pgmoon * lots of other smaller changes All of this code is in master I just need to write all the documentation for it, sigh. The biggest site I've created with lapis is itch.io: http://itch.io http://itch.io it's currently 43,209 lines of code (excluding libraries). I run it on a really crappy 10 dollar a month vps, memory and cpu usage is tiny. I've handled well over a million requests per day in the past with no issues. The app communicates with a lot of third party payment services over http in the server, because of the non-blocking nature of openresty throughput is not affected when these services are slow (eg. paypal). Another interesting lapis project is MoonRocks: http://rocks.moonscript.org/ http://rocks.moonscript.org/ it's a public lua module hosting site. It's opensource so you can check out what a developed lapis application looks like: https://github.com/leafo/moonrocks-site https://github.com/leafo/moonrocks-site Thanks for checking out lapis!

statenjason 12y ago
Thanks for the great documentation. It makes lapis stand out from just some weekend project.
egeozcan 12y ago
Awesome work! By the way, does pgmoon support arrays in postgres?
- leafo 12y ago
  I was just experimenting this today, select queries will be able to deserialize array results into lua tables automatically. Same for json values as well.
pspeter3 12y ago
With all of these projects, I'm unsure of the relationship between Lua and MoonScript. Is it similar to the relationship between JavaScript and CoffeeScript? Also, does it matter which one is used ultimately?

reinhardt 12y ago

Performance aside, does anyone have experience with how Lapis/OpenResty compare to, say, Django or Rails in terms of stability, included batteries, 3rd party libraries and overall developer productivity and production readiness? I guess this overlaps a lot with the respective question for lua vs python/ruby ecosystems.

xt 12y ago
The community for Lua in general, and specifically for Lapis is much smaller.
shuzchen 12y ago
As a person that uses Django to make a living (at my full-time job, and for my freelance work) and who has tinkered with a lapis/openresty site on my spare time (http://graphcake.com http://graphcake.com), I'd say productivity is on the side of Django. There are too many included batteries (form validation, django ORM more powerful than lapis ORM, staticfiles) and high quality 3rd party libraries (task queue (celery, python-rq), image processing, caching) that I'd hate to reimplement. That said, performance is totally on the side of lapis/openresty. The performance of my side project totally depends on how fast postgres responds to the needed queries.

avargas 12y ago

Lua within Ngnix is amazingly fast. I developed a custom solution for a client last year that basically did bot detection (similar to CloudFlare). It's amazing what you can do within nginx without touching an external fastcgi server or proxy.

dubcanada 12y ago

I've been experimenting with this with a fairly large website. The lack of MySQL support is the only thing keeping me away.

taf2 12y ago
You migh look at this nginx module wiki.nginx.org/HttpDrizzleModule. It should be possible to access mysql this way with lua as well

xt 12y ago

If someone is eager to try Lapis I have a Dockerfile available here: https://github.com/torhve/lapis-docker https://github.com/torhve/lapis-docker it includes a readme for getting started with Openresty in general.

_mikz 12y ago
For plain openresty you can try https://github.com/3scale/docker-openresty https://github.com/3scale/docker-openresty It does not have fancy readme, but has batteries included. Luarocks works, supervisor is set up, cron is there...

frik 12y ago

Interesting, and I found it in the techempower benchmark too: http://www.techempower.com/benchmarks/#section=data-r9&hw=i7&test=fortune http://www.techempower.com/benchmarks/#section=data-r9&hw=i7...

pspeter3 12y ago
I don't get how it does so badly in the queries and writes section when compared to node.js while the serving plaintext and json is so high.

deleted 12y ago

[deleted]

thinxer 12y ago

I don't understand. How is Lua's performance compared to NodeJS, or compiled languages such as Java or Go ? Does it make sense to stuck everything into a (used to be ) reverse proxy server ?

otikik 12y ago
> How is Lua's performance compared to NodeJS, or compiled languages such as Java or Go This might help: http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ > Does it make sense to stuck everything into a (used to be ) reverse proxy server ? Just like putting stuff into Apache or whatever other webserver you like.

martijn_himself 12y ago

Absolutely amazing work. I have been meaning to give this a go for ages.