4 ms·

While we're on the subject, does anyone here have experience using Clojure with GAE? I'm considering that configuration for a useful-but-not-mission-critical ap

by dhess 17y ago

While we're on the subject, does anyone here have experience using Clojure with GAE? I'm considering that configuration for a useful-but-not-mission-critical application, and wondering if Clojure on GAE is robust enough for that.

DTrejo 17y ago
Yes, someone please tell us about their experiences?
jeandenis 17y ago
I've been using Clojure w/ GAE to build an enterprise knowledge-base webservice for about 3 months now. It's my first serious project with Clojure on the GAE although I've been playing around with it since last summer. (1) On GAE you don't get to use some of the good parts of Clojure because you can't spawn any threads. So the rationale for Clojure has to be something else than agents and the Clojure concurrency model. I'd say use Clojure if you want functional programming and all of the Lispy goodness: code as data, macros... For me Lispyness is a huge + so it was a no brainer. (2) One of the things I really like about traditional Clojure web-development (using Ring or Compojure) is that if you set things up properly you can experiment with and patch a live system using the REPL. I haven't been able to set that up just yet on the GAE. On your development server using the GAE jars, however, you can do it (see http://www.hackers-with-attitude.com/2009/08/intertactive-programming-with-clojure.html http://www.hackers-with-attitude.com/2009/08/intertactive-pr... for some guidance). There is nothing like being a C-c C-c away from updating your development server app :) I can't emphasize enough taking the time to get Slime, your GAE development server, lein, and Clojure well-integrated. Your development cycle will make good ol' Java GAE development using the Eclipse plug-in (which I used to think was pretty sweet) feel really really clunky. (See also http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html http://elhumidor.blogspot.com/2009/04/clojure-on-google-appe... for some pointers on setting up Clojure on GAE) (3) GAE's datastore works extremely well with Clojure because it's schema-less. You can store arbitrary hashmaps and structmaps in GAE with zero effort -- it's a match made in heaven. There are a couple of branches of appengine-clj (http://github.com/duelinmarkers/appengine-clj http://github.com/duelinmarkers/appengine-clj is the original, http://github.com/r0man/appengine-clj http://github.com/r0man/appengine-clj is a pretty good branch) that you can look at to get a feel what I mean. (4) With GAE's transaction support and automatic index generation (it's good enough most of the time), you can dynamically create types (kinds), put them in the datastore, and query on them without having to do any of the traditional NoSQL stuff (thinking queries first, denormalizing ahead of time, etc.). It's very sweet. I've done some Clojure+Cassandra and Clojure+GAE is a lot more painless (as long as you can live with the other limitations of the GAE) -- although things are getting much easier in Cassandra world with Hector + automatic index generators. (5) In terms of performance and robustness, nothing about Clojure's held me up. But since I haven't launched yet, that doesn't mean anything. I don't know of any "high-traffic" Clojure+GAE apps out there -- if there are any, I'd love to get some numbers on costs (CPU, especially) to see if there is a significant difference with similar java projects. (Actually, the guys behind https://the-deadline.appspot.com https://the-deadline.appspot.com, see http://www.slideshare.net/smartrevolution/how-a-clojure-pet-project-turned-into-a-fullblown-cloudcomputing-webapp http://www.slideshare.net/smartrevolution/how-a-clojure-pet-..., just launched three days ago -- you may want to shoot them an email, they seem nice enough) My guess is that this is a non-issue. Clojure holds up really well against Java and JVM languages like Scala in terms of performance (especially if you give type-hints), and I'd expect the same thing to be true on GAE. (6) General Clojure limitation with web development: Clojure web frameworks are light. You don't get to start with anything as nice as Scala's Lift. You'll have to work and build your own tools -- the good thing is that Clojure is a blast and macros will, in the longer term, allow you to do much cooler stuff. Compojure is a great light framework to start with. Also look at Enlive for templating. On the flip-side Clojure immutable nature works very naturally with request / response. (7) For the OP: before you get on GAE, spend some time reading about its limitations: - exploding indexes (makes implementing text search very tough -- there is some irony in there I think) - no threads (it kills me, I want to use pmap) - not real message queues. GAE is a pull/request-based system, not a push system. It is, however, getting much better (tasks, comet on the coming features list). Still, if you are building a real-time system that needs constant background processing, I'd think twice right now. - if GAE goes down, you're down until they fix things, and will be totally helpless. - some of the GAE goodness/crack means you're locked in (e.g., index building -- to move away you'll have to build equivalent facilities for whatever DB you move to). Although you can protect yourself by being smart in how you abstract away the GAE. (8) Can't think of anything else. If there is interest, I have been thinking about writing some blog posts about my experience setting up a Clojure+GAE development environment, working with the data store, working the the GAE API, etc. Let me know here and I'll get a first post out of the way by week's end.
- fuzzythinker 17y ago
  This is the most detailed and helpful post I read on HN. I wish I can up vote it multiple times.
  - jeandenis 17y ago
    Thanks, I appreciate it. First post for me after being a longtime reader so it's nice to know it was helpful to some.
- smartrevolution 17y ago
  Hi. I am from above mentioned TheDeadline-Team! We can definitly recommend Google App Engine. The SDK is really cool and you don't have to care about managing servers. Scaling is easy. Its (almost) automatic. The datastore is a natural fit for Clojure datastructures, but you have to learn how to structure your data for a distributed key-value store: When you structure your data, you have to think about what has to be in the same transaction (entity-group) and at the same time you have to think about reducing data-dependencies to minimize contention. But when you have lots of data, this is what you have to do anyway. For me, it is much more fun than using an RDBMS. (Read this: http://code.google.com/intl/de-DE/appengine/docs/python/datastore/transactions.html http://code.google.com/intl/de-DE/appengine/docs/python/data...) We did the whole system in Clojure, so you really can write large systems using Clojure/GAE. It works! (Although the UI is simple, TheDeadline is a complex piece of software e.g. containing a self-written rule-based expert-system, a production system, very similar to LISA http://lisa.sourceforge.net/. http://lisa.sourceforge.net/.) The functional paradigm means less code and makes it easier to build powerful abstractions. You don't have to write lots of boiler-plate code like in Java and Clojure is more concises and much more powerful than Python. (Stop! I don't want to start a programming language discussion now! :) ). We will soon post some Clojure examples and some free software libraries for GAE. If you are interested, just follow our blog H.W.A. (http://www.hackers-with-attitude.com http://www.hackers-with-attitude.com) and/or try out http://the-deadline.appspot.com http://the-deadline.appspot.com to get the "feeling". ;)
- frognibble 17y ago
  The one problem with using Clojure on App Engine is that the time for a "loading request" is at least 5 seconds. It makes your app look bad for the random user stuck on the loading request.

by dhess 17y ago

DTrejo 17y ago

Yes, someone please tell us about their experiences?

jeandenis 17y ago

I've been using Clojure w/ GAE to build an enterprise knowledge-base webservice for about 3 months now. It's my first serious project with Clojure on the GAE although I've been playing around with it since last summer. (1) On GAE you don't get to use some of the good parts of Clojure because you can't spawn any threads. So the rationale for Clojure has to be something else than agents and the Clojure concurrency model. I'd say use Clojure if you want functional programming and all of the Lispy goodness: code as data, macros... For me Lispyness is a huge + so it was a no brainer. (2) One of the things I really like about traditional Clojure web-development (using Ring or Compojure) is that if you set things up properly you can experiment with and patch a live system using the REPL. I haven't been able to set that up just yet on the GAE. On your development server using the GAE jars, however, you can do it (see http://www.hackers-with-attitude.com/2009/08/intertactive-programming-with-clojure.html http://www.hackers-with-attitude.com/2009/08/intertactive-pr... for some guidance). There is nothing like being a C-c C-c away from updating your development server app :) I can't emphasize enough taking the time to get Slime, your GAE development server, lein, and Clojure well-integrated. Your development cycle will make good ol' Java GAE development using the Eclipse plug-in (which I used to think was pretty sweet) feel really really clunky. (See also http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html http://elhumidor.blogspot.com/2009/04/clojure-on-google-appe... for some pointers on setting up Clojure on GAE) (3) GAE's datastore works extremely well with Clojure because it's schema-less. You can store arbitrary hashmaps and structmaps in GAE with zero effort -- it's a match made in heaven. There are a couple of branches of appengine-clj (http://github.com/duelinmarkers/appengine-clj http://github.com/duelinmarkers/appengine-clj is the original, http://github.com/r0man/appengine-clj http://github.com/r0man/appengine-clj is a pretty good branch) that you can look at to get a feel what I mean. (4) With GAE's transaction support and automatic index generation (it's good enough most of the time), you can dynamically create types (kinds), put them in the datastore, and query on them without having to do any of the traditional NoSQL stuff (thinking queries first, denormalizing ahead of time, etc.). It's very sweet. I've done some Clojure+Cassandra and Clojure+GAE is a lot more painless (as long as you can live with the other limitations of the GAE) -- although things are getting much easier in Cassandra world with Hector + automatic index generators. (5) In terms of performance and robustness, nothing about Clojure's held me up. But since I haven't launched yet, that doesn't mean anything. I don't know of any "high-traffic" Clojure+GAE apps out there -- if there are any, I'd love to get some numbers on costs (CPU, especially) to see if there is a significant difference with similar java projects. (Actually, the guys behind https://the-deadline.appspot.com https://the-deadline.appspot.com, see http://www.slideshare.net/smartrevolution/how-a-clojure-pet-project-turned-into-a-fullblown-cloudcomputing-webapp http://www.slideshare.net/smartrevolution/how-a-clojure-pet-..., just launched three days ago -- you may want to shoot them an email, they seem nice enough) My guess is that this is a non-issue. Clojure holds up really well against Java and JVM languages like Scala in terms of performance (especially if you give type-hints), and I'd expect the same thing to be true on GAE. (6) General Clojure limitation with web development: Clojure web frameworks are light. You don't get to start with anything as nice as Scala's Lift. You'll have to work and build your own tools -- the good thing is that Clojure is a blast and macros will, in the longer term, allow you to do much cooler stuff. Compojure is a great light framework to start with. Also look at Enlive for templating. On the flip-side Clojure immutable nature works very naturally with request / response. (7) For the OP: before you get on GAE, spend some time reading about its limitations: - exploding indexes (makes implementing text search very tough -- there is some irony in there I think) - no threads (it kills me, I want to use pmap) - not real message queues. GAE is a pull/request-based system, not a push system. It is, however, getting much better (tasks, comet on the coming features list). Still, if you are building a real-time system that needs constant background processing, I'd think twice right now. - if GAE goes down, you're down until they fix things, and will be totally helpless. - some of the GAE goodness/crack means you're locked in (e.g., index building -- to move away you'll have to build equivalent facilities for whatever DB you move to). Although you can protect yourself by being smart in how you abstract away the GAE. (8) Can't think of anything else. If there is interest, I have been thinking about writing some blog posts about my experience setting up a Clojure+GAE development environment, working with the data store, working the the GAE API, etc. Let me know here and I'll get a first post out of the way by week's end.

fuzzythinker 17y ago
This is the most detailed and helpful post I read on HN. I wish I can up vote it multiple times.
- jeandenis 17y ago
  Thanks, I appreciate it. First post for me after being a longtime reader so it's nice to know it was helpful to some.
smartrevolution 17y ago
Hi. I am from above mentioned TheDeadline-Team! We can definitly recommend Google App Engine. The SDK is really cool and you don't have to care about managing servers. Scaling is easy. Its (almost) automatic. The datastore is a natural fit for Clojure datastructures, but you have to learn how to structure your data for a distributed key-value store: When you structure your data, you have to think about what has to be in the same transaction (entity-group) and at the same time you have to think about reducing data-dependencies to minimize contention. But when you have lots of data, this is what you have to do anyway. For me, it is much more fun than using an RDBMS. (Read this: http://code.google.com/intl/de-DE/appengine/docs/python/datastore/transactions.html http://code.google.com/intl/de-DE/appengine/docs/python/data...) We did the whole system in Clojure, so you really can write large systems using Clojure/GAE. It works! (Although the UI is simple, TheDeadline is a complex piece of software e.g. containing a self-written rule-based expert-system, a production system, very similar to LISA http://lisa.sourceforge.net/. http://lisa.sourceforge.net/.) The functional paradigm means less code and makes it easier to build powerful abstractions. You don't have to write lots of boiler-plate code like in Java and Clojure is more concises and much more powerful than Python. (Stop! I don't want to start a programming language discussion now! :) ). We will soon post some Clojure examples and some free software libraries for GAE. If you are interested, just follow our blog H.W.A. (http://www.hackers-with-attitude.com http://www.hackers-with-attitude.com) and/or try out http://the-deadline.appspot.com http://the-deadline.appspot.com to get the "feeling". ;)
frognibble 17y ago
The one problem with using Clojure on App Engine is that the time for a "loading request" is at least 5 seconds. It makes your app look bad for the random user stuck on the loading request.