13 ms·

Using Scala to handle exponential growth at a startup

62 points by bhanks 14y ago

dkhenry 14y ago
Its nice to see another shop switching to Scala. I have been advocating the same change where I work ( currently PHP and really feeling the pinch of what it can do ). Looks like I now have another data point to use as a success story for transitioning away.
- tikhonj 14y ago
  You're not yet invested in the JVM, so have you considered some alternatives like Haskell or OCaml? I've found these to be simpler and more productive. The main worry would be about libraries, but that really depends on what you're doing. If it's web development, both Haskell and OCaml have top-notch frameworks that are very good for asynchronous code. OCaml also has a very good JavaScript compiler, so you can use it on the front-end. (Haskell can also be compiled to JS, but I've never used it that way.)
  - dkhenry 14y ago
    I have looked at Haskell, but the problem is getting the team migrated over. Its a lot easier to migrate a team to Scala then to Haskell. Also the library selection for the JVM is unparallelled.
mylons 14y ago
I don't really feel like they actually get into the specifics of how scala actually benefited them outside of what everyone already knows about scala from a really high level. Bummer. I'm using scala at a small startup.
- lbolla 14y ago
  No. But their "Hey Jude" diagram is worth a watch! https://www.lucidchart.com/pages/examples/flowchart_software https://www.lucidchart.com/pages/examples/flowchart_software
- mason55 14y ago
  No, not at all. There was almost no meat to this article.
pramodbiligiri 14y ago
Would love to see a more detailed post on how Scala helps you specifically.
- bhanks 14y ago
  Been getting that feedback from others. If we did a followup article what specifics would be most interesting to know?
  - pramodbiligiri 14y ago
    - How did Scala make it easier to design scalable abstractions - both for performance and for features? - What kind of tooling do you use? Which version of Scala? - Any tips around hiring and ramping up a team? Thanks for the offer. Edit: formatting
yumraj 14y ago
Since Scala is also a JVM language, from scalability perspective can someone expand on what it offers that Java, or any other JVM language, doesn't. I'm not asking about language semantics since I believe that to be a personal preference, rather scalability and performance.
- pivo 14y ago
  Just a guess (and I didn't look at the link) but possibly the Akka library. http://akka.io/ http://akka.io/
  - killface 14y ago
    Akka can be used in Java just fine :)
- eddieplan9 14y ago
  Off the top of my head, Immutability means less locking, which is big enemy of scalability; Actor Model support in the standard library also makes concurrency much easier to deal with. Wikipedia [1] mentions that Scala "actors may also be distributed or combined with software transactional memory". [1] https://en.wikipedia.org/wiki/Scala_(programming_language)#Concurrency https://en.wikipedia.org/wiki/Scala_(programming_language)#C...
- pkulak 14y ago
  As far as I know, Scala can't give you _better_ performance than Java, but I also can't see it being too far off. I've never really seen the benefits of Scala. Everyone whines about Java syntax, but it's really not that bad. Verbose, sure, but it just doesn't seem like time spent typing is where my productivity goes. Maybe I'm just getting old, but I no longer care about language syntax as much as I used to. A preprocessor to give me Java 8 lambda syntax would probably solve all the issues I have with Java right now.
  - killface 14y ago
    I think the benefit is that people don't want to say they're Java programmers. It's like a hipster thing. Most modern scala programmers don't even use the language correctly, or couldn't write "hello world" if it wasn't in the play framework.
jumby 14y ago
I'm so confused. Some diagramming app no one has heard of has to scale to 'exponential' growth and figure out: * [not] loading a lot of session data on every request * storing data across shards * parallel processing * a services approach Wow, I think something is over-architected. Curious: does exponential growth mean 2 concurrent users to 4? If they have more than 100 concurrent users, I would be highly skeptical. And, if they can't solve that problem with any language, I would also be highly skeptical.
- dmgrow 14y ago
  Lucidchart employee here. Though we don't publicly release total user stats, this recent post about 500,000 installs through the Chrome Web Store gives a hint: https://www.lucidchart.com/blog/2012/10/31/500000-chrome-web-store-installs-what-weve-learned/ https://www.lucidchart.com/blog/2012/10/31/500000-chrome-web... Even with 550,000 installs there, the Chrome Web Store is just one of a number of strong channels for Lucidchart and contributes a small minority of our user base.
  - jumby 14y ago
    So, how many requests / second is your load balancer receiving from outside?
- brown9-2 14y ago
  This seems like an unfair reaction from the outside. There is no way you can know their use case and traffic better than they can. What does this add to the conversation anyway?
- skrebbel 14y ago
  Why do you think no one has ever heard of an app just because you haven't? In general, your comment feels rather agressive for no good reason. That said, i have my doubts about the term 'exponential'. I forgive financial journalists who think that it means the same as 'explosive', but a programmer should know better. Unless, of course, Lucidchart does have exponential growth, in which case i guess congrats are in order.
brown9-2 14y ago
A bit off-topic but I'm curious what people familiar/experienced with other Java web app frameworks think of the Play framework?
- paullth 14y ago
  I work in an enterprise-ish setting and regularly develop with Spring MVC, Struts and Rails on the server side. More recently have been using Play 2.0/Scala (I must admit I have not had a Play app make it to production yet). I find it more productive, like rails in dev mode you can make a change, refresh the browser and watch what happens. This kicks arse compared to hot deploy/in place deploy with Java, or even to JRebel. The main advantage I find over the others in terms of framework features is the non blocking/asynchronous constructs provided out of the box. For anyone new to it, it's also worth noting that from 2.1 (almost out) you can integrate with Spring, and that there seems to be a healthy collection of 3rd part modules for it, ie for stuff like oauth. I could go on for ages, but essentially I prefer it to the other frameworks Im familiar with...
- RyanZAG 14y ago
  The choices really come down to (imo anyway), 1. JSP 2. GWT/GAE/AppScale 3. Play JSP is just awful - taking PHP and trying to fit it into Java somehow. It's just terrible and needs to be phased out ASAP. (EDIT: Probably too harsh.. JSP works in the same way PHP works: it powers most of the web, and it does a grudgingly good job. The ease of plopping in some dynamic content into am html website made by a designer should not be discounted lightly for small websites. Probably best to just use PHP and not JSP in this use case though...) GWT/GAE/AppScale mix is my preference. You can share code across server and client. Building RESTful services is a dream with Jersey on the server/thick client, and RestyGWT in the browser. The GWT compiler produces some very well performing javascript (only a bit slower than highly optimized libraries, but feels faster than nearly all webapp javascript). GWT also has incredibly good tooling support in automatic sprite sheet creation, compiled CSS styles, uibinder for declarative layouts, etc. The downsides are (1) java language, so no functional programming and (2) compile time can get nasty when you head in the millions-lines-of-code arena. GWT team is seemingly working on speeding up the dev cycle with SuperDevMode and that. Play framework is most similar to the ruby on rails approach and works incredibly well for your standard webapp. The support for Comet and WebSockets are particularly nice and making very dynamic 'push' webapps is the real strongest point. The tooling can get pretty annoying at times with features/bugs, but it is steadily improving. The biggest downside to Play is that it is in a 'hipster phase' currently with the release of Play 2. This means both a huge surge in interest, but that same surge can be a curse when it drops out of 'hipster' mode in the future and loses devs. Numerous promising frameworks/languages have hit this problem and never really recover when the 'fashion' changes. GWT, for example, has long since fallen out of fashion but continues to be heavily developed because of the huge and profitable existing user base inside Google (Adwords, etc), guaranteeing it active development for awhile to come.

62 points by bhanks 14y ago

Its nice to see another shop switching to Scala. I have been advocating the same change where I work ( currently PHP and really feeling the pinch of what it can do ). Looks like I now have another data point to use as a success story for transitioning away.

tikhonj 14y ago
You're not yet invested in the JVM, so have you considered some alternatives like Haskell or OCaml? I've found these to be simpler and more productive. The main worry would be about libraries, but that really depends on what you're doing. If it's web development, both Haskell and OCaml have top-notch frameworks that are very good for asynchronous code. OCaml also has a very good JavaScript compiler, so you can use it on the front-end. (Haskell can also be compiled to JS, but I've never used it that way.)
- dkhenry 14y ago
  I have looked at Haskell, but the problem is getting the team migrated over. Its a lot easier to migrate a team to Scala then to Haskell. Also the library selection for the JVM is unparallelled.

mylons 14y ago

I don't really feel like they actually get into the specifics of how scala actually benefited them outside of what everyone already knows about scala from a really high level. Bummer. I'm using scala at a small startup.

lbolla 14y ago
No. But their "Hey Jude" diagram is worth a watch! https://www.lucidchart.com/pages/examples/flowchart_software https://www.lucidchart.com/pages/examples/flowchart_software
mason55 14y ago
No, not at all. There was almost no meat to this article.

pramodbiligiri 14y ago

Would love to see a more detailed post on how Scala helps you specifically.

bhanks 14y ago
Been getting that feedback from others. If we did a followup article what specifics would be most interesting to know?
- pramodbiligiri 14y ago
  - How did Scala make it easier to design scalable abstractions - both for performance and for features? - What kind of tooling do you use? Which version of Scala? - Any tips around hiring and ramping up a team? Thanks for the offer. Edit: formatting

yumraj 14y ago

Since Scala is also a JVM language, from scalability perspective can someone expand on what it offers that Java, or any other JVM language, doesn't. I'm not asking about language semantics since I believe that to be a personal preference, rather scalability and performance.

pivo 14y ago
Just a guess (and I didn't look at the link) but possibly the Akka library. http://akka.io/ http://akka.io/
- killface 14y ago
  Akka can be used in Java just fine :)
eddieplan9 14y ago
Off the top of my head, Immutability means less locking, which is big enemy of scalability; Actor Model support in the standard library also makes concurrency much easier to deal with. Wikipedia [1] mentions that Scala "actors may also be distributed or combined with software transactional memory". [1] https://en.wikipedia.org/wiki/Scala_(programming_language)#Concurrency https://en.wikipedia.org/wiki/Scala_(programming_language)#C...
pkulak 14y ago
As far as I know, Scala can't give you _better_ performance than Java, but I also can't see it being too far off. I've never really seen the benefits of Scala. Everyone whines about Java syntax, but it's really not that bad. Verbose, sure, but it just doesn't seem like time spent typing is where my productivity goes. Maybe I'm just getting old, but I no longer care about language syntax as much as I used to. A preprocessor to give me Java 8 lambda syntax would probably solve all the issues I have with Java right now.
- killface 14y ago
  I think the benefit is that people don't want to say they're Java programmers. It's like a hipster thing. Most modern scala programmers don't even use the language correctly, or couldn't write "hello world" if it wasn't in the play framework.

jumby 14y ago

I'm so confused. Some diagramming app no one has heard of has to scale to 'exponential' growth and figure out: * [not] loading a lot of session data on every request * storing data across shards * parallel processing * a services approach Wow, I think something is over-architected. Curious: does exponential growth mean 2 concurrent users to 4? If they have more than 100 concurrent users, I would be highly skeptical. And, if they can't solve that problem with any language, I would also be highly skeptical.

dmgrow 14y ago
Lucidchart employee here. Though we don't publicly release total user stats, this recent post about 500,000 installs through the Chrome Web Store gives a hint: https://www.lucidchart.com/blog/2012/10/31/500000-chrome-web-store-installs-what-weve-learned/ https://www.lucidchart.com/blog/2012/10/31/500000-chrome-web... Even with 550,000 installs there, the Chrome Web Store is just one of a number of strong channels for Lucidchart and contributes a small minority of our user base.
- jumby 14y ago
  So, how many requests / second is your load balancer receiving from outside?
brown9-2 14y ago
This seems like an unfair reaction from the outside. There is no way you can know their use case and traffic better than they can. What does this add to the conversation anyway?
skrebbel 14y ago
Why do you think no one has ever heard of an app just because you haven't? In general, your comment feels rather agressive for no good reason. That said, i have my doubts about the term 'exponential'. I forgive financial journalists who think that it means the same as 'explosive', but a programmer should know better. Unless, of course, Lucidchart does have exponential growth, in which case i guess congrats are in order.

brown9-2 14y ago

A bit off-topic but I'm curious what people familiar/experienced with other Java web app frameworks think of the Play framework?

paullth 14y ago
I work in an enterprise-ish setting and regularly develop with Spring MVC, Struts and Rails on the server side. More recently have been using Play 2.0/Scala (I must admit I have not had a Play app make it to production yet). I find it more productive, like rails in dev mode you can make a change, refresh the browser and watch what happens. This kicks arse compared to hot deploy/in place deploy with Java, or even to JRebel. The main advantage I find over the others in terms of framework features is the non blocking/asynchronous constructs provided out of the box. For anyone new to it, it's also worth noting that from 2.1 (almost out) you can integrate with Spring, and that there seems to be a healthy collection of 3rd part modules for it, ie for stuff like oauth. I could go on for ages, but essentially I prefer it to the other frameworks Im familiar with...
RyanZAG 14y ago
The choices really come down to (imo anyway), 1. JSP 2. GWT/GAE/AppScale 3. Play JSP is just awful - taking PHP and trying to fit it into Java somehow. It's just terrible and needs to be phased out ASAP. (EDIT: Probably too harsh.. JSP works in the same way PHP works: it powers most of the web, and it does a grudgingly good job. The ease of plopping in some dynamic content into am html website made by a designer should not be discounted lightly for small websites. Probably best to just use PHP and not JSP in this use case though...) GWT/GAE/AppScale mix is my preference. You can share code across server and client. Building RESTful services is a dream with Jersey on the server/thick client, and RestyGWT in the browser. The GWT compiler produces some very well performing javascript (only a bit slower than highly optimized libraries, but feels faster than nearly all webapp javascript). GWT also has incredibly good tooling support in automatic sprite sheet creation, compiled CSS styles, uibinder for declarative layouts, etc. The downsides are (1) java language, so no functional programming and (2) compile time can get nasty when you head in the millions-lines-of-code arena. GWT team is seemingly working on speeding up the dev cycle with SuperDevMode and that. Play framework is most similar to the ruby on rails approach and works incredibly well for your standard webapp. The support for Comet and WebSockets are particularly nice and making very dynamic 'push' webapps is the real strongest point. The tooling can get pretty annoying at times with features/bugs, but it is steadily improving. The biggest downside to Play is that it is in a 'hipster phase' currently with the release of Play 2. This means both a huge surge in interest, but that same surge can be a curse when it drops out of 'hipster' mode in the future and loses devs. Numerous promising frameworks/languages have hit this problem and never really recover when the 'fashion' changes. GWT, for example, has long since fallen out of fashion but continues to be heavily developed because of the huge and profitable existing user base inside Google (Adwords, etc), guaranteeing it active development for awhile to come.