13 ms·
Reasonable Scala Compiler
- bearforcenine 9y agoI am a bit concerned that between the Scala, Dotty, Typelevel, and now the Reasonable Scala Compiler, that the Scala community might become fragmented. Hopefully the rsc's claim of "making our findings available to the Scala community at large" holds true. Edit: should have keep reading before commenting. In their related work document: >we are definitely not declaring Lightbend Scala dead. To the contrary, we view ourselves as complementary to Lightbend Scala and hope that our results will create new opportunities for the official compiler. We are planning to keep in contact with Lightbend to discuss our findings and facilitate technology transfer.
- foolfoolz 9y agoscala and dotty will be one and the same in the next major version. the rest are niche tools. no one should be afraid to develop their own compiler or tool chain for fear of fragmentation. the lessons learned can only help the trunk
- hota_mazi 9y agoRegardless of Scala, it's an absolutely insane idea to develop your own language to write your apps. Either stick to the real Scala or switch to Java or Kotlin, but writing your own language is a failure waiting to happen. I give this project a few more months before it gets canceled.
- carlosdp 9y agoThat's... not true. You mention Kotlin, that was developed by JetBrains as a better language to develop Intellij and plugins. Hack/HHVM was developed by Facebook to modernize PHP. There are plenty of cases where organizations successfully developed their own languages/compilers that gave them advantages. It's insane only if you have no good reason to do it. This sounds like they have a need: compile times are slow.
- alexandercrohde 9y agoyou think compile times are so slow that they're saving time by writing a whole language? I'd like to see the numbers on that...
- carlosdp 9y agoWe're talking about a large organization that probably does 1000s or 10000s of builds a day, so yea it might
- sk5t 9y agoJetBrains' business and expertise is language stuff, lexers, compilers--isn't it fair to give them a pass on making a language? One needn't extend the same consideration to, say, fogcreek...
- hasenj 9y ago> Hack/HHVM was developed by Facebook to modernize PHP. Yes but why use PHP in the first place? It's not the right tool for something like Facebook.
- Joeri 9y agoIn 2003 PHP was the most sensible choice for rapid prototyping of a web app and for early facebook it was more important to iterate and experiment than build it right. Even today it still holds its own, but there are plenty of alternatives (rails, node.js, ...)
- yawaramin 9y agoIt's not a new language, it's a reduced subset of the same language.
- hota_mazi 9y agoYou're playing with words. They are creating their own language which is a subset of Scala, and they decide which features to keep based on whether they are fast enough to compile. The point is: developing your own language to ship your apps is crazy.
- deleted 9y ago[deleted]
- swsieber 9y agoI think there is a significant difference in that developing g your own language requires a rewrite to use new language, not a tweak to existing code.. If you insist on them "creating a new language" then say that they created a new language to speed up development, not to ship their apps. They were already shopping their apps. This is an optimization that helps them speed up development on their existing codebase.
- sulam 9y agoInsane choices start to look very rational at sufficient scale.
- hasenj 9y agoNot to mention they can also decide laters to add new features to this new language, further diverging from Scala.
- hota_mazi 9y agoThey are already diverging from Scala since they are only implementing a subset of it: > We are planning to start small with a trivial subset of Scala and then gradually add features, carefully measuring their impact on compilation performance It will be a while before they reach parity with Scala, and I'll bet the project will be canceled before they even come close.
- mintplant 9y agoFunny you should mention Kotlin, as it's the result of JetBrains doing just that: developing their own language to write their apps. Mozilla represents another successful example with Rust. I'm of the opinion that developing new, purpose-built languages (or extending existing ones) is a strategy that isn't given as much consideration as it should.
- Throaway786 9y agoJetbrains is a compiler/IDE company, they have the expertise to do such task. What will Twitter gain from this endeavor?
- Scarbutt 9y agodeveloper productivity.
- notfried 9y agoMore like developer retention.
- chairmanwow 9y agoZING
- sulam 9y ago1) Twitter certainly has the right engineers to do this. 2) If they can speed up Scala compile times by 5 minutes a day (extremely conservative -- compilation times tend towards 1s per file), across even 1,000 developers (Twitter had around 2000 when I was there) that immediately funds a 5+ person development team. This isn't the sort of thing you throw a huge team at, anyway. 3) It helps with recruiting talent who is looking to work in environments that have strong type safety, or who just want to work for companies willing to invest time in making their software engineers more productive.
- hasenj 9y ago
- foolfoolz 9y ago> it's insane to write your own language > switch to Java or kotlin ah yes rewriting in a new language, the classic not-insane approach
- hota_mazi 9y agoI offered several alternatives and I made it clear they belong in a spectrum. Don't make it sound like I claimed it was binary. If the options are "Keep writing in Scala", "Port to Kotlin" and "Write a Scala compiler", the latter is clearly the worst and most costly of all.
- zeckalpha 9y agoYou posted this on hacker news, which is essentially the only app written in its language.
- addisonj 9y agoIt will be interesting to see what the subset is. I have been writing scala for 2 years, mostly in spark and flink and as much as I love parts of it, overall, the language is just not the most approachable. Some of the features are super valuable every once in a while, but I might gladly give some of them up for a streamlined language that is easier to get people started on (while not just writing imperative style Java with Scala syntax) My other curiosity is compatibly. If I can compile a subset of my project much faster that doesn't need all of Scala, that seems like an okay place to be in.
- adamnemecek 9y agoCheck out Kotlin. I picked it up in like half a day. It wasn't idiomatic Kotlin but w/e. It can compile to JS and you can use just about any JS lib from Kotlin (which is insanity). You'll be able to write ios apps with it soon enough. Scala can be too hard sometimes.
- brianwawok 9y agoYou can use JS from scala also.
- alextheparrot 9y agoHe's using Spark and Flink... Given that I'm also in that big data space, I'm certain the tooling provided by Kotlin is strictly worse than Scala for those applications.
- afghanPower 9y agoClojure(Script) can do the same.
- MrBuddyCasino 9y ago> It will be interesting to see what the subset is. Totally. Its interesting that they approach this from a compile speed angle, I wonder how much they are also considering the mental load of certain features. This might become an interesting project for companies already doing Scala. New ones at this point will probably prefer Kotlin.
- needusername 9y agoI'm a bit confused. I have been told multiple times by Scala users that Scala compile times are a non-issue.
- sulam 9y agoDepends on the size of the code base, how good your incremental compilation is, and what percentage of the code changes frequently. At Twitter the values are: large, great, and relatively high.
- davedx 9y agoI'm an intermediate Scala user. Scala compile times hurt my productivity significantly. Our best Scala developer says his approach is "I know the language so well, I usually write the whole feature in one go before compiling. That way I avoid the slow compile times." Which feels to me like you will miss one of the advantages of having a compiler in the first place - that it helps you write correct code. To be fair, in an IDE you do get some of the compiler feedback in real time. But not for the whole code base, if you're working on things that have high coupling with other modules then it can be really painful. (I'm looking at you, Spray serialization!! Ugh)
- eip 9y agoI don't have any issues with compile times. But I don't work on an underpowered Crapbook Pro. If you are serious about your work and time than use serious hardware. You don't see see F1 racers show up on race day with a Scooty Puff Jr..
- lmm 9y agoThey're not a complete non-issue. They're just much less of an issue than not using Scala. (And bear in mind that Twitter has literally the largest Scala codebase in the world. Almost all scala programmers work on something an order of magnitude smaller, with correspondingly smaller compile times)
- qingwei91 9y agoIt depends on project size, twitter likely have some really huge scala project, but many organization are now going for microservices, so compile time is less of an issue
- Throaway786 9y agoTwitter is not performing well as a business. It is also at almost all time low. Should it be working on a new language/compiler?
- adamnemecek 9y agoIt's because their devs are spending too much time compiling their code /s
- monksy 9y agoor maintaining their monorepo
- daxfohl 9y agoMaybe it was a business deal. If you start seeing Martin Odersky retweeting Justin Bieber a lot going forward, you'll know what's up.
- jmnicolas 9y agoDoesn't Scala run on the JVM ? Are they talking about JIT times or are there other Scala 'environment' that are not semi-interpreted ?
- ptx 9y agoThey are talking about compilation of source code to Java byte code, i.e. the same kind of compiler as javac from the JDK. (There is also Scala Native, but that doesn't seem to be what this is about.)
- maemre 9y agoThis is the compiler from Scala to Java bytecode. It does type checking, some optimizations (like inlining, removing some instances of boxing, or converting tail-recursive functions to loops), and converts all Scala-specific features to some Java constructs (e.g. traits become interfaces, classes etc.; lazy values become methods and accompanying values; lambdas get lifted to the class they are defined in). The JIT works on Java bytecode. Java also has a similar compiler (javac) from Java -> bytecode.
- jankotek 9y ago> Identify a subset of Scala that can be compiled with reasonable speed I am afraid of Scala fragmentation. There is no way new compiler can be 100% compatible. Hopefully it will be tested on major projects, or there will be some official Scala Language spec. It would be nice to add a switch into old scalac, which downgrades its features, to make it compatible with new scalac. I believe Scala has too many features, and it will be very difficult to fix compilation speed. But there could be decend boost just by rewritting compiler from scratch. If anything we will get better error messages. Compilation speed was major reason why Jetbrains started Kotlin. Disclaimer: I worked with Scalac / IDE back in 2009. Kotlin fanatic.
- skrebbel 9y ago> Hopefully it will be tested on major projects Twitter has pretty big resources. It's probably feasible for them to spin up a build farm that runs their compiler against all active Scala projects on GitHub, measuring error count and compilation speed as they go along.
- harpocrates 9y ago> There is no way new compiler can be 100% compatible. Hopefully it will be tested on major projects, or there will be some official Scala Language spec. I, for one, think that an official language spec would do the language a lot of good. As it stands, it is sometimes difficult to separate bugs from features in scalac...
- dtech 9y agoI am not sure how complete/correct it is, but there is one for 2.9 [1] Martin Odersky & co have been focusing most of their effort the last few years on Dotty (aka Scala 3), which succeeded in giving in giving Scala a proven theoretical basis in exchange for a few esoteric typesystem features they couldn't prove. [2] [1]: http://www.scala-lang.org/old/sites/default/files/linuxsoft_archives/docu/files/ScalaReference.pdf http://www.scala-lang.org/old/sites/default/files/linuxsoft_... [2] http://www.scala-lang.org/blog/2016/02/03/essence-of-scala.html http://www.scala-lang.org/blog/2016/02/03/essence-of-scala.h...
- majidazimi 9y agoThis truly means we still lack a programming language with: 1. reasonable support of object oriented programming. 2. reasonable support of functional programming. 3. solid concurrency features. 4. runs comparable to native code. 5. simple to learn. Even though it is not difficult to create such a language but we see a new language poping out every now and then and none of them try to solve these issues.
- adamnemecek 9y agoRust, swift and kotlin once they have the llvm backend will fit the bill.
- jonathanyc 9y agoRust is very neat, but it lacks OO features.
- adamnemecek 9y agoThat depends on your definition of oo.
- mintplant 9y agoMany would call that a feature in itself.
- AJRF 9y agoSwift is awesome but lacks atomicity guarantees and it allows some race conditions even when using gcd. https://github.com/apple/swift/blob/master/docs/proposals/Concurrency.rst https://github.com/apple/swift/blob/master/docs/proposals/Co...
- davedx 9y agoI wouldn't say Rust is simple to learn. The borrow checker alone is quite a steep hill (though the compiler errors are very good). In my limited experience Rust and Swift both suffer from boilerplate/ceremony you have in many other OOP languages (C# too). Static languages for you I guess - there are benefits too...
- joefkelley 9y agoMy opinion of this changed after reading this comment by the developer: https://www.reddit.com/r/scala/comments/6ubuix/twitter_announces_reasonable_scala_compiler_an/dlrxs6m/ https://www.reddit.com/r/scala/comments/6ubuix/twitter_annou... Sounds like they are intending it as a research effort and hope to make progress with the entire Scala community, not fork it.
- amelius 9y agoWill they use this compiler only for prototyping/debugging/testing? Or also for production work?
- ece 9y agoOverall, I feel like with the success of Go and Kotlin, having an easy to learn language definitely wins. Faster runtime speed might be a bigger 'market' than wanting fast compile times, which matters if you have a large enough codebase. I do wish Scala native was further along, and if the language was going to get simpler, it would focus more on native speed than compilation time. Quite a few Scala compiler releases have made big strides in compilation times, to the extent that java vs scala compilation times I feel are the same or well worth the cost, so this has been and will be an ongoing effort. The Java group is also working on being native, so we will likely see Java native when it gets there.
- danarmak 9y ago> The Java group is also working on being native, so we will likely see Java native when it gets there. I haven't heard about this. Can you give a reference please?
- ece 9y agoThey are working on AOT compilation: https://www.youtube.com/watch?v=Xybzyv8qbOc&index=241&list=WL https://www.youtube.com/watch?v=Xybzyv8qbOc&index=241&list=W...
- danarmak 9y agoIIUC, this is about AOT but still using the JVM. This is a fix for slow warm-up, but in longer running processes the eventual behavior isn't different from JIT. Which is important in its own right, but very different from a hypothetical 'Java native' in the sense of 'Scala native', without a JVM at all.
- ece 9y agoI believe you are right. Azul has released a JIT compiler based on llvm: http://www.infoworld.com/article/3193078/java/azul-introduces-llvm-compiler-to-java-runtime.html http://www.infoworld.com/article/3193078/java/azul-introduce... Also: https://www.excelsiorjet.com/ https://www.excelsiorjet.com/ and there seem to be open source projects like vmkit, https://github.com/ReadyTalk/avian https://github.com/ReadyTalk/avian that are somewhere in between.
- JediPig 9y agoScala just sucks. It was a research project that turned into a bad idea. I hope Scala dies
- JediPig 9y agoScala just sucks. It was a research project that turned into a bad idea. I hope Scala dies
- jgalt212 9y agoSo Clojure is slow to start and Scala is slow to compile. Is Scala also slow to start? As compared to a Java program to performs the equivalent functions.
- iLemming 9y agoClojure doesn't have slow start-up, Leiningen and Boot-clj make it feel slow. But once you start a REPL you can keep it running. I keep mine running for weeks.
- jgalt212 9y agoI know. I've often heard this, but for me that's a bit tricky b/c I'm worried that my current definition of "foo" is not what it is in my .clj file b/c I may have monkey-patched it in the REPL and forgot I did that.
- Others 9y agoI'm interested in what they said about reducing the compiler to 4 passes. Dotty takes a really interesting approach to this with 'miniphases' (https://infoscience.epfl.ch/record/228518/files/paper.pdf https://infoscience.epfl.ch/record/228518/files/paper.pdf). I think it'll be interesting to compare these two methods when both compilers are production ready.
- harveywi 9y agoI'd like to see a Scala compiler that works more like a proof assistant like Coq. I want the compiler's job to be to help me quickly write correct code. That should be first-class functionality. Batch-compiling the same source files over and over again should be just a neat trick that the compiler can do on the side.
- wpietri 9y agoHuzzah! Maybe this is already in the works, but I would love to see a compiler that stays hot in RAM and works incrementally. RAM is cheap nowadays, and the number one reason I want a fast compiler is to speed up my feedback cycle between making a change and evaluating the effects of the change. So I want something that squats on a large chunk of RAM and works in parallel with me, updating with every edit I make.
- walkingolof 9y agoSBT always runs Scalac in the same VM, so it will be hot and JIT compiled after a few passes. Intellij implemented a SBT Shell in 2017.1, so it will keep the same SBT session going between passes also.
- wpietri 9y agoThat's a good start, but won't the compiler still be batch-oriented? By which I mean doing a ton of I/O and activity only when I hit go? I'm talking about something much more real time. I want the parser to keep everything hot, updating every time I hit a key. And whenever I get to anything that the parser approves of, downstream stages, which also keep everything hot, dynamically update all of their structures, revising object files as they go. Batch orientation is great, and I understand why compilers started there. But I want something that moves beyond that. It's sort of the difference between a batch-oriented rendering system like TeX or early word processing versus a modern WYSIWYG word processing setup, where we burn resources prodigiously to give users much faster feedback loops.
- speedkills 9y agoThe Dotty compiler is going down the route you are looking for.
- sorokod 9y agoGradle?
- edem 9y agoWriting a reasonable compiler would need a reasonable language, no? Scala had its chance in the last 13 years now it is time for this abomination to go.
- DeepRote 9y agoScala is a great language. There's room to shoot yourself in the foot, but maybe you should watch your muzzle more if you're having that problem. Imo the biggest obstacle in scalaverse is sbt and implicits. 10000% the two things I have to devote extra time to when showing somebody the ropes.