11 ms·

Policy – A fork of the Scala compiler

• nemothekid 12y ago
  Can someone with some familiarity with Scala community explain why someone who seems to be the largest contributor to Scala felt the need to create a fork? It seems completely justified due to his credentials, but he doesn't give any specific reasons why he created this fork.
  • cakoose 12y ago
    Paul Phillips' 2013 rant/talk: https://www.youtube.com/watch?v=TS1lpKBMkgg https://www.youtube.com/watch?v=TS1lpKBMkgg
    • deleted 12y ago
      [deleted]
      • jfim 12y ago
        It turns into a rant about how he'd wish the compiler and the editor and the source control should be all talking with one another. It's funny, because there's actually a language just like that: Smalltalk. The development environment is the same as the runtime environment, it has its own source control (Monticello), its own editor and a very regular language.
        • teacup50 12y ago
          I can assure you of two things: 1) Paul is not ignorant of the existence of Smalltalk 2) Smalltalk is not a replacement for the fruits of modern language research. In fact, the academic work around Smalltalk is nothing more than unsupported opinion, and it's not even remotely useful as a source of research.
          • seanmcdirmid 12y ago
            Parent was out of line but there is still plenty of modern academic research occurring around Smalltalk. Not to mention Gilad's Newspeak (which may or may not be academic, you'd have to ask him).
            • teacup50 12y ago
              Can you think of a single example that: 1) Posits new conjectures 2) Proves those conjectures 3) Those conjectures can be used to formulate new conjectures that conform to items 1-3? I've read a great deal of smalltalk literature, and I can't recall any papers that would qualify. There's plenty of empirical exploration of ambiguous hypothesis, but nothing that actually provides anyone in the field anything on which they could actually build.
        • zak_mc_kracken 12y ago
          Smalltalk is dynamically typed so it's a non starter for most of the Scala and Haskell folks. Same for Lisp.
          • pjmlp 12y ago
            Well, there was StrongTalk as well.
        • pjmlp 12y ago
          Smalltalk lost the spot in the industry thanks to Java, sadly I doubt it will hardly get a second chance. Back in the mid-90's, before Java was released to the world, I was using VisualWorks for an university project. IBM was a big enterprise player with Visual Age for Smalltalk. A small startup (Animorphic) was making Smalltalk faster (StrongTalk). There were, of course, other companies searching their piece of the pie. When Sun started pushing Java to the world, many in the Smalltalk world shifted direction. Visual Age for Smalltalk architecture became the foundation of Eclipse. Sun eventually bought Animorphic and Hotspot was born out of StrongTalk VM ashes. Slowly, other Smalltalk players joined the party.
        • jibal 12y ago
          "It's funny, because there's actually a language just like that: Smalltalk." I don't see how that's "funny" or relevant, as there are many other things about Scala that Smalltalk isn't just like.
    • rch 12y ago
      The slide "What I'm after" may be found around the 38 minute mark.
    • a8da6b0c91d 12y ago
      It's hard to decide if his deep emotional investment is inspiring or frightening.
      • ionforce 12y ago
        Why can't it be both??
        • extempore 12y ago
          I also vote for both.
          • jroper 12y ago
            If in doubt, order everything on the menu.
    • jpgvm 12y ago
      After watching this talk I don't care what he comes up with in the future, I want to use it. Excellent talk.
      • mdaniel 12y ago
        Are you familiar with JetBrains MPS? http://www.jetbrains.com/mps/ http://www.jetbrains.com/mps/ It sounds very much like what he is describing.
        • jpgvm 12y ago
          I don't think an IDE is the answer to the problem. I think making Scala more correct and more modular is the answer. Hygienic macros and syntactic extensions are more what I had in mind. Rust seems to be a solid contender here but it's much more of a systems language.
        • extempore 12y ago
          Yes, MPS and "language workbenches" in general are chasing something similar. Ideas like these are entirely in the details though, and so far I haven't found anything which does it for me. I could easily be missing it, in part because systems like MPS have a high up-front investment, and I have regretted such investments more often than not.
    • jbza 12y ago
      That guy is a maniac. Much of the stuff that he was ranting off about are textbook examples of why functional programming and imperative programming ought to be separated from each other
  • nathancahill 12y ago
    > "The leadership of scala has shown itself unfit stewards of the core technologies and the total price paid by scala programmers (actual and potential) is too high. I failed for half a decade to change that from the inside. Now I'll try from the outside."
    • rch 12y ago
      That's a strongly worded statement for a front page, but I wish I knew something more about the situation. What progress is being held back? I've not written a line of Scala myself.
      • tormeh 12y ago
        Internals, mostly, I think. Correctness. According to Phillips, the internals of scalac are a complete mess, and the runtime generated code is also non-ideal. Basically, all sorts of software best practices are being broken to achieve performance and ensure Scala is a superset of Java.
  • dkarapetyan 12y ago
    The type systems broke him.
  • gtani 12y ago
    there is in fact more forking happening: https://news.ycombinator.com/item?id=8258255 https://news.ycombinator.com/item?id=8258255 https://news.ycombinator.com/item?id=8261896 https://news.ycombinator.com/item?id=8261896 _____________________________ Many (most?) of the debates regarding the scalac roadmap (and attendant tooling) happened in public on the internals mailing list. The simplest explanation is that nobody agreed on allocation of resources, which were mostly a few Typesafe employees and some (largely unknown but very talented) EPFL grad students. A sample thread https://groups.google.com/forum/#!msg/scala-internals/9Ts3GLsXuOg/PeWytVtp568J https://groups.google.com/forum/#!msg/scala-internals/9Ts3GL...
    • taeric 12y ago
      That thread doesn't exactly inspire confidence in choosing Scala as a platform, to be honest. With quotes such as "There's still plenty of very basic stuff in the compiler which doesn't work," I'm honestly not sure how anyone could look forward to a dive. Don't get me wrong. I know that it isn't necessarily the case that "basic" represents "affects plenty of people." But, especially when it was used in a "hits more people than a vocal group that is complaining about a plugin that needs attention," one does get a sense that you will be hit by these things sooner than later.
      • frowaway001 12y ago
        The thing is that it's much worse everywhere else, the Scala devs just work with it in a more transparent way.
        • zak_mc_kracken 12y ago
          Your ability of denial is quite impressive but looking at your posting history, it's actually not surprising at all. You seem to have quite a thing about Scala and hating pretty much anything that's not it. Java is at least as transparent as Scala in that respect: every single feature that was shipped as part of Java 8 has been discussed and decided openly on mailing-lists. Everyone knew exactly what was going to be in Java 8 (including what was not going to be shipped) which is a far cry from Scala's operating mode of adding features to the language and then retroactively creating SIPs to give them an air of legitimacy. I'm siding squarely with Paul Phillips there: I don't have a lot of faith in Typesafe to carry Scala forward any more.
  • zak_mc_kracken 12y ago
    He gives a few hints on the github page and in the many presentations he's been giving these past months since he left Typesafe. The short version is that he lost trust in the Typesafe team and he thinks they are not competent to carry Scala moving forward. Like he says, he tried to change things from the inside, he couldn't, so now he's trying from the outside.
    • extempore 12y ago
      Typesafe hasn't lost my confidence. Typesafe has been responsible for important progress which is already being taken for granted. People either have forgotten how low assurance was in the 2.8 days, or they weren't using scala yet. The problem is on the other side of the Atlantic. Scala is unfixable now no matter the abilities of the people working on it because of the bad decisions which have been made to this point and the unwillingness to revisit them even when the failings were grossly apparent. Organizations modeled on dictatorships can work fine but they're painfully vulnerable to the dictator's blind spots. As the years go by, the dictator becomes accustomed to being able to settle any question by fiat. He becomes less and less concerned with evaluating the true worth of his own ideas. As long as his power is unquestioned and there is a steady supply of people to do the work, why should he? He's the Decider. He decides. As I told him at some point "That well does run dry eventually" and indeed it did, for me at least.
      • brudgers 12y ago
        Want to express my thanks for all you have done for the Scala and JVM communities so far. Looking forward to seeing what's next.
      • zak_mc_kracken 12y ago
        Thank you for all your contributions, Paul, you have been instrumental in helping me improve my understanding of various programming language and functional concepts. I'll be keeping a close eye on what you decide to do next, I have no doubt it will be extremely interesting.
      • dustingetz 12y ago
  • jibal 12y ago
    "he doesn't give any specific reasons why he created this fork" This is somewhat like asking why Richard Dawkins just assumes the Theory of Evolution without justification in his tweets. Googling Paul Phillips Scala brings up plenty of relevant discussion.
• wittekm 12y ago
  The "A Fork!? You Should Improve Scalac!" section is delightful.
• djrogers 12y ago
  Shouldn't this be called Standard Scala? Too soon?
  • nine_k 12y ago
    Will eventually become Common Sca.
  • asuffield 12y ago
    Or "Enhanced Scala Compiler System"?
• tormeh 12y ago
  Why are parallel collections an "albatross"? I can understand it if he plans to replace the functionality you get by writing "list.par.map(func).seq" with "list.parmap(func)" or something, though. Parallel list functions are one of my favorite features in Scala, and I think it would be a serious loss if they disappeared. Also, I wonder what the two Scala forkers are thinking about Dotty (the next-gen Typesafe scala compiler) and how they imagine merging their ideas in.
  • extempore 12y ago
    It's not a question of whether they should exist at all but of whether they should infiltrate the collections hierarchy to the extent that someone has to maintain code which looks like this. And there's plenty of it: this is only an example. trait ParIterableViewLike[+T, +Coll <: Parallel, +CollSeq, +This <: ParIterableView[T, Coll, CollSeq] with ParIterableViewLike[T, Coll, CollSeq, This, ThisSeq], +ThisSeq <: IterableView[T, CollSeq] with IterableViewLike[T, CollSeq, ThisSeq]] extends GenIterableView[T, Coll] with GenIterableViewLike[T, Coll, This] with ParIterable[T] with ParIterableLike[T, This, ThisSeq]
    • vidarh 12y ago
      As a recovering former C++ programmer, that reminds me of the verbosity of C++ meta programming...
      • jkbyc 12y ago
        Which language have you moved to?
    • tormeh 12y ago
      Well, that's a pretty good reason to fix something, I agree. But what are your thoughts about Dotty? I assume it's insufficient for you, but is Typesafe going in the right direction here, is it not relevant to what you are doing or will it make things harder?
• vfclists 12y ago
  It appears to be a fork of the compiler not the language itself. I don't know much about Scala but I believe that most of its facilities are in the libraries rather than the language itself. Since he seems to know more about compiling Scala than everyone else I guess it must be a good thing. Lets hope others get to fill the inevitable gaps.
  • saryant 12y ago
    He started on his own standard library a while ago: https://github.com/paulp/psp-std https://github.com/paulp/psp-std
  • zak_mc_kracken 12y ago
    > It appears to be a fork of the compiler not the language itself. They are the same thing. There is no spec for Scala (no recent one anyway) so the specification is the Scala compiler. And based on everything that Paul has been saying these past months, the compiler is very close to being unmaintainable.
    • eeperson 12y ago
      There actually is a recent spec, although I'm not sure if it still needs some updating. http://www.scala-lang.org/files/archive/spec/2.11/ http://www.scala-lang.org/files/archive/spec/2.11/
      • extempore 12y ago
        There is "a spec" yes. Now let's see someone implement a compatible implementation based on that. We're not talking about fiddling at the boundaries here. For instance type inference is completely unspecified. There is no aspect of scala compilation which is not heavily influenced by how type inference unfolds, yet there is nothing. And that is the tip of the iceberg. If you're always hearing about how amazingly small scala's specification is, now you know why.
• ENGNR 12y ago
  Wait.. is it called policy? That's going to be a nightmare to Google Almost as bad as 'play' (which has largely fallen back to 'playframework')
  • whalesalad 12y ago
    Thank goodness you can search "golang" for things related to the go language. That's sometimes a nightmare too.
    • stefantalpalaru 12y ago
      Unfortunately this SEO justified change has permeated the community and now people call the language "golang" instead of "Go".
      • Tuna-Fish 12y ago
        s/Unfortunately/Fortunately/ Go is a terrible, terrible name.
        • gress 12y ago
          Golang is a terrible name.
          • Tuna-Fish 12y ago
            Agree. Still vastly better than go.
  • wtetzner 12y ago
    Yeah, maybe he should call it Paulicy.
    • gress 12y ago
      Pauliz
  • 12y ago
• benjaminjackman 12y ago
  Heh +1 to `trailing commas! I think that's the only "feature".` I don't know why they were removed in the first place.
  • frowaway001 12y ago
    Because the one thing Scala is missing is more ways to do the same thing.
    • benjaminjackman 12y ago
      Same things? I don't know what you mean, It was a language feature back in the 2.7 days I think? I was a long time ago, hard to remember when they got rid of it. It's really nice for this: val color = List( "Red", "Yellow", "Green", ) If I want to add a 4th element I just ctrl+alt+down, I don't have to go back up and add the comma at the end of the last line, I spend too much time juggling commas in lists for my liking. Some programmers work around it like this (which breaks head-inserting, but that's typically a lot less common) : List("Yellow" ,"Red" ,"Green" ) Which is pretty ugly (to me). It's one of those minor QOL improvements where I don't see what the downside is. Frankly I wish every language had this feature.
      • frowaway001 12y ago
        > It's one of those minor QOL improvements where I don't see what the downside is. That's a nice example why 99.9% of developers out there should never design a language.
        • 1ris 12y ago
          Great explanation!
        • equup 12y ago
          You really like to insult everyone who disagrees with you, don't you?
• saosebastiao 12y ago
  Can anyone more familiar with the matter tell me what is wrong with the parallel collections? Does he plan to reimplement them in a more maintainable way? Or do away with them altogether? All the talk about scala makes me wonder what the language would have looked like if Java compatibility was never a goal. I'm no language designer and I'm probably wrong but some things that come to mind are 1) no nulls, 2) no distinction between classes and their companion objects, 3) no distinction between traits and abstract classes.
  • bkirwi 12y ago
    For a short answer to the first question, see his slides on the collection library[0], particularly number 42. The gist is that the Scala collections have too many dimensions (mutable/immutable, sequential/parallel, strict/lazy), all of which share code -- but it turns out that the right implementation for a lazy immutable parallel collection is not necessarily good for a strict mutable sequential collection, and juggling all these variations makes it much trickier to modify or reason about the code. He seems to be fine with the existence of parallel collections, but probably as something that lives on top of the standard library instead of being layered into it. To your second paragraph -- I believe that #2 might have found its way into a Scala-from-scratch, but the others certainly wouldn't. I think the big difference would be in the type system, which could be much simpler if it didn't have to interop with Java's weird generics. [0] http://www.slideshare.net/extempore/a-scala-corrections-library http://www.slideshare.net/extempore/a-scala-corrections-libr...
  • deleted 12y ago
    [deleted]
• cpeterso 12y ago
  Is the name "Policy" a pun on "Paul SC" (Paul's Scala Compiler)? Or is there another meaning?
  • mjt0229 12y ago
    That's what I thought, and it's clever.
• morsch 12y ago
  What the hell is going on? We're in the middle of a major investment into Scala, and now there's forks left and right, both amicable and not-so-amicable. This doesn't inspire confidence, although I'm still classifying it as growing pains/healthy lively ecosystem. I really don't want to stick with Java 8.
  • village-idiot 12y ago
    Clojure isn't forking...
    • Pacabel 12y ago
      But it also doesn't offer many of the benefits and much of the functionality that Scala offers. A familiar C-like syntax, static typing, and so on are very important for a lot of Scala's users.
  • noelwelsh 12y ago
    Nothing really is going on. The Typelevel "fork" of Scala would better be called an extension or an experimental playground. They have broadly the same goal as Odersky for Scala but want to get there faster. They intend to remain compatible with Typesafe's compiler and Typesafe intend to merge their changes back. Paul's fork is him exploring some ideas and not intended for widespread use. To quote the README: "It's still in development, in every imaginable sense. I would prefer not to publish code in this condition ... However since typelevel has also forked the scala compiler, publishing now and giving them an opportunity to exploit my work seems the lesser evil."
• nimrody 12y ago
  asfq1
• nimrody 12y ago
  asfq1
• jongraehl 12y ago
  Paul, why do you say it's wrong that Float(Long.max - Int.max) is equal to Float(Long.max)? Float has less than 32 significant binary digits, and Long.max - Int.max has the same MSBs as Long.max. You would need to use Double to see the difference. I'm not sure what implicit conversions/promotions have to do with that - you want a warning or error when going from Long to Float? That would be reasonable. You can even lose significant digits between Int->Float and Long->Double even though they have the same total number of bits, so you could require those to be explicit. (from a slide in https://www.youtube.com/watch?v=TS1lpKBMkgg https://www.youtube.com/watch?v=TS1lpKBMkgg )
  • extempore 12y ago
    > Paul, why do you say it's wrong that Float(Long.max - Int.max) is equal to Float(Long.max)? I don't. You have inserted coercions. I'm aware of how many bits things have. It's hard to make it through five years on a compiler without gaining some awareness of such details. I consider the introduction of lossy type conversions into the global scope to be a kind of insanity. People should be recoiling from such practices, not looking for ways to justify them.
• jendap 12y ago
  I'm so happy to see this! Thank you Paul! Paul may have been hurting perception of Scala community by his rants a bit lately. But the fork is a greatly welcomed! All the things Paul said make sense. The fork can prove the value of those ideas. Simpler compiler => more compiler developers => more productive => better compiler => better for the whole community! As for Typesafe. There are doing great job. Just two releases ago Scala was not compatible even between minor releases. Stability, documentation, training, driving the community, good roadmap for 2.12, etc. Great job! Plus they are build a great framework around akka and play which is gaining traction. Policy is actually not really a fork. Forks are, imho, projects which diverge code base one day and there's really no sharing between them anymore. Paul is merging from scalac all the time. It really is just vastly simplified scalac with some wise decisions like modularizing some aspects (like parallel collections). BTW: Anybody worried about a friction while trying to push a bold change in a team of more than one person? Have you ever worked in a team of more than one? :-) Just do it. Prove it. Dismiss the FUD. Fist questions on my mind: 1) Future? Is this repo intended to pull from scalac or diverge more? 2) Dropping reflection? Does it means removal of macros as well? For the time being or for good? 3) How about leaf node invariance in collections (and other collection stuff you talked about)?
  • extempore 12y ago
    I will continue to merge from scalac for as long as I can, because with no meaningful specification it is the only way to retain compatibility. I didn't say I'm dropping reflection, I said I'm dropping the scala-reflect layer. It's a gigantic and useless abstraction penalty. I'll fold the pseudo-compiler into the compiler. We'll see about collections. Here's where it stands: https://github.com/paulp/psp-std https://github.com/paulp/psp-std
• charlieflowers 12y ago
  I just watched the talk, and it is full of insight. Highly recommended even if you've never used Scala (as I have not). Some samples (some are quotes, some are paraphrased): * "The compiler [should be] designed around inputs which are Diff[T]s against an in-memory understanding of the last successful compilation.... As a result, both Repl and Debugger are superfluous as separate tools." (emphasis mine) * "I want to programmatically generate ASTs and feed them in" * The AST (the tree) is fundamental, not the language. For example, if you have an AST you're working with, and you hate curly braces, theoretically you could see the same tree "rendered" as text without those curly braces. * "True power is in restriction" & "unnecessary expressiveness is the enemy" -- whatever your language lets you "say", the "other side" (the compiler or interpreter) has to be able to understand. So unlimited language means horribly over-complicated compiler that cannot do nearly as much optimization as you'd like. * The more free your code is of "over-precision", the more powerful that code is, because the compiler can do more with it. Example: the well known example of an imperative for-loop versus the functional "map". * "If you want fast, start with comprehensible."
  • charlieflowers 12y ago
    In fact, towards the end of the talk, I was thinking "Wow, he sounds almost as ambitious as Bret Victor." Then, the next slide was a picture and some quotes from Bret Victor. :)
  • seanmcdirmid 12y ago
    > "The compiler [should be] designed around inputs which are Diff[T]s against an in-memory understanding of the last successful compilation.... As a result, both Repl and Debugger are superfluous as separate tools." (emphasis mine) We did something like this before with scalac back in 2007. But they eventually decided to use lazy rather than incremental evaluation after I left the project (I admit the tech was too crazy and under developed for the time). But the approach wound up working very well, check out: http://research.microsoft.com/en-us/people/smcdirm/managedtime.aspx http://research.microsoft.com/en-us/people/smcdirm/managedti... I'd imagine it would work well for Scala today (it took about 6 months to retrofit scalac in 2007, and most of this was just polishing positions for memoization + getting rid of object identities). However, TypeSafe's current approach to reactive programming seems to revolve around Rx and actors, which isn't very useful in a compiler or live interpreter. > The AST (the tree) is fundamental, not the language. For example, if you have an AST you're working with, and you hate curly braces, theoretically you could see the same tree "rendered" as text without those curly braces. Ya, but all of this can be done in the IDE. Also, projectional editing isn't quite their yet; I prefer hacks like "Little Braces" for Visual Studio that simply shrink brace-only lines to very small font sizes :) Of course, I think syntax is "language", and having multiple languages flying around isn't great for community building.
• cryptos 12y ago
  What is the goal of Paul Philips? He is obviously frustrated by his experiences at Typesafe, but what does he really want? Show us that he can do better? Create an alternative Scala? This seems not be realistic and he seems to be realistic enough to know it. All this sounds like emotional actions from a disappointed ex-lover. Paul, if you read this: why don't you do something completely different? Leave Scala and your personal rage behind.
• 0x434D53 12y ago
  I don't get that outcry and panic inside the Scala community. We should give the 2 forks a chance and see what it leads to. There is still the chance then to reunite the community behind "one" scala. Also they could have great influence on Typesafe Scala. As far as i know the compiler (which is not very much): The 2 forks will anyway not move away from Typesafe scala at huge speed. Let's come here again in 6 months and see where we stand.