10 ms·

Skip – A programming language to skip the things you have already computed

283 points by pbowyer 8y ago

zellyn 8y ago
Apparently this was a research project. Anyone have links to papers, conclusions, results? Should be interesting…
- tnowacki 8y ago
  No papers unfortunately. As far as results, we saw some great numbers for the effectiveness of the recomputation. The language is self hosted. And the type checker is currently incremental. On my machine the initial type checking of the compiler itself is in the ballpark of ~40s. Changing a file and getting new type errors returns in <0.5s
- vjeux 8y ago
  We posted about some of those in the blog over the years. http://skiplang.com/blog http://skiplang.com/blog Some of the interesting ones: - An overview of how memoization works and the MVCC model behind the scenes: http://skiplang.com/blog/2017/01/04/how-memoization-works.html http://skiplang.com/blog/2017/01/04/how-memoization-works.ht... - How pattern matching is implemented and the tricks to make goto work in JavaScript: http://skiplang.com/blog/2017/11/15/simulating-goto-in-javascript.html http://skiplang.com/blog/2017/11/15/simulating-goto-in-javas... - The work done on making error messages much more helpful by understanding common idioms from other programming languages: http://skiplang.com/blog/2017/11/20/fixing-the-syntax-barrier.html http://skiplang.com/blog/2017/11/20/fixing-the-syntax-barrie... - The macro syntax that elegantly solves a lot of use cases where dynamism is commonly used http://skiplang.com/blog/2018/07/24/macros.html http://skiplang.com/blog/2018/07/24/macros.html
  - zackmorris 8y ago
    Thank you, I feel like Skip is very close to the language I've always wanted. Its MVCC looks very similar to Clojure's, I'm having trouble finding a good link about it: https://sw1nn.com/blog/2012/04/11/clojure-stm-what-why-how/ https://sw1nn.com/blog/2012/04/11/clojure-stm-what-why-how/ The gist of it is that instead of tracking multiple separate locations in memory for values, the software transactional memory (STM) references data by value. So if you assign the value {a: 42, b: 24} to two variables x and y, that value is only stored in memory in one place that the variables both point to. Then if one of the variables changes something, for example y.b = 25, this big tree structure works like copy-on-write and makes copies of branches when mutations occur. So internally, a: 42 is one reference and b: (24 or 25) is another reference. So rather than using 4 cells of memory, we've only used 3. This frees the developer from having to micromanage memory resources and makes a lot of other things like concurrency "just work" without locks. Do I have this correct?
    - vjeux 8y ago
      Your explanation looks correct! The practical application is having a webserver that runs a bunch of requests in parallel but that can all share the same memoization cache. We don't want to lock the full memoization cache and therefore block all the other requests when one thread writes a new value.
    - Serow225 8y ago
      FWIW, MATLAB does a lot of this kind of optimization in its internal implementation of the core datatypes. There are a lot of smart compiler/JIT people that have worked hard on it over the years to optimize the internal guts despite the naive design of the language due to its age and some poor design decisions made ages ago that cannot be changed due to extreme backwards compatibility requirements. Kind of like javascript and V8/SpiderMonkey :)
      - zackmorris 8y ago
        Thank you, I didn't know that MATLAB did that. I'm a huge fan of it and hope that someday other matrix programming frameworks like TensorFlow, OpenCL and CUDA move to a similar general-purpose syntax that frees the developer from having to deal with the boilerplate associated with old OpenGL-style buffer representations.
thedirt0115 8y ago
Anyone have some insider knowledge on the history of this project? I'm curious as to how a big multi-year create-a-language research project like this gets pitched, accepted, and finally sunset -- Was it a full-time project? Were there deadlines? Was this considered a success or a failure?
- grawprog 8y ago
  The page itself mentions the project took place from 2015-2018. There's also what looks like a devlog. I'm not sure how far back it goes but it may have the details you're looking for.
falardea 8y ago
"supports ergonomic asynchronous computation"... because you wouldn't want any of your computations getting carpal tunnel or anything.
timdorr 8y ago
> The Skip project concluded in 2018 and Skip is no longer under active development at Facebook. Just a warning if you're interested in using this.
- jverlaguet 8y ago
  I would like to add that I intend to maintain it and build a community around it in outside of FB as it is mentioned on the front-page. "The language, compiler and libraries are maintained as a side project by Julien Verlaguet, the main designer of the language."
  - fusiongyro 8y ago
    How does your language compare to the glut of modern systems-ish languages like Rust, Go, Swift, Julia, Nim? I see a lot of C++ influence here. Your docs don't explain how you make parallelism ergonomic, but the "async" keyword is getting popular. Why would I choose to invest in your language over one of these other ones that has more momentum?
    - aylmao 8y ago
      > Why would I choose to invest in your language over one of these other ones that has more momentum? FWIW I don't think anyone is claiming you should, especially given it's not under active development (:
      - jverlaguet 8y ago
        That's not exactly true. It's not under active development at Facebook. I still intend to work on Skip as it is said on the front-page: "The language, compiler and libraries are maintained as a side project by Julien Verlaguet, the main designer of the language." And I am looking for people to help ;-)
        dmix 8y ago
        This was an unfortunate choice of wording, that maybe should be updated: > The Skip project concluded in 2018 and Skip is no longer under active development at Facebook. This makes it sound like the project was killed off entirely, as it mentions its 'concluded'.
chrisaycock 8y ago
How are the side effects tracked? I couldn't find it in a brief read of the source code. The documentation mentions sequence points at which previous side effects are guaranteed to have finished. But what actually tracks whether IO or state changes have occurred?
- tnowacki 8y ago
  IO is managed through very specific native extensions where updates to that IO source are able to be tracked outside of the language and fed back in. For example, we don't have anything that lets you read from command line or stdin. But you can open and watch a file. Any changes to that file will be tracked by the runtime. Other non deterministic computation, like randomness or time is outright banned in the tracked environment. (There is an opt-in 'untracked' modifier if you want to write code outside of the reactive environment). For other tricky behavior, like mutability/mutations on an object, the type system tracks the mutability mode of any object. Only objects that are fully immutable ('frozen' in the language) can be memoized inputs/outputs to a function.
  - X6S1x6Okd1st 8y ago
    > For example, we don't have anything that lets you read from command line or stdin. But you can open and watch a file. whats the difference between opening a file and watching it and watching stdin?
    - TheDong 8y ago
      inotify and seek works on most files. FIFO file descriptors, not so much. If you can't seek backwards, you pretty much have to cache the whole contents in memory, which seems wildly inefficient.
adamnemecek 8y ago
The link to specification is broken.
- tnowacki 8y ago
  Where do you see a link to that? We had a specification in the works, but it fell horribly out of date. I wasn't sure that we kept it around
  - adamnemecek 8y ago
    It's in the footer. CTRL + F for specification.
    - tnowacki 8y ago
      I'll look into removing it. If you want an overview of language features/design. I'd start with the docs: http://skiplang.com/docs/hello_world.html http://skiplang.com/docs/hello_world.html
      - adamnemecek 8y ago
        Can I email you?
        vjeux 8y ago
        You didn't ask me personally but feel free to email me, I've helped with the open sourcing effort: vjeux@fb.com
      - mncharity 8y ago
        FWIW, http://skiplang.com/docs/type_definitions.html http://skiplang.com/docs/type_definitions.html says "A class can define a type, and later redefine it in on of [sic] it's children." Thanks for sharing your work.
lifthrasiir 8y ago
I always had my go-to questions for commercially sponsored programming language projects after having done some of them: How did them start? Did them pivot from, say, a personal project or were them thoroughly planned within the company? How much resource was spent and how was the (current) fate of the project determined?
X6S1x6Okd1st 8y ago
> Skip also features a code-formatter to ensure consistent code style and a tool for running codemods. Nice! I love this trend.
- vjeux 8y ago
  Yeah, before joining the Skip team I worked on Prettier. Once you have an automatic formatter, you can't really go back, so I ported the core algorithm of prettier to Skip and built all the formatter for it. My biggest surprise was that the language has been designed by people that have been working on languages for their entire lives so it was dead simple to write the formatter for it compared to JavaScript. It only took me ~2 weeks to get it in a place where we could reasonably convert the entire codebase to it. It took Prettier ~6 months to get to that point.
  - X6S1x6Okd1st 8y ago
    Yeah my first experience was with gofmt. I am currently working in both large and old java code bases so I have needed to only use tools that auto format lines that are staged for committing, but it was one of the first things I set up.
  - mattdeboard 8y ago
    I recently read a couple github threads in prettier's repo that said prettier won't do import optimization/organization, nor object property organization (e.g. {c: 3, b: 2} -> {b: 2, c: 3}) because it has potential to cause AST changes and could break things for people. Do you think it's feasible to have a "jsfmt" tool or is the technical hurdle too great?
    - vjeux 8y ago
      A lot of people have successfully coupled eslint with prettier, where they enable eslint rules that do those transforms with autofix.
X6S1x6Okd1st 8y ago
Does skip target js just through emscripten? Does it have good support for browser interaction?
- vjeux 8y ago
  We have js printed at the "OuterIST" phase right now. It's the phase just before we sent to the native backend which will generate llvm. You can go to the playground ( http://skiplang.com/playground/ http://skiplang.com/playground/ ) and click "JavaScript" to see the output. We talked about doing it at an earlier phase in order to output JavaScript closer to what the input code was, in a very similar fashion as BuckleScript does. But we never gotten around to as we focused on the native backend. We also have a prototype of a wasm backend using emscriptem. Right now none of them are really usable as is but there's a lot of potential there.
qwerty456127 8y ago
Just introduce a syntax to define immutable variables (e.g. like "val" in Scala) and to mark particular functions as pure (easy to implement manually as a decorator in Python, I've been using it a lot) and memoization becomes a seemingly easy task. Why a new language? By the way it seems very sad to me that the majority of imperative and hybrid (functional×imperative) languages lack syntax for immutable variables: in just so many cases I introduce a variable to store an intermediate result of an operation an don't mean to re-assign it ever after, it's nice of a developer to define this intention explicitly and of a compiler to raise an error when the variable is re-assigned accidentally - this simple feature is among the reasons why Scala programs usually are comparably easy to debug and run as expected as soon as they get compiled successfully.
- vjeux 8y ago
  The problem that the language is trying to solve is essentially building safe caches. Once a value is in the cache it cannot be changed from inside the program (the solution we opted for is to make it deeply immutable) and externally (if it depends on some external data, we need to invalidate the cache when it changes). The big challenge is how do you maintain those guarantees while having a language that product engineers would want to write in. You can decide to not allow mutability in your language but developers need to think about code in a very very different way than what they do now. It’s also going to be challenging to convert/translate existing codebases and patterns to it. So we decided to allow both mutability and immutability. But it poses very interesting challenges. For example in the standard library, what is the type of x.map(y -> z). Which pieces are mutable or immutable. If you are not precise enough in your type system, then you are going to type it as “i don’t know (readonly)” or “mutable” but then you can’t pass the result of this to a memoized function, so you lost the point of the language. The other interesting aspect is that the backend relies on the fact that the invariant are true in order to output optimized code and garbage collector. So we cannot “cheat” like many gradual typed languages like Hack, TypeScript, C# where they can just say, actually this type is wrong but i’m just going to ignore it and it’ll throw an exception at runtime if the programmer got it wrong. It would segfault in Skip case.
  - 8y ago
joedevon 8y ago
Sounds a bit like netkernel to me. http://www.1060research.com/products/ http://www.1060research.com/products/
jhallenworld 8y ago
The IBM DataPower XSLT compiler has automatic memoization, I remember the compiler engineers implementing this. Of course XSLT is a functional language, so it's not so hard.. Bob Morgan was behind it.
mpartel 8y ago
This could be really useful for game development! Certain kinds of networked games are built as a model, deterministically updated by clock ticks and commands from the server, and a view layer on top of that. And even model-level objects often monitor each other for changes. Seems like a perfect fit. Game objects often have graph-like (not tree-like) dependencies though. E.g. two characters may want to move towards each other, creating a pointer cycle. Ideally I'd want one to be "magically" updated if the other disappears or changes state. I wonder if Skip has a good answer there, or if a good language-level answer is even theoretically possible.
- vjeux 8y ago
  Game development was a major driver for the design of Skip but for another reason: the GC. The big problem with languages with GC is that it is impredictable: when the amount of allocated memory globally passes a certain threshold, then a GC pass happens and it’s unclear how much time it’ll take and you may miss your frame. The idea of Skip is that GC always or never triggers for some functions. So if while you're developing your game/app the time it takes to GC is within the bounds you wanted, then it’ll keep behaving this way in production. If it is too expensive for your use case, then you can optimize that specific function (and what it calls) using a profiler. You don’t have to think about the context of the entire app to figure out how to reduce GC pauses.
- jverlaguet 8y ago
  It's definitely possible. You need to "break the cycle" by adding an additional layer. You can encode any graph as an array of objects were you replaced the pointers with indexes within the array. Of course, the granularity (how big the arrays are) is up to you, and there is a tradeoff here. The other very important thing to note is that Skip has a memory model suitable for that use case. Every function has a GC overhead, but that overhead is non-contextual. Meaning, the GC will collect the memory for only one function (instead of the entire heap). This is possible thanks to the guarantees of the type-system.
  - mpartel 8y ago
    So, say I have: `characters = [A, B, ...]` and I want to maintain for each character `x` the invariant: `x.target = (first y in 'characters' where y.score > x.score)` Can Skip maintain such a self-referential invariant automatically? Does the answer depend on A and B's mutability? (This "graph as array" escape hatch seems to come up in many language designs with compile-time pointer analysis. It's useful, maybe even essential, but you tend to lose some nice language features/guarantees with it, in my limited experience.) The GC thing sounds excellent!
qwerty456127 8y ago
By the way, I've got a question to programming language engineers and people keeping track of emerging and experimental languages: is there a language where everything (or almost everything, excluding elementary types and structs perhaps) is an "actor" and every class method call is an asynchronous message passing? Together with an idea of a heavily-memoized language (which Skip is meant to be an implementation of) this idea won't leave my mind for years since I've first learnt about the actor model.
- bitexploder 8y ago
  Well, Obj-C used true message passing to call methods. And SmallTalk is also a True OO language in that sense. I wonder if SmallTalk isn’t a good starting point for such a language.
- scroot 8y ago
  I believe that some Smalltalkers were able to turn objects into things that were "like actors" when implementing Croquet / OpenCobalt [1]. These systems relied upon creating their own sense of "time" via a protocol named TeaTime [2]. I have tried to get more information on the actual implementation of TeaTime but have come up short so far. [1] https://en.wikipedia.org/wiki/Croquet_Project https://en.wikipedia.org/wiki/Croquet_Project [2] http://www.vpri.org/pdf/tr2003001_croq_collab.pdf http://www.vpri.org/pdf/tr2003001_croq_collab.pdf
- jitl 8y ago
  On the subject of actor-concurrency languages, you should check out Pony. In Pony, actors are very common, but they are not the ubiquitous data-type. Actors exist along side regular classes in the language. See https://tutorial.ponylang.io/types/actors.html https://tutorial.ponylang.io/types/actors.html
- hobofan 8y ago
  There was Axum[0], a research language by Microsoft. [0]: https://en.wikipedia.org/wiki/Axum_(programming_language) https://en.wikipedia.org/wiki/Axum_(programming_language)
mrkgnao 8y ago
Can anyone comment on the similarities with (apparently somewhat less sophisticated) laziness as in Haskell, or the memoisation/caching done by the Nix package manager, or (perhaps most interestingly) the approach of the Funflow library for Haskell[1]? [1]: https://www.tweag.io/posts/2018-07-10-funflow-make.html https://www.tweag.io/posts/2018-07-10-funflow-make.html
thom 8y ago
Incremental computation is a fascinating area of research, I've followed Matthew Hammer's work on Adapton[1] for a while, as well as Daco Harkes work on IceDust[2]. The issues are obviously quite deep (i.e. how do can you change a single value and recalculate quicksort without the effects of the change fanning out massively?) whether you're targeting existing languages and runtimes or building from scratch. I would very much like to see this as an area of active development both in programming languages and distributed computing platforms. Current streaming platforms are great and it's getting easier and easier to create declarative data flow pipelines, doing real-time processing, splitting things into windows etc. But on top of that I imagine incorporating incremental computation, allowing you to keep a durable history of the event stream (or at least manageable parts of it) in a way that automagically allows incremental recalculation of upstream data when an underlying fact is retracted or updated. [1] http://adapton.org/ http://adapton.org/ [2] https://dcharkes.github.io/ https://dcharkes.github.io/
- thom 8y ago
  Of course if that already exists then I'd love to hear about it! There seem to be all sorts of hybrid stream/batch systems but none with quite this focus. Effectively I want stuff that streams into a time series database in realtime, with the opportunity to edit or delete events later. On top of that there would be a processing pipeline with familiar streaming paradigms (transforming data, windowing data arbitrarily, aggregating on top of that) with the additional magic that the necessary events/windows/calculations re-fire if some underlying data changes. All in a magically efficient and transparent way.
  - frankmcsherry 8y ago
    Shameless plug for something that does all of these things (perhaps not exactly as you want, but ..) https://github.com/frankmcsherry/differential-dataflow https://github.com/frankmcsherry/differential-dataflow
    - thom 8y ago
      Wonderful! I feel bad that I didn't follow the work after hearing about Naiad originally. Where might I go to read more about the fault tolerance story - I note that's an ongoing area of research in the README. Also the link to the Kafka adapter is dead, is that work ongoing or does the story end at the capture/replay API? (Happy to bombard you with email if that's any more manageable for you!) Reading the 'when not to use Timely Dataflow' section on sorting, are there any ideas from Nominal Adapton that are relevant in this distributed setting, when we're talking about small updates or inserts?
bausshf 8y ago
You should consider https for your site.
deleted 8y ago
[deleted]

283 points by pbowyer 8y ago

zellyn 8y ago

Apparently this was a research project. Anyone have links to papers, conclusions, results? Should be interesting…

tnowacki 8y ago
No papers unfortunately. As far as results, we saw some great numbers for the effectiveness of the recomputation. The language is self hosted. And the type checker is currently incremental. On my machine the initial type checking of the compiler itself is in the ballpark of ~40s. Changing a file and getting new type errors returns in <0.5s
vjeux 8y ago
We posted about some of those in the blog over the years. http://skiplang.com/blog http://skiplang.com/blog Some of the interesting ones: - An overview of how memoization works and the MVCC model behind the scenes: http://skiplang.com/blog/2017/01/04/how-memoization-works.html http://skiplang.com/blog/2017/01/04/how-memoization-works.ht... - How pattern matching is implemented and the tricks to make goto work in JavaScript: http://skiplang.com/blog/2017/11/15/simulating-goto-in-javascript.html http://skiplang.com/blog/2017/11/15/simulating-goto-in-javas... - The work done on making error messages much more helpful by understanding common idioms from other programming languages: http://skiplang.com/blog/2017/11/20/fixing-the-syntax-barrier.html http://skiplang.com/blog/2017/11/20/fixing-the-syntax-barrie... - The macro syntax that elegantly solves a lot of use cases where dynamism is commonly used http://skiplang.com/blog/2018/07/24/macros.html http://skiplang.com/blog/2018/07/24/macros.html
- zackmorris 8y ago
  Thank you, I feel like Skip is very close to the language I've always wanted. Its MVCC looks very similar to Clojure's, I'm having trouble finding a good link about it: https://sw1nn.com/blog/2012/04/11/clojure-stm-what-why-how/ https://sw1nn.com/blog/2012/04/11/clojure-stm-what-why-how/ The gist of it is that instead of tracking multiple separate locations in memory for values, the software transactional memory (STM) references data by value. So if you assign the value {a: 42, b: 24} to two variables x and y, that value is only stored in memory in one place that the variables both point to. Then if one of the variables changes something, for example y.b = 25, this big tree structure works like copy-on-write and makes copies of branches when mutations occur. So internally, a: 42 is one reference and b: (24 or 25) is another reference. So rather than using 4 cells of memory, we've only used 3. This frees the developer from having to micromanage memory resources and makes a lot of other things like concurrency "just work" without locks. Do I have this correct?
  - vjeux 8y ago
    Your explanation looks correct! The practical application is having a webserver that runs a bunch of requests in parallel but that can all share the same memoization cache. We don't want to lock the full memoization cache and therefore block all the other requests when one thread writes a new value.
  - Serow225 8y ago
    FWIW, MATLAB does a lot of this kind of optimization in its internal implementation of the core datatypes. There are a lot of smart compiler/JIT people that have worked hard on it over the years to optimize the internal guts despite the naive design of the language due to its age and some poor design decisions made ages ago that cannot be changed due to extreme backwards compatibility requirements. Kind of like javascript and V8/SpiderMonkey :)
    - zackmorris 8y ago
      Thank you, I didn't know that MATLAB did that. I'm a huge fan of it and hope that someday other matrix programming frameworks like TensorFlow, OpenCL and CUDA move to a similar general-purpose syntax that frees the developer from having to deal with the boilerplate associated with old OpenGL-style buffer representations.

thedirt0115 8y ago

Anyone have some insider knowledge on the history of this project? I'm curious as to how a big multi-year create-a-language research project like this gets pitched, accepted, and finally sunset -- Was it a full-time project? Were there deadlines? Was this considered a success or a failure?

grawprog 8y ago
The page itself mentions the project took place from 2015-2018. There's also what looks like a devlog. I'm not sure how far back it goes but it may have the details you're looking for.

falardea 8y ago

"supports ergonomic asynchronous computation"... because you wouldn't want any of your computations getting carpal tunnel or anything.

timdorr 8y ago

> The Skip project concluded in 2018 and Skip is no longer under active development at Facebook. Just a warning if you're interested in using this.

jverlaguet 8y ago
I would like to add that I intend to maintain it and build a community around it in outside of FB as it is mentioned on the front-page. "The language, compiler and libraries are maintained as a side project by Julien Verlaguet, the main designer of the language."
- fusiongyro 8y ago
  How does your language compare to the glut of modern systems-ish languages like Rust, Go, Swift, Julia, Nim? I see a lot of C++ influence here. Your docs don't explain how you make parallelism ergonomic, but the "async" keyword is getting popular. Why would I choose to invest in your language over one of these other ones that has more momentum?
  - aylmao 8y ago
    > Why would I choose to invest in your language over one of these other ones that has more momentum? FWIW I don't think anyone is claiming you should, especially given it's not under active development (:
    - jverlaguet 8y ago
      That's not exactly true. It's not under active development at Facebook. I still intend to work on Skip as it is said on the front-page: "The language, compiler and libraries are maintained as a side project by Julien Verlaguet, the main designer of the language." And I am looking for people to help ;-)
      - dmix 8y ago
        This was an unfortunate choice of wording, that maybe should be updated: > The Skip project concluded in 2018 and Skip is no longer under active development at Facebook. This makes it sound like the project was killed off entirely, as it mentions its 'concluded'.

chrisaycock 8y ago

How are the side effects tracked? I couldn't find it in a brief read of the source code. The documentation mentions sequence points at which previous side effects are guaranteed to have finished. But what actually tracks whether IO or state changes have occurred?

tnowacki 8y ago
IO is managed through very specific native extensions where updates to that IO source are able to be tracked outside of the language and fed back in. For example, we don't have anything that lets you read from command line or stdin. But you can open and watch a file. Any changes to that file will be tracked by the runtime. Other non deterministic computation, like randomness or time is outright banned in the tracked environment. (There is an opt-in 'untracked' modifier if you want to write code outside of the reactive environment). For other tricky behavior, like mutability/mutations on an object, the type system tracks the mutability mode of any object. Only objects that are fully immutable ('frozen' in the language) can be memoized inputs/outputs to a function.
- X6S1x6Okd1st 8y ago
  > For example, we don't have anything that lets you read from command line or stdin. But you can open and watch a file. whats the difference between opening a file and watching it and watching stdin?
  - TheDong 8y ago
    inotify and seek works on most files. FIFO file descriptors, not so much. If you can't seek backwards, you pretty much have to cache the whole contents in memory, which seems wildly inefficient.

adamnemecek 8y ago

The link to specification is broken.

tnowacki 8y ago
Where do you see a link to that? We had a specification in the works, but it fell horribly out of date. I wasn't sure that we kept it around
- adamnemecek 8y ago
  It's in the footer. CTRL + F for specification.
  - tnowacki 8y ago
    I'll look into removing it. If you want an overview of language features/design. I'd start with the docs: http://skiplang.com/docs/hello_world.html http://skiplang.com/docs/hello_world.html
    - adamnemecek 8y ago
      Can I email you?
      - vjeux 8y ago
        You didn't ask me personally but feel free to email me, I've helped with the open sourcing effort: vjeux@fb.com
    - mncharity 8y ago
      FWIW, http://skiplang.com/docs/type_definitions.html http://skiplang.com/docs/type_definitions.html says "A class can define a type, and later redefine it in on of [sic] it's children." Thanks for sharing your work.

lifthrasiir 8y ago

I always had my go-to questions for commercially sponsored programming language projects after having done some of them: How did them start? Did them pivot from, say, a personal project or were them thoroughly planned within the company? How much resource was spent and how was the (current) fate of the project determined?

X6S1x6Okd1st 8y ago

> Skip also features a code-formatter to ensure consistent code style and a tool for running codemods. Nice! I love this trend.

vjeux 8y ago
Yeah, before joining the Skip team I worked on Prettier. Once you have an automatic formatter, you can't really go back, so I ported the core algorithm of prettier to Skip and built all the formatter for it. My biggest surprise was that the language has been designed by people that have been working on languages for their entire lives so it was dead simple to write the formatter for it compared to JavaScript. It only took me ~2 weeks to get it in a place where we could reasonably convert the entire codebase to it. It took Prettier ~6 months to get to that point.
- X6S1x6Okd1st 8y ago
  Yeah my first experience was with gofmt. I am currently working in both large and old java code bases so I have needed to only use tools that auto format lines that are staged for committing, but it was one of the first things I set up.
- mattdeboard 8y ago
  I recently read a couple github threads in prettier's repo that said prettier won't do import optimization/organization, nor object property organization (e.g. {c: 3, b: 2} -> {b: 2, c: 3}) because it has potential to cause AST changes and could break things for people. Do you think it's feasible to have a "jsfmt" tool or is the technical hurdle too great?
  - vjeux 8y ago
    A lot of people have successfully coupled eslint with prettier, where they enable eslint rules that do those transforms with autofix.

X6S1x6Okd1st 8y ago

Does skip target js just through emscripten? Does it have good support for browser interaction?

vjeux 8y ago
We have js printed at the "OuterIST" phase right now. It's the phase just before we sent to the native backend which will generate llvm. You can go to the playground ( http://skiplang.com/playground/ http://skiplang.com/playground/ ) and click "JavaScript" to see the output. We talked about doing it at an earlier phase in order to output JavaScript closer to what the input code was, in a very similar fashion as BuckleScript does. But we never gotten around to as we focused on the native backend. We also have a prototype of a wasm backend using emscriptem. Right now none of them are really usable as is but there's a lot of potential there.

qwerty456127 8y ago

Just introduce a syntax to define immutable variables (e.g. like "val" in Scala) and to mark particular functions as pure (easy to implement manually as a decorator in Python, I've been using it a lot) and memoization becomes a seemingly easy task. Why a new language? By the way it seems very sad to me that the majority of imperative and hybrid (functional×imperative) languages lack syntax for immutable variables: in just so many cases I introduce a variable to store an intermediate result of an operation an don't mean to re-assign it ever after, it's nice of a developer to define this intention explicitly and of a compiler to raise an error when the variable is re-assigned accidentally - this simple feature is among the reasons why Scala programs usually are comparably easy to debug and run as expected as soon as they get compiled successfully.

vjeux 8y ago
The problem that the language is trying to solve is essentially building safe caches. Once a value is in the cache it cannot be changed from inside the program (the solution we opted for is to make it deeply immutable) and externally (if it depends on some external data, we need to invalidate the cache when it changes). The big challenge is how do you maintain those guarantees while having a language that product engineers would want to write in. You can decide to not allow mutability in your language but developers need to think about code in a very very different way than what they do now. It’s also going to be challenging to convert/translate existing codebases and patterns to it. So we decided to allow both mutability and immutability. But it poses very interesting challenges. For example in the standard library, what is the type of x.map(y -> z). Which pieces are mutable or immutable. If you are not precise enough in your type system, then you are going to type it as “i don’t know (readonly)” or “mutable” but then you can’t pass the result of this to a memoized function, so you lost the point of the language. The other interesting aspect is that the backend relies on the fact that the invariant are true in order to output optimized code and garbage collector. So we cannot “cheat” like many gradual typed languages like Hack, TypeScript, C# where they can just say, actually this type is wrong but i’m just going to ignore it and it’ll throw an exception at runtime if the programmer got it wrong. It would segfault in Skip case.
- 8y ago

joedevon 8y ago

Sounds a bit like netkernel to me. http://www.1060research.com/products/ http://www.1060research.com/products/

jhallenworld 8y ago

The IBM DataPower XSLT compiler has automatic memoization, I remember the compiler engineers implementing this. Of course XSLT is a functional language, so it's not so hard.. Bob Morgan was behind it.

mpartel 8y ago

This could be really useful for game development! Certain kinds of networked games are built as a model, deterministically updated by clock ticks and commands from the server, and a view layer on top of that. And even model-level objects often monitor each other for changes. Seems like a perfect fit. Game objects often have graph-like (not tree-like) dependencies though. E.g. two characters may want to move towards each other, creating a pointer cycle. Ideally I'd want one to be "magically" updated if the other disappears or changes state. I wonder if Skip has a good answer there, or if a good language-level answer is even theoretically possible.

vjeux 8y ago
Game development was a major driver for the design of Skip but for another reason: the GC. The big problem with languages with GC is that it is impredictable: when the amount of allocated memory globally passes a certain threshold, then a GC pass happens and it’s unclear how much time it’ll take and you may miss your frame. The idea of Skip is that GC always or never triggers for some functions. So if while you're developing your game/app the time it takes to GC is within the bounds you wanted, then it’ll keep behaving this way in production. If it is too expensive for your use case, then you can optimize that specific function (and what it calls) using a profiler. You don’t have to think about the context of the entire app to figure out how to reduce GC pauses.
jverlaguet 8y ago
It's definitely possible. You need to "break the cycle" by adding an additional layer. You can encode any graph as an array of objects were you replaced the pointers with indexes within the array. Of course, the granularity (how big the arrays are) is up to you, and there is a tradeoff here. The other very important thing to note is that Skip has a memory model suitable for that use case. Every function has a GC overhead, but that overhead is non-contextual. Meaning, the GC will collect the memory for only one function (instead of the entire heap). This is possible thanks to the guarantees of the type-system.
- mpartel 8y ago
  So, say I have: `characters = [A, B, ...]` and I want to maintain for each character `x` the invariant: `x.target = (first y in 'characters' where y.score > x.score)` Can Skip maintain such a self-referential invariant automatically? Does the answer depend on A and B's mutability? (This "graph as array" escape hatch seems to come up in many language designs with compile-time pointer analysis. It's useful, maybe even essential, but you tend to lose some nice language features/guarantees with it, in my limited experience.) The GC thing sounds excellent!

qwerty456127 8y ago

By the way, I've got a question to programming language engineers and people keeping track of emerging and experimental languages: is there a language where everything (or almost everything, excluding elementary types and structs perhaps) is an "actor" and every class method call is an asynchronous message passing? Together with an idea of a heavily-memoized language (which Skip is meant to be an implementation of) this idea won't leave my mind for years since I've first learnt about the actor model.

bitexploder 8y ago
Well, Obj-C used true message passing to call methods. And SmallTalk is also a True OO language in that sense. I wonder if SmallTalk isn’t a good starting point for such a language.
scroot 8y ago
I believe that some Smalltalkers were able to turn objects into things that were "like actors" when implementing Croquet / OpenCobalt [1]. These systems relied upon creating their own sense of "time" via a protocol named TeaTime [2]. I have tried to get more information on the actual implementation of TeaTime but have come up short so far. [1] https://en.wikipedia.org/wiki/Croquet_Project https://en.wikipedia.org/wiki/Croquet_Project [2] http://www.vpri.org/pdf/tr2003001_croq_collab.pdf http://www.vpri.org/pdf/tr2003001_croq_collab.pdf
jitl 8y ago
On the subject of actor-concurrency languages, you should check out Pony. In Pony, actors are very common, but they are not the ubiquitous data-type. Actors exist along side regular classes in the language. See https://tutorial.ponylang.io/types/actors.html https://tutorial.ponylang.io/types/actors.html
hobofan 8y ago
There was Axum[0], a research language by Microsoft. [0]: https://en.wikipedia.org/wiki/Axum_(programming_language) https://en.wikipedia.org/wiki/Axum_(programming_language)

mrkgnao 8y ago

Can anyone comment on the similarities with (apparently somewhat less sophisticated) laziness as in Haskell, or the memoisation/caching done by the Nix package manager, or (perhaps most interestingly) the approach of the Funflow library for Haskell[1]? [1]: https://www.tweag.io/posts/2018-07-10-funflow-make.html https://www.tweag.io/posts/2018-07-10-funflow-make.html

thom 8y ago

Incremental computation is a fascinating area of research, I've followed Matthew Hammer's work on Adapton[1] for a while, as well as Daco Harkes work on IceDust[2]. The issues are obviously quite deep (i.e. how do can you change a single value and recalculate quicksort without the effects of the change fanning out massively?) whether you're targeting existing languages and runtimes or building from scratch. I would very much like to see this as an area of active development both in programming languages and distributed computing platforms. Current streaming platforms are great and it's getting easier and easier to create declarative data flow pipelines, doing real-time processing, splitting things into windows etc. But on top of that I imagine incorporating incremental computation, allowing you to keep a durable history of the event stream (or at least manageable parts of it) in a way that automagically allows incremental recalculation of upstream data when an underlying fact is retracted or updated. [1] http://adapton.org/ http://adapton.org/ [2] https://dcharkes.github.io/ https://dcharkes.github.io/

thom 8y ago
Of course if that already exists then I'd love to hear about it! There seem to be all sorts of hybrid stream/batch systems but none with quite this focus. Effectively I want stuff that streams into a time series database in realtime, with the opportunity to edit or delete events later. On top of that there would be a processing pipeline with familiar streaming paradigms (transforming data, windowing data arbitrarily, aggregating on top of that) with the additional magic that the necessary events/windows/calculations re-fire if some underlying data changes. All in a magically efficient and transparent way.
- frankmcsherry 8y ago
  Shameless plug for something that does all of these things (perhaps not exactly as you want, but ..) https://github.com/frankmcsherry/differential-dataflow https://github.com/frankmcsherry/differential-dataflow
  - thom 8y ago
    Wonderful! I feel bad that I didn't follow the work after hearing about Naiad originally. Where might I go to read more about the fault tolerance story - I note that's an ongoing area of research in the README. Also the link to the Kafka adapter is dead, is that work ongoing or does the story end at the capture/replay API? (Happy to bombard you with email if that's any more manageable for you!) Reading the 'when not to use Timely Dataflow' section on sorting, are there any ideas from Nominal Adapton that are relevant in this distributed setting, when we're talking about small updates or inserts?

bausshf 8y ago

You should consider https for your site.

deleted 8y ago

[deleted]