5 ms·

Scrapscript: A functional, content-addressable programming language

193 points by luu 2y ago

surprisetalk 2y ago
Main site/community: • https://scrapscript.org https://scrapscript.org • https://scrapscript.discourse.group https://scrapscript.discourse.group • https://news.scrapscript.org https://news.scrapscript.org Max's brilliant dev insights: • https://bernsteinbear.com/blog/scrapscript https://bernsteinbear.com/blog/scrapscript • https://bernsteinbear.com/blog/scrapscript-baseline https://bernsteinbear.com/blog/scrapscript-baseline • https://bernsteinbear.com/blog/scrapscript-tricks https://bernsteinbear.com/blog/scrapscript-tricks I posted a brief update here this morning: • https://scrapscript.discourse.group/t/2024-07-23-design-development-update/85 https://scrapscript.discourse.group/t/2024-07-23-design-deve... Stay tuned! Lots of cool stuff coming :) Some sneak previews: • https://taylor.town/sle-2024-cfp https://taylor.town/sle-2024-cfp • https://taylor.town/unsound-2024-cfp https://taylor.town/unsound-2024-cfp
emmanueloga_ 2y ago
> Scrapscript solves the software sharability problem. > it’s JSON with types and functions and hashed references > it’s tiny Haskell with extreme syntactic consistency Content-addressability is one of the main features of Unison [1]. How do the languages compare? Why did you feel the need to create a new language (especially since both have a strong Haskell flavor). I'm guessing lots of people will have these questions so it could be good to mention Unison somewhere in your materials. The language seems to have big goals, so it doesn't seem that you are working on it "just for fun" :-). Are there other "content-addressable" languages out there other than these two? -- 1: https://www.unison-lang.org/ https://www.unison-lang.org/
- surprisetalk 2y ago
  Great questions and observations! -- > Why did you feel the need to create a new language? Well when I started on this language in 2017-2019(?), I definitely wasn't aware of Unison. I remember talking to Richard Feldman about scrapscript when he first started working on Roc (fun fact: I was the first person Feldman added to the private repo!). I spent a long time on the initial Scrapscript implementations, trying to compete in some similar applications as Roc, and Feldman just absolutely crushed it. I felt (and still feel) that Roc is doing a great job at the low-level "platform" experience I yearn for. [1] https://www.roc-lang.org/ https://www.roc-lang.org/ Eventually I met some Unison folks at Strange Loop (2019?), and we had very different inspirations and goals. That seems to still be the case. There are some hard-to-articulate things I want out of the web, and Unison's architecture doesn't seem suited for it. Anyway, Scrapscript only got a bunch of attention a year or two when I started working on it in public a bit more. I think defeating alcoholism also made me much more productive haha -- > How do the languages compare? From what I understand, Unison has very little in common with Scrapscript. "Content-addressibilty" is really more of a mechanism than a "feature". Kinda like how AI is not really a "feature". You can't just sprinkle SHA on something and go raise venture capital haha I think one major difference is that Unison opted for a more traditional git-based paradigm and then built some stellar dev tools on top of that. So far, Scrapscript is a bit more ambitious in its plans. So go try out Unison! And come back and play with Scrapscript too, when it's more mature :)
  - emmanueloga_ 2y ago
    Congrats on all the impressive progress!
  - the_duke 2y ago
    > From what I understand, Unison has very little in common with Scrapscript. I think you should read up on Unison a bit more, I see a lot of overlap. Both identify code with hashes, and implement code sharing through hash based syncing. Builtin serialization is also present in both, and both are very Haskell inspired. One difference I see is the sharing mechanism, where Unison has a local sqlite repository and tooling for uploading, while Scrapscript seems to be built more around IPFS. But that seems like an implementation detail rather than a fundemantal difference. Unison supports algebraic effects, aka abilities with handlers, which is a much more powerful and general concept than the platforms in Roc.
    - actionfromafar 2y ago
      I think ergonomics and practicality could use more consideration when discussing new languages. A language exists in an ecosystem. At a glance to me, Scrapscript looks very traditional and easy to grok from a regular programmers perspective, there are build steps, you can use an IDE or editor etc. Unison feels more like "Smalltalk Squeak" and Scrapscript feels more like "GNU Smalltalk". Come live in our new world, or integrate new stuff into your existing world kind of vibe.
  - efnx 2y ago
    > I think defeating alcoholism also made me much more productive haha Just out of curiosity and because we all have different limits - how much do you think you used to drink, generally, and what helped you quit?
    - surprisetalk 2y ago
      Before I quit, it was ~6 drinks per day minimum. Here's the essay I wrote when I decided to quit: [1] https://taylor.town/1000-weekends https://taylor.town/1000-weekends Here was my 8-week progress report: [2] https://taylor.town/8-weekends https://taylor.town/8-weekends Right now I'm 96 weeks in! Going great! If you need any support/encouragement, always feel free to email me at hello@taylor.town :)
- AlotOfReading 2y ago
  I've had an unnamed content-addressable language in my private repos for awhile now. The language was an exploration of what it would look like if an entire language was transactional, and any failures (power loss, etc) could be mitigated by simply restarting execution at the last line of code executed. Content addressability helped solve the issue of ensuring state was consistent even in the presence of restarted I/O transactions.
tekknolagi 2y ago
Hi! It's me, I'm one of the interpreter and compiler authors. Happy to answer questions
- silasdavis 2y ago
  What is the runtime thing I run on a machine in order to support shipping code around? How does that work? Unison makes reference to a node I also could find scant details of rust.
  - tekknolagi 2y ago
    I'm not 100% sure what you're asking. The package manager equivalent is in development but currently looks like a tiny wrapper around a hash->serialized_program database.
  - agentultra 2y ago
    I think you need Python to be running the script on each machine.
    - tekknolagi 2y ago
      You can also compile a scrap to C (and then probably from there to native, or apparently with pnut you can now compile to shell...)
- bruce343434 2y ago
  What exactly does it mean to "address content", and how do I do it in the syntax of scrapscript?
- Vivtek 2y ago
  This might be a stupid question, but by "content-addressable" do you mean that namespaces are eliminated? This thing where you store each individual version of a function indexed by hash, that's content addressability?
  - 2y ago
hombre_fatal 2y ago
i’ve liked this from the start but never dove too deep.
- efilife 2y ago
  Same, good that op posted it as a reminder
jeffrey597doss 2y ago
[flagged]
lifeisstillgood 2y ago
So, (and this is five minutes playing here no deep dive), content-addressable is taken Seriously here. Each expression is a hash of (its ancestors - not too sure) but the implication is sort of the git dag is built into the language. Looks like that has deepmimplications for releases, deployments, inspecting code chnages There is an implication that my-labcorp is using scrapscript. Which is interesting as it seems to be a health based platform - tracking and dealing with test results, X-ray images etc etc So I love committing to one of the most basic CS concepts (cryptographic hashes) And I love the implications for code management. What I am not sure is why that helps with things like labcorp platform - I also love those implications (encrypt and store and just shuffle that data) because cannot any language use the concepts ?
- noduerme 2y ago
  I don't think I fully understand the way versioning and encapsulation happen by hashing in this, but I agree that conceptually it sounds very cool. Maybe the first new language I'd take a stab at in a long time just for the novelty of it.
fire_lake 2y ago
I am interested in this as a replacement for bash scripts The killer feature would be pulling in other scripts as libraries (they may be published anywhere) but I can lock down the hash of these dependencies inside of my script file. Is this possible?
- IshKebab 2y ago
  Deno can do this, and some other languages but Deno is the nicest I've used for this purpose so far. There's a great list here https://dbohdan.com/scripts-with-dependencies https://dbohdan.com/scripts-with-dependencies but be aware that a lot of those are through third party tools and therefore don't have proper IDE support etc.
- surprisetalk 2y ago
  Yep! In scrapscript, you can use the hashes directly to lock down the versions of any code directly. There's a few reasons why this might be annoying to deal with in practice, so we're working on a few different ways of managing versions throughout your script
- abathur 2y ago
  FWIW we can do this* with bash in the nix ecosystem (not that this knowledge should deter you from checking out scrapscript). * you said locking the hash in the script, and in the nix approach these are generally locked in either a lockfile or in the nix expression. Can elaborate if you're curious.
  - fire_lake 2y ago
    Multiple files adds just a tiny bit of friction when moving things around, putting on Gist, sending on Slack, etc. I would love for the Nix lock to somehow be embedded in the script.
    - abathur 2y ago
      You can inline the script within nix and get them all in one file (but that implies buy-in on nix, or maintaining 2 copies of the script). At the risk of overstepping (it sounds like you might be a little interested) here are a few links in to some basics just-in-case. For context, I extracted my bashrc into its own repo, and it in turn sources bash libraries for my shell history and git status plugins. It's also leaning on some libraries that are pulled in through those two. (To be clear, this is just a general example of the flake pattern with a separate lockfile. I can find or synthesize something with the inline script pattern if you ask.) - Source statements: https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f44a721ac7860f054745155ce/bashrc#L28-L31 https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f4... - Nix expression responsible for resolving and ~linking these two dependencies in to the final script: https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f44a721ac7860f054745155ce/bashrc.nix#L12-L25 https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f4... - Flake inputs for the two dependencies. Note that these can be specified as just the repo or a branch and updated by updating the lock through Nix, or they can be ~pinned to a specific rev: https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f44a721ac7860f054745155ce/flake.nix#L11-L22 https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f4... - A gist with examples of the entire chain of scripts that get resolved together here: https://gist.github.com/abathur/09f06f1f34b003376017d7a5c031a13b https://gist.github.com/abathur/09f06f1f34b003376017d7a5c031... These all get handled separately, so they can technically have divergent implementations of the same external command (for example, one library could depend on libressl and one could depend on openssl--and each resolved library would independently point to the correct one.
deleted 2y ago
[deleted]
winternewt 2y ago
The web site appears to claim that this solves interoperability between systems. I don't understand how. What enables one system to understand content coming out of another? For example, let's say you have two medical devices sharing patient data. Will scrapscript make it possible for them to create a mapping between the information models of the two systems?
postepowanieadm 2y ago
It compiles to python?
- surprisetalk 2y ago
  Compiler is written in python. It compiles to native code
bwestergard 2y ago
How does Scrapscript compare to Dhall?
johnkozak 2y ago
Another niche that ScrapScript might fit nicely in is NixOS/Guix type specification languages (e.g. nix)
zubairq 2y ago
As the author of another language with uses content addressable code for all source code (also to aid in shareable code - https://yazz.com/app/homepage.html https://yazz.com/app/homepage.html) I feel that it is really good to see that projects such as ScrapScript use this technique. Well done guys!
munro 2y ago
Lots of amazing ideas here! One more that would be great is 'sandboxing' if a program can use network/filesystem/etc, then I would feel safe to run any random scrap if I knew I would get some sort of oauth-looking screen that said what it wanted to access.
dang 2y ago
Related: Some Tricks from the Scrapscript Compiler - https://news.ycombinator.com/item?id=40933250 https://news.ycombinator.com/item?id=40933250 - July 2024 (6 comments) Scrapscript - https://news.ycombinator.com/item?id=40613756 https://news.ycombinator.com/item?id=40613756 - June 2024 (1 comment) A Baseline Scrapscript Compiler - https://news.ycombinator.com/item?id=40551583 https://news.ycombinator.com/item?id=40551583 - June 2024 (4 comments) scrapscript.py - https://news.ycombinator.com/item?id=39104504 https://news.ycombinator.com/item?id=39104504 - Jan 2024 (75 comments) Show HN: Scrapscript “guide”, proposals, and community chat - https://news.ycombinator.com/item?id=36519579 https://news.ycombinator.com/item?id=36519579 - June 2023 (2 comments) Show HN: ScrapScript – A tiny functional language for sharable software - https://news.ycombinator.com/item?id=35712163 https://news.ycombinator.com/item?id=35712163 - April 2023 (135 comments)
ilaksh 2y ago
Is there a way to search the large scrap yard?

193 points by luu 2y ago

Main site/community: • https://scrapscript.org https://scrapscript.org • https://scrapscript.discourse.group https://scrapscript.discourse.group • https://news.scrapscript.org https://news.scrapscript.org Max's brilliant dev insights: • https://bernsteinbear.com/blog/scrapscript https://bernsteinbear.com/blog/scrapscript • https://bernsteinbear.com/blog/scrapscript-baseline https://bernsteinbear.com/blog/scrapscript-baseline • https://bernsteinbear.com/blog/scrapscript-tricks https://bernsteinbear.com/blog/scrapscript-tricks I posted a brief update here this morning: • https://scrapscript.discourse.group/t/2024-07-23-design-development-update/85 https://scrapscript.discourse.group/t/2024-07-23-design-deve... Stay tuned! Lots of cool stuff coming :) Some sneak previews: • https://taylor.town/sle-2024-cfp https://taylor.town/sle-2024-cfp • https://taylor.town/unsound-2024-cfp https://taylor.town/unsound-2024-cfp

emmanueloga_ 2y ago

> Scrapscript solves the software sharability problem. > it’s JSON with types and functions and hashed references > it’s tiny Haskell with extreme syntactic consistency Content-addressability is one of the main features of Unison [1]. How do the languages compare? Why did you feel the need to create a new language (especially since both have a strong Haskell flavor). I'm guessing lots of people will have these questions so it could be good to mention Unison somewhere in your materials. The language seems to have big goals, so it doesn't seem that you are working on it "just for fun" :-). Are there other "content-addressable" languages out there other than these two? -- 1: https://www.unison-lang.org/ https://www.unison-lang.org/

surprisetalk 2y ago
Great questions and observations! -- > Why did you feel the need to create a new language? Well when I started on this language in 2017-2019(?), I definitely wasn't aware of Unison. I remember talking to Richard Feldman about scrapscript when he first started working on Roc (fun fact: I was the first person Feldman added to the private repo!). I spent a long time on the initial Scrapscript implementations, trying to compete in some similar applications as Roc, and Feldman just absolutely crushed it. I felt (and still feel) that Roc is doing a great job at the low-level "platform" experience I yearn for. [1] https://www.roc-lang.org/ https://www.roc-lang.org/ Eventually I met some Unison folks at Strange Loop (2019?), and we had very different inspirations and goals. That seems to still be the case. There are some hard-to-articulate things I want out of the web, and Unison's architecture doesn't seem suited for it. Anyway, Scrapscript only got a bunch of attention a year or two when I started working on it in public a bit more. I think defeating alcoholism also made me much more productive haha -- > How do the languages compare? From what I understand, Unison has very little in common with Scrapscript. "Content-addressibilty" is really more of a mechanism than a "feature". Kinda like how AI is not really a "feature". You can't just sprinkle SHA on something and go raise venture capital haha I think one major difference is that Unison opted for a more traditional git-based paradigm and then built some stellar dev tools on top of that. So far, Scrapscript is a bit more ambitious in its plans. So go try out Unison! And come back and play with Scrapscript too, when it's more mature :)
- emmanueloga_ 2y ago
  Congrats on all the impressive progress!
- the_duke 2y ago
  > From what I understand, Unison has very little in common with Scrapscript. I think you should read up on Unison a bit more, I see a lot of overlap. Both identify code with hashes, and implement code sharing through hash based syncing. Builtin serialization is also present in both, and both are very Haskell inspired. One difference I see is the sharing mechanism, where Unison has a local sqlite repository and tooling for uploading, while Scrapscript seems to be built more around IPFS. But that seems like an implementation detail rather than a fundemantal difference. Unison supports algebraic effects, aka abilities with handlers, which is a much more powerful and general concept than the platforms in Roc.
  - actionfromafar 2y ago
    I think ergonomics and practicality could use more consideration when discussing new languages. A language exists in an ecosystem. At a glance to me, Scrapscript looks very traditional and easy to grok from a regular programmers perspective, there are build steps, you can use an IDE or editor etc. Unison feels more like "Smalltalk Squeak" and Scrapscript feels more like "GNU Smalltalk". Come live in our new world, or integrate new stuff into your existing world kind of vibe.
- efnx 2y ago
  > I think defeating alcoholism also made me much more productive haha Just out of curiosity and because we all have different limits - how much do you think you used to drink, generally, and what helped you quit?
  - surprisetalk 2y ago
    Before I quit, it was ~6 drinks per day minimum. Here's the essay I wrote when I decided to quit: [1] https://taylor.town/1000-weekends https://taylor.town/1000-weekends Here was my 8-week progress report: [2] https://taylor.town/8-weekends https://taylor.town/8-weekends Right now I'm 96 weeks in! Going great! If you need any support/encouragement, always feel free to email me at hello@taylor.town :)
AlotOfReading 2y ago
I've had an unnamed content-addressable language in my private repos for awhile now. The language was an exploration of what it would look like if an entire language was transactional, and any failures (power loss, etc) could be mitigated by simply restarting execution at the last line of code executed. Content addressability helped solve the issue of ensuring state was consistent even in the presence of restarted I/O transactions.

tekknolagi 2y ago

Hi! It's me, I'm one of the interpreter and compiler authors. Happy to answer questions

silasdavis 2y ago
What is the runtime thing I run on a machine in order to support shipping code around? How does that work? Unison makes reference to a node I also could find scant details of rust.
- tekknolagi 2y ago
  I'm not 100% sure what you're asking. The package manager equivalent is in development but currently looks like a tiny wrapper around a hash->serialized_program database.
- agentultra 2y ago
  I think you need Python to be running the script on each machine.
  - tekknolagi 2y ago
    You can also compile a scrap to C (and then probably from there to native, or apparently with pnut you can now compile to shell...)
bruce343434 2y ago
What exactly does it mean to "address content", and how do I do it in the syntax of scrapscript?
Vivtek 2y ago
This might be a stupid question, but by "content-addressable" do you mean that namespaces are eliminated? This thing where you store each individual version of a function indexed by hash, that's content addressability?
- 2y ago

hombre_fatal 2y ago

i’ve liked this from the start but never dove too deep.

efilife 2y ago
Same, good that op posted it as a reminder

jeffrey597doss 2y ago

[flagged]

lifeisstillgood 2y ago

So, (and this is five minutes playing here no deep dive), content-addressable is taken Seriously here. Each expression is a hash of (its ancestors - not too sure) but the implication is sort of the git dag is built into the language. Looks like that has deepmimplications for releases, deployments, inspecting code chnages There is an implication that my-labcorp is using scrapscript. Which is interesting as it seems to be a health based platform - tracking and dealing with test results, X-ray images etc etc So I love committing to one of the most basic CS concepts (cryptographic hashes) And I love the implications for code management. What I am not sure is why that helps with things like labcorp platform - I also love those implications (encrypt and store and just shuffle that data) because cannot any language use the concepts ?

noduerme 2y ago
I don't think I fully understand the way versioning and encapsulation happen by hashing in this, but I agree that conceptually it sounds very cool. Maybe the first new language I'd take a stab at in a long time just for the novelty of it.

fire_lake 2y ago

I am interested in this as a replacement for bash scripts The killer feature would be pulling in other scripts as libraries (they may be published anywhere) but I can lock down the hash of these dependencies inside of my script file. Is this possible?

IshKebab 2y ago
Deno can do this, and some other languages but Deno is the nicest I've used for this purpose so far. There's a great list here https://dbohdan.com/scripts-with-dependencies https://dbohdan.com/scripts-with-dependencies but be aware that a lot of those are through third party tools and therefore don't have proper IDE support etc.
surprisetalk 2y ago
Yep! In scrapscript, you can use the hashes directly to lock down the versions of any code directly. There's a few reasons why this might be annoying to deal with in practice, so we're working on a few different ways of managing versions throughout your script
abathur 2y ago
FWIW we can do this* with bash in the nix ecosystem (not that this knowledge should deter you from checking out scrapscript). * you said locking the hash in the script, and in the nix approach these are generally locked in either a lockfile or in the nix expression. Can elaborate if you're curious.
- fire_lake 2y ago
  Multiple files adds just a tiny bit of friction when moving things around, putting on Gist, sending on Slack, etc. I would love for the Nix lock to somehow be embedded in the script.
  - abathur 2y ago
    You can inline the script within nix and get them all in one file (but that implies buy-in on nix, or maintaining 2 copies of the script). At the risk of overstepping (it sounds like you might be a little interested) here are a few links in to some basics just-in-case. For context, I extracted my bashrc into its own repo, and it in turn sources bash libraries for my shell history and git status plugins. It's also leaning on some libraries that are pulled in through those two. (To be clear, this is just a general example of the flake pattern with a separate lockfile. I can find or synthesize something with the inline script pattern if you ask.) - Source statements: https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f44a721ac7860f054745155ce/bashrc#L28-L31 https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f4... - Nix expression responsible for resolving and ~linking these two dependencies in to the final script: https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f44a721ac7860f054745155ce/bashrc.nix#L12-L25 https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f4... - Flake inputs for the two dependencies. Note that these can be specified as just the repo or a branch and updated by updating the lock through Nix, or they can be ~pinned to a specific rev: https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f44a721ac7860f054745155ce/flake.nix#L11-L22 https://github.com/abathur/bashrc.nix/blob/e7c437578e2a623f4... - A gist with examples of the entire chain of scripts that get resolved together here: https://gist.github.com/abathur/09f06f1f34b003376017d7a5c031a13b https://gist.github.com/abathur/09f06f1f34b003376017d7a5c031... These all get handled separately, so they can technically have divergent implementations of the same external command (for example, one library could depend on libressl and one could depend on openssl--and each resolved library would independently point to the correct one.

deleted 2y ago

[deleted]

winternewt 2y ago

The web site appears to claim that this solves interoperability between systems. I don't understand how. What enables one system to understand content coming out of another? For example, let's say you have two medical devices sharing patient data. Will scrapscript make it possible for them to create a mapping between the information models of the two systems?

postepowanieadm 2y ago

It compiles to python?

surprisetalk 2y ago
Compiler is written in python. It compiles to native code

bwestergard 2y ago

How does Scrapscript compare to Dhall?

johnkozak 2y ago

Another niche that ScrapScript might fit nicely in is NixOS/Guix type specification languages (e.g. nix)

zubairq 2y ago

As the author of another language with uses content addressable code for all source code (also to aid in shareable code - https://yazz.com/app/homepage.html https://yazz.com/app/homepage.html) I feel that it is really good to see that projects such as ScrapScript use this technique. Well done guys!

munro 2y ago

Lots of amazing ideas here! One more that would be great is 'sandboxing' if a program can use network/filesystem/etc, then I would feel safe to run any random scrap if I knew I would get some sort of oauth-looking screen that said what it wanted to access.

dang 2y ago

Related: Some Tricks from the Scrapscript Compiler - https://news.ycombinator.com/item?id=40933250 https://news.ycombinator.com/item?id=40933250 - July 2024 (6 comments) Scrapscript - https://news.ycombinator.com/item?id=40613756 https://news.ycombinator.com/item?id=40613756 - June 2024 (1 comment) A Baseline Scrapscript Compiler - https://news.ycombinator.com/item?id=40551583 https://news.ycombinator.com/item?id=40551583 - June 2024 (4 comments) scrapscript.py - https://news.ycombinator.com/item?id=39104504 https://news.ycombinator.com/item?id=39104504 - Jan 2024 (75 comments) Show HN: Scrapscript “guide”, proposals, and community chat - https://news.ycombinator.com/item?id=36519579 https://news.ycombinator.com/item?id=36519579 - June 2023 (2 comments) Show HN: ScrapScript – A tiny functional language for sharable software - https://news.ycombinator.com/item?id=35712163 https://news.ycombinator.com/item?id=35712163 - April 2023 (135 comments)

ilaksh 2y ago

Is there a way to search the large scrap yard?