7 ms·

Spritely Goblins v0.13.0: Object persistence and easier IO

62 points by paroneayea 2y ago

cooljoseph 2y ago
How fast is Goblins? I've been considering using it for a project, but I'm worried that it would be too slow. Would it, e.g., be fast enough to serve as a database?
- tonyg 2y ago
  This depends heavily on the project you have in mind. How long is a piece of string?
waldrews 2y ago
Why don't we see more general purpose computing environments where the whole interpreter state is persistent and serializable by default? The only ones I'm aware of is R's .RData mechanism, some Smalltalk environments, and solutions at the container/VM level. What if an interpreter wrote every event down to a variable change to a database-like WAL, making it resumable after a crash? Downside: it would be slow, and there would need to be provision for transaction boundaries and for access to network sockets and other non-serializable resources. Upside: develop incrementally in the Smalltalk REPL style, never worry about the data layer at all, backup or give developers a copy of the whole application state - for utilities or LOB apps of limited scope or complexity, why not?
- cess11 2y ago
  You'd still need a mechanism for persisting separately, or you'd end up with a broken image/"data layer" due to post-crash persists. It's common to do something similar by writing most of an application as functions over a database and static files, and using scaffolding tools to reinitialise the database when things break.
- tonyg 2y ago
  It wouldn't necessarily be slow! There's this wonderful paper from Usenix OSDI 2014 [1] in which the authors present an eidetic system: "a computer system [should] provide the ability to recall any past state that existed on the computer, and further, [should] be able to provide the lineage of any byte in a current or past state." They implemented and evaluated it, too: "Preliminary data from several weeks of continuous use [show that] storage requirements for 4 or more years of usage can be satisfied by adding a 4 TB hard drive to the system. Further, the performance overhead on almost all workloads we measured was under 8%." [1]: Devecsery, David, MIchael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. “Eidetic Systems.” In Proc. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/devecsery https://www.usenix.org/conference/osdi14/technical-sessions/...
- mike_hearn 2y ago
  You generally only want to serialize a part of the program state. I built a workflow engine in the past on the JVM that uses serializable continuations, a bit like what Goblins is doing. The important part is not the VM-level engineering, the important part is defining the boundaries of what gets serialized. If you actually checkpoint everything then you can't upgrade your software anymore, because developers think about program upgrades as reloading from a blank slate, not hot-patching a running program in memory. OK maybe some Lisp devs think in the latter way, but most people don't, and the moment you upgrade a library from one version to another of course there is no instructions for how to incrementally patch your way in memory from one to another. It's also not clear why you'd want to. A lot of program state is plumbing. Logging frameworks, HTTP stacks, thread pools, system configuration, JIT compiler state, etc. There's no point in saving that to disk. It belongs in transient RAM naturally.
traverseda 2y ago
I know that guile/scheme is uniquely well suited to this, but I'd be interested in hearing why you're using it instead of, say, writing a python library? Is it like a smalltalk thing with really good message-passing, are you getting some guarantees about serializability that are impossible in a more mainstream language? Do you just like it?
- tonyg 2y ago
  There's rather a lot of new syntax, and, more importantly, exploration and experimentation with new syntax involved. Python syntax is... not easy to extend. Let alone on an experimental basis.
dang 2y ago
Surprisingly little discussion in the past! But I found these two: Spritely Goblins v0.11.0, time travel distributed debugger and more - https://news.ycombinator.com/item?id=35864365 https://news.ycombinator.com/item?id=35864365 - May 2023 (1 comment) Growing a Networked Garden with Spritely Goblins - https://news.ycombinator.com/item?id=34031136 https://news.ycombinator.com/item?id=34031136 - Dec 2022 (2 comments) Maybe we should convert this thread to a discussion of the project as a whole, rather than of a point release?
- andrewflnr 2y ago
  People will feel free to comment on the project as a whole anyway, so why go to the bother of making a special thing out of it? :) I am surprised you didn't find more discussion, though. I'm pretty sure I first heard of the project and have seen it here multiple times. I think it's more often (or in the past) just "Spritely". If your search included the word "Goblins" instead of just "Spritely", that might have narrowed it down excessively.
kgeist 2y ago
Reminded me of a game engine of mine. Years ago I was experimenting with a time travel game concept. It had a virtual machine with green threads whose state could be suspended and serialized into a single binary blob at any time. I basically repurposed the GC's reachability analysis to find all objects reachable from the roots (VM's stack and global variables). IIRC it had also some sort of topological sort when serializing so that most references could be restored in one pass. It also serialized the stack itself and the current instruction pointer. "Manual persistence" they mention is actually quite straightforward to do if you're in full control of the VM. Additionally, it also serialized the physics engine's state as well, so that the forces applied were fully restored (important in a game). The concept was about being able to return to any point in time and have N versions of yourself working together to solve a problem. NPCs and objects had running scripts attached and that's why I needed the whole thing. It also allowed to have proper game saves out of the box. Oh, those were the fun times, and now I'm writing boring enterprise stuff :)
- BaculumMeumEst 2y ago
  You may enjoy this talk about the implementation of Braid's rewind: https://www.gdcvault.com/play/1012210/The-Implementation-of-Rewind-in https://www.gdcvault.com/play/1012210/The-Implementation-of-...
ryukafalz 2y ago
This is great! I've been following the Spritely project on the sidelines for a while now and have built a couple toy applications with Goblins; persistence and IO were often my biggest pain points. Looking forward to giving this a try soon!
cxr 2y ago
This blog post commits the common sin of showing a screencapture of a terminal emulator session that moves at the speed of thought of the creator rather than presenting things in a reasonable way for the person actually consuming it (i.e. the one in the audience who's trying to work out, without the creator's a priori understanding of what they're doing, the relative significance/insignificance of whatever's flashing on the screen). This happens constantly with software demos. It shouldn't. It's such a silly thing to mess up and takes more effort than just not including a screencapture whatsoever.
- ryukafalz 2y ago
  The only thing in that screencapture that happens outside of the space shooter game is relaunching the game after closing it. If you know that pressing the up arrow and enter to run the last command is a common thing, I don't think that's too hard to follow. Granted, I've played the space shooter in question before, so I do have some context :)
  - cxr 2y ago
    > The only thing in that screencapture that happens outside of the space shooter game is relaunching the game after closing it. If you know that pressing the up arrow and enter to run the last command is a common thing, I don't think that's too hard to follow. Working out that that's all that's happening (not to mentioning ask the inevitable, "Why is there a screencapture included here at all?") takes about 2–10x the amount of time it takes to read this description that says that's what it's showing.
    - yetihehe 2y ago
      Depends, I work daily with console, so I just reasoned that a sequence of: - exit game - console shows prompt for new command - command magically appears fully defined - game starts again is a normal "quit, recall with arrow up, restart". It's pretty evident and intuitive for most developers working with console applications and didn't require any thinking why those things are represented so. It's the first time I've heard of spritely goblins or seen that video.

62 points by paroneayea 2y ago

How fast is Goblins? I've been considering using it for a project, but I'm worried that it would be too slow. Would it, e.g., be fast enough to serve as a database?

tonyg 2y ago
This depends heavily on the project you have in mind. How long is a piece of string?

waldrews 2y ago

Why don't we see more general purpose computing environments where the whole interpreter state is persistent and serializable by default? The only ones I'm aware of is R's .RData mechanism, some Smalltalk environments, and solutions at the container/VM level. What if an interpreter wrote every event down to a variable change to a database-like WAL, making it resumable after a crash? Downside: it would be slow, and there would need to be provision for transaction boundaries and for access to network sockets and other non-serializable resources. Upside: develop incrementally in the Smalltalk REPL style, never worry about the data layer at all, backup or give developers a copy of the whole application state - for utilities or LOB apps of limited scope or complexity, why not?

cess11 2y ago
You'd still need a mechanism for persisting separately, or you'd end up with a broken image/"data layer" due to post-crash persists. It's common to do something similar by writing most of an application as functions over a database and static files, and using scaffolding tools to reinitialise the database when things break.
tonyg 2y ago
It wouldn't necessarily be slow! There's this wonderful paper from Usenix OSDI 2014 [1] in which the authors present an eidetic system: "a computer system [should] provide the ability to recall any past state that existed on the computer, and further, [should] be able to provide the lineage of any byte in a current or past state." They implemented and evaluated it, too: "Preliminary data from several weeks of continuous use [show that] storage requirements for 4 or more years of usage can be satisfied by adding a 4 TB hard drive to the system. Further, the performance overhead on almost all workloads we measured was under 8%." [1]: Devecsery, David, MIchael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. “Eidetic Systems.” In Proc. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/devecsery https://www.usenix.org/conference/osdi14/technical-sessions/...
mike_hearn 2y ago
You generally only want to serialize a part of the program state. I built a workflow engine in the past on the JVM that uses serializable continuations, a bit like what Goblins is doing. The important part is not the VM-level engineering, the important part is defining the boundaries of what gets serialized. If you actually checkpoint everything then you can't upgrade your software anymore, because developers think about program upgrades as reloading from a blank slate, not hot-patching a running program in memory. OK maybe some Lisp devs think in the latter way, but most people don't, and the moment you upgrade a library from one version to another of course there is no instructions for how to incrementally patch your way in memory from one to another. It's also not clear why you'd want to. A lot of program state is plumbing. Logging frameworks, HTTP stacks, thread pools, system configuration, JIT compiler state, etc. There's no point in saving that to disk. It belongs in transient RAM naturally.

traverseda 2y ago

I know that guile/scheme is uniquely well suited to this, but I'd be interested in hearing why you're using it instead of, say, writing a python library? Is it like a smalltalk thing with really good message-passing, are you getting some guarantees about serializability that are impossible in a more mainstream language? Do you just like it?

tonyg 2y ago
There's rather a lot of new syntax, and, more importantly, exploration and experimentation with new syntax involved. Python syntax is... not easy to extend. Let alone on an experimental basis.

dang 2y ago

Surprisingly little discussion in the past! But I found these two: Spritely Goblins v0.11.0, time travel distributed debugger and more - https://news.ycombinator.com/item?id=35864365 https://news.ycombinator.com/item?id=35864365 - May 2023 (1 comment) Growing a Networked Garden with Spritely Goblins - https://news.ycombinator.com/item?id=34031136 https://news.ycombinator.com/item?id=34031136 - Dec 2022 (2 comments) Maybe we should convert this thread to a discussion of the project as a whole, rather than of a point release?

andrewflnr 2y ago
People will feel free to comment on the project as a whole anyway, so why go to the bother of making a special thing out of it? :) I am surprised you didn't find more discussion, though. I'm pretty sure I first heard of the project and have seen it here multiple times. I think it's more often (or in the past) just "Spritely". If your search included the word "Goblins" instead of just "Spritely", that might have narrowed it down excessively.

kgeist 2y ago

Reminded me of a game engine of mine. Years ago I was experimenting with a time travel game concept. It had a virtual machine with green threads whose state could be suspended and serialized into a single binary blob at any time. I basically repurposed the GC's reachability analysis to find all objects reachable from the roots (VM's stack and global variables). IIRC it had also some sort of topological sort when serializing so that most references could be restored in one pass. It also serialized the stack itself and the current instruction pointer. "Manual persistence" they mention is actually quite straightforward to do if you're in full control of the VM. Additionally, it also serialized the physics engine's state as well, so that the forces applied were fully restored (important in a game). The concept was about being able to return to any point in time and have N versions of yourself working together to solve a problem. NPCs and objects had running scripts attached and that's why I needed the whole thing. It also allowed to have proper game saves out of the box. Oh, those were the fun times, and now I'm writing boring enterprise stuff :)

BaculumMeumEst 2y ago
You may enjoy this talk about the implementation of Braid's rewind: https://www.gdcvault.com/play/1012210/The-Implementation-of-Rewind-in https://www.gdcvault.com/play/1012210/The-Implementation-of-...

ryukafalz 2y ago

This is great! I've been following the Spritely project on the sidelines for a while now and have built a couple toy applications with Goblins; persistence and IO were often my biggest pain points. Looking forward to giving this a try soon!

cxr 2y ago

This blog post commits the common sin of showing a screencapture of a terminal emulator session that moves at the speed of thought of the creator rather than presenting things in a reasonable way for the person actually consuming it (i.e. the one in the audience who's trying to work out, without the creator's a priori understanding of what they're doing, the relative significance/insignificance of whatever's flashing on the screen). This happens constantly with software demos. It shouldn't. It's such a silly thing to mess up and takes more effort than just not including a screencapture whatsoever.

ryukafalz 2y ago
The only thing in that screencapture that happens outside of the space shooter game is relaunching the game after closing it. If you know that pressing the up arrow and enter to run the last command is a common thing, I don't think that's too hard to follow. Granted, I've played the space shooter in question before, so I do have some context :)
- cxr 2y ago
  > The only thing in that screencapture that happens outside of the space shooter game is relaunching the game after closing it. If you know that pressing the up arrow and enter to run the last command is a common thing, I don't think that's too hard to follow. Working out that that's all that's happening (not to mentioning ask the inevitable, "Why is there a screencapture included here at all?") takes about 2–10x the amount of time it takes to read this description that says that's what it's showing.
  - yetihehe 2y ago
    Depends, I work daily with console, so I just reasoned that a sequence of: - exit game - console shows prompt for new command - command magically appears fully defined - game starts again is a normal "quit, recall with arrow up, restart". It's pretty evident and intuitive for most developers working with console applications and didn't require any thinking why those things are represented so. It's the first time I've heard of spritely goblins or seen that video.