7 ms·

Pstore: Ruby Built-In Hash Persistence

102 points by hstaab 4y ago

kayodelycaon 4y ago
Note, this is a wrapper around Ruby’s Marshal class.
- thunderbong 4y ago
  Mentioned in the linked article.
fny 4y ago
I do a lot of ML and AI work nowadays... I miss Ruby a lot especially the its culture around ergonomics.
- prescriptivist 4y ago
  I recently had the need to build an internal system that distributed workloads across many workers via a client/server model. I did the proof-of-concept using druby [1] and it turned out to be so simple and stable that we just ran with it. It'd been years since I had used that library and instinctively I assumed we'd get the prototype out and then rebuild it using some sort of web service and utilize a high concurrency web server but druby just worked! [1] https://github.com/ruby/drb https://github.com/ruby/drb
  - thunderbong 4y ago
    drb is awesome. I've had the good fortune to be able to use it once. The simplicity of it compared to anything else is amazing.
  - iovrthoughtthis 4y ago
    how have i never seen this!! :D
- cdiamand 4y ago
  There have been some interesting ML gems rolled in the past few years: https://ankane.org/new-ml-gems https://ankane.org/new-ml-gems Any thoughts on what the Ruby community would need to build in order for it to become an attractive tool for AI work?
  - mattnewton 4y ago
    My guess is some kind of corporate sponsorship. Someone with deep pockets to maintain it, encourage new apis keeping up with the latest papers, and make sure it works out of the box with the accelerator people want to use this month.
    - pmontra 4y ago
      The web framework part is basically sponsored by 37signals https://37signals.com/32 https://37signals.com/32 Maybe that's why Ruby is best known for Ruby on Rails.
  - waffle_ss 4y ago
    As a primarily Ruby dev I'd prefer the AI/ML ecosystem not be split-brained between two languages that are semantically 90% the same thing. Just learn Python and integrate the models into your Rails (or whatever) apps.
  - fny 4y ago
    A huge cultural shift. People in scientific computing speak Python and R. Something would need to happen that makes Ruby far more attractive. Say performance parity with Crystal or Nim.
    - mattnewton 4y ago
      I think it’s more than that, Julia exists and adoption is still slow. Lua and torch were plenty fast and they were still replaced by pytorch. I think to compete with python you need at least a fraction of the de-facto corporate sponsorship for python in the ML space.
- brightball 4y ago
  It keeps me coming back. Ruby itself is just such an enabler.
- green_on_black 4y ago
  Have you tried Scala?
  - fny 4y ago
    It's more of a cultural thing. People tend to write Ruby in a literate fashion and think critically about their APIs. Scala devs get a little over their skis sometimes playing with language features.
mrinterweb 4y ago
I would think this would have limited usefulness for most web applications as the latest trend for web apps is to think of the deployed code as ephemeral, and local files are not something devs often rely on. I guess if you're mounting block storage or some other virtual file system that would be another thing. For non-web applications, this could be a simplistic replacement for what people often use sqlite for. The readme doesn't talk much about concurrent access to the store other than the transactions, so concurrent operations may also be a limitation.
aartav 4y ago
pstore has been a built-in with Ruby stdlib for as long as ruby has existed, so _over_ 20 years.
- mperham 4y ago
  I'm assuming it pre-dates Rubygems because it really should be a gem. I can't speak for Japan but few people in the Western world seem to use it.
  - brunno 4y ago
    There was a time when some stuff was being extracted (removed) from Ruby core and becoming gems and I really tought PStore and YAML::Store were going to be among those, but no, they decided to keep them in core. So maybe there are some important enough use cases that justify it being there. Or maybe it would be a hard task that didn't justify the effort.
    - byroot 4y ago
      Many parts of the stdlib are being slowly gemified, that's the case of `pstore` too hence why it has it's own repo. It's now no longer technically stdlib, but a "default gem", a gem that is installed by default with ruby, see: https://stdgems.org/ https://stdgems.org/ Since a few years every version remove one or two rarely used default gems. The Ruby core team just doesn't like big breaking changes.
- tyingq 4y ago
  Pstore also uses Marshal behind the scenes, so I assume has similar caveats you see in other comments on this thread.
3pt14159 4y ago
Don't use this. Marshal has too many issues. If you really need persistence and can't use something like Postgres, use the Ox gem instead. It's more reliable between versions of Ruby and easier to parse from other languages if you ever have to.
- jrochkind1 4y ago
  > too many issues Such as?
  - woodruffw 4y ago
    Marshal is Ruby's version of pickle in Python: it serializes arbitrary objects, which means that correct deserialization requires arbitrary code execution. This is bad enough on its own, but it also makes pivoting a file read/write primitive into code execution much easier.
    - Rafert 4y ago
      https://github.com/ruby/psych https://github.com/ruby/psych defaults to only loading permitted classes since 4.0 so that seems less of a concern now?
      - jrochkind1 4y ago
        `psych`, used for YAML, is a different thing than Marshal. pstore uses Marshal. https://ruby-doc.org/core-2.6.3/Marshal.html https://ruby-doc.org/core-2.6.3/Marshal.html. I don't believe psych will be involved with pstore. I'm honestly not sure, though, how much I should be worried about the fact that someone who has write access to my database can maybe escalate that to an arbitrary code execution if I use pstore. Literally not sure. Write access to my DB seems pretty disastrous already...
    - solarkraft 4y ago
      Pickle is fine (in a pinch). It's not meant for untrusted data.
why-el 4y ago
Interesting. Transactionality is implemented via a regular thread lock, this means in a concurrent Rails app where this library is used in a hot path you might suffer some contention. Best is to use for marshaling data in non-hot paths such as stand alone scripts or app start up. I only say this because it's quite different from expectations around transactions in an SQL sense.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]

102 points by hstaab 4y ago

kayodelycaon 4y ago

Note, this is a wrapper around Ruby’s Marshal class.

thunderbong 4y ago
Mentioned in the linked article.

fny 4y ago

I do a lot of ML and AI work nowadays... I miss Ruby a lot especially the its culture around ergonomics.

prescriptivist 4y ago
I recently had the need to build an internal system that distributed workloads across many workers via a client/server model. I did the proof-of-concept using druby [1] and it turned out to be so simple and stable that we just ran with it. It'd been years since I had used that library and instinctively I assumed we'd get the prototype out and then rebuild it using some sort of web service and utilize a high concurrency web server but druby just worked! [1] https://github.com/ruby/drb https://github.com/ruby/drb
- thunderbong 4y ago
  drb is awesome. I've had the good fortune to be able to use it once. The simplicity of it compared to anything else is amazing.
- iovrthoughtthis 4y ago
  how have i never seen this!! :D
cdiamand 4y ago
There have been some interesting ML gems rolled in the past few years: https://ankane.org/new-ml-gems https://ankane.org/new-ml-gems Any thoughts on what the Ruby community would need to build in order for it to become an attractive tool for AI work?
- mattnewton 4y ago
  My guess is some kind of corporate sponsorship. Someone with deep pockets to maintain it, encourage new apis keeping up with the latest papers, and make sure it works out of the box with the accelerator people want to use this month.
  - pmontra 4y ago
    The web framework part is basically sponsored by 37signals https://37signals.com/32 https://37signals.com/32 Maybe that's why Ruby is best known for Ruby on Rails.
- waffle_ss 4y ago
  As a primarily Ruby dev I'd prefer the AI/ML ecosystem not be split-brained between two languages that are semantically 90% the same thing. Just learn Python and integrate the models into your Rails (or whatever) apps.
- fny 4y ago
  A huge cultural shift. People in scientific computing speak Python and R. Something would need to happen that makes Ruby far more attractive. Say performance parity with Crystal or Nim.
  - mattnewton 4y ago
    I think it’s more than that, Julia exists and adoption is still slow. Lua and torch were plenty fast and they were still replaced by pytorch. I think to compete with python you need at least a fraction of the de-facto corporate sponsorship for python in the ML space.
brightball 4y ago
It keeps me coming back. Ruby itself is just such an enabler.
green_on_black 4y ago
Have you tried Scala?
- fny 4y ago
  It's more of a cultural thing. People tend to write Ruby in a literate fashion and think critically about their APIs. Scala devs get a little over their skis sometimes playing with language features.

mrinterweb 4y ago

I would think this would have limited usefulness for most web applications as the latest trend for web apps is to think of the deployed code as ephemeral, and local files are not something devs often rely on. I guess if you're mounting block storage or some other virtual file system that would be another thing. For non-web applications, this could be a simplistic replacement for what people often use sqlite for. The readme doesn't talk much about concurrent access to the store other than the transactions, so concurrent operations may also be a limitation.

aartav 4y ago

pstore has been a built-in with Ruby stdlib for as long as ruby has existed, so _over_ 20 years.

mperham 4y ago
I'm assuming it pre-dates Rubygems because it really should be a gem. I can't speak for Japan but few people in the Western world seem to use it.
- brunno 4y ago
  There was a time when some stuff was being extracted (removed) from Ruby core and becoming gems and I really tought PStore and YAML::Store were going to be among those, but no, they decided to keep them in core. So maybe there are some important enough use cases that justify it being there. Or maybe it would be a hard task that didn't justify the effort.
  - byroot 4y ago
    Many parts of the stdlib are being slowly gemified, that's the case of `pstore` too hence why it has it's own repo. It's now no longer technically stdlib, but a "default gem", a gem that is installed by default with ruby, see: https://stdgems.org/ https://stdgems.org/ Since a few years every version remove one or two rarely used default gems. The Ruby core team just doesn't like big breaking changes.
tyingq 4y ago
Pstore also uses Marshal behind the scenes, so I assume has similar caveats you see in other comments on this thread.

3pt14159 4y ago

Don't use this. Marshal has too many issues. If you really need persistence and can't use something like Postgres, use the Ox gem instead. It's more reliable between versions of Ruby and easier to parse from other languages if you ever have to.

jrochkind1 4y ago
> too many issues Such as?
- woodruffw 4y ago
  Marshal is Ruby's version of pickle in Python: it serializes arbitrary objects, which means that correct deserialization requires arbitrary code execution. This is bad enough on its own, but it also makes pivoting a file read/write primitive into code execution much easier.
  - Rafert 4y ago
    https://github.com/ruby/psych https://github.com/ruby/psych defaults to only loading permitted classes since 4.0 so that seems less of a concern now?
    - jrochkind1 4y ago
      `psych`, used for YAML, is a different thing than Marshal. pstore uses Marshal. https://ruby-doc.org/core-2.6.3/Marshal.html https://ruby-doc.org/core-2.6.3/Marshal.html. I don't believe psych will be involved with pstore. I'm honestly not sure, though, how much I should be worried about the fact that someone who has write access to my database can maybe escalate that to an arbitrary code execution if I use pstore. Literally not sure. Write access to my DB seems pretty disastrous already...
  - solarkraft 4y ago
    Pickle is fine (in a pinch). It's not meant for untrusted data.

why-el 4y ago

Interesting. Transactionality is implemented via a regular thread lock, this means in a concurrent Rails app where this library is used in a hot path you might suffer some contention. Best is to use for marshaling data in non-hot paths such as stand alone scripts or app start up. I only say this because it's quite different from expectations around transactions in an SQL sense.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]