4 ms·

Actually this is interesting - whhy Feather vs sqlite. I would love to know the answer! But coming back to the anti-pattern : well, obviously the authors have

Actually this is interesting - whhy Feather vs sqlite. I would love to know the answer! But coming back to the anti-pattern : well, obviously the authors have the power to not spend time on something. But I'm trying to figure out why it's an anti-pattern in general. Snapshotting execution state is probably the ideal goal, but saving intermediate data structures is a decent convenience feature. Now if that's restricted by the limitations of the format itself (no multiple frames in a single file), then we are back to thinking that HDF5/sqlite may indeed be the better format.

hadley 11y ago
Basically because you should be encoding state in code, not data. If you store data between sessions, it's easy to lose the code that you use to create it and then later on you can't recreate it. It is convenient to save your complete workspace but I've seen too many cases where it's contributed to lack of reproducibility to spend my time working on it.
- sandGorgon 11y ago
  So there's a use case difference. I create models from remote data sources - this is incremental on a daily basis and takes quite a bit of time. So I snapshot the workspace after I do a run and do some experiments. Now - for me, saving the workspace is a convenience feature, NOT a programming feature. This is what I mean by thou-shall-not. My use case is very well defined and I'm not stupid. And I completely knows the pitfalls of what you talk about - but a philosophical opposition is what hurts me (and lots of devs like me)
  - hadley 11y ago
    I hope I didn't come across as "thou shalt not" - it's just never going to be high in my priority list. (And even for your use case I would think you'd be better off keeping the models in a list and saving that. Then other random stuff in your evn won't get carried along for the ride)
    - sandGorgon 11y ago
      Oh no you did not! That was polite musing. Thank you for the reply - I still hope you change your mind. Because people do have genuine, but different needs ;)

hadley 11y ago

Basically because you should be encoding state in code, not data. If you store data between sessions, it's easy to lose the code that you use to create it and then later on you can't recreate it. It is convenient to save your complete workspace but I've seen too many cases where it's contributed to lack of reproducibility to spend my time working on it.

sandGorgon 11y ago
So there's a use case difference. I create models from remote data sources - this is incremental on a daily basis and takes quite a bit of time. So I snapshot the workspace after I do a run and do some experiments. Now - for me, saving the workspace is a convenience feature, NOT a programming feature. This is what I mean by thou-shall-not. My use case is very well defined and I'm not stupid. And I completely knows the pitfalls of what you talk about - but a philosophical opposition is what hurts me (and lots of devs like me)
- hadley 11y ago
  I hope I didn't come across as "thou shalt not" - it's just never going to be high in my priority list. (And even for your use case I would think you'd be better off keeping the models in a list and saving that. Then other random stuff in your evn won't get carried along for the ride)
  - sandGorgon 11y ago
    Oh no you did not! That was polite musing. Thank you for the reply - I still hope you change your mind. Because people do have genuine, but different needs ;)