5 ms·

Make It Ephemeral: Software Should Decay and Lose Data

31 points by BerislavLopac 2y ago

maxbond 2y ago
The physical world does impose itself on software in some ways. Software is prone to going out of fashion, becoming disused and unmaintained, and eventually forgotten. Peering at a long dead document format in a hex editor is the digital equivalent of wiping dust and grime from the wall of a forgotten tomb.
Arainach 2y ago
This is a strange and awful take. Physical objects decay to the fundamental forces of nature, but that is a flaw, not something we should emulate. Software is a tool. Software is infrastructure. Degradation is desirable in neither. Data has metadata - creation time, access time, etc. - that can power workflows to identify and deal with old data if that is desirable, but in most cases they are not desirable. I haven't looked at the directory full of my wedding pictures in a decade, but I don't want my photo editing software to decide they're irrelevant and delete or hide them. Having access to old documents and data is often important and beneficial. It's why people look at inscriptions in old books, why they keep photo albums, why finding a forgotten item when moving furniture can be delightful. It's why we're able to identify trends. Most crucially, it is incredibly difficult to predict in advance which data will be useful in the future. Proactively discarding things when storage is incredibly cheap is a net negative.
- rpmisms 2y ago
  You can view software as tools or art, it is arguably both.
- TZubiri 2y ago
  Disagree, I'm with Kay on this one, life is much more powerful than current machines. When was the last time your mind ran into an out of storage issue and crashed? You talk about metadata being important, yet more than half of the storage used today isn't used to store books, or medical records or whatever business case, but traffic logs and virtualized package dependencies and shit like that.
  - fragmede 2y ago
    It's called a senior moment, and you'll have them too if you've never had one before. In a middle of a conversation, you'll just forget what you were saying because you got distracted.
    - TZubiri 2y ago
      First of all that's memory, not storage. And second, it's a graceful decay, you are not completely shut down from storing imporant information for later use. If in that moment something memorable happens, you will remember that.
  - castwide 2y ago
    The disagreement here seems to be about what kind of data should be considered ephemeral. Traffic logs from a decade ago? A ten-year-old package lock? I'm sure I don't care anymore. Documents I wrote in the 90s? Yeah, I might still want them. I might not need them everyday, but I can say from experience that I've needed to track down files that are more than a decade old, and I'm glad I was able to find them.
    - TZubiri 2y ago
      We strictly talking documents? It's very unlikely they will suffer from any decay since they are so lightweight. And to the extent that it's important you probably accessed it or shared it sometime in that period. A video from the 90s that you never read or accessed in that period? Yes, a candidate for deletion, or at least a decay. For example, if it was a video, the transcript might be saved, but the video might be lost. We may also get some written interpretation on the visual elements of the video.
      - castwide 2y ago
        I'm talking documents in a very general sense. It might be a text file, audio, video, a software installer, source code, or anything at all. > A video from the 90s that you never read or accessed in that period? Yes, a candidate for deletion, or at least a decay. For example, if it was a video, the transcript might be saved, but the video might be lost. We may also get some written interpretation on the visual elements of the video. My point is that the video shouldn't decay, either. It might not be playable in modern media software for a variety of reasons, but that doesn't automatically make it a candidate for deletion.
- the_mitsuhiko 2y ago
  Author here. > Physical objects decay to the fundamental forces of nature, but that is a flaw, not something we should emulate. That is explicitly called out in the article and I do not agree with you. Useful information is replicated, less useful information tends to die out. That we threw away that concept entirely in the digital world without a good replacement has some pretty strong downsides from my experience. And if you read what I wrote in more detail you can see that I'm very open to the idea of soft deletes and intentional information hiding.
  - CyberDildonics 2y ago
    [flagged]
    - the_mitsuhiko 2y ago
      I don't see how the headline is "inflammatory". You might disagree with the headline and the content and that is fine, however both headline and content reflects what I believe a core capability of (at least a significant number) of software products should be.
      - CyberDildonics 2y ago
        You think a core capability of software should be that it loses your data? That's like saying a core capability of shoes should be tacks under the sole. Of course you don't really think that. What software do you use that loses your data. Don't say something that cycles data out of a cache or lets you delete data, because that wouldn't be losing it.
        deleted 2y ago
        [deleted]
    - seo-speedwagon 2y ago
      No one forced them to write a comment without reading the article. And it’s not a small point that the author corrected them on; multiple paragraphs addressed thoughts like archiving to make searching old things intentional, and soft deletes. Also I didn’t read a scolding tone in the author’s response. And calling it an “inflammatory headline” is quite a stretch: this is a post tagged “thoughts” in someone’s personal blog, not a headline in the New York Times.
- Barrin92 2y ago
  >It's why people look at inscriptions in old books, why they keep photo albums, why finding a forgotten item when moving furniture can be delightful. It's why we're able to identify trends. I think it's often the opposite. Drawing confusing or bad analogies to the past rests on the false assumption that the future resembles the past, and the reason people love old photo albums or forgotten items is usually a kind of nostalgia, not because they hold relevance towards the future. This is also why forgetting or deliberately wiping the slate clean is fundamentally a good thing, it forces people to rebuild things from first principles without being bogged down in false assumptions carried over from the past, it's why the people who build new things tend to be young rather than old. If past information was so valuable in determining the future most visionaries would be 60. Hanging on to old information is a form of digital hoarding.
- deleted 2y ago
  [deleted]
totallykvothe 2y ago
Methinks the mindfulness cult has gone too far
TZubiri 2y ago
Thought about this. I agree 100%. The incumbent belief in software that machines should always remember (embedded in postgresql logo for example) has a dystopian sense of inmortality desires. I would like to play with the idea of a filesystem layer solution that tiers out information based on staleness. The closest I've seen to this is AWS S3 tiers, you've got standard S# at 2.2 cents per GB month, then infrequent, then glacier and some tiers in between.
deleted 2y ago
[deleted]
tracerbulletx 2y ago
I mean, retention policies exist.
orbital-decay 2y ago
That's a very feel-good "why" that doesn't answer on "how", which is 99% of the work. How exactly do you select what to forget?
- batch12 2y ago
  The attempts I've seen at this are usually in support of a corporate data classification and retention policy and are pretty hard to pull off successfully. The attempts I've seen usually involve adding metadata or tagging the data either manually or using automated classifiers.
donpark 2y ago
Decay and losses are just two of the many constraints that could be mixed like paints to create new systems. Limitations, artificial or not, are not always bad. Walls, for example, can be seen as limitation or protection depending on how it's used.
paulproteus 2y ago
For this reason, my browser's Downloads folder is /tmp.
- TacticalCoder 2y ago
  > For this reason, my browser's Downloads folder is /tmp OK. But my desktop reaches uptime of six months very easily (its not common that a security update mandating a reboot happens with Linux). Once in a while I clean my browser's Download folder. When I do that the very first thing I do, before the mess creeps in everywhere in "proper" directories is to deduplicate the browser's Download folder. The amount of time I'll have bankingstatement20241103.pdf, bankingstatement20241104-1.pdf (which should be really be "-2" btw instead of that zero-based non-sense but I digress), bankingstatement20241104-2.pdf, etc. in my Download folder is insane.
crabmusket 2y ago
What are some concrete examples of this? - Linear automatically cancels tickets which haven't been touched in 6 months - Trello would visually "age" cards, making them increasingly yellow and tattered - Datadog dashboards have a "popularity" score based on usage
- erik_seaberg 2y ago
  As jwz points out in "The CADT Model," abandoning reported bugs is just telling your users not to bother reporting anything.
  - crabmusket 2y ago
    In my experience, things don't get done just because they are hanging about in the backlog. Most auto-closed issues in my small team are of the varieties "the PM had a nice idea one day" and "long standing annoyance which we would need to re-architect to solve and we'll get to it one day". And I don't even necessarily disagree with the CADT post, but... most users are not, for better or worse, so easily dissuaded.
bsnnkv 2y ago
This was an excellent read and it resonates with me a lot. I have written here before[1] about the tyranny of "read it later" apps and how liberating it felt to ditch them and move to a "read it now or read it never" (RINORIN) mindset. Although this articles talks about digital ephemera in different contexts such as shared spaces, wikis, runbooks etc., I believe the underlying idea is the same. [1]: https://news.ycombinator.com/item?id=41991376 https://news.ycombinator.com/item?id=41991376
- erik_seaberg 2y ago
  Not everything important is equally urgent. https://en.wikipedia.org/wiki/Getting_Things_Done https://en.wikipedia.org/wiki/Getting_Things_Done recognizes the need for a trustworthy "I know I am going to get back to this at the right time" process, so that I don't continually reserve a bit of my brain worrying about forgetting.
keyle 2y ago
It's already ephemeral and decays by itself. Honestly as a polyglot, only a Go project can run without being touched for 5 years. The rest of them get vulnerabilities discovered in the packages, which requires you to upgrade, which changes the feature set and takes re-work. Also design ages, UX paradigms change over time, look at the "web 2.0" look today and the "web 1.0" of before. Skeuomorphic design aged as fast as Garage and Jungle music did in the 90s. This will be the same with flat design and neuomorphic design in 5-10 years, if there is even an UI then. All in all, software does decay and we lose data every day. Try go through your oldest bookmarks, many sites will be gone. Or run software that used to run on windows 98 or DOS or amiga or classic mac, you need to jump through hoops, the UX will be terrible, that's aging. Software doesn't need artificial aging, it erodes well all by itself. Even fast moving code in a startup changes so much from week to week, your merges can be difficult. That's like a river bed redefining itself during a rain storm. As humans go through time, so does software as an extension.
- dig1 2y ago
  > Honestly as a polyglot, only a Go project can run without being touched for 5 years. Go CVE archive [1] clearly indicate otherwise. What distinguishes a Go project from any other programming language? [1] https://pkg.go.dev/vuln/list https://pkg.go.dev/vuln/list
  - im_dario 2y ago
    > What distinguishes a Go project from any other programming language? My experience with Go, as user and open source maintainer, is that many libraries just work even if untouched for several years. There is a culture around building small libraries, with minimal usage of dependencies beyond the standard library. This leads to most CVEs existing in the standard library, which combined with Go's commitment to backward compatibility, makes improving your security posture simple by just upgrading the toolchain.
booleandilemma 2y ago
I don't think I could disagree more. He compares software to "the physical world", but photos in photo albums don't just disappear because they get old, and if I put something in a filing cabinet I know it'll be there in 1 year or 10 years.
- maxbond 2y ago
  That's because you took the care to put them in the photo album where they would be protected. If you left it sitting naked on your desk it would fade, accrue coffee stains, etc. Same goes with the filing cabinet. That's what Ronacher is suggesting - save the things that people care about, discard the things that are no longer relevant. I don't think it's a dogmatic position either, I don't think the suggestion is to always do this but to do this far more often than it's done now (almost never). And they are pretty explicit that they think this should be a default which can be overriden, and that you should be able to save things indefinitely if you so choose.
tqi 2y ago
> I'm referring to making a deliberate design decision to discard data at a later time. I'm willing to bet that your cloud storage or SaaS applications likely serve as dumping grounds for outdated, forgotten files and artifacts. This doesn’t have to be the case. IMO there is no way to do this programmatically that satisfies all of your users. What the author describes is organizational debt, and the only way to deal with it effectively is to have the organizational will to periodically clean up that debt. It's not easy, and any shortcut comes with it's own drawbacks (see corporate data retention policies that inevitably nuke documentation or context).

31 points by BerislavLopac 2y ago

The physical world does impose itself on software in some ways. Software is prone to going out of fashion, becoming disused and unmaintained, and eventually forgotten. Peering at a long dead document format in a hex editor is the digital equivalent of wiping dust and grime from the wall of a forgotten tomb.

Arainach 2y ago

This is a strange and awful take. Physical objects decay to the fundamental forces of nature, but that is a flaw, not something we should emulate. Software is a tool. Software is infrastructure. Degradation is desirable in neither. Data has metadata - creation time, access time, etc. - that can power workflows to identify and deal with old data if that is desirable, but in most cases they are not desirable. I haven't looked at the directory full of my wedding pictures in a decade, but I don't want my photo editing software to decide they're irrelevant and delete or hide them. Having access to old documents and data is often important and beneficial. It's why people look at inscriptions in old books, why they keep photo albums, why finding a forgotten item when moving furniture can be delightful. It's why we're able to identify trends. Most crucially, it is incredibly difficult to predict in advance which data will be useful in the future. Proactively discarding things when storage is incredibly cheap is a net negative.

rpmisms 2y ago
You can view software as tools or art, it is arguably both.
TZubiri 2y ago
Disagree, I'm with Kay on this one, life is much more powerful than current machines. When was the last time your mind ran into an out of storage issue and crashed? You talk about metadata being important, yet more than half of the storage used today isn't used to store books, or medical records or whatever business case, but traffic logs and virtualized package dependencies and shit like that.
- fragmede 2y ago
  It's called a senior moment, and you'll have them too if you've never had one before. In a middle of a conversation, you'll just forget what you were saying because you got distracted.
  - TZubiri 2y ago
    First of all that's memory, not storage. And second, it's a graceful decay, you are not completely shut down from storing imporant information for later use. If in that moment something memorable happens, you will remember that.
- castwide 2y ago
  The disagreement here seems to be about what kind of data should be considered ephemeral. Traffic logs from a decade ago? A ten-year-old package lock? I'm sure I don't care anymore. Documents I wrote in the 90s? Yeah, I might still want them. I might not need them everyday, but I can say from experience that I've needed to track down files that are more than a decade old, and I'm glad I was able to find them.
  - TZubiri 2y ago
    We strictly talking documents? It's very unlikely they will suffer from any decay since they are so lightweight. And to the extent that it's important you probably accessed it or shared it sometime in that period. A video from the 90s that you never read or accessed in that period? Yes, a candidate for deletion, or at least a decay. For example, if it was a video, the transcript might be saved, but the video might be lost. We may also get some written interpretation on the visual elements of the video.
    - castwide 2y ago
      I'm talking documents in a very general sense. It might be a text file, audio, video, a software installer, source code, or anything at all. > A video from the 90s that you never read or accessed in that period? Yes, a candidate for deletion, or at least a decay. For example, if it was a video, the transcript might be saved, but the video might be lost. We may also get some written interpretation on the visual elements of the video. My point is that the video shouldn't decay, either. It might not be playable in modern media software for a variety of reasons, but that doesn't automatically make it a candidate for deletion.
the_mitsuhiko 2y ago
Author here. > Physical objects decay to the fundamental forces of nature, but that is a flaw, not something we should emulate. That is explicitly called out in the article and I do not agree with you. Useful information is replicated, less useful information tends to die out. That we threw away that concept entirely in the digital world without a good replacement has some pretty strong downsides from my experience. And if you read what I wrote in more detail you can see that I'm very open to the idea of soft deletes and intentional information hiding.
- CyberDildonics 2y ago
  [flagged]
  - the_mitsuhiko 2y ago
    I don't see how the headline is "inflammatory". You might disagree with the headline and the content and that is fine, however both headline and content reflects what I believe a core capability of (at least a significant number) of software products should be.
    - CyberDildonics 2y ago
      You think a core capability of software should be that it loses your data? That's like saying a core capability of shoes should be tacks under the sole. Of course you don't really think that. What software do you use that loses your data. Don't say something that cycles data out of a cache or lets you delete data, because that wouldn't be losing it.
      - deleted 2y ago
        [deleted]
  - seo-speedwagon 2y ago
    No one forced them to write a comment without reading the article. And it’s not a small point that the author corrected them on; multiple paragraphs addressed thoughts like archiving to make searching old things intentional, and soft deletes. Also I didn’t read a scolding tone in the author’s response. And calling it an “inflammatory headline” is quite a stretch: this is a post tagged “thoughts” in someone’s personal blog, not a headline in the New York Times.
Barrin92 2y ago
>It's why people look at inscriptions in old books, why they keep photo albums, why finding a forgotten item when moving furniture can be delightful. It's why we're able to identify trends. I think it's often the opposite. Drawing confusing or bad analogies to the past rests on the false assumption that the future resembles the past, and the reason people love old photo albums or forgotten items is usually a kind of nostalgia, not because they hold relevance towards the future. This is also why forgetting or deliberately wiping the slate clean is fundamentally a good thing, it forces people to rebuild things from first principles without being bogged down in false assumptions carried over from the past, it's why the people who build new things tend to be young rather than old. If past information was so valuable in determining the future most visionaries would be 60. Hanging on to old information is a form of digital hoarding.
deleted 2y ago
[deleted]

totallykvothe 2y ago

Methinks the mindfulness cult has gone too far

TZubiri 2y ago

Thought about this. I agree 100%. The incumbent belief in software that machines should always remember (embedded in postgresql logo for example) has a dystopian sense of inmortality desires. I would like to play with the idea of a filesystem layer solution that tiers out information based on staleness. The closest I've seen to this is AWS S3 tiers, you've got standard S# at 2.2 cents per GB month, then infrequent, then glacier and some tiers in between.

deleted 2y ago

[deleted]

tracerbulletx 2y ago

I mean, retention policies exist.

orbital-decay 2y ago

That's a very feel-good "why" that doesn't answer on "how", which is 99% of the work. How exactly do you select what to forget?

batch12 2y ago
The attempts I've seen at this are usually in support of a corporate data classification and retention policy and are pretty hard to pull off successfully. The attempts I've seen usually involve adding metadata or tagging the data either manually or using automated classifiers.

donpark 2y ago

Decay and losses are just two of the many constraints that could be mixed like paints to create new systems. Limitations, artificial or not, are not always bad. Walls, for example, can be seen as limitation or protection depending on how it's used.

paulproteus 2y ago

For this reason, my browser's Downloads folder is /tmp.

TacticalCoder 2y ago
> For this reason, my browser's Downloads folder is /tmp OK. But my desktop reaches uptime of six months very easily (its not common that a security update mandating a reboot happens with Linux). Once in a while I clean my browser's Download folder. When I do that the very first thing I do, before the mess creeps in everywhere in "proper" directories is to deduplicate the browser's Download folder. The amount of time I'll have bankingstatement20241103.pdf, bankingstatement20241104-1.pdf (which should be really be "-2" btw instead of that zero-based non-sense but I digress), bankingstatement20241104-2.pdf, etc. in my Download folder is insane.

crabmusket 2y ago

What are some concrete examples of this? - Linear automatically cancels tickets which haven't been touched in 6 months - Trello would visually "age" cards, making them increasingly yellow and tattered - Datadog dashboards have a "popularity" score based on usage

erik_seaberg 2y ago
As jwz points out in "The CADT Model," abandoning reported bugs is just telling your users not to bother reporting anything.
- crabmusket 2y ago
  In my experience, things don't get done just because they are hanging about in the backlog. Most auto-closed issues in my small team are of the varieties "the PM had a nice idea one day" and "long standing annoyance which we would need to re-architect to solve and we'll get to it one day". And I don't even necessarily disagree with the CADT post, but... most users are not, for better or worse, so easily dissuaded.

bsnnkv 2y ago

This was an excellent read and it resonates with me a lot. I have written here before[1] about the tyranny of "read it later" apps and how liberating it felt to ditch them and move to a "read it now or read it never" (RINORIN) mindset. Although this articles talks about digital ephemera in different contexts such as shared spaces, wikis, runbooks etc., I believe the underlying idea is the same. [1]: https://news.ycombinator.com/item?id=41991376 https://news.ycombinator.com/item?id=41991376

erik_seaberg 2y ago
Not everything important is equally urgent. https://en.wikipedia.org/wiki/Getting_Things_Done https://en.wikipedia.org/wiki/Getting_Things_Done recognizes the need for a trustworthy "I know I am going to get back to this at the right time" process, so that I don't continually reserve a bit of my brain worrying about forgetting.

keyle 2y ago

It's already ephemeral and decays by itself. Honestly as a polyglot, only a Go project can run without being touched for 5 years. The rest of them get vulnerabilities discovered in the packages, which requires you to upgrade, which changes the feature set and takes re-work. Also design ages, UX paradigms change over time, look at the "web 2.0" look today and the "web 1.0" of before. Skeuomorphic design aged as fast as Garage and Jungle music did in the 90s. This will be the same with flat design and neuomorphic design in 5-10 years, if there is even an UI then. All in all, software does decay and we lose data every day. Try go through your oldest bookmarks, many sites will be gone. Or run software that used to run on windows 98 or DOS or amiga or classic mac, you need to jump through hoops, the UX will be terrible, that's aging. Software doesn't need artificial aging, it erodes well all by itself. Even fast moving code in a startup changes so much from week to week, your merges can be difficult. That's like a river bed redefining itself during a rain storm. As humans go through time, so does software as an extension.

dig1 2y ago
> Honestly as a polyglot, only a Go project can run without being touched for 5 years. Go CVE archive [1] clearly indicate otherwise. What distinguishes a Go project from any other programming language? [1] https://pkg.go.dev/vuln/list https://pkg.go.dev/vuln/list
- im_dario 2y ago
  > What distinguishes a Go project from any other programming language? My experience with Go, as user and open source maintainer, is that many libraries just work even if untouched for several years. There is a culture around building small libraries, with minimal usage of dependencies beyond the standard library. This leads to most CVEs existing in the standard library, which combined with Go's commitment to backward compatibility, makes improving your security posture simple by just upgrading the toolchain.

booleandilemma 2y ago

I don't think I could disagree more. He compares software to "the physical world", but photos in photo albums don't just disappear because they get old, and if I put something in a filing cabinet I know it'll be there in 1 year or 10 years.

maxbond 2y ago
That's because you took the care to put them in the photo album where they would be protected. If you left it sitting naked on your desk it would fade, accrue coffee stains, etc. Same goes with the filing cabinet. That's what Ronacher is suggesting - save the things that people care about, discard the things that are no longer relevant. I don't think it's a dogmatic position either, I don't think the suggestion is to always do this but to do this far more often than it's done now (almost never). And they are pretty explicit that they think this should be a default which can be overriden, and that you should be able to save things indefinitely if you so choose.

tqi 2y ago

> I'm referring to making a deliberate design decision to discard data at a later time. I'm willing to bet that your cloud storage or SaaS applications likely serve as dumping grounds for outdated, forgotten files and artifacts. This doesn’t have to be the case. IMO there is no way to do this programmatically that satisfies all of your users. What the author describes is organizational debt, and the only way to deal with it effectively is to have the organizational will to periodically clean up that debt. It's not easy, and any shortcut comes with it's own drawbacks (see corporate data retention policies that inevitably nuke documentation or context).