18 ms·

Spacedrive – a cross-platform file explorer, powered by a distributed filesystem

310 points by mmmmkay 4y ago

hunterb123 4y ago
Dupe: https://news.ycombinator.com/item?id=31170815 https://news.ycombinator.com/item?id=31170815
shaunregenbaum 4y ago
This is genius. Why didn't Dropbox do this?
- teawrecks 4y ago
  For the same reason all streaming services and game storefronts don't provide a generalized API to be used with 3rd party front ends. They all want to pretend that they are the only service you'll ever need/use, and their UX isn't designed for practicality, it's designed for marketing.
- api 4y ago
  All economic incentives for the past 20+ years have been away from "personal" computing because people flatly refuse to pay for software but for whatever reason have less aversion to paying for services. I have to repeat just to emphasize how irrational this is. People will refuse to pay $50 once for software but will happily pay $20-$50 per month for a service that offers less performance, no privacy, worse security, no continuity if you stop using it, fuzzy ownership of your own data, and a system that utterly vaporizes if the company goes away. The aversion to paying for software is intense too. People will happily sign up for SaaS but will go through insane contortions to pirate software or just flatly refuse to use it if it's not capital-F FOSS. Even OSS that you pay for is not okay. The most important thing about FOSS is not freedom, openness, or source, but "free" as in beer... but this same aversion to paying does not transfer to services. People will also pay $10+ dollars for a single cup of coffee at Starbucks every day. But god forbid they pay that much once for an app they use every day. Markets are incredibly irrational.
  - nittanymount 4y ago
    good points !
  - haswell 4y ago
    Is a cup of coffee at Starbucks really $10+ these days? I know that place is expensive, but has it gotten that expensive without me noticing? The thing I would primarily challenge is the assertion that this is fundamentally irrational behavior. Software is a means to an end. People don't spend money to acquire software, they spend money to solve a particular problem. There are two things that come to mind about subscriptions vs. the standalone pricing models of the recent past: 1) Before I'm willing to pay, you need to prove to me that your software will solve my problem. That proof doesn't come by installing the software, it comes after I've used it long enough to see if it does what I need it to do. Time or feature-limited trials attempted to solve this, but in a time-limited trial, I might solve my problem within the time period and never need to buy it (think unzipping a file). Or in a feature limited trial, I might get the general sense that the thing will work for me, but the software now holds my problem hostage, and I think there's a psychological component to why that feels "bad". 2) Sufficiently advanced software will be proportionally expensive, and out of reach for the average person. For this, I think about apps meant for writers, media and content creation, password management apps, etc. Apps that provide tremendous value, and therefore could command a high price, but now the average user is priced out. Subscriptions provide two things: 1) A feeling of safety coming from the fact that I can both use all aspects of the software, but also choose to just cancel the thing if it doesn't work or me. 2) The ability to spread out the cost over time. Maybe I can't afford $1,000 for <media creation software of choice>, but I can afford $20/month right now. You might argue that someone good with their finances should realize that they will eventually pay $1,000 either way, and eventually more, and they should just drop $1,000 now, but that is also a somewhat privileged position that most of us in the SW industry get to enjoy. It's also rational to opt for a solution that is lower risk, i.e. spending $1,000 for something that might not solve my problem could be considered irrational. I generally agree that people continue to struggle with the idea of paying for software. But I don't think that's fundamentally the problem. What people really want is a service that just handles their problems for them, and the software is just a means through which they interact with that service. The fact that people are willing to pay for services at least tells us that people are willing to pay for something. This is a hint that the model might be the problem, not the fact that money is involved. And I think this is what has ultimately led to the change in licensing models for apps like 1Password. I think that we as a development community need to accept the maybe uncomfortable truth that continuing to try to sell software in the traditional ways is itself irrational. It's the ultimate "the user is using it wrong". This is not a phenomenon that is unique to software either. Just look at the popularity of low-friction installment plans for purchases of many physical products. It's common across most major retailers to either plop down your $1,000 now, or sign up for monthly payments to spread it out over time.
smm11 4y ago
This is potentially awesome, but if I could just get a decent front-end to OneDrive on everything, I'd be happy.
- jbverschoor 4y ago
  https://mountainduck.io/ https://mountainduck.io/ oh.. front end.. well it's ok as a backend
AndrewUnmuted 4y ago
- indiv0 4y ago
  Are you boycotting operating rooms while you’re at it? I’ve never seen such an overreaction to such a non-issue before. How do people get to a state where they can unironically say things like this with a straight face?
  - AndrewUnmuted 4y ago
    Been in an OR since after you were born, OP? The OR physicians are in literal spacesuits these days.
- lazyier 4y ago
  I don't get it. You don't like that he is wearing a mask in the github profile? Are you serious? Am I missing something?
  - AndrewUnmuted 4y ago
- Fauntleroy 4y ago
  Can you explain how simply having a mask on in his profile picture is "propaganda"? Furthermore, how is having a github profile picture "anti-scientific"?
  - Melatonic 4y ago
    Hilariously thats not even the owner of the Github. Just a contributor !
    - AndrewUnmuted 4y ago
      Yeah it's super-funny! I'm having a ball right now!
poetril 4y ago
The creator streams his development over on twitch[0]. 0: https://twitch.tv/jamiepinelive https://twitch.tv/jamiepinelive
- deleted 4y ago
  [deleted]
- InCityDreams 4y ago
  ...streams his mealtime, too. No thanks - misophonic, me.
  - jamiepinelive 4y ago
    I do apologize, I also hate watching others eat. I should mute during mealtimes!
samstave 4y ago
FEATURE REQUEST PLEASE: As an (ahem) "hoarder" as you so kindly put in the .MD... Here is a feature I would like: Tag-routing... I want to create a folder with content tags. Then, when I go to download something, instead/in addition, I can just put in the tag for the media type and it lands in the right bucket. So /media/docs vs media/movies media/pdfs etc... When I download a thing - just let me calss tag it as movie and it will be placed in movie. etc... Make a browser extension that takes on downloading, and allows for more precise downloading, tagging, sorting of content before it ever hits a disk I own... Like this https://youtu.be/c0eTHm8NEqE?t=79 https://youtu.be/c0eTHm8NEqE?t=79
- danieltanfh95 4y ago
  ikr, me too, which is why I built http://ikomi.space/ http://ikomi.space/ , wip tho
- contr-error 4y ago
  git-annex has metadata driven views[0], which might be what you're looking for. When you're looking at a view, copying or moving a file to a folder "tagname", adds the tag "tagname" to that file. That seems easily scriptable to tag by media type/extension... [0] https://git-annex.branchable.com/tips/metadata_driven_views/ https://git-annex.branchable.com/tips/metadata_driven_views/
deleted 4y ago
[deleted]
anderspitman 4y ago
I love all the cool backend remote storage projects we have, but what I really want is a simple, well-specified app protocol for filesystem operations. WebDAV is too crufty. I'm bullish on Solid but I think there's room for something much simpler. I've done a bit of work[0] in this space but keep getting distracted with other parts of the self-hosted storage stack. [0]: https://gemdrive.io/ https://gemdrive.io/
- qbasic_forever 4y ago
  9p? https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol) I believe most VMs use 9p as a low level protocol for syncing filesystems between host and VM or container.
  - anderspitman 4y ago
    9p won't work for web apps. Personally I think we just need a small layer on top of standard HTTP for most use cases. It's really not a very hard technical problem, we just need to agree on a protocol. But it's a chicken-egg situation. App developers aren't going to develop against a protocol that no one uses, and no one's going to use a protocol that doesn't have any apps.
    - qbasic_forever 4y ago
      I'd tunnel it over a websocket, or there's no reason why you couldn't translate its verbs into a REST API. It's a super simple protocol with not a lot of complexity. It sounds like you're after a federated storage protocol though, i.e. something that two totally different apps can use to allow a user to transfer data between them. This is a much, much bigger leap in complexity and requires a very strong authentication and authorization story. IMHO look into the fediverse and whatever is going on with storage and files there.
      - mike_hearn 4y ago
        Isn't translating filesystem operation verbs into an HTTP based protocol what WebDAV does?
rufugee 4y ago
Spacedrive accounts for every file you own, uniquely fingerprinting and extracting metadata so you can sort, tag, backup Oh...how I've longed for this. How I've wanted to carve out the time to do exactly this to service my own needs (as a hoarder). I am so excited to use this one it's stable enough to trust fingerprinting to the extent that you can dedupe across devices!
LeoPanthera 4y ago
Previous thread: https://news.ycombinator.com/item?id=31170815 https://news.ycombinator.com/item?id=31170815 My comment from the previous thread: I'm not sure if it actually does work this way, but this would be amazing if I could install a small daemon on my NAS/fileserver to do the scanning and indexing, which the graphical client could then pull the results of.
- sixothree 4y ago
  My simple question from previous thread, asking where the indexes were stored and who has access to them, went unanswered. Not a good look.
xoa 4y ago
One feature a few of us toyed with for years that I'd love to see tried seriously someday (or does it already exist?) would be intelligent tag trees or maps that stuff could be plugged into, ideally that could also be shared and mixed too. More specific tags would have parent/child/sibling relationships with more generic tags, and a single tag would then pull in all the others as far as searching. So for example if I have a photo of one of my cats, and I tag it Cat Name having set that up, "calico" or "tabby", breed, plain old "cat" and anything else associated with the specific tag would all be hits. And up the stack, if I tag a picture "calico" it should show up under the "cat" tag as well automatically with no further effort. Conditionals could be incorporated too, like if there is a character who in a series has long hair, tagging "short hair" could automatically pull in "alternative hairstyle". My big problem with typical tagging is that it's a lot of grunt work to be comprehensive. AI can help but has its own issues. While getting the UI/UX right would take some work, a universal system-wide multi-dimensional tag map could allow a lot more metadata a lot more easily. Even more so with sharing, communities could curate tag maps around areas of interest and one could stay synced with tag maps of "aircraft" or "trees" or the like which would bring in lots of technical depth to explore in a distributed fashion. Eventually maybe community AI recognition models could be an optional part of that too to help identify specific given subjects (and one would be free to then not have other kinds of things identified or shared).
- jewel 4y ago
  My homemade photo organization software has the tags as trees and it works great for exactly the example you've given, as well as "plant" being a parent of "flower" which is a parent of "rose". In addition to better searches, it also allows for better tagging, because a lazy tagger might tag a photo as "food" and I can come back later and fix it to be "fruit pie". There was an open ontology database somewhere I might integrate so that it has a lot of prefilled tags. That'd be the place to start for photos or anything dealing with the physical world.
user_7832 4y ago
Sightly tangential, but does anyone know of any good system that integrates everything across devices? I'm talking backups/data management, clipboards, maybe notifications too etc. Bonus if music/video can be streamed and mouse/keyboards can be used cross device. KDE Connect is nice and can do some of these, and many backup utilities like rclone or duplicati (which I've used in the past) can do some, but many features are not great. The closest I can think of is Apple's walled garden where you store everything on iCloud, but a) the massive restrictions on iOS, and more recently on MacOS (see: kext restrictions) are forced and b) it still isn't perfect and requires an iCloud subscription if you have more than 5gb of data you want synchronized. I'm curious to know if any HN person knows of such software (assuming I can run a server to host the backend). (In real life I still have an "unlimited" GDrive account from my university that I use to sync but it's far from perfect.)
obphuscate 4y ago
Thanks goodness I clicked on the link out of shear curiosity with this title - I might not have known it was written in Rust otherwise.
linsomniac 4y ago
Kinda want to put in my e-mail address to be notified when I can download it to try.
pabs3 4y ago
Sounds a bit like Hurd's translators. https://www.gnu.org/software/hurd/hurd/translator.html https://www.gnu.org/software/hurd/hurd/translator.html
salutonmundo 4y ago
thats called sshfs
oefrha 4y ago
> File discovery - Scan devices, drives and cloud accounts to build a directory of all files with metadata. How well does this scale? For instance, I have millions of objects totaling >10TB on Google Drive. Tools that attempt to index them all tend to break down because (1) the files/list API has a page size limit of 1000; (2) accidentally nonlinear complexity.

310 points by mmmmkay 4y ago

hunterb123 4y ago

Dupe: https://news.ycombinator.com/item?id=31170815 https://news.ycombinator.com/item?id=31170815

shaunregenbaum 4y ago

This is genius. Why didn't Dropbox do this?

teawrecks 4y ago
For the same reason all streaming services and game storefronts don't provide a generalized API to be used with 3rd party front ends. They all want to pretend that they are the only service you'll ever need/use, and their UX isn't designed for practicality, it's designed for marketing.
api 4y ago
All economic incentives for the past 20+ years have been away from "personal" computing because people flatly refuse to pay for software but for whatever reason have less aversion to paying for services. I have to repeat just to emphasize how irrational this is. People will refuse to pay $50 once for software but will happily pay $20-$50 per month for a service that offers less performance, no privacy, worse security, no continuity if you stop using it, fuzzy ownership of your own data, and a system that utterly vaporizes if the company goes away. The aversion to paying for software is intense too. People will happily sign up for SaaS but will go through insane contortions to pirate software or just flatly refuse to use it if it's not capital-F FOSS. Even OSS that you pay for is not okay. The most important thing about FOSS is not freedom, openness, or source, but "free" as in beer... but this same aversion to paying does not transfer to services. People will also pay $10+ dollars for a single cup of coffee at Starbucks every day. But god forbid they pay that much once for an app they use every day. Markets are incredibly irrational.
- nittanymount 4y ago
  good points !
- haswell 4y ago
  Is a cup of coffee at Starbucks really $10+ these days? I know that place is expensive, but has it gotten that expensive without me noticing? The thing I would primarily challenge is the assertion that this is fundamentally irrational behavior. Software is a means to an end. People don't spend money to acquire software, they spend money to solve a particular problem. There are two things that come to mind about subscriptions vs. the standalone pricing models of the recent past: 1) Before I'm willing to pay, you need to prove to me that your software will solve my problem. That proof doesn't come by installing the software, it comes after I've used it long enough to see if it does what I need it to do. Time or feature-limited trials attempted to solve this, but in a time-limited trial, I might solve my problem within the time period and never need to buy it (think unzipping a file). Or in a feature limited trial, I might get the general sense that the thing will work for me, but the software now holds my problem hostage, and I think there's a psychological component to why that feels "bad". 2) Sufficiently advanced software will be proportionally expensive, and out of reach for the average person. For this, I think about apps meant for writers, media and content creation, password management apps, etc. Apps that provide tremendous value, and therefore could command a high price, but now the average user is priced out. Subscriptions provide two things: 1) A feeling of safety coming from the fact that I can both use all aspects of the software, but also choose to just cancel the thing if it doesn't work or me. 2) The ability to spread out the cost over time. Maybe I can't afford $1,000 for <media creation software of choice>, but I can afford $20/month right now. You might argue that someone good with their finances should realize that they will eventually pay $1,000 either way, and eventually more, and they should just drop $1,000 now, but that is also a somewhat privileged position that most of us in the SW industry get to enjoy. It's also rational to opt for a solution that is lower risk, i.e. spending $1,000 for something that might not solve my problem could be considered irrational. I generally agree that people continue to struggle with the idea of paying for software. But I don't think that's fundamentally the problem. What people really want is a service that just handles their problems for them, and the software is just a means through which they interact with that service. The fact that people are willing to pay for services at least tells us that people are willing to pay for something. This is a hint that the model might be the problem, not the fact that money is involved. And I think this is what has ultimately led to the change in licensing models for apps like 1Password. I think that we as a development community need to accept the maybe uncomfortable truth that continuing to try to sell software in the traditional ways is itself irrational. It's the ultimate "the user is using it wrong". This is not a phenomenon that is unique to software either. Just look at the popularity of low-friction installment plans for purchases of many physical products. It's common across most major retailers to either plop down your $1,000 now, or sign up for monthly payments to spread it out over time.

smm11 4y ago

This is potentially awesome, but if I could just get a decent front-end to OneDrive on everything, I'd be happy.

jbverschoor 4y ago
https://mountainduck.io/ https://mountainduck.io/ oh.. front end.. well it's ok as a backend

AndrewUnmuted 4y ago

indiv0 4y ago
Are you boycotting operating rooms while you’re at it? I’ve never seen such an overreaction to such a non-issue before. How do people get to a state where they can unironically say things like this with a straight face?
- AndrewUnmuted 4y ago
  Been in an OR since after you were born, OP? The OR physicians are in literal spacesuits these days.
lazyier 4y ago
I don't get it. You don't like that he is wearing a mask in the github profile? Are you serious? Am I missing something?
- AndrewUnmuted 4y ago
Fauntleroy 4y ago
Can you explain how simply having a mask on in his profile picture is "propaganda"? Furthermore, how is having a github profile picture "anti-scientific"?
- Melatonic 4y ago
  Hilariously thats not even the owner of the Github. Just a contributor !
  - AndrewUnmuted 4y ago
    Yeah it's super-funny! I'm having a ball right now!

poetril 4y ago

The creator streams his development over on twitch[0]. 0: https://twitch.tv/jamiepinelive https://twitch.tv/jamiepinelive

deleted 4y ago
[deleted]
InCityDreams 4y ago
...streams his mealtime, too. No thanks - misophonic, me.
- jamiepinelive 4y ago
  I do apologize, I also hate watching others eat. I should mute during mealtimes!

samstave 4y ago

FEATURE REQUEST PLEASE: As an (ahem) "hoarder" as you so kindly put in the .MD... Here is a feature I would like: Tag-routing... I want to create a folder with content tags. Then, when I go to download something, instead/in addition, I can just put in the tag for the media type and it lands in the right bucket. So /media/docs vs media/movies media/pdfs etc... When I download a thing - just let me calss tag it as movie and it will be placed in movie. etc... Make a browser extension that takes on downloading, and allows for more precise downloading, tagging, sorting of content before it ever hits a disk I own... Like this https://youtu.be/c0eTHm8NEqE?t=79 https://youtu.be/c0eTHm8NEqE?t=79

danieltanfh95 4y ago
ikr, me too, which is why I built http://ikomi.space/ http://ikomi.space/ , wip tho
contr-error 4y ago
git-annex has metadata driven views[0], which might be what you're looking for. When you're looking at a view, copying or moving a file to a folder "tagname", adds the tag "tagname" to that file. That seems easily scriptable to tag by media type/extension... [0] https://git-annex.branchable.com/tips/metadata_driven_views/ https://git-annex.branchable.com/tips/metadata_driven_views/

deleted 4y ago

[deleted]

anderspitman 4y ago

I love all the cool backend remote storage projects we have, but what I really want is a simple, well-specified app protocol for filesystem operations. WebDAV is too crufty. I'm bullish on Solid but I think there's room for something much simpler. I've done a bit of work[0] in this space but keep getting distracted with other parts of the self-hosted storage stack. [0]: https://gemdrive.io/ https://gemdrive.io/

qbasic_forever 4y ago
9p? https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol) I believe most VMs use 9p as a low level protocol for syncing filesystems between host and VM or container.
- anderspitman 4y ago
  9p won't work for web apps. Personally I think we just need a small layer on top of standard HTTP for most use cases. It's really not a very hard technical problem, we just need to agree on a protocol. But it's a chicken-egg situation. App developers aren't going to develop against a protocol that no one uses, and no one's going to use a protocol that doesn't have any apps.
  - qbasic_forever 4y ago
    I'd tunnel it over a websocket, or there's no reason why you couldn't translate its verbs into a REST API. It's a super simple protocol with not a lot of complexity. It sounds like you're after a federated storage protocol though, i.e. something that two totally different apps can use to allow a user to transfer data between them. This is a much, much bigger leap in complexity and requires a very strong authentication and authorization story. IMHO look into the fediverse and whatever is going on with storage and files there.
    - mike_hearn 4y ago
      Isn't translating filesystem operation verbs into an HTTP based protocol what WebDAV does?

rufugee 4y ago

Spacedrive accounts for every file you own, uniquely fingerprinting and extracting metadata so you can sort, tag, backup Oh...how I've longed for this. How I've wanted to carve out the time to do exactly this to service my own needs (as a hoarder). I am so excited to use this one it's stable enough to trust fingerprinting to the extent that you can dedupe across devices!

LeoPanthera 4y ago

Previous thread: https://news.ycombinator.com/item?id=31170815 https://news.ycombinator.com/item?id=31170815 My comment from the previous thread: I'm not sure if it actually does work this way, but this would be amazing if I could install a small daemon on my NAS/fileserver to do the scanning and indexing, which the graphical client could then pull the results of.

sixothree 4y ago
My simple question from previous thread, asking where the indexes were stored and who has access to them, went unanswered. Not a good look.

xoa 4y ago

One feature a few of us toyed with for years that I'd love to see tried seriously someday (or does it already exist?) would be intelligent tag trees or maps that stuff could be plugged into, ideally that could also be shared and mixed too. More specific tags would have parent/child/sibling relationships with more generic tags, and a single tag would then pull in all the others as far as searching. So for example if I have a photo of one of my cats, and I tag it Cat Name having set that up, "calico" or "tabby", breed, plain old "cat" and anything else associated with the specific tag would all be hits. And up the stack, if I tag a picture "calico" it should show up under the "cat" tag as well automatically with no further effort. Conditionals could be incorporated too, like if there is a character who in a series has long hair, tagging "short hair" could automatically pull in "alternative hairstyle". My big problem with typical tagging is that it's a lot of grunt work to be comprehensive. AI can help but has its own issues. While getting the UI/UX right would take some work, a universal system-wide multi-dimensional tag map could allow a lot more metadata a lot more easily. Even more so with sharing, communities could curate tag maps around areas of interest and one could stay synced with tag maps of "aircraft" or "trees" or the like which would bring in lots of technical depth to explore in a distributed fashion. Eventually maybe community AI recognition models could be an optional part of that too to help identify specific given subjects (and one would be free to then not have other kinds of things identified or shared).

jewel 4y ago
My homemade photo organization software has the tags as trees and it works great for exactly the example you've given, as well as "plant" being a parent of "flower" which is a parent of "rose". In addition to better searches, it also allows for better tagging, because a lazy tagger might tag a photo as "food" and I can come back later and fix it to be "fruit pie". There was an open ontology database somewhere I might integrate so that it has a lot of prefilled tags. That'd be the place to start for photos or anything dealing with the physical world.

user_7832 4y ago

Sightly tangential, but does anyone know of any good system that integrates everything across devices? I'm talking backups/data management, clipboards, maybe notifications too etc. Bonus if music/video can be streamed and mouse/keyboards can be used cross device. KDE Connect is nice and can do some of these, and many backup utilities like rclone or duplicati (which I've used in the past) can do some, but many features are not great. The closest I can think of is Apple's walled garden where you store everything on iCloud, but a) the massive restrictions on iOS, and more recently on MacOS (see: kext restrictions) are forced and b) it still isn't perfect and requires an iCloud subscription if you have more than 5gb of data you want synchronized. I'm curious to know if any HN person knows of such software (assuming I can run a server to host the backend). (In real life I still have an "unlimited" GDrive account from my university that I use to sync but it's far from perfect.)

obphuscate 4y ago

Thanks goodness I clicked on the link out of shear curiosity with this title - I might not have known it was written in Rust otherwise.

linsomniac 4y ago

Kinda want to put in my e-mail address to be notified when I can download it to try.

pabs3 4y ago

Sounds a bit like Hurd's translators. https://www.gnu.org/software/hurd/hurd/translator.html https://www.gnu.org/software/hurd/hurd/translator.html

salutonmundo 4y ago

thats called sshfs

oefrha 4y ago

> File discovery - Scan devices, drives and cloud accounts to build a directory of all files with metadata. How well does this scale? For instance, I have millions of objects totaling >10TB on Google Drive. Tools that attempt to index them all tend to break down because (1) the files/list API has a page size limit of 1000; (2) accidentally nonlinear complexity.