11 ms·

Nix × IPFS – Milestone 1

229 points by Fnoord 6y ago

nmfisher 6y ago
Interesting use case for IPFS, which has often felt like an (admittedly cool) solution in search of a problem. Can someone enlighten me as to real-world examples where actually reproducible builds are critical?
- arianvanp 6y ago
  For one. They allow for distributed incremental builds. Improving your developer productivity. IT doesn't matter anymore where an object file is compiled. You just need to know its hash, ask if it exists, or otherwise compile yourself. One problem with NixOS currently is that certain dependencies in our package tree are very painful to change. If we touch glibc, we need to recompile 60.000 packages and it takes A LOT of compute power to do that. With reproducible, content-addressed builds we can do things like early cutoff optimisations; making these changes less painful. SImple example; if somebody just changed a source code comment in glibc; then we get the same build artifact, and can skip building 60.000 packages. If you do this at the source code level instead of the package level (like Bazel) then somebody can change how glibc does domain resolution; but packages that don't depend on that don't need to be recompiled either.
  - teacpde 6y ago
    I understand why changing comments doesn’t matter, but how do you decide which dependencies don’t need to be rebuilt when certain part of the source code is changed?
    - kevincox 6y ago
      The most "proper" way to do this is splitting the result into interface + implementation. For example if you are dynamically linking to a library instead of being given access to the entire library you are given a simplified "signature" which has the minimal amount of info required to linking. This interface would change when you add or remove a function, but not when a function body changes. The example was given for linking but this can be done for many steps of the build process.
    - Ericson2314 6y ago
      Nix does this simply and stupidly --- if you can observ it, it's part of the cache key. This is wonderful because it's totally sound. All efficiency methods (basically, hiding things, as the other comment says) is left to the build plan itself. Switching to Nix for batch jobs is switching from DOS to an OS with actual process isolation. A complete paradigm shift.
      - sterlind 6y ago
        It does still require trust, though. You can lie about the contents since the hash only covers the build inputs, so someone could break the sandboxing. Of course, since builds are reproducible you can catch this if you build it yourself and compare the actual content-hashes.
        Ericson2314 6y ago
        I would say the trust issues with build remotes and substitutors are fairly independent of the caching policy? The caching model is about when the drv changes ((drv, output) is the cache key, basically), the trust model is one one can download something to avoiding build to fill in the value for the new cache key. Note also that with the new floating content-addressed derivations, the trust stuff is much easier to think about.
- Taek 6y ago
  Aren't reproducible builds pretty well established as being important for security? How do you know that the developers are providing you with the binaries that they claim? Most people do not compile their own software, they used signed binary distributions being given to them. If you have reproducible builds, third parties can more easily verify independently that the code being distributed is the same as the code published in the open source repositories. This doesn't just keep dev teams honest, it also provides defenses against situations like those where a hacker gains control of the website being used for distribution. I believe reproducible builds have helped to catch this real life issue at least once in the cryptocurrency community (with Monero). To me, that's enough to justify their existence.
- chpatrick 6y ago
  Our team uses Nix. It lets us share a binary cache of modified/own packages between the developers, CI, and our production machines. If you push something to CI that needs some new dependencies, the CI machine will build them and when you want to get a local dev environment or deploy, the binaries will be fetched from the CI machine. We can also send the binaries between machines to save compile time. This is possible because we have confidence that it doesn't matter which machine built them because the environment is so strictly controlled.
- Foxboron 6y ago
  Confusingly enough, when NixOS talks about reproducible builds they are not talking about https://reproducible-builds.org/ https://reproducible-builds.org/. They are talking about reproducible systems. NixOS doesn't solve reproducible builds as-in "bit-for-bit identical binary distributed files", but it does solve "same input" -> "functionally same system". This is the core idea of NixOS and what makes the package manager and the spawned ecosystem (like Guix) quite neat. I also think this isn't any more controversial then say having your infrastructure reproducible with terraform and ansible/salt. It gives you machines as cattle, and not pets, with the added flexibility that gives you. NixOS is just a complete package where in other systems there are several components accomplishing the goal. As to why reproducible builds is important is to ensure a strong connection between the upstream source code and the distributed binary build. This gives you confidence that everything going into the build has been declared and that the binary can be reproduced bit-for-bit identical if you so wish. This can be both important in a supply chain process. I recently saw tailscale provided pre-compiled binaries without signatures. What does those binaries contain? They are not signed, so the releases could very well be unauthenticated and someone could have compromised the server distributing the binaries. So I tried reproducing the tailscale binaries without much success as the build process is proprietary. https://github.com/tailscale/tailscale/issues/779 https://github.com/tailscale/tailscale/issues/779 So what does the tailscale binaries contain? I can't get a bit-for-bit identical binary without putting into a great deal of effort so for all intents and purposes they can very well be proprietary. Now I don't think the tailscale people are malicious, nor that the binaries have been replaced. But it's a real-world example I recently went through.
  - solarkraft 6y ago
    Even when the binaries are signed you'd still have to trust Tailscale they didn't modify the code before compilation, maybe putting in a simple back door. They probably don't, but you can't really know. When you have binary reproducible builds such a thing would immediately be obvious and hell would break loose if they attempted it.
  - Ericson2314 6y ago
    The floating content-addressed derivations we've worked on as part of this Nix × IPFS work get's us a lot closer to reproducible builds by making non-determinstic builds far more noticiable. See these: https://github.com/NixOS/rfcs/pull/62 https://github.com/NixOS/rfcs/pull/62 https://github.com/NixOS/nix/issues/4087 https://github.com/NixOS/nix/issues/4087
- zelly 6y ago
  There are much more real-world use cases for reproducible builds than for IPFS. It saves time and money. It's not just academic. Reproducible builds (as in bit-for-bit) means I can trust a (secure) checksum of a binary I downloaded somewhere, let's say from IPFS. This is defense-in-depth, adding another layer of security to prevent sabotage. Reproducibility means the whole system can reliably be built from source and I don't need to manually make sure the system has the correct constellation of implicit dependencies and their version numbers. This saves time and money in deployment and will pay off in a very big way 10 years down the line when that code is still in use somewhere.
arianvanp 6y ago
Does IPFS actually solve the problem they set out here though? IPFS is a distributed CDN; but not very good for storing things persistently or reliably from my experience. At the moment; the nixos cache is stored in very durable and reliable S3 storage; with very high durability guarantees. Why is that not good enough? sure it's centralised. But IPFS doesn't offer distributed durability; it offers a CDN. It doesn't seem to address this issue the authors seem to claim it solves. (To me) I still think it's _super cool_. And once builds are also content-addressed and not just fixed-output derivations; having trustless content delivery is a valuable addition. But IPFS doesnt' seem like a robust answer for the "what if we lose access to the source code problem" given it's not a durable storage system In this case; projects like https://www.softwareheritage.org/ https://www.softwareheritage.org/ and https://sfconservancy.org/ https://sfconservancy.org/ seem better bets to solve the source code access issue
- the_duke 6y ago
  I imagine you would ensure the persistence of the data you care about by either running your own IPFS nodes that pin the data, or by using a pinning service like Pinata [1]. [1] https://pinata.cloud/ https://pinata.cloud/
  - TheOtherWW 6y ago
    The problem with Pinata is that they charge $150 per TB for their Individual plan. That's nearly 6.5x the cost of storing the data on S3. Sure it works, but that high barrier to entry pricing scares off a lot of people. Why not just use S3? Meanwhile, we're still waiting for Filecoin to launch, and networks such as Sia have seized that opportunity and created great things like Skynet [1]. Skynet itself still has some overhead if you want to ensure data persistence and availability, but the cost is orders of magnitude lower. In addition, new layer 2 providers have emerged to address those gaps, such as Filebase [2]. They provide S3 compatible object storage that is backed by decentralized cloud storage. You get high availability (of the storage layer), geo-redundancy, and less than S3 pricing out of the box. It is this type of offering where we are going to see the most impact and adoption as the underlying technology not only makes things more efficient, but cheaper too. [1] https://siasky.net/ https://siasky.net/ [2] https://filebase.com/ https://filebase.com/
imhoguy 6y ago
Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and seeds it.
- Avamander 6y ago
  I don't get why machines running the same Linux distribution couldn't actually do the same. BT2 would indeed work very well for this.
  - e12e 6y ago
    Nixos/guix might be a good fit for rootfs on ipfs? Comes with a local cache, and you could overlay local changes (eg: /etc/passwd)?
    - takeda 6y ago
      Actually not exactly rootfs on ipfs, but with NixOS is fairly easy to make it wipe data on reboot, here's an example article[1]. The idea is to only keep state we care about and rebuild the system on every boot. [1] https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings
- solarkraft 6y ago
  > I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution 2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.
  - whyrusleeping 6y ago
    Some links without comment: - https://www.npmjs.com/package/ipfs-npm https://www.npmjs.com/package/ipfs-npm - https://github.com/ipfs-shipyard/apt-on-ipfs https://github.com/ipfs-shipyard/apt-on-ipfs - https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain...
iElectric2 6y ago
See also https://cachix.org https://cachix.org for a hosted binary cache.
majewsky 6y ago
Is it down for everyone or just me? Firefox is showing PR_CONNECT_ABORTED_ERROR when trying to navigate to the link.
- Shared404 6y ago
  It works for me, but I'll drop this just to be safe: https://outline.com/vWwGas https://outline.com/vWwGas
ArchD 6y ago
IPFS is not censorship resistant. If people start using it to subvert tyranny, politically neutral things like Nix's use of IPFS can get collaterally damaged in its infancy by state power.
- jFriedensreich 6y ago
  Its slightly more censorship resistant than S3 but yes, that seems to not have been on the list of problems they try to solve. On the other hand a list of all the systems that solve the problems in a half decent performant manner and are also censorship resistant would be 0 to my knowledge.
- PureParadigm 6y ago
  I think censorship resistance should be handled at a different abstraction layer. IPFS can focus on peer-to-peer immutable file sharing and some other tool can tunnel your traffic to avoid censorship. You could run IPFS over a VPN or something like that.
vander_elst 6y ago
Does anyone know if there are authz features planned for ipfs? Like having acls based on certificates or something...
- e12e 6y ago
  As far as I'm aware the "official" ipfs line is ipfs host and share data - if you want to protect it; encrypt it.
ingenieroariel 6y ago
This is a big deal, congratulations on the milestone. Really looking forward to trying out Reflex FRP / Obelisk from IPFS, in particular the focus on making the source available because fetching Nix packages from semi-public caches and not being able to rebuild them is something that has bothered me for a while.
alexmingoia 6y ago
One thing I’ve always wondered, is how DHTs like BitTorrent and IPFS prevent Sybil attacks. If anyone wanted to block a file in IPFS, could they generate node ids close to the filehash and return empty peer lists?
- whyrusleeping 6y ago
  Yeah, sybill attacks are pretty hard to get around. IPFS itself doesnt entirely depend on a DHT though, it uses it when it can, but also falls back to a more gossip style approach of just asking peers you connect to for the data. So if you wait long enough, and the content is out there, you're very likely to end up finding it as your node makes and receives random connections throughout the network. This does still mean that the sybill attacker can severely degrade the quality of service though, so other solutions are getting looked into. The 'easiest' one that comes to mind is just forming an incentivized DHT of some kind, with the simplest start to that being a requirement that all DHT server peers stake some funds. There are a lot of weird things in this problem space, and its still pretty young IMO
adsjhdashkj 6y ago
Tangent, what's the best way to use Nix-like features in a mainstream distro? Eg, i've been evaluating PopOS recently because i want a Mac equivalent OS. Ie, i don't want driver/config issues. However one problem i never see solved from any "normal" distros is reproducible systems. Hypothetically if i used Nix as a Desktop my config would be bulletproof. .. but then i'm going through a fair amount of work configuring everything when, as established, i want none of it. So i (as a user) seem to want some middleground between no effort installations / configurations of my Desktop, with reproducible snapshots/states/configs/something. You can use Nixpkgs on other distros/OSs, so maybe that is enough. But is there a better way?
- MarcScott 6y ago
  I just bit the bullet and use NixOS as my Desktop. There's a bit of work after initial install, to setup your config files the way you want, but after that you're all set. I've had no problems transferring my configs when I change hardware, and end up with an identical environment that I had on my previous computer.
  - kevincox 6y ago
    I recently switched as well and am very happy. I don't think that re-configuring takes much longer that the average distro would especially because a lot of services you can just say `services.foobar.enable = true` and get a decent default config. You only need to spend your time configuring the things that you are picky about. And yes, the fact that I can configure all of my systems with a shared set of configs and have consistent and reproducible environments is fantastic. I wrote up my thoughts about the switch here: https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/ https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/
  - adsjhdashkj 6y ago
    Sidenote, i'm taking the plunge into NixOS, but notably using Flakes. While i've dabbled in Nix in the past some things felt more odd to me than perhaps they should have. Primarily the fact that it felt like all these configs were spread out and i didn't understand how Nix wanted me to version them. Flakes (from early tests) seems to make this very clear. As it starts from a repo, so i'm attempting to make all my configs, including Home Manager (of which also feels weird lol) start from the Flakes installation. Flakes also solves the reproducibility problem that i didn't get why NixOS had. So far it's really neat.

229 points by Fnoord 6y ago

Interesting use case for IPFS, which has often felt like an (admittedly cool) solution in search of a problem. Can someone enlighten me as to real-world examples where actually reproducible builds are critical?

arianvanp 6y ago
For one. They allow for distributed incremental builds. Improving your developer productivity. IT doesn't matter anymore where an object file is compiled. You just need to know its hash, ask if it exists, or otherwise compile yourself. One problem with NixOS currently is that certain dependencies in our package tree are very painful to change. If we touch glibc, we need to recompile 60.000 packages and it takes A LOT of compute power to do that. With reproducible, content-addressed builds we can do things like early cutoff optimisations; making these changes less painful. SImple example; if somebody just changed a source code comment in glibc; then we get the same build artifact, and can skip building 60.000 packages. If you do this at the source code level instead of the package level (like Bazel) then somebody can change how glibc does domain resolution; but packages that don't depend on that don't need to be recompiled either.
- teacpde 6y ago
  I understand why changing comments doesn’t matter, but how do you decide which dependencies don’t need to be rebuilt when certain part of the source code is changed?
  - kevincox 6y ago
    The most "proper" way to do this is splitting the result into interface + implementation. For example if you are dynamically linking to a library instead of being given access to the entire library you are given a simplified "signature" which has the minimal amount of info required to linking. This interface would change when you add or remove a function, but not when a function body changes. The example was given for linking but this can be done for many steps of the build process.
  - Ericson2314 6y ago
    Nix does this simply and stupidly --- if you can observ it, it's part of the cache key. This is wonderful because it's totally sound. All efficiency methods (basically, hiding things, as the other comment says) is left to the build plan itself. Switching to Nix for batch jobs is switching from DOS to an OS with actual process isolation. A complete paradigm shift.
    - sterlind 6y ago
      It does still require trust, though. You can lie about the contents since the hash only covers the build inputs, so someone could break the sandboxing. Of course, since builds are reproducible you can catch this if you build it yourself and compare the actual content-hashes.
      - Ericson2314 6y ago
        I would say the trust issues with build remotes and substitutors are fairly independent of the caching policy? The caching model is about when the drv changes ((drv, output) is the cache key, basically), the trust model is one one can download something to avoiding build to fill in the value for the new cache key. Note also that with the new floating content-addressed derivations, the trust stuff is much easier to think about.
Taek 6y ago
Aren't reproducible builds pretty well established as being important for security? How do you know that the developers are providing you with the binaries that they claim? Most people do not compile their own software, they used signed binary distributions being given to them. If you have reproducible builds, third parties can more easily verify independently that the code being distributed is the same as the code published in the open source repositories. This doesn't just keep dev teams honest, it also provides defenses against situations like those where a hacker gains control of the website being used for distribution. I believe reproducible builds have helped to catch this real life issue at least once in the cryptocurrency community (with Monero). To me, that's enough to justify their existence.
chpatrick 6y ago
Our team uses Nix. It lets us share a binary cache of modified/own packages between the developers, CI, and our production machines. If you push something to CI that needs some new dependencies, the CI machine will build them and when you want to get a local dev environment or deploy, the binaries will be fetched from the CI machine. We can also send the binaries between machines to save compile time. This is possible because we have confidence that it doesn't matter which machine built them because the environment is so strictly controlled.
Foxboron 6y ago
Confusingly enough, when NixOS talks about reproducible builds they are not talking about https://reproducible-builds.org/ https://reproducible-builds.org/. They are talking about reproducible systems. NixOS doesn't solve reproducible builds as-in "bit-for-bit identical binary distributed files", but it does solve "same input" -> "functionally same system". This is the core idea of NixOS and what makes the package manager and the spawned ecosystem (like Guix) quite neat. I also think this isn't any more controversial then say having your infrastructure reproducible with terraform and ansible/salt. It gives you machines as cattle, and not pets, with the added flexibility that gives you. NixOS is just a complete package where in other systems there are several components accomplishing the goal. As to why reproducible builds is important is to ensure a strong connection between the upstream source code and the distributed binary build. This gives you confidence that everything going into the build has been declared and that the binary can be reproduced bit-for-bit identical if you so wish. This can be both important in a supply chain process. I recently saw tailscale provided pre-compiled binaries without signatures. What does those binaries contain? They are not signed, so the releases could very well be unauthenticated and someone could have compromised the server distributing the binaries. So I tried reproducing the tailscale binaries without much success as the build process is proprietary. https://github.com/tailscale/tailscale/issues/779 https://github.com/tailscale/tailscale/issues/779 So what does the tailscale binaries contain? I can't get a bit-for-bit identical binary without putting into a great deal of effort so for all intents and purposes they can very well be proprietary. Now I don't think the tailscale people are malicious, nor that the binaries have been replaced. But it's a real-world example I recently went through.
- solarkraft 6y ago
  Even when the binaries are signed you'd still have to trust Tailscale they didn't modify the code before compilation, maybe putting in a simple back door. They probably don't, but you can't really know. When you have binary reproducible builds such a thing would immediately be obvious and hell would break loose if they attempted it.
- Ericson2314 6y ago
  The floating content-addressed derivations we've worked on as part of this Nix × IPFS work get's us a lot closer to reproducible builds by making non-determinstic builds far more noticiable. See these: https://github.com/NixOS/rfcs/pull/62 https://github.com/NixOS/rfcs/pull/62 https://github.com/NixOS/nix/issues/4087 https://github.com/NixOS/nix/issues/4087
zelly 6y ago
There are much more real-world use cases for reproducible builds than for IPFS. It saves time and money. It's not just academic. Reproducible builds (as in bit-for-bit) means I can trust a (secure) checksum of a binary I downloaded somewhere, let's say from IPFS. This is defense-in-depth, adding another layer of security to prevent sabotage. Reproducibility means the whole system can reliably be built from source and I don't need to manually make sure the system has the correct constellation of implicit dependencies and their version numbers. This saves time and money in deployment and will pay off in a very big way 10 years down the line when that code is still in use somewhere.

arianvanp 6y ago

Does IPFS actually solve the problem they set out here though? IPFS is a distributed CDN; but not very good for storing things persistently or reliably from my experience. At the moment; the nixos cache is stored in very durable and reliable S3 storage; with very high durability guarantees. Why is that not good enough? sure it's centralised. But IPFS doesn't offer distributed durability; it offers a CDN. It doesn't seem to address this issue the authors seem to claim it solves. (To me) I still think it's _super cool_. And once builds are also content-addressed and not just fixed-output derivations; having trustless content delivery is a valuable addition. But IPFS doesnt' seem like a robust answer for the "what if we lose access to the source code problem" given it's not a durable storage system In this case; projects like https://www.softwareheritage.org/ https://www.softwareheritage.org/ and https://sfconservancy.org/ https://sfconservancy.org/ seem better bets to solve the source code access issue

the_duke 6y ago
I imagine you would ensure the persistence of the data you care about by either running your own IPFS nodes that pin the data, or by using a pinning service like Pinata [1]. [1] https://pinata.cloud/ https://pinata.cloud/
- TheOtherWW 6y ago
  The problem with Pinata is that they charge $150 per TB for their Individual plan. That's nearly 6.5x the cost of storing the data on S3. Sure it works, but that high barrier to entry pricing scares off a lot of people. Why not just use S3? Meanwhile, we're still waiting for Filecoin to launch, and networks such as Sia have seized that opportunity and created great things like Skynet [1]. Skynet itself still has some overhead if you want to ensure data persistence and availability, but the cost is orders of magnitude lower. In addition, new layer 2 providers have emerged to address those gaps, such as Filebase [2]. They provide S3 compatible object storage that is backed by decentralized cloud storage. You get high availability (of the storage layer), geo-redundancy, and less than S3 pricing out of the box. It is this type of offering where we are going to see the most impact and adoption as the underlying technology not only makes things more efficient, but cheaper too. [1] https://siasky.net/ https://siasky.net/ [2] https://filebase.com/ https://filebase.com/

imhoguy 6y ago

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and seeds it.

Avamander 6y ago
I don't get why machines running the same Linux distribution couldn't actually do the same. BT2 would indeed work very well for this.
- e12e 6y ago
  Nixos/guix might be a good fit for rootfs on ipfs? Comes with a local cache, and you could overlay local changes (eg: /etc/passwd)?
  - takeda 6y ago
    Actually not exactly rootfs on ipfs, but with NixOS is fairly easy to make it wipe data on reboot, here's an example article[1]. The idea is to only keep state we care about and rebuild the system on every boot. [1] https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings
solarkraft 6y ago
> I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution 2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.
- whyrusleeping 6y ago
  Some links without comment: - https://www.npmjs.com/package/ipfs-npm https://www.npmjs.com/package/ipfs-npm - https://github.com/ipfs-shipyard/apt-on-ipfs https://github.com/ipfs-shipyard/apt-on-ipfs - https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain...

iElectric2 6y ago

See also https://cachix.org https://cachix.org for a hosted binary cache.

majewsky 6y ago

Is it down for everyone or just me? Firefox is showing PR_CONNECT_ABORTED_ERROR when trying to navigate to the link.

Shared404 6y ago
It works for me, but I'll drop this just to be safe: https://outline.com/vWwGas https://outline.com/vWwGas

ArchD 6y ago

IPFS is not censorship resistant. If people start using it to subvert tyranny, politically neutral things like Nix's use of IPFS can get collaterally damaged in its infancy by state power.

jFriedensreich 6y ago
Its slightly more censorship resistant than S3 but yes, that seems to not have been on the list of problems they try to solve. On the other hand a list of all the systems that solve the problems in a half decent performant manner and are also censorship resistant would be 0 to my knowledge.
PureParadigm 6y ago
I think censorship resistance should be handled at a different abstraction layer. IPFS can focus on peer-to-peer immutable file sharing and some other tool can tunnel your traffic to avoid censorship. You could run IPFS over a VPN or something like that.

vander_elst 6y ago

Does anyone know if there are authz features planned for ipfs? Like having acls based on certificates or something...

e12e 6y ago
As far as I'm aware the "official" ipfs line is ipfs host and share data - if you want to protect it; encrypt it.

ingenieroariel 6y ago

This is a big deal, congratulations on the milestone. Really looking forward to trying out Reflex FRP / Obelisk from IPFS, in particular the focus on making the source available because fetching Nix packages from semi-public caches and not being able to rebuild them is something that has bothered me for a while.

alexmingoia 6y ago

One thing I’ve always wondered, is how DHTs like BitTorrent and IPFS prevent Sybil attacks. If anyone wanted to block a file in IPFS, could they generate node ids close to the filehash and return empty peer lists?

whyrusleeping 6y ago
Yeah, sybill attacks are pretty hard to get around. IPFS itself doesnt entirely depend on a DHT though, it uses it when it can, but also falls back to a more gossip style approach of just asking peers you connect to for the data. So if you wait long enough, and the content is out there, you're very likely to end up finding it as your node makes and receives random connections throughout the network. This does still mean that the sybill attacker can severely degrade the quality of service though, so other solutions are getting looked into. The 'easiest' one that comes to mind is just forming an incentivized DHT of some kind, with the simplest start to that being a requirement that all DHT server peers stake some funds. There are a lot of weird things in this problem space, and its still pretty young IMO

adsjhdashkj 6y ago

Tangent, what's the best way to use Nix-like features in a mainstream distro? Eg, i've been evaluating PopOS recently because i want a Mac equivalent OS. Ie, i don't want driver/config issues. However one problem i never see solved from any "normal" distros is reproducible systems. Hypothetically if i used Nix as a Desktop my config would be bulletproof. .. but then i'm going through a fair amount of work configuring everything when, as established, i want none of it. So i (as a user) seem to want some middleground between no effort installations / configurations of my Desktop, with reproducible snapshots/states/configs/something. You can use Nixpkgs on other distros/OSs, so maybe that is enough. But is there a better way?

MarcScott 6y ago
I just bit the bullet and use NixOS as my Desktop. There's a bit of work after initial install, to setup your config files the way you want, but after that you're all set. I've had no problems transferring my configs when I change hardware, and end up with an identical environment that I had on my previous computer.
- kevincox 6y ago
  I recently switched as well and am very happy. I don't think that re-configuring takes much longer that the average distro would especially because a lot of services you can just say `services.foobar.enable = true` and get a decent default config. You only need to spend your time configuring the things that you are picky about. And yes, the fact that I can configure all of my systems with a shared set of configs and have consistent and reproducible environments is fantastic. I wrote up my thoughts about the switch here: https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/ https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/
- adsjhdashkj 6y ago
  Sidenote, i'm taking the plunge into NixOS, but notably using Flakes. While i've dabbled in Nix in the past some things felt more odd to me than perhaps they should have. Primarily the fact that it felt like all these configs were spread out and i didn't understand how Nix wanted me to version them. Flakes (from early tests) seems to make this very clear. As it starts from a repo, so i'm attempting to make all my configs, including Home Manager (of which also feels weird lol) start from the Flakes installation. Flakes also solves the reproducibility problem that i didn't get why NixOS had. So far it's really neat.