13 ms·

IPFS and Their Gateways

140 points by JoachimS 4y ago

xd1936 4y ago
This is silly. You could say the same for files CURLd from servers over http today. Did you know that when you download files from the internet, you're putting yourself at risk of the server tampering with your file before serving it to you??
- remram 4y ago
  The server giving you the wrong file is not "tampering". The word refers to something in the middle making changes. For the web, TLS provides tamper-resistance. Accessing the web through a HTTP proxy would allow tampering, similarly to accessing IPFS through a gateway.
- sneak 4y ago
  http urls are not content-addressed. ipfs urls are. a better analogy than http urls would be git commit hashes.
  - IggleSniggle 4y ago
    Then like with git, couldn't you cryptographically sign ipfs urls to have a public-trust guarantee of provenance? Either GPG signed or perhaps a CA?
- jstanley 4y ago
  But the point of IPFS is to do better than this. The point of IPFS is that the server can't tamper with the content because it would break the hash, but if you're using an untrusted proxy then it is just as bad as using an untrusted proxy for your normal web browsing.
neilalexander 4y ago
> the gateway is in full control and can inspect and tamper with the data as much as it likes Inspect, yes, but tamper? IPFS content IDs are hashes, so it seems like it should be possible to verify what you have been sent if you really cared enough to do so.
- danpalmer 4y ago
  Possible, yes, but probably the most widely used IPFS implementation – the Brave browser – does not implement this yet. They say they plan to in the future.
  - cowtools 4y ago
    I would think that the browser plugin + daemon is the most common implementation.
- georgyo 4y ago
  IPFS content IDs are hashes of the DAG of the content. That DAG chucks up the data and includes it directly in itself. There isn't a tool today that can verify an IPFS address without actually downloading that file's DAG, which is basically downloading the entire file.
  - dark-star 4y ago
    That's what he said: you can veify the file's hash after you downloaded it, to make sure it wasn't tampered with...
    - sneak 4y ago
      The merkle root of the DAG over the chunks is not related to the file's hash. Imagine if I gave you a file out of a git repo, and the long commit hash. It's insufficient on its own to verify integrity, you need the intermediate nodes. With ipfs you can infer that it used the reference chunker and try to reproduce the DAG but there is no guarantee it will work. Better to just speak ipfs to the p2p net and get the DAG.
      - georgyo 4y ago
        Exactly, and it is even worse than git. Git the Merkel root is a list of directories and hashs of whole files. Ipfs, an individual file can be made up of many trees and those trees can have trees. Example: https://explore.ipld.io/#/explore/QmbGtJg23skhvFmu9mJiePVByhfzu5rwo74MEkVDYAmF5T https://explore.ipld.io/#/explore/QmbGtJg23skhvFmu9mJiePVByh... That is a single file, but good luck verify that file without actually downloading the entire dag tree which includes the file. Worse, that same file can be chucked and bucketed infinitely number of ways. So if I download that file and the re-add it to IPFS, I will very likely get a _different_ address. And that address will likely share _none_ of the of the nodes from the previous address. Meaning even though the same file is in IPFS, they are completely distinct and don't actually help seed each other.
teddyh 4y ago
The same argument basically applies to DoH (DNS over HTTPS).
- georgyo 4y ago
  The argument is that the user must choose their gateway, not that https gateways are bad in general. With DoH generally must both opt in and must choose your provider.
  - teddyh 4y ago
    I’ve seen a lot of defaults to use Google or Cloudflare as resolvers.
    - snvzz 4y ago
      I do unbound -> tor -> cloudflare.
      - LinuxBender 4y ago
        I've never tried that. Would you be willing to post the extended unbound stats, just the median group of latency timing? Here is one of my unbound nodes that does DoT to other DoT nodes I control that then do more DoT to other resolvers. I use a cron job to pre-cache most things I use so the latency isn't a big deal. I'm just curious what yours looks like. If you don't have it set to go to logs then it can be pulled with "unbound-control stats_noreset|grep histo" 0.004096 0.008192 1 0.008192 0.016384 3 0.016384 0.032768 2 0.032768 0.065536 121 0.065536 0.131072 156 0.131072 0.262144 205 0.262144 0.524288 304 0.524288 1.000000 98 1.000000 2.000000 141 2.000000 4.000000 24
- jeroenhd 4y ago
  With ODoH and DNSSEC this risk is mostly mitigated. It's a shame DNSSEC is often missing client implementation (and that the .com sphere barely implements it at all) but the risks have been mitigated pretty well in my opinion.
janandonly 4y ago
So basically CURL will get support for IPFS by using a proxy service (which is bad for user privacy). And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy. Brave browser [1] at least has a native support for IPFS and no need for proxies at all. > Other IPFS privacy concerns: Brave, the browser, has actual native IPFS support (not using any gateway) and they have an informative page called How does IPFS Impact my Privacy? [1] https://brave.com/brave-integrates-ipfs/ https://brave.com/brave-integrates-ipfs/
- danpalmer 4y ago
  This blog post about cURL is saying that they will not do this, and will require a manual configuration of a gateway. > By default, Brave will load the URI being requested via a public HTTP gateway; however, it will also show an infobar that asks you if you’d like to use a local node to resolve IPFS URIs. So Brave by-default uses a public gateway.
- nl 4y ago
  > And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy. It literally says: I have insisted, in the PR for ipfs support to curl, that the IPFS URL handling code should not automatically follow such gateway redirects, as I believe that adds even more risk to the user so if a user wants to allow this operation, it should be opt-in.
- bananskalhalk 4y ago
  Max redirs in curl defaults to 50. Most browsers default to around 20. 30X (in the article) is a range of status response codes. You would probably benefit by rereading the article.
londons_explore 4y ago
IPFS isn't fit for purpose. It needs censorship-resistance added. It needs user privacy added. Both of those could be enabled by combining data blocks semi-randomly. Ie. rather than downloading the data you're looking for, you download a block of data containing what you're looking for and some other data. Then nobody knows what it was you were looking at. Some combined blocks might contain a few bytes of data from thousands of files. For censorship resistance, rather than users pinning individual files, the network needs to assign them blocks they must store (or be kicked off the network). Some of those blocks will be combined blocks which contain a few bytes some users may not wish to store - and that's fine, it means IPFS isn't for them. Remember that data can be encrypted by content hash, so there is no way to know what you're storing unless you go scouring the public internet for a key to decrypt it.
- sneak 4y ago
  ipfs is a transport, not a storage system. you don't store things "on ipfs" any more than you store things "on http". http and tcp, other transport systems, also do not have censorship resistance. it's the wrong place for it. you can use ipfs to build a censorship resistant system, just like you can with tcp, but that is the purview of a higher layer.
  - londons_explore 4y ago
    A new version of IPFS would be both transport system and storage. I don't think it's possible to reasonably to maintain layering while adding privacy and censorship resistance.
    - vorpalhex 4y ago
      This is sort of being upset at your mop because it makes a bad vacuum. These goals are not directly part of ipfs.
- hickimsedenolan 4y ago
  Or you could just use Freenet, thus not re-inventing the wheel.
ilaksh 4y ago
It almost sounds like part of his concern is that the web server could view the content you are requesting. But this is always true for IPFS data since it's public and available to anyone with the CID. Also you can use the CID (which is a hash), to verify the content hasn't been tampered with.
- zaik 4y ago
  If I understand correctly, IPFS will even re-advertise blocks you have downloaded on the DHT. So per default your whole browsing history is public on the IPFS network.
  - lijogdfljk 4y ago
    Yea, IPFS has a horrible issue with representing what it .. is. People, even in this thread, keep spouting privacy.. when i'm not sure if i've ever seen that as a goal. If anything content addressability makes privacy remarkably difficult.
Nemo157 4y ago
A long time ago Cloudflare was experimenting with support for E2E integrity via their public gateway: https://blog.cloudflare.com/e2e-integrity/ https://blog.cloudflare.com/e2e-integrity/ It looks like they've taken the FF addon down and archived the source repo though.
waynenilsen 4y ago
This article made a false statement > the gateway is in full control and can inspect and tamper with the data as much as it likes. And there is no way for the client to know or detect if it is happening. Regarding data tampering; IPFS uses content based addressing, the point is that the client _can_ know by computing a message digest of the incoming data. The verification is fast and cheap.
- danpalmer 4y ago
  > The verification is fast and cheap. And yet the most widely used IPFS client, Brave, does not implement this. The blog post is technically wrong, but I think it's worth accounting for the general state of the ecosystem in these decisions, and in whether to include IPFS in cURL, and it sounds like the general state of the ecosystem is not up to the standards that cURL should reasonably expect from a protocol asking for inclusion.
  - jstanley 4y ago
    Brave uses its own local gateway running go-ipfs. go-ipfs verifies the content, so Brave does not have to. You only have to verify the content yourself if it is not being verified by someone you trust (i.e. a local gateway that you're running yourself).
    - danpalmer 4y ago
      That's not what their documentation says. It says that it uses a public gateway, and prompts the user asking if they would like to use a local gateway, at which point it will download go-ipfs and run that. That's a fairly smooth UX to get a local gateway running, but I wouldn't bet money on it being used by the majority. Most people take the path of least resistance, and it's up to projects like Brave or cURL to ensure that the path is safe, which in this case it is not for Brave.
- Nemo157 4y ago
  The hash you have is computed from the hash of the content and other data. You would need to send that additional data out-of-band to allow the client to compute the overall hash and verify it. (Not to mention with urls like ipfs://<hash>/some/path you only have the hash of some arbitrary parent node, so there's even more additional data necessary to be able to verify that the content at that path under the hash you have is valid).
kordlessagain 4y ago
pastel-mature-herring~> What are the issues listed in these comments about IPFS? victorious-capuchin|> The main issues seem to be that IPFS is not yet up to the standards that cURL should reasonably expect from a protocol asking for inclusion. In particular, there are concerns about the lack of censorship resistance and user privacy.
seaourfreed 4y ago
The HERO will be when IPFS CLIs/Libraries have the hash verification included. CLI & Library open source devs, please be our heros. We want IPFS and security (both)
- ranger_danger 4y ago
  The _real_ hero will be when locating a random IPFS file doesn't take 10+ minutes.
trumpablehump 4y ago
Great to see the proposed fix in ffmpeg. Using a remote gateway should really not be a default but something the user has to consciously and deliberately configure.
sascha_sl 4y ago
I will never run an IPFS gateway as long as there are no robust legal protections against someone provoking a downloads (and with IPFS, subsequent uploads) of content that's illegal to have on your hard drive (a universal example is CSAM). And believe it or not - occasionally such content is used as a weapon. A Twitch streamer with an open Dropbox learned the hard way. It's used regularly on _that_ kind of imageboard to derail threads. I know because a naive version of me briefly ran an archive of a supposedly worksafe board about video games a decade ago and had their door knocked down at gunpoint about a year later. I don't know how the Fediverse still exists, considering that most implementations are very eager to mirror all media they encounter. I know HTTP can provoke the same caching, but for that there's actually much more robust legal precedent.
deleted 4y ago
[deleted]

140 points by JoachimS 4y ago

xd1936 4y ago

This is silly. You could say the same for files CURLd from servers over http today. Did you know that when you download files from the internet, you're putting yourself at risk of the server tampering with your file before serving it to you??

remram 4y ago
The server giving you the wrong file is not "tampering". The word refers to something in the middle making changes. For the web, TLS provides tamper-resistance. Accessing the web through a HTTP proxy would allow tampering, similarly to accessing IPFS through a gateway.
sneak 4y ago
http urls are not content-addressed. ipfs urls are. a better analogy than http urls would be git commit hashes.
- IggleSniggle 4y ago
  Then like with git, couldn't you cryptographically sign ipfs urls to have a public-trust guarantee of provenance? Either GPG signed or perhaps a CA?
jstanley 4y ago
But the point of IPFS is to do better than this. The point of IPFS is that the server can't tamper with the content because it would break the hash, but if you're using an untrusted proxy then it is just as bad as using an untrusted proxy for your normal web browsing.

neilalexander 4y ago

> the gateway is in full control and can inspect and tamper with the data as much as it likes Inspect, yes, but tamper? IPFS content IDs are hashes, so it seems like it should be possible to verify what you have been sent if you really cared enough to do so.

danpalmer 4y ago
Possible, yes, but probably the most widely used IPFS implementation – the Brave browser – does not implement this yet. They say they plan to in the future.
- cowtools 4y ago
  I would think that the browser plugin + daemon is the most common implementation.
georgyo 4y ago
IPFS content IDs are hashes of the DAG of the content. That DAG chucks up the data and includes it directly in itself. There isn't a tool today that can verify an IPFS address without actually downloading that file's DAG, which is basically downloading the entire file.
- dark-star 4y ago
  That's what he said: you can veify the file's hash after you downloaded it, to make sure it wasn't tampered with...
  - sneak 4y ago
    The merkle root of the DAG over the chunks is not related to the file's hash. Imagine if I gave you a file out of a git repo, and the long commit hash. It's insufficient on its own to verify integrity, you need the intermediate nodes. With ipfs you can infer that it used the reference chunker and try to reproduce the DAG but there is no guarantee it will work. Better to just speak ipfs to the p2p net and get the DAG.
    - georgyo 4y ago
      Exactly, and it is even worse than git. Git the Merkel root is a list of directories and hashs of whole files. Ipfs, an individual file can be made up of many trees and those trees can have trees. Example: https://explore.ipld.io/#/explore/QmbGtJg23skhvFmu9mJiePVByhfzu5rwo74MEkVDYAmF5T https://explore.ipld.io/#/explore/QmbGtJg23skhvFmu9mJiePVByh... That is a single file, but good luck verify that file without actually downloading the entire dag tree which includes the file. Worse, that same file can be chucked and bucketed infinitely number of ways. So if I download that file and the re-add it to IPFS, I will very likely get a _different_ address. And that address will likely share _none_ of the of the nodes from the previous address. Meaning even though the same file is in IPFS, they are completely distinct and don't actually help seed each other.

teddyh 4y ago

The same argument basically applies to DoH (DNS over HTTPS).

georgyo 4y ago
The argument is that the user must choose their gateway, not that https gateways are bad in general. With DoH generally must both opt in and must choose your provider.
- teddyh 4y ago
  I’ve seen a lot of defaults to use Google or Cloudflare as resolvers.
  - snvzz 4y ago
    I do unbound -> tor -> cloudflare.
    - LinuxBender 4y ago
      I've never tried that. Would you be willing to post the extended unbound stats, just the median group of latency timing? Here is one of my unbound nodes that does DoT to other DoT nodes I control that then do more DoT to other resolvers. I use a cron job to pre-cache most things I use so the latency isn't a big deal. I'm just curious what yours looks like. If you don't have it set to go to logs then it can be pulled with "unbound-control stats_noreset|grep histo" 0.004096 0.008192 1 0.008192 0.016384 3 0.016384 0.032768 2 0.032768 0.065536 121 0.065536 0.131072 156 0.131072 0.262144 205 0.262144 0.524288 304 0.524288 1.000000 98 1.000000 2.000000 141 2.000000 4.000000 24
jeroenhd 4y ago
With ODoH and DNSSEC this risk is mostly mitigated. It's a shame DNSSEC is often missing client implementation (and that the .com sphere barely implements it at all) but the risks have been mitigated pretty well in my opinion.

janandonly 4y ago

So basically CURL will get support for IPFS by using a proxy service (which is bad for user privacy). And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy. Brave browser [1] at least has a native support for IPFS and no need for proxies at all. > Other IPFS privacy concerns: Brave, the browser, has actual native IPFS support (not using any gateway) and they have an informative page called How does IPFS Impact my Privacy? [1] https://brave.com/brave-integrates-ipfs/ https://brave.com/brave-integrates-ipfs/

danpalmer 4y ago
This blog post about cURL is saying that they will not do this, and will require a manual configuration of a gateway. > By default, Brave will load the URI being requested via a public HTTP gateway; however, it will also show an infobar that asks you if you’d like to use a local node to resolve IPFS URIs. So Brave by-default uses a public gateway.
nl 4y ago
> And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy. It literally says: I have insisted, in the PR for ipfs support to curl, that the IPFS URL handling code should not automatically follow such gateway redirects, as I believe that adds even more risk to the user so if a user wants to allow this operation, it should be opt-in.
bananskalhalk 4y ago
Max redirs in curl defaults to 50. Most browsers default to around 20. 30X (in the article) is a range of status response codes. You would probably benefit by rereading the article.

londons_explore 4y ago

IPFS isn't fit for purpose. It needs censorship-resistance added. It needs user privacy added. Both of those could be enabled by combining data blocks semi-randomly. Ie. rather than downloading the data you're looking for, you download a block of data containing what you're looking for and some other data. Then nobody knows what it was you were looking at. Some combined blocks might contain a few bytes of data from thousands of files. For censorship resistance, rather than users pinning individual files, the network needs to assign them blocks they must store (or be kicked off the network). Some of those blocks will be combined blocks which contain a few bytes some users may not wish to store - and that's fine, it means IPFS isn't for them. Remember that data can be encrypted by content hash, so there is no way to know what you're storing unless you go scouring the public internet for a key to decrypt it.

sneak 4y ago
ipfs is a transport, not a storage system. you don't store things "on ipfs" any more than you store things "on http". http and tcp, other transport systems, also do not have censorship resistance. it's the wrong place for it. you can use ipfs to build a censorship resistant system, just like you can with tcp, but that is the purview of a higher layer.
- londons_explore 4y ago
  A new version of IPFS would be both transport system and storage. I don't think it's possible to reasonably to maintain layering while adding privacy and censorship resistance.
  - vorpalhex 4y ago
    This is sort of being upset at your mop because it makes a bad vacuum. These goals are not directly part of ipfs.
hickimsedenolan 4y ago
Or you could just use Freenet, thus not re-inventing the wheel.

ilaksh 4y ago

It almost sounds like part of his concern is that the web server could view the content you are requesting. But this is always true for IPFS data since it's public and available to anyone with the CID. Also you can use the CID (which is a hash), to verify the content hasn't been tampered with.

zaik 4y ago
If I understand correctly, IPFS will even re-advertise blocks you have downloaded on the DHT. So per default your whole browsing history is public on the IPFS network.
- lijogdfljk 4y ago
  Yea, IPFS has a horrible issue with representing what it .. is. People, even in this thread, keep spouting privacy.. when i'm not sure if i've ever seen that as a goal. If anything content addressability makes privacy remarkably difficult.

Nemo157 4y ago

A long time ago Cloudflare was experimenting with support for E2E integrity via their public gateway: https://blog.cloudflare.com/e2e-integrity/ https://blog.cloudflare.com/e2e-integrity/ It looks like they've taken the FF addon down and archived the source repo though.

waynenilsen 4y ago

This article made a false statement > the gateway is in full control and can inspect and tamper with the data as much as it likes. And there is no way for the client to know or detect if it is happening. Regarding data tampering; IPFS uses content based addressing, the point is that the client _can_ know by computing a message digest of the incoming data. The verification is fast and cheap.

danpalmer 4y ago
> The verification is fast and cheap. And yet the most widely used IPFS client, Brave, does not implement this. The blog post is technically wrong, but I think it's worth accounting for the general state of the ecosystem in these decisions, and in whether to include IPFS in cURL, and it sounds like the general state of the ecosystem is not up to the standards that cURL should reasonably expect from a protocol asking for inclusion.
- jstanley 4y ago
  Brave uses its own local gateway running go-ipfs. go-ipfs verifies the content, so Brave does not have to. You only have to verify the content yourself if it is not being verified by someone you trust (i.e. a local gateway that you're running yourself).
  - danpalmer 4y ago
    That's not what their documentation says. It says that it uses a public gateway, and prompts the user asking if they would like to use a local gateway, at which point it will download go-ipfs and run that. That's a fairly smooth UX to get a local gateway running, but I wouldn't bet money on it being used by the majority. Most people take the path of least resistance, and it's up to projects like Brave or cURL to ensure that the path is safe, which in this case it is not for Brave.
Nemo157 4y ago
The hash you have is computed from the hash of the content and other data. You would need to send that additional data out-of-band to allow the client to compute the overall hash and verify it. (Not to mention with urls like ipfs://<hash>/some/path you only have the hash of some arbitrary parent node, so there's even more additional data necessary to be able to verify that the content at that path under the hash you have is valid).

kordlessagain 4y ago

pastel-mature-herring~> What are the issues listed in these comments about IPFS? victorious-capuchin|> The main issues seem to be that IPFS is not yet up to the standards that cURL should reasonably expect from a protocol asking for inclusion. In particular, there are concerns about the lack of censorship resistance and user privacy.

seaourfreed 4y ago

The HERO will be when IPFS CLIs/Libraries have the hash verification included. CLI & Library open source devs, please be our heros. We want IPFS and security (both)

ranger_danger 4y ago
The _real_ hero will be when locating a random IPFS file doesn't take 10+ minutes.

trumpablehump 4y ago

Great to see the proposed fix in ffmpeg. Using a remote gateway should really not be a default but something the user has to consciously and deliberately configure.

sascha_sl 4y ago

I will never run an IPFS gateway as long as there are no robust legal protections against someone provoking a downloads (and with IPFS, subsequent uploads) of content that's illegal to have on your hard drive (a universal example is CSAM). And believe it or not - occasionally such content is used as a weapon. A Twitch streamer with an open Dropbox learned the hard way. It's used regularly on _that_ kind of imageboard to derail threads. I know because a naive version of me briefly ran an archive of a supposedly worksafe board about video games a decade ago and had their door knocked down at gunpoint about a year later. I don't know how the Fediverse still exists, considering that most implementations are very eager to mirror all media they encounter. I know HTTP can provoke the same caching, but for that there's actually much more robust legal precedent.

deleted 4y ago

[deleted]