4 ms·

An approach like https://github.com/benbjohnson/hashfs https://github.com/benbjohnson/hashfs allows file names to be updated at runtime to be content hashed. Th

by ajayvk 3y ago

An approach like https://github.com/benbjohnson/hashfs https://github.com/benbjohnson/hashfs allows file names to be updated at runtime to be content hashed. This removes the need for the extra "304 Not Modified" API calls from the client. This content hash based file renaming is usually done using a build step which renames files. For applications where the static file serving and HTTP request processing are done in the same application, this can be done in memory without a build step for file renames. I am using that approach in my project https://github.com/claceio/clace https://github.com/claceio/clace. It removes the need for a build step while making aggressive static file caching possible.

codetrotter 3y ago
I use content hashes for some of the images in parts of my site. And I use the IPFS scheme for it, and have the path be under /ipfs/ or some such. And so you could find the same file on IPFS if anyone served it there, as the content hash in the url tells you what to look for. Even though on my side I’ve done this all completely manually, so much so that it’s literally just me calculating the IPFS hash on my machine one time and then having symlinks with those content hashes so that /ipfs/ directory on my sites contains content that is served by their IPFS content hash, even though my server does not run an IPFS node or anything. A very interesting side effect of this is that one time I loaded one of my pages, the web browser actually picked up on the pattern and offered to load those files over actual IPFS!
- EGreg 3y ago
  Wait what? Browsers now offer to load over IPFS? I was wondering how you can trust an IFPS gateway, does does browser verify the file is legit using some checksum? Maybw subresource integrity supports IPFS content hashing or something? How does it generate cid anyway ? https://docs.ipfs.tech/concepts/content-addressing/#cids-are-not-file-hashes https://docs.ipfs.tech/concepts/content-addressing/#cids-are... How would you use SRI here to verify the cid (and not an additional out-of-band hash) make sure the gateway isn’t returning some crap to, say, inject malicious JS?
  - codetrotter 3y ago
    So in my case, I am not using SRI. But I am using the CID as the name in the path. Using the example file from your link, I would host it as something like https://mywebsite.example.com/ipfs/QmPK1s3pNYLi9ERiq3BDxKa4XosgWwFRQUydHUtz4YgpqB https://mywebsite.example.com/ipfs/QmPK1s3pNYLi9ERiq3BDxKa4X... And this was enough for that particular browser I was using to recognize that this file can be attempted to be retrieved directly from IPFS
    - EGreg 3y ago
      Yea but the gateway can be compromised, so that is insecure
      - codetrotter 3y ago
        Brave wrote in 2021 that they had plans to verify the CID. Not sure if they have added that or not yet. But either way, I run Brave with a local IPFS gateway on my laptop. So I think when it said that it could retrieve those files via IPFS for me, it meant in my case that those files would be retrieved from actual IPFS and not via a public IPFS gateway hosted on the web
infogulch 3y ago
I've also been dissatisfied with http caching not utilizing content hashes enough. If you're using server side templating one issue is that it's not efficient to calculate the hash while you're running the template, it would need to be precalculated to be efficient enough to use. So I wrote https://github.com/infogulch/xtemplate https://github.com/infogulch/xtemplate to scan all assets at startup to precalculate the hash for templates that use it, and if a request comes in with a query parameter ?hash=sha384-xyz and it matches then it gives it a 1 year immutable Cache-Control header automatically. If a file x.ext has a matching x.ext.gz/x.ext.zst/x.ext.br file then (after hashing the content to make sure it matches) client requests that support it are sent a compressed version streamed directly from disk with sendfile2. I call this "Optimal asset serving" (a bit bold perhaps).

by ajayvk 3y ago

codetrotter 3y ago

I use content hashes for some of the images in parts of my site. And I use the IPFS scheme for it, and have the path be under /ipfs/ or some such. And so you could find the same file on IPFS if anyone served it there, as the content hash in the url tells you what to look for. Even though on my side I’ve done this all completely manually, so much so that it’s literally just me calculating the IPFS hash on my machine one time and then having symlinks with those content hashes so that /ipfs/ directory on my sites contains content that is served by their IPFS content hash, even though my server does not run an IPFS node or anything. A very interesting side effect of this is that one time I loaded one of my pages, the web browser actually picked up on the pattern and offered to load those files over actual IPFS!

EGreg 3y ago
Wait what? Browsers now offer to load over IPFS? I was wondering how you can trust an IFPS gateway, does does browser verify the file is legit using some checksum? Maybw subresource integrity supports IPFS content hashing or something? How does it generate cid anyway ? https://docs.ipfs.tech/concepts/content-addressing/#cids-are-not-file-hashes https://docs.ipfs.tech/concepts/content-addressing/#cids-are... How would you use SRI here to verify the cid (and not an additional out-of-band hash) make sure the gateway isn’t returning some crap to, say, inject malicious JS?
- codetrotter 3y ago
  So in my case, I am not using SRI. But I am using the CID as the name in the path. Using the example file from your link, I would host it as something like https://mywebsite.example.com/ipfs/QmPK1s3pNYLi9ERiq3BDxKa4XosgWwFRQUydHUtz4YgpqB https://mywebsite.example.com/ipfs/QmPK1s3pNYLi9ERiq3BDxKa4X... And this was enough for that particular browser I was using to recognize that this file can be attempted to be retrieved directly from IPFS
  - EGreg 3y ago
    Yea but the gateway can be compromised, so that is insecure
    - codetrotter 3y ago
      Brave wrote in 2021 that they had plans to verify the CID. Not sure if they have added that or not yet. But either way, I run Brave with a local IPFS gateway on my laptop. So I think when it said that it could retrieve those files via IPFS for me, it meant in my case that those files would be retrieved from actual IPFS and not via a public IPFS gateway hosted on the web

infogulch 3y ago

I've also been dissatisfied with http caching not utilizing content hashes enough. If you're using server side templating one issue is that it's not efficient to calculate the hash while you're running the template, it would need to be precalculated to be efficient enough to use. So I wrote https://github.com/infogulch/xtemplate https://github.com/infogulch/xtemplate to scan all assets at startup to precalculate the hash for templates that use it, and if a request comes in with a query parameter ?hash=sha384-xyz and it matches then it gives it a 1 year immutable Cache-Control header automatically. If a file x.ext has a matching x.ext.gz/x.ext.zst/x.ext.br file then (after hashing the content to make sure it matches) client requests that support it are sent a compressed version streamed directly from disk with sendfile2. I call this "Optimal asset serving" (a bit bold perhaps).