5 ms·
Hashlookup.circl.lu API – hash lookup of known files
- kfarr 5y agoCurious if IPFS file hash is compatible with this list?
- atilimcetin 5y agoMost probably no. Instead of calculating the hash of whole file, IPFS chunks a file, calculates the hash of each chunk and turns them into a DAG of hashes (Merkle DAG). Also because there are multiple ways to construct the Merkle DAG, it's possible that same file can have different hashes.
- kfarr 5y agoThanks appreciate the info!
- datenarsch 5y agoisn't that just VirusTotal only without the convenience of having the files virus checked?
- atatatat 5y agolol, if-then checks for maliciousness
- CodesInChaos 5y agoWhy use hashes which aren't collision resistant for the lookup, instead of say SHA-256?
- pseudoramble 5y agoA simple practical reason is that the volume of known files that has MD5 or SHA1 hashes is huge compared to SHA256. It's preferable to have something collision-resistant, but also if your goal is to simply be able to identify files, and MD5 or SHA1 checksum can be fine. That will change over time, but realistically tons of AV scanners and tools like this need to support MD5 or SHA1. Of course, if you're generating new files that you know about or checking existing known good files, one would include the SHA256 and SHA512 with it.
- 9dev 5y agoIsn’t that something the `Accept` header would be perfect for?
- pseudoramble 5y agoSorry, I don't follow what you mean. The accept header can specify mimetypes and negotiate what the response can look like. So maybe you could stretch it to say "I'm only accepting SHA256" somehow. That might improve the API, but then you're indicating to the API to return 404's more often since it won't be able to find files with that hash. Maybe there are other things you could do instead though? I'm not sure.
- 9dev 5y agoWell, I think you could definitely stretch it that way, because the Accept header is absolutely intended for situations like this: It lets you specify precisely which content formats you're willing to accept, and in which particular order. Accept: text/html, application/xhtml+xml, application/xml;q=0.9, image/webp, */*;q=0.8 No reason why that couldn't read like this instead: Accept: application/vnd.hash+sha256, application/vnd.hash+sha1;q=0.9, application/vnd.hash+md5;q=0.8 That would indicate to the server that you'd like to receive (preferably) SHA256, with a fallback to SHA1, and another fallback to MD5, if none of the other two is available. If the server isn't able to provide any of those, it returns a "406 Not Acceptable" response. HTTP covers all of this out of the box, people just tend to overlook the more advanced stuff.
- jnwatson 5y agoSHA256 is the current standard for this use case. Please use and encourage the use of that.
- pseudoramble 5y agoFair enough! I over-spoke in my previous comment (I don't seem to able to edit it anymore). Thanks for pointing that out. Using SHA256 or greater is what should be used for generating new hashes, especially if you can update old entries and files with SHA256's. Many databases still have way more MD5/SHA1 hashes which won't be updated to include the SHA256, which is why you'll see them be used often.
- kzrdude 5y agoWhich files will be in the database?
- ickyforce 5y ago> the API is also described as an OpenAPI. The response formats are not described so not very useful.