4 ms·

What happened to hashcash proposals to email protocols?

by fiatjaf 10y ago

chriswarbo 10y ago
The problem with hashcash for email is that there's no way for the sender and receiver to negotiate the amount of work to be proved, so they must correctly guess how much to perform/require. The most sensible requirement to guess is zero, since all existing email is sent without any proof of work, and the value of interoperability is higher than blocking spam. After all, if you're willing to lose interoperability, you can block 100% of spam by just not using email at all! Note that it may be possible to have servers negotiate an amount of work to perform, but having someone else (the server) perform the work defeats the point of the system. The client must be told to do the work, but it's a core feature of email that it's not realtime: messages can be composed offline, and queued up by servers, so by the time a "needs more work" response is received, there's no way for the server to send a message back to the client.
- mikeash 10y ago
  Could the work requirement be advertised as part of the destination's MX record or something? I know that in theory you're supposed to be able to send the message to your upstream server and let it worry about how to deliver, but in practice it seems like these days you can count on being able to look up the target's DNS info. Another problem is finding a quantity of work which is high enough to stop spam, but low enough not to stop legitimate traffic. You want people to be able to send email from their RPiZero or original iPhone, but stop spammers who may have massive botnets, or at least decent PCs.
  - chriswarbo 10y ago
    I can't think of a way to make this work with email, but I do think a proof of work like hashcash would be useful for Web APIs instead of using API keys: - Give a 402 reponse if hashcash isn't included or isn't enough, with a link to a price URL - GETting a price URL may return different values depending on identifying information like IP address, user agent, etc. - Prices can be adjusted based on server load, whether we recognise this agent as malicious or benevolent, etc. - Different types of request can have different prices, e.g. GETting a specific resource could be cheaper than searching or performing some expensive computation. There would also need to be a mechanism to avoid replays, which would make things slightly less RESTful. I haven't thought of anything more useful here than an increasing request ID.
    - mikeash 10y ago
      I've done a crappy implementation for my blog to fight spammy comments. It works OK, probably more because it's a totally custom thing that isn't worth time for spammers to fight, rather than because it's actually effective. The way I did it is: - When the user focuses in a comment field, the page makes a request to the server asking for initial parameters. - The server returns the number of leading zeroes required, the number of distinct hashes it needs, and a salt to use. (This is just a constant in my code right now, but could be varied based on client specifics.) - The page then crunches on the work as the user types their comment. The submit button is disabled until it's complete. - The proof of work is submitted to the server along with the comment. The server then checks to see if it's good and accepts or rejects. A properly working client should never be rejected (since it fetches the required parameters in advance) so the rejection doesn't have to be too fancy. - Replays are prevented by storing the salts in a database, and deleting them once they're used for a comment. I changed the standard hashcash technique a bit by requiring the client to submit multiple distinct hashes. Requiring only one hash works fine, but results in a lot of variance in how long it takes to compute the proof of work. You might tune it for an average of 30 seconds, but a decent percentage of clients will get it in 1 second, or will take 60 seconds. By, for example, requiring 7 fewer leading zeroes but requiring 128 distinct hashes, you get the same average but with a lot less variance. You can also display a semi-accurate progress indicator this way. The downside, of course, is that you have to send more data and the server has to do some extra work to verify.
      - fiatjaf 10y ago
        Is that code public and in Python/Go/Javascript?
        mikeash 10y ago
        My web site is a hacked-together thing that's been gradually accumulating since the late 90s, so it's kind of ugly. The hashcash code is not very modular, either. If you'd like to see it anyway, I pulled out the relevant parts here: https://gist.github.com/mikeash/daf02b6cc3017560f930515c3c17af43 https://gist.github.com/mikeash/daf02b6cc3017560f930515c3c17... The comment-inline.js file is directly embedded in the HTML for the comments area, and is the glue code between the actual UI and the hashcash computation code. The hashcash.js file is where all the client-side work happens, and it handles the actual hashing, making multiple attempts, checking to see if an attempt produced a good result, and such. Then commentsubmit.py handles the server side by returning hashcash parameters when requested, and checking the provided hashcash for validity when submitting. I have a brief blog post about it here, which you can also use to see the system in action: https://www.mikeash.com/pyblog/testing-hashcash-based-anti-spam-measures.html https://www.mikeash.com/pyblog/testing-hashcash-based-anti-s... If you have any questions, comments, or criticisms, please feel free to get in touch.
  - chriswarbo 10y ago
    > Could the work requirement be advertised as part of the destination's MX record or something? The frustrating part is that only the good guys would follow such a scheme. As long as there's a fallback available, no matter how deprecated, spammers will use it to avoid such payment schemes. I suppose it could be used as a strong signal for spam filters, in addition to other security/authentication schemes which have been adopted. > Another problem is finding a quantity of work which is high enough to stop spam, but low enough not to stop legitimate traffic. Yes, this is another problem which negotiation could alleviate. These days there would also be the option of offloading the actual email sending to a service (either run on your own server, or "in the cloud"). Of course this just shifts the problem, but at least such services can provide arbitrary APIs, and hence can restrict their hashcash power to authenticated users, or require users to provide proof of work in a way which can be negotiated down, e.g. by building up a reputation over time. Not sure how botsnets can be tackled, but raising the amount of effort even a small amount would hopefully make a lot of spam-based scams unprofitable. I'm sure the problem would just shift then, e.g. to using the botnets for more DDoS attacks or other more lucrative schemes.
- inimino 10y ago
  Instead of hashcash why haven't we solved the problem by just attaching real money to every email? The convention would be that you "accept" the payment only if it is spam. $1 gets you into the inbox. If it isn't spam then after 48 hours the token just expires and the sender keeps their dollar. I assume the money transfer can be solved today using Bitcoin, and it wouldn't be a terribly complicated protocol.
  - chriswarbo 10y ago
    > $1 gets you into the inbox. This is exactly the same problem faced by hashcash, just wrapped up in more layers of complication. If emails require a $1 attachment to get into your inbox, you will receive no emails, since nobody is currently sending any emails with $1 attached. Just rewrite my above comment, but replace "hashcash" with "dollars". > I assume the money transfer can be solved today using Bitcoin, and it wouldn't be a terribly complicated protocol. You do realise that Bitcoin itself is a complicated protocol built on top of hashcash, right? Keypairs, blockchains, mining, etc. is just adding unnecessary complexity to this problem; not to mention your choice of the US dollar as the denomination, which requires an exchange rate, etc.
    - inimino 10y ago
      Hashcash is pure waste, this at least benefits the one whose attention is wasted by the spam. > You do realise that Bitcoin itself is a complicated protocol built on top of hashcash, right? Of course, but it already exists. > If emails require a $1 attachment to get into your inbox, you will receive no emails, since nobody is currently sending any emails with $1 attached. Yes, but assume there are people sending bulk email legitimately who have delivery issues. All you need is one major provider to accept it and then there is a benefit for senders. DKIM and SPF prove that the deployment problem can be overcome, even for relatively weak attacks on the spam problem.
bbcbasic 10y ago
The problem: My chromebook is competing with the spammer's ASIC. What's the appropriate difficulty level?

What happened to hashcash proposals to email protocols?

by fiatjaf 10y ago

What happened to hashcash proposals to email protocols?

The problem with hashcash for email is that there's no way for the sender and receiver to negotiate the amount of work to be proved, so they must correctly guess how much to perform/require. The most sensible requirement to guess is zero, since all existing email is sent without any proof of work, and the value of interoperability is higher than blocking spam. After all, if you're willing to lose interoperability, you can block 100% of spam by just not using email at all! Note that it may be possible to have servers negotiate an amount of work to perform, but having someone else (the server) perform the work defeats the point of the system. The client must be told to do the work, but it's a core feature of email that it's not realtime: messages can be composed offline, and queued up by servers, so by the time a "needs more work" response is received, there's no way for the server to send a message back to the client.

mikeash 10y ago
Could the work requirement be advertised as part of the destination's MX record or something? I know that in theory you're supposed to be able to send the message to your upstream server and let it worry about how to deliver, but in practice it seems like these days you can count on being able to look up the target's DNS info. Another problem is finding a quantity of work which is high enough to stop spam, but low enough not to stop legitimate traffic. You want people to be able to send email from their RPiZero or original iPhone, but stop spammers who may have massive botnets, or at least decent PCs.
- chriswarbo 10y ago
  I can't think of a way to make this work with email, but I do think a proof of work like hashcash would be useful for Web APIs instead of using API keys: - Give a 402 reponse if hashcash isn't included or isn't enough, with a link to a price URL - GETting a price URL may return different values depending on identifying information like IP address, user agent, etc. - Prices can be adjusted based on server load, whether we recognise this agent as malicious or benevolent, etc. - Different types of request can have different prices, e.g. GETting a specific resource could be cheaper than searching or performing some expensive computation. There would also need to be a mechanism to avoid replays, which would make things slightly less RESTful. I haven't thought of anything more useful here than an increasing request ID.
  - mikeash 10y ago
    I've done a crappy implementation for my blog to fight spammy comments. It works OK, probably more because it's a totally custom thing that isn't worth time for spammers to fight, rather than because it's actually effective. The way I did it is: - When the user focuses in a comment field, the page makes a request to the server asking for initial parameters. - The server returns the number of leading zeroes required, the number of distinct hashes it needs, and a salt to use. (This is just a constant in my code right now, but could be varied based on client specifics.) - The page then crunches on the work as the user types their comment. The submit button is disabled until it's complete. - The proof of work is submitted to the server along with the comment. The server then checks to see if it's good and accepts or rejects. A properly working client should never be rejected (since it fetches the required parameters in advance) so the rejection doesn't have to be too fancy. - Replays are prevented by storing the salts in a database, and deleting them once they're used for a comment. I changed the standard hashcash technique a bit by requiring the client to submit multiple distinct hashes. Requiring only one hash works fine, but results in a lot of variance in how long it takes to compute the proof of work. You might tune it for an average of 30 seconds, but a decent percentage of clients will get it in 1 second, or will take 60 seconds. By, for example, requiring 7 fewer leading zeroes but requiring 128 distinct hashes, you get the same average but with a lot less variance. You can also display a semi-accurate progress indicator this way. The downside, of course, is that you have to send more data and the server has to do some extra work to verify.
    - fiatjaf 10y ago
      Is that code public and in Python/Go/Javascript?
      - mikeash 10y ago
        My web site is a hacked-together thing that's been gradually accumulating since the late 90s, so it's kind of ugly. The hashcash code is not very modular, either. If you'd like to see it anyway, I pulled out the relevant parts here: https://gist.github.com/mikeash/daf02b6cc3017560f930515c3c17af43 https://gist.github.com/mikeash/daf02b6cc3017560f930515c3c17... The comment-inline.js file is directly embedded in the HTML for the comments area, and is the glue code between the actual UI and the hashcash computation code. The hashcash.js file is where all the client-side work happens, and it handles the actual hashing, making multiple attempts, checking to see if an attempt produced a good result, and such. Then commentsubmit.py handles the server side by returning hashcash parameters when requested, and checking the provided hashcash for validity when submitting. I have a brief blog post about it here, which you can also use to see the system in action: https://www.mikeash.com/pyblog/testing-hashcash-based-anti-spam-measures.html https://www.mikeash.com/pyblog/testing-hashcash-based-anti-s... If you have any questions, comments, or criticisms, please feel free to get in touch.
- chriswarbo 10y ago
  > Could the work requirement be advertised as part of the destination's MX record or something? The frustrating part is that only the good guys would follow such a scheme. As long as there's a fallback available, no matter how deprecated, spammers will use it to avoid such payment schemes. I suppose it could be used as a strong signal for spam filters, in addition to other security/authentication schemes which have been adopted. > Another problem is finding a quantity of work which is high enough to stop spam, but low enough not to stop legitimate traffic. Yes, this is another problem which negotiation could alleviate. These days there would also be the option of offloading the actual email sending to a service (either run on your own server, or "in the cloud"). Of course this just shifts the problem, but at least such services can provide arbitrary APIs, and hence can restrict their hashcash power to authenticated users, or require users to provide proof of work in a way which can be negotiated down, e.g. by building up a reputation over time. Not sure how botsnets can be tackled, but raising the amount of effort even a small amount would hopefully make a lot of spam-based scams unprofitable. I'm sure the problem would just shift then, e.g. to using the botnets for more DDoS attacks or other more lucrative schemes.
inimino 10y ago
Instead of hashcash why haven't we solved the problem by just attaching real money to every email? The convention would be that you "accept" the payment only if it is spam. $1 gets you into the inbox. If it isn't spam then after 48 hours the token just expires and the sender keeps their dollar. I assume the money transfer can be solved today using Bitcoin, and it wouldn't be a terribly complicated protocol.
- chriswarbo 10y ago
  > $1 gets you into the inbox. This is exactly the same problem faced by hashcash, just wrapped up in more layers of complication. If emails require a $1 attachment to get into your inbox, you will receive no emails, since nobody is currently sending any emails with $1 attached. Just rewrite my above comment, but replace "hashcash" with "dollars". > I assume the money transfer can be solved today using Bitcoin, and it wouldn't be a terribly complicated protocol. You do realise that Bitcoin itself is a complicated protocol built on top of hashcash, right? Keypairs, blockchains, mining, etc. is just adding unnecessary complexity to this problem; not to mention your choice of the US dollar as the denomination, which requires an exchange rate, etc.
  - inimino 10y ago
    Hashcash is pure waste, this at least benefits the one whose attention is wasted by the spam. > You do realise that Bitcoin itself is a complicated protocol built on top of hashcash, right? Of course, but it already exists. > If emails require a $1 attachment to get into your inbox, you will receive no emails, since nobody is currently sending any emails with $1 attached. Yes, but assume there are people sending bulk email legitimately who have delivery issues. All you need is one major provider to accept it and then there is a benefit for senders. DKIM and SPF prove that the deployment problem can be overcome, even for relatively weak attacks on the spam problem.

bbcbasic 10y ago

The problem: My chromebook is competing with the spammer's ASIC. What's the appropriate difficulty level?