5 ms·

TL;DR: use URI instead. URL is behaving this way due to backwards compatibility. --- To quote a JDK bug ticket: https://bugs.java.com/bugdatabase/view_bug.do?

by yzmtf2008 7y ago

TL;DR: use URI instead. URL is behaving this way due to backwards compatibility. --- To quote a JDK bug ticket: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4434494 https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4434494 > We are well aware of the problem with URL.equals and URL.hashCode. The cause of the problem is due to the existing spec and implementation, where we will try to compare the two URLs by resolving the host IP addresses, instead of just doing a string comparison. Because hashCode has to maintain certain relationships with equals, namely if two objects are equal, they should have the same hashCode, the implementation of hashCode also tries to resolve the host in the URL into an IP address. As a result, we are facing problems with http virtual hosting, as described in the Description part, and performance hit due to DNS name resolutions. > Unfortunately, changing the behavior now would break backward compatibility in a serious way, plus Java Security mechanism depends on it in some parts of the implementation. We can't change it now. > However, to address URI parsing in general, we introduced a new class called URI in Merlin (jdk1.4). People are encouraged to use URI for parsing and URI comparison, and leave URL class for accessing the URI itself, getting at the protocol handler, interacting with the protocol etc. So, at present, we don't plan on changing the URL.equals/hashCode behavior and we will leave the bug open until Tiger, when we re-investigate our options.

tuwtuwtuwtuw 7y ago
Out of curiosity do you get a compiler warning when you use URL.equals?
- ivan_gammel 7y ago
  No and probably it will never happen in javac. Some IDEs may try to do static code analysis to identify possible uses of URL.equals (e.g. Map<? extends URL, ?>), but I doubt anyone would invest in this feature.
  - matsemann 7y ago
    Looks like IntelliJ has it: - equals() or hashCode() called on java.net.URL object - Map or Set may contain java.net.URL objects https://www.jetbrains.com/help/idea/list-of-java-inspections.html https://www.jetbrains.com/help/idea/list-of-java-inspections...
    - bspammer 7y ago
      Doesn't look like it's turned on by default though, which seems strange. Presumably if you care enough to go into the settings to enable that specific warning you're unlikely to make the mistake in the first place. It's exactly the type of non-obvious gotcha that would be nice to have your IDE warn you about out of the box.
      - ShinTakuya 7y ago
        I agree and disagree. I mean, the thing is that outside of personal projects it's not uncommon to inherit a codebase from other people, sometimes quite a large codebase. Such a warning can be useful when eliminating boo-boos (acting as an lint I guess).
        tuwtuwtuwtuw 7y ago
        Wouldn't you want warnings when an inherited code base does weird things with possible incorrect behaviors as side effect?
        ShinTakuya 7y ago
        Well I would consider DNS resolution on comparisons/hashing to be an incorrect behaviour as a side effect.
        ivan_gammel 7y ago
        In this case Sonar will do a better job. IDE warnings are good for fixing them immediately - when they accumulate it’s very easy to stop paying any attention.
jefftk 7y ago
I see why this was useful, though it still wasn't a good idea. This looks like it was added in ~1995, at which point (a) it was common for multiple hostnames to refer to the same server and (b) they were guaranteed to act identically because there was no HOST: header yet.
- Slartie 7y ago
  Your b) was actually only ever true for the HTTP protocol, but URLs are not strictly limited to that protocol. They are not limited to any finite set of protocols at all. You could easily make up your own protocol which differentiates between host names in a similar way as modern HTTP does. And you could have made that in 1995.
  - jefftk 7y ago
    You could have made one in 1995, but looking over https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml https://www.iana.org/assignments/protocol-numbers/protocol-n... I think HTTP/1.1 was the first protocol to send the host name like this. (Though I could be missing something, since that's a lot of protocols.)
- inetknght 7y ago
  Your 'a' case is still common.
  - edw 7y ago
    Perversely, a well-timed DNS record update and cache expiry could make a URL not-equal to another URL made from the same string of characters.
    - throwawaymath 7y ago
      That's the essence of a DNS rebinding attack, which can be used to bypass server-side request forgery vulnerability mitigations.
hinkley 7y ago
When I was a Java developer ~8 years ago, "Use java.net.URI" was a whole sentence in my vernacular. Not just as a replacement for URL, but also for all of the dumb string concatenation people were doing to build urls and query parameters. It was already ancient advice then, but I kept finding myself having to teach it to a new group of people, usually after some incident had happened (because they won't listen until something has already broken)
- bradgessler 7y ago
  Yes, I’m constantly giving advice to Ruby and JS developers that “URLs are not strings” and to use URI libraries for manipulation. This is also true for phone numbers, emails, IPs, etc. Hell, even Rails maintainers rejected a PR I opened that tried to get rails link helpers to work with URI objects.
Quekid5 7y ago
I'm really curious as to why they don't make it, say, protected where the JVM wouldn't actually barf at runtime (or would it? I'm a bit too enmeshed in Scala bincompat at the moment), but where compiling from source would error out. If I'm wrong: Why not invent a 'protection level' where this were a feasible thing... even if just to be able to remove old APIs and really force people to stop just adding @SuppressWarnings-deprecation all over the place. That's not a solution to anything.
cryptonector 7y ago
This is nonsense. The spec is wrong. There is no chance I would consider two URLs with equal local parts and query parameters, but different authorities resolving to the same addresses as equal -- that would be WRONG. The authority could easily provide different semantics for the same URL based on differing Host: values.
- romwell 7y ago
  They know this is nonsense and that the spec is wrong. It's there for backwards compatibility, which is not nonsense. It keeps the old code running.
  - cryptonector 7y ago
    Sometimes you need to break backwards compatibility for semantics. This is one of those cases.
chopin 7y ago
I wish that was part of the Javadoc. I just looked up the Javadoc for jdk8 and URL, URL.equals and URL.hashCode and I would have no idea that it does DNS requests. I am a very seasoned Java dev (since 2001) and I am sure I read at one time about this but forgot. But I do use URL from time to time though and I am pretty sure to have used it with Map or Set somewhere. A clear warning at top of URL, URL(String) would be good to make it recognizable in most IDEs when using it.
jillesvangurp 7y ago
You should also use a gradle or maven plugin called forbidden-apis to prevent people using deprecated stuff like this. I've been adding this to pretty much any Java/Kotlin project for years with some custom patterns to also prevent people using shadowed packages that some libraries expose. In this case, for forbidding java.net.URL.equals is probably a good idea. Might already be in the default patterns. Also sounds like something a good static code analyzer should be able to detect.

tuwtuwtuwtuw 7y ago

Out of curiosity do you get a compiler warning when you use URL.equals?

ivan_gammel 7y ago
No and probably it will never happen in javac. Some IDEs may try to do static code analysis to identify possible uses of URL.equals (e.g. Map<? extends URL, ?>), but I doubt anyone would invest in this feature.
- matsemann 7y ago
  Looks like IntelliJ has it: - equals() or hashCode() called on java.net.URL object - Map or Set may contain java.net.URL objects https://www.jetbrains.com/help/idea/list-of-java-inspections.html https://www.jetbrains.com/help/idea/list-of-java-inspections...
  - bspammer 7y ago
    Doesn't look like it's turned on by default though, which seems strange. Presumably if you care enough to go into the settings to enable that specific warning you're unlikely to make the mistake in the first place. It's exactly the type of non-obvious gotcha that would be nice to have your IDE warn you about out of the box.
    - ShinTakuya 7y ago
      I agree and disagree. I mean, the thing is that outside of personal projects it's not uncommon to inherit a codebase from other people, sometimes quite a large codebase. Such a warning can be useful when eliminating boo-boos (acting as an lint I guess).
      - tuwtuwtuwtuw 7y ago
        Wouldn't you want warnings when an inherited code base does weird things with possible incorrect behaviors as side effect?
        ShinTakuya 7y ago
        Well I would consider DNS resolution on comparisons/hashing to be an incorrect behaviour as a side effect.
      - ivan_gammel 7y ago
        In this case Sonar will do a better job. IDE warnings are good for fixing them immediately - when they accumulate it’s very easy to stop paying any attention.

jefftk 7y ago

I see why this was useful, though it still wasn't a good idea. This looks like it was added in ~1995, at which point (a) it was common for multiple hostnames to refer to the same server and (b) they were guaranteed to act identically because there was no HOST: header yet.

Slartie 7y ago
Your b) was actually only ever true for the HTTP protocol, but URLs are not strictly limited to that protocol. They are not limited to any finite set of protocols at all. You could easily make up your own protocol which differentiates between host names in a similar way as modern HTTP does. And you could have made that in 1995.
- jefftk 7y ago
  You could have made one in 1995, but looking over https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml https://www.iana.org/assignments/protocol-numbers/protocol-n... I think HTTP/1.1 was the first protocol to send the host name like this. (Though I could be missing something, since that's a lot of protocols.)
inetknght 7y ago
Your 'a' case is still common.
- edw 7y ago
  Perversely, a well-timed DNS record update and cache expiry could make a URL not-equal to another URL made from the same string of characters.
  - throwawaymath 7y ago
    That's the essence of a DNS rebinding attack, which can be used to bypass server-side request forgery vulnerability mitigations.

hinkley 7y ago

When I was a Java developer ~8 years ago, "Use java.net.URI" was a whole sentence in my vernacular. Not just as a replacement for URL, but also for all of the dumb string concatenation people were doing to build urls and query parameters. It was already ancient advice then, but I kept finding myself having to teach it to a new group of people, usually after some incident had happened (because they won't listen until something has already broken)

bradgessler 7y ago
Yes, I’m constantly giving advice to Ruby and JS developers that “URLs are not strings” and to use URI libraries for manipulation. This is also true for phone numbers, emails, IPs, etc. Hell, even Rails maintainers rejected a PR I opened that tried to get rails link helpers to work with URI objects.

Quekid5 7y ago

I'm really curious as to why they don't make it, say, protected where the JVM wouldn't actually barf at runtime (or would it? I'm a bit too enmeshed in Scala bincompat at the moment), but where compiling from source would error out. If I'm wrong: Why not invent a 'protection level' where this were a feasible thing... even if just to be able to remove old APIs and really force people to stop just adding @SuppressWarnings-deprecation all over the place. That's not a solution to anything.

cryptonector 7y ago

This is nonsense. The spec is wrong. There is no chance I would consider two URLs with equal local parts and query parameters, but different authorities resolving to the same addresses as equal -- that would be WRONG. The authority could easily provide different semantics for the same URL based on differing Host: values.

romwell 7y ago
They know this is nonsense and that the spec is wrong. It's there for backwards compatibility, which is not nonsense. It keeps the old code running.
- cryptonector 7y ago
  Sometimes you need to break backwards compatibility for semantics. This is one of those cases.

chopin 7y ago

I wish that was part of the Javadoc. I just looked up the Javadoc for jdk8 and URL, URL.equals and URL.hashCode and I would have no idea that it does DNS requests. I am a very seasoned Java dev (since 2001) and I am sure I read at one time about this but forgot. But I do use URL from time to time though and I am pretty sure to have used it with Map or Set somewhere. A clear warning at top of URL, URL(String) would be good to make it recognizable in most IDEs when using it.

jillesvangurp 7y ago

You should also use a gradle or maven plugin called forbidden-apis to prevent people using deprecated stuff like this. I've been adding this to pretty much any Java/Kotlin project for years with some custom patterns to also prevent people using shadowed packages that some libraries expose. In this case, for forbidding java.net.URL.equals is probably a good idea. Might already be in the default patterns. Also sounds like something a good static code analyzer should be able to detect.