10 ms·

Java.sun.com is down again - breaking bad apps across the land

97 points by zorlem 14y ago

This is a notice to fellow HN readers. This morning around 8:00 UTC Nagios alerted me, that the thread count on a number of Apache Tomcat servers I support for a client started to rise dramatically. I've discovered that a library that is part of the application was trying to fetch DTDs like http://java.sun.com/dtd/properties.dtd from java.sun.com. The servers are unreachable so each request thread was taking 30+ seconds to time out. I've worked around the problem by putting an iptables rule for 192.9.162.55 port 80, that would reject the request immediately and informed to client to look for a permanent solution. In case the library can't be fixed (I don't know who developed it), I'm planning on putting the relevant DTD files on a local HTTP server and redirect all requests for java.sun.com to that virtual host (eg. through relevant entries in the hosts file). Check your application servers and the software you use if it processes XML documents. One way to check: $ netstat -ant | grep -E '192.9.162.55:80[[:space:]]+SYN_SENT' | wc -l 184 $ SYN_SENT is because the 192.9.162.55 (java.sun.com) is not responding ATM. If it becomes reachable again, just s/[[:space::]]+SYN_SENT// . Relevant URLs for more information: http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic/ http://www.oasis-open.org/committees/entity/spec-2001-08-06.html And a previous discussion here about the same problem: http://news.ycombinator.com/item?id=3094075

rshm 14y ago
virtualbox.org is down as well.
- kia 14y ago
  For virtualbox.org it's planned maintenance from April 27th to April 30th. The announcement was on the main page.
  - eagsalazar 14y ago
    Planned downtime is handled like this? It isn't hard to put up a temporary page. Just taking it down, even with notice is really poor form.
    - bdunbar 14y ago
      At times like this I think of Lily Tomlin's Ernestine character: "We don't care. We don't have to. We're Oracle!" (I've been dealing with Oracle for a few years. It started with just database stuff, but they kept buying applications I supported, now they own Solaris ... anyway.)
      - re_todd 14y ago
        I got a job where we don't deal with Oracle at all, life is so much better! I'd recommend it to anyone. Eat your veggies, exercise regularly, and work in an Oracle-free workplace .... this is the secret to happiness!
        bdunbar 14y ago
        work in an Oracle-free workplace .... this is the secret to happiness! In Big Enterprise the alternative [1] to Oracle is Microsoft. You're darned if you do and darned if you don't. [1] Don't even mention open source. Not going to fly at BE, in my experience.
      - beedogs 14y ago
        When they bought Sun, the quality of Sun service dropped to the point where I can't imagine why anyone still buys their kit. It was remarkable.
        bdunbar 14y ago
        Sun support was remarkable. About five years ago I had a critical problem ... There I was in the data center at 3 a.m., trying to figure out why my mirrored drive server wasn't booting on it's surviving disk. I was groggy as heck, and even basic vi commands required a lot of thought. Actual thinking took more effort. The Support Engineer walked me through even the basic stuff "Okay, now 'yank-yank put' to copy that line" And a few minutes later the server booted and all was well. We're moving as quickly as possible away from Solaris, to Linux. But service quality isn't the driver - it never is. The problem is cost.
  - quink 14y ago
    Hang on... WTH??? The most popular virtualisation software out there that's full-featured and free to use... is shutting down their website for three days for planned maintenance? This is something that would have happened in 1993. Maybe. Between this and java.sun.com being offline it's pretty much the biggest red flag to stay away from Oracle as far as possible I could imagine.
    - llimllib 14y ago
      And the coursera compilers class is distributing its dev environment as a virtualbox image. I'm lucky I have a copy already.
      - harshreality 14y ago
        Downloads are still there. http://download.virtualbox.org/virtualbox/4.1.14/ http://download.virtualbox.org/virtualbox/4.1.14/
        HaakonKL 14y ago
        Worst case scenario, you'd get it from your package manager. There's likely a few downloads on torrent sites for the download as well. Since it's FLOSS it should be legal to grab it from torrents anyway.
        checkers 14y ago
        http://techiest.blogspot.com/ http://techiest.blogspot.com/
    - jsprinkles 14y ago
      Could be any number of things, operationally, and could also have a buffer built-in to the maintenance window to avoid unexpected issues. Take a breath, you have more important things to worry about.
ryandvm 14y ago
They're probably busy switching everything over to an Oracle stack...
blinkingled 14y ago
As a best practice applications should reference dtds from local filesystem. Most sane data centers would have outbound (App->Internet) access locked down - only needed hosts/ports are allowed after the application developer specifically requests for it.
- MBCook 14y ago
  At the least, the program could use a singleton to fetch and cache the DTDs. To just pull it over the internet every time you need it is, ignoring the practical problems, just flat out wasteful.
  - sixcorners 14y ago
    Does the DTD have the right headers set to allow clients to cache it?
    - zorlem 14y ago
      I'm not sure the situation with java.sun.com, but those provided by w3c do have a 90 days expiration (according to one of the links I've posted). In all cases, since the DTDs are more or less versioned through their filenames, with quite a minimal rate of changes, caching them (even if not outright saving them forever) should be the default action.
- neild 14y ago
  Sadly, if you use Python's batteries-included XML tools, this is virtually impossible to do. See http://bugs.python.org/issue2124 http://bugs.python.org/issue2124 for some discussion.
  - lucian1900 14y ago
    Those tools suck. lxml is better.
sxtxixtxcxh 14y ago
you can pass -c to grep to get a count, you don't need to pipe it to `wc`
- zorlem 14y ago
  thanks :) the "| wc -l" was tacked in for the submission :)
soc88 14y ago
I like how Oracle educates developers about the proper handling of DTD's. (They didn't break it by accident for the third time already, right? RIGHT?!)
- HaakonKL 14y ago
  Of course not. Oracle would never do anything bad to a developer community. To be honest though, if this have already happened TWICE do people really have any excuse for using a server that goes down a lot for something important? Why would you NOT just download the stuff on some separate server and at most run some cronjob to keep it up to date? Or am I just being stupid?
  - zorlem 14y ago
    See the link to the w3c - several times they've started delivering 503 HTTP error codes, hoping that the applications would start to break. It didn't have a big effect, either because the application didn't actually use the DTD they've retrieved for anything or because they broke in a non-obvious manner (like with the servers I'm administering). Had the outage been shorter or if I wasn't monitoring the Tomcat JVMs it could've stayed under the radar. That's one of the reasons I've made this submission.
ShabbyDoo 14y ago
Around 2005, I was semi-forced to use Xalan/Xerces (the Apache reference implementation of SAX, DOM, XPath, XSLT, etc.) for a project. These libraries were included in the JDK [edited from orig post] To make sure that these libraries did not attempt to talk servers outside my company's control, I had to dig through the code and implement "neutered" forms of schema look-up interfaces, etc. I can't recall exact details. The default behavior was promiscuity and presumption, and making sure that these libraries didn't strike-up conversations with random servers was not trivial or terribly well documented. So, I'm not surprised by the current state of affairs.
ShabbyDoo 14y ago
More generally, it ought to be easier to constrain apps running on the JVM to declared sandboxes. I once looked at the Java security model and found it to be totally inadequate for such purposes as it seemed to have been designed for ensuring that desktops could not be compromised by rogue applets running in browsers. Specifically, I was surprised by the coarse-grainedness of the security settings. Want to limit access to the network by disallowing use of the Socket class? Done. Want to only allow access to a whitelist of hosts? No dice. No filesystem access at all? Easy. Limit the app to only reading and/or writing to certain directories? Not a chance. I want to define whitelists for each environment in which my app will run -- development, QA, production, etc. To which hosts may it connect? Where may it access files? What else might I wish to constrain as way of avoiding inadvertent dependencies? Particular queues/topics on messaging buses? Database schemas within a particular server (network restrictions are too coarse for this)? When asking this question, I'm not trying to protect myself from rogue developers with malevolent intentions -- I just want to avoid a scenario like the one described by the OP. Recently, I started-up the Java app upon which I am currently working and watched its network behavior via Microsoft's Wireshark-esque network monitoring tool. It turns out that EHCache now asks one of Terracotta's servers for the most recent EHCache version number so that it can spit an out-of-date warning in the logs. Benign and useful, but I still had to spend a few minutes in the EHCache source to make sure that, if Terracotta's servers were down, our app would still start-up. Should one do this at the OS level (jails, perhaps)? I'm not limiting this idea to just Java apps, but I'm really only an expert in the Java space. I also argue that the whitelist would help codify inter-app dependencies in large IT environments. A few years ago, the large IT shop for which I worked did a disaster recovery drill where they literally deployed 10's of apps in an IBM-provided datacenter as a dry run. One thing they learned was that a particular production app was erroneously configured to log certain audit events to a server in a QA environment (which was not part of the disaster recovery plan for obvious reasons). Whitelists would have prevented this issue.
- tomjen3 14y ago
  Coun't you do this by writting your own security manager?
- zorlem 14y ago

97 points by zorlem 14y ago

rshm 14y ago

virtualbox.org is down as well.

kia 14y ago
For virtualbox.org it's planned maintenance from April 27th to April 30th. The announcement was on the main page.
- eagsalazar 14y ago
  Planned downtime is handled like this? It isn't hard to put up a temporary page. Just taking it down, even with notice is really poor form.
  - bdunbar 14y ago
    At times like this I think of Lily Tomlin's Ernestine character: "We don't care. We don't have to. We're Oracle!" (I've been dealing with Oracle for a few years. It started with just database stuff, but they kept buying applications I supported, now they own Solaris ... anyway.)
    - re_todd 14y ago
      I got a job where we don't deal with Oracle at all, life is so much better! I'd recommend it to anyone. Eat your veggies, exercise regularly, and work in an Oracle-free workplace .... this is the secret to happiness!
      - bdunbar 14y ago
        work in an Oracle-free workplace .... this is the secret to happiness! In Big Enterprise the alternative [1] to Oracle is Microsoft. You're darned if you do and darned if you don't. [1] Don't even mention open source. Not going to fly at BE, in my experience.
    - beedogs 14y ago
      When they bought Sun, the quality of Sun service dropped to the point where I can't imagine why anyone still buys their kit. It was remarkable.
      - bdunbar 14y ago
        Sun support was remarkable. About five years ago I had a critical problem ... There I was in the data center at 3 a.m., trying to figure out why my mirrored drive server wasn't booting on it's surviving disk. I was groggy as heck, and even basic vi commands required a lot of thought. Actual thinking took more effort. The Support Engineer walked me through even the basic stuff "Okay, now 'yank-yank put' to copy that line" And a few minutes later the server booted and all was well. We're moving as quickly as possible away from Solaris, to Linux. But service quality isn't the driver - it never is. The problem is cost.
- quink 14y ago
  Hang on... WTH??? The most popular virtualisation software out there that's full-featured and free to use... is shutting down their website for three days for planned maintenance? This is something that would have happened in 1993. Maybe. Between this and java.sun.com being offline it's pretty much the biggest red flag to stay away from Oracle as far as possible I could imagine.
  - llimllib 14y ago
    And the coursera compilers class is distributing its dev environment as a virtualbox image. I'm lucky I have a copy already.
    - harshreality 14y ago
      Downloads are still there. http://download.virtualbox.org/virtualbox/4.1.14/ http://download.virtualbox.org/virtualbox/4.1.14/
      - HaakonKL 14y ago
        Worst case scenario, you'd get it from your package manager. There's likely a few downloads on torrent sites for the download as well. Since it's FLOSS it should be legal to grab it from torrents anyway.
        checkers 14y ago
        http://techiest.blogspot.com/ http://techiest.blogspot.com/
  - jsprinkles 14y ago
    Could be any number of things, operationally, and could also have a buffer built-in to the maintenance window to avoid unexpected issues. Take a breath, you have more important things to worry about.

ryandvm 14y ago

They're probably busy switching everything over to an Oracle stack...

blinkingled 14y ago

As a best practice applications should reference dtds from local filesystem. Most sane data centers would have outbound (App->Internet) access locked down - only needed hosts/ports are allowed after the application developer specifically requests for it.

MBCook 14y ago
At the least, the program could use a singleton to fetch and cache the DTDs. To just pull it over the internet every time you need it is, ignoring the practical problems, just flat out wasteful.
- sixcorners 14y ago
  Does the DTD have the right headers set to allow clients to cache it?
  - zorlem 14y ago
    I'm not sure the situation with java.sun.com, but those provided by w3c do have a 90 days expiration (according to one of the links I've posted). In all cases, since the DTDs are more or less versioned through their filenames, with quite a minimal rate of changes, caching them (even if not outright saving them forever) should be the default action.
neild 14y ago
Sadly, if you use Python's batteries-included XML tools, this is virtually impossible to do. See http://bugs.python.org/issue2124 http://bugs.python.org/issue2124 for some discussion.
- lucian1900 14y ago
  Those tools suck. lxml is better.

sxtxixtxcxh 14y ago

you can pass -c to grep to get a count, you don't need to pipe it to `wc`

zorlem 14y ago
thanks :) the "| wc -l" was tacked in for the submission :)

soc88 14y ago

I like how Oracle educates developers about the proper handling of DTD's. (They didn't break it by accident for the third time already, right? RIGHT?!)

HaakonKL 14y ago
Of course not. Oracle would never do anything bad to a developer community. To be honest though, if this have already happened TWICE do people really have any excuse for using a server that goes down a lot for something important? Why would you NOT just download the stuff on some separate server and at most run some cronjob to keep it up to date? Or am I just being stupid?
- zorlem 14y ago
  See the link to the w3c - several times they've started delivering 503 HTTP error codes, hoping that the applications would start to break. It didn't have a big effect, either because the application didn't actually use the DTD they've retrieved for anything or because they broke in a non-obvious manner (like with the servers I'm administering). Had the outage been shorter or if I wasn't monitoring the Tomcat JVMs it could've stayed under the radar. That's one of the reasons I've made this submission.

ShabbyDoo 14y ago

Around 2005, I was semi-forced to use Xalan/Xerces (the Apache reference implementation of SAX, DOM, XPath, XSLT, etc.) for a project. These libraries were included in the JDK [edited from orig post] To make sure that these libraries did not attempt to talk servers outside my company's control, I had to dig through the code and implement "neutered" forms of schema look-up interfaces, etc. I can't recall exact details. The default behavior was promiscuity and presumption, and making sure that these libraries didn't strike-up conversations with random servers was not trivial or terribly well documented. So, I'm not surprised by the current state of affairs.

ShabbyDoo 14y ago

More generally, it ought to be easier to constrain apps running on the JVM to declared sandboxes. I once looked at the Java security model and found it to be totally inadequate for such purposes as it seemed to have been designed for ensuring that desktops could not be compromised by rogue applets running in browsers. Specifically, I was surprised by the coarse-grainedness of the security settings. Want to limit access to the network by disallowing use of the Socket class? Done. Want to only allow access to a whitelist of hosts? No dice. No filesystem access at all? Easy. Limit the app to only reading and/or writing to certain directories? Not a chance. I want to define whitelists for each environment in which my app will run -- development, QA, production, etc. To which hosts may it connect? Where may it access files? What else might I wish to constrain as way of avoiding inadvertent dependencies? Particular queues/topics on messaging buses? Database schemas within a particular server (network restrictions are too coarse for this)? When asking this question, I'm not trying to protect myself from rogue developers with malevolent intentions -- I just want to avoid a scenario like the one described by the OP. Recently, I started-up the Java app upon which I am currently working and watched its network behavior via Microsoft's Wireshark-esque network monitoring tool. It turns out that EHCache now asks one of Terracotta's servers for the most recent EHCache version number so that it can spit an out-of-date warning in the logs. Benign and useful, but I still had to spend a few minutes in the EHCache source to make sure that, if Terracotta's servers were down, our app would still start-up. Should one do this at the OS level (jails, perhaps)? I'm not limiting this idea to just Java apps, but I'm really only an expert in the Java space. I also argue that the whitelist would help codify inter-app dependencies in large IT environments. A few years ago, the large IT shop for which I worked did a disaster recovery drill where they literally deployed 10's of apps in an IBM-provided datacenter as a dry run. One thing they learned was that a particular production app was erroneously configured to log certain audit events to a server in a QA environment (which was not part of the disaster recovery plan for obvious reasons). Whitelists would have prevented this issue.

tomjen3 14y ago
Coun't you do this by writting your own security manager?
zorlem 14y ago