13 ms·
Finally, an instance made for Java!
by vegancap 10y ago
Finally, an instance made for Java!
- granos 10y agoI dislike developing in Java. I am not a fanboy by any stretch of the imagination. That being said, someone who takes the time to understand how the JVM works and how to configure their processes with a proper operator's mindset can do amazing things in terms of resource usage. It's easy to poke at Java for being a hog when in reality its just poor coding and operating practices that lead to bloated runtime behavior.
- yongjik 10y ago> ...can do amazing things in terms of resource usage. Sorry, but you just made my day. :P
- placeybordeaux 10y agoFor a long time I wondered if it was a failing of the language or the culture. After spending 4 days trying to diagnose a problem with hbase given the two errors "No region found" and "No table provided" and finally figuring out it was due to a version mismatch I now believe it is the culture. At the very least you should be printing a WARN when you connect to an incompatible version.
- electrum 10y agoBad code is bad code, no matter what language it is written in.
- sk5t 10y agoBad logging is not endemic to the Java development mindset... in fact I'd give Java the nod for the _option_ to generate tremendously large quantities of log data when requested, above almost any other platform, excepting maybe Windows if you figure out how to turn on ETL. If you want bad logging, look at most PHP projects... For really thoughtful logging, the Apache HTTP client and HikariCP are good Java examples.
- breischl 10y agoDefinitely the culture. It seems to be a Java axiom that you should never use one object when eight will do. I've never seen another community that would actually have a builder for the configuration for the factory for the settings for a class.
- mpd 10y agoI assume you've never had the pleasure of configuring a Sendmail installation?
- ddalex 10y agoActually configuring sendmail is a piececake these days thanks to Google and countless engineers documenting in excruciating detail the problems they faced.
- granos 10y agoThis is over a decade old now, but is still very appropriate http://discuss.joelonsoftware.com/?joel.3.219431.12 http://discuss.joelonsoftware.com/?joel.3.219431.12
- Kristine1975 10y agoSo much this. Back in 2001 I used IntelliJ IDEA on a PC with 128MB of RAM. It worked perfectly, and it was the first IDE I used that checked my code while I was writing it. The much less evolved JBuilder on the other hand stopped every couple seconds for garbage collection. Both were written in Java. And don't get me started on Forte (developed by Sun itself, no less). It was even slower and more memory-hungry than JBuilder.
- abraae 10y agoI love Java. We shifted from c++ a year after it arrived on the scene. Since then, I've never needed to learn a new language in any depth. To me, that's a good thing and shows the longevity of the language.
- sievebrain 10y agoYou jest, but think about how unbelievably painful it'd be to write a program that uses >1TB of RAM in C++ .... any bug that causes a segfault, div by zero, or really any kind of crash at all would mean you'd have to reload the entire dataset into RAM from scratch. That's gonna take a while no matter what. You could work around it by using shared memory regions and the like but then you're doing a lot of extra work. With a managed language and a bit of care around exception handling, you can write code that's pretty much invincible without much effort because you can't corrupt things arbitrarily. Also, depending on the dataset in question you might find that things shrink. The latest HotSpots can deduplicate strings in memory as they garbage collect. If your dataset has a lot of repeated strings then you effectively get an interning scheme for free. I don't know if G1 can really work well with over 1TB of heap, though. I've only ever heard of it going up to a few hundred gigabytes.
- Kristine1975 10y ago>With a managed language and a bit of care around exception handling, you can write code that's pretty much invincible without much effort because you can't corrupt things arbitrarily. The JVM has crashed on me in the past (as in hard crash, not a Java exception). Less often than the C++ programs I write do? Yes, but I of course I wouldn't test a program on a 1TB dataset before ironing out all the kinks. >The latest HotSpots can deduplicate strings in memory as they garbage collect Obviously when working with huge datasets I would implement some kind of string deduplication myself. Most likely even a special string class an memory allocation scheme optimized for write-once, read-many access and cache friendliness. Or I would use memory mapping for the input file and let the OS's virtual memory management sort it out.
- tosseraccount 10y agoUse shared memory.
- 0xfaded 10y agommap is not "a lot of extra work".
- 10y ago
- scaleout1 10y agoWhen you suddenly realize that your "big" data is not really that big!. Who needs a Hadoop/Spark cluster when you can run one of these bad boys
- tracker1 10y agoThat was kind of my thought as well... I worked on a small-mid sized classifieds site (about 10-12 unique visitors a month on average) and even then the core dataset was about 8-10GB, with some log-like data hitting around 4-5GB/month. This is freakishly huge. I don't know enough about different platforms to even digest how well you can even utilize that much memory. Though it would be a first to genuinely have way more hardware than you'll likely ever need for something. IIRC, the images for the site were closer to 7-8TB, but I don't know how typical that is for other types of sites, and caching every image on the site in memory is pretty impractical... just the same... damn.
- samstave 10y agoHeh, but I wonder what the default per account limits are on launching these... prolly (1) per account.
- saosebastiao 10y agoAll I can think about is the 30 minute garbage collection pauses.
- tracker1 10y agoI know that you are probably going to be modded into oblivion, but can Java address this much memory in a single application? I'm genuinely curious, as I would assume, depending on the OS that you'd have to run several (many) processes in order to even address that much ram effectively. Still really cool to see something like this, I didn't even know you could get close to 2TB of ram in a single server at any kind of scale.
- wmfiv 10y agoNot using the out of the box solutions. But while I haven't done this personally my understanding is Azul Zing will allow you to efficiently use multi TB heaps in Java.
- astral303 10y agoJava can address 32GB heaps with compressedoops flag enabled. After that flag is off, you can address as much as 64 bits will allow. http://stackoverflow.com/questions/2093679/max-memory-for-64bit-java http://stackoverflow.com/questions/2093679/max-memory-for-64... Do a little research before implying that there's no way that Java can address gigantic heaps.
- tracker1 10y agoI wasn't implying or assuming anything, I was genuinely asking... I'm more familiar with windows, than other OSes, but iirc, windows apps can only get 4GB per process. (Maybe that was just 32bit windows apps).
- tracker1 10y agoI was incorrect... Then again, I've never needed to address more than a couple gb of ram. [1] http://stackoverflow.com/a/11892191/43906 http://stackoverflow.com/a/11892191/43906
- Kristine1975 10y agoJust 32bit apps. 64bit apps can go much higher.
- stcredzero 10y agoActually, as far as VMs go, the JVM is fairly spare in comparison with earlier versions of Ruby and Python -- on a per object basis. (Because of its Smalltalk roots. Yes, I had to get that in there. Drink!) That said, I've seen those horrors of cargo-cult imitation of the Gang of Four patterns, resulting in my having to instantiate 7 freaking objects to send one JMS message. If practice in recent decades has taught us anything, it's that performance is found in intelligently using the cache. In a multi-core concurrent world, our tools should be biased towards pass by value, allocation on the stack/avoiding allocating on the heap, and avoiding chasing pointers and branching just to facilitate code organization. EDIT: Or, as placybordeaux puts it more succinctly in a nephew comment, "VM or culture? It's the culture." EDIT: It just occurred to me -- Programming suffers from a worship of Context-Free "Clever"! Whether or not a particular pattern or decision is smart is highly dependent on context. (In the general sense, not the function call one.) The difficulty with programming, is that often context is very involved and hard to convey in media. As a result, a whole lot of arguments are made for or against patterns/paradigms/languages using largely context free examples. This is why we end up in so many meaningless arguments akin to, "What is the ultimate bladed weapon?" That's simply a meaningless question, because the effectiveness of such items is very highly dependent on context. (Look up Matt Easton on YouTube.) The analogy works in terms of the degree of fanboi nonsense.
- 0xmohit 10y agoand Scala too. Scala _beats_ Java in most of the benchmarks: http://benchmarksgame.alioth.debian.org/u64q/scala.html http://benchmarksgame.alioth.debian.org/u64q/scala.html
- igouy 10y ago> _beats_ Java Not according to that data!
- aaronkrolik 10y agoA small word of caution: I'd strongly recommend against using a huge java heap size. Java GC is stop the world, and a huge java heap size can lead to hour long gc sessions. It's much better to store data in a memory mapped file that is off heap, and access accordingly. Still very fast.
- Xorlev 10y agoGood advice. Even with G1GC it's hard to run heaps that large. However, not to be overly pedantic, Java GC has many different algorithms and many avoid STW collection for as long as possible and do concurrent collection until it's no longer possible. I don't think it's fair to just call it stop the world.
- jerven 10y agoAzul C4 will collect with stop the world due to GC no longer than 40ms. You might want to think beyond Oracle/OpenJDK (altough openjdk with Shendoah GC will be interesting in the future) if you are dealing with this kind of heap sizes.