4 ms·
Scheduling jobs by distributing docker container? Looks really heavyweight. There is a good reason most of Map-Reduce frameworks are written on top of JVM - it
by rkwasny 12y ago
Scheduling jobs by distributing docker container? Looks really heavyweight.
There is a good reason most of Map-Reduce frameworks are written on top of JVM - it is VERY easy to serialize Java/Scala code ship it to remote host and execute there.
This is not trivial in compiled languages ( I believe some hacks for Haskell exists )
What we need in Big Data space is more people that understand how computers really work. Some effort must be put into supporting TCP/IP Stacks in Userspace using new 10GBe cards or support for IB verbs. Also the long forgotten art of writing zero-copy code.
- why-el 12y ago> Some effort must be put into supporting TCP/IP Stacks in Userspace using new 10GBe cards or support for IB verbs Obviously my understanding of networks is not at that level yet. What would a layman do to become somewhat vested in this? Any books or resrouces?
- chaotic-good 12y agoThis technique usually called 'kernel bypass'. http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html http://highscalability.com/blog/2013/5/13/the-secret-to-10-m... and http://lukego.github.io/blog/2013/01/04/kernel-bypass-networking/ http://lukego.github.io/blog/2013/01/04/kernel-bypass-networ...
- why-el 12y agoThank you. :)
- codygman 12y agoI wouldn't consider StaticPointers a hack. I believe this is the feature you are talking about.