5 ms·

Systems Programming at Twitter (2012)

57 points by lelf 13y ago

amirouche 13y ago
no a single word about the "data bus" they use. This would be better entitled something like "Scala crash course by a twitter system engineer"
- lumpypua 13y ago
  They don't use a global data bus AFAICT. It's all driven by rpc through finagle (i.e. thrift). Finding service endpoints is likely done through zookeeper or some equivalent.
  - deleted 13y ago
    [deleted]
  - alecthomas 13y ago
    That is correct. The service discovery is called ServerSets and is available here: https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/ServerSetImpl.java https://github.com/twitter/commons/blob/master/src/java/com/...
    - amirouche 13y ago
      I understand that there is finagle and also a service discovery backed by zookeeper. I fail to understand how twitter's data stays consistent without something like linkedin's databus. Doesn't twitter manage a single point of thruth or a single version of thruth and propagate changes to secondary databases like social graph & search indexes?
YZF 13y ago
Twitter seems like the poster child for Scala. I've played with Scala a little and it definitely has a lot of things going for it yet I wonder: - Is Scala used exclusively for systems programming in Twitter or are there other languages in the mix? - The majority of other big software players seem to be not using it for various reasons. Is that a concern? - How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)? - Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters? - Is it easier to hire people for Scala positions? Harder? The same? - How do engineers in Twitter with a lot of experience in other platforms and languages feel about Scala after having used it for a while? E.g. vs. C++, Go, Java, C# just to throw a few names into the hat there. Do people tend to just write C++ in Scala or Java in Scala or whatever or do they adopt to new paradigms with ease? I know we always say that a good engineer can pick up any new language and to some extent that's true but I think expertise in a specific language builds up over time. Maybe an analogy would be a musical instrument. It seems some of the other languages have larger pools of talent, more mature tools, etc. I guess it's hard to be a new language in this world...
- meowface 13y ago
  I've heard many complaints about Scala being too much of a "kitchen sink" language; too many paradigms and too many features from each paradigm. It still seems like the best language that uses the JVM, though.
  - RussianCow 13y ago
    Clojure seems to be picking up some steam, though it's a dynamic language and thus appeals to a different crowd.
  - danieldk 13y ago
    There's also Kotlin, which is largely a better Java. And Java 8 is an enormous improvement too, since it supports lambdas, default methods, streams, etc.
- yummyfajitas 13y ago
  Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters? Actually no - having a good concurrent garbage collector actually helps a lot. Consider any message passing system. Messages come into the system, you create an immutable object, and send a reference to that object to other subsystems. Eventually, when each subsystem has processed that object, it is garbage collected. In C++ you need to manually handle tracking the object and deleting it when it's no longer needed. This dramatically hinders the sort of systems you can reliably build. Further thoughts on this here: http://www.chrisstucchio.com/blog/2013/why_not_python.html http://www.chrisstucchio.com/blog/2013/why_not_python.html As for low level performance, the JVM is actually very fast - i.e., nearly as fast as C. The main places where it becomes an issue is that you can't always control the cache locality of objects. class Foo { val bar: Bar val y: Int } In C/C++ you can demand that foo.bar be stored in the same page. On the JVM you can't.
- virtualwhys 13y ago
  May I ask what you think Scala has going for it? All of your questions undermine rather than point out the strengths of the language and the platform on which it runs. > The majority of other big software players seem to be not using it for various reasons. Is that a concern? Twitter, LinkedIn, Sony, FourSquare, Tumblr, Amazon, UBS, NASA, The Guardian, etc., is that a concern? On the flipside it's not likely that Facebook uses F# as its primary language, and that Apple does not use Haskell; in neither case is this a cause for concern ;-) > How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)? So, your asking 2 questions in one: how long before new hires become productive in Scala, and how long before they develop mastery of the language. The former, very quickly, write Java without semicolons, and the latter, depends on the abilities of the new employee, could be in as little as 3 months to learn the ropes, corner cases, etc., but for average programmers I'd say at the very least 6 months, and more likely a year or more. The biggest hindrance in learning Scala, IMO, is not the language itself, but the build system, SBT. God help you, there are some hard yards to wade through at first, especially if you have trouble getting the build *.scala files recognized by the IDE (i.e. blind coding plain text files with no idea what <+= is vs. <<=, or % vs. %%, or that these types of symbolic method pointers are even available to begin with). > Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters? What? The JVM is a performance beast, perhaps if you drop down to straight C or go with C++ you'll get better performance, but you'll do so at a price (i.e. reduced maintainability and flexability). > Is it easier to hire people for Scala positions? Harder? The same? Compared to what? > Twitter seems like the poster child for Scala sounds like a backhanded complement...
readme 13y ago
I don't understand how this is about systems programming. Would someone care to clarify?
- dbrower 13y ago
  I take it to imply the 'system' they are programming is the distributed/concurrent complex they are running. To other people, 'system' is below that, and this is application programming, or "middleware" It seems like at any level, you write the system, and applications are above. -dB
  - seanmcdirmid 13y ago
    What else does the systems community do these days besides concurrent distributed systems?
    - weland 13y ago
      Power management and flash/SSD-aware filesystems are also quite the rage.
      - seanmcdirmid 13y ago
        Ya: distributed computing, storage, and power are hot. Where did all the kernel research go?
        weland 13y ago
        Much of it happens with different objectives, or at different levels, in the context of computing, storage and power management. See Akaros for example: http://akaros.cs.berkeley.edu/akaros-web/overview.php http://akaros.cs.berkeley.edu/akaros-web/overview.php .
- snarfy 13y ago
  I'm also confused. I have a book on my bookshelf titled "Systems Programming", and it's about writing kernels and runtimes in C. Can someone explain to me when the definition went from kernel level to server duct tape? I'm guessing it's about the same time the term 'full stack developer' started showing up.
noname123 13y ago
Hi, I'm currently in the researching & prototyping phase of what's the best and will-be-the-most-community supported distributed real time distributed framework based on the reactor pattern? So far, I've got: Queuing Framework (rolling your own streams): 1) ZeroMQ (language agnostic, so have to roll your own serialization and services, seems like Google Protocol Buffer is the best way to go) 2) RabbitMQ (relies on a central message broker, with lots of built-in bindings) Event-Based Frameworks: 2) Akka (JVM) 3) RxExtensions (.net) but have RxCpp and RxJava ports 4) Reactor (JVM) Stream-Based Frameworks 4) Twitter Storm (Java-ish, but suppose to be language agnostic?) 5) S4 (Java-ish, but suppose to have bindings?) Can any peeps help me out on your opinion what is the best or give me a better one? Currently, I'm having fun multiplexing sockets, dereferencing C pointers and seg-faulting, so any suggestion is welcome.
- noelwelsh 13y ago
  IF you're looking at RabbitMQ you should take a look at Kafka. Overall, I think looking for one framework to-rule-them-all is a mistake. Different components demand different tradeoffs.
- tegeek 13y ago
  You would like Storm http://storm.incubator.apache.org/ http://storm.incubator.apache.org/
- eweise 13y ago
  Check out Amazon Kinesis for messaging.
wging 13y ago
Is anyone else having trouble viewing this on a mobile device?
jeffreyrogers 13y ago
The content here is interesting, but I don't like the way the information is displayed. Most of the time when I read HN I skim the articles I'm interested since it's not worth my time to dive into all the details. This layout makes it pretty difficult to do that efficiently.

57 points by lelf 13y ago

no a single word about the "data bus" they use. This would be better entitled something like "Scala crash course by a twitter system engineer"

lumpypua 13y ago
They don't use a global data bus AFAICT. It's all driven by rpc through finagle (i.e. thrift). Finding service endpoints is likely done through zookeeper or some equivalent.
- deleted 13y ago
  [deleted]
- alecthomas 13y ago
  That is correct. The service discovery is called ServerSets and is available here: https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/ServerSetImpl.java https://github.com/twitter/commons/blob/master/src/java/com/...
  - amirouche 13y ago
    I understand that there is finagle and also a service discovery backed by zookeeper. I fail to understand how twitter's data stays consistent without something like linkedin's databus. Doesn't twitter manage a single point of thruth or a single version of thruth and propagate changes to secondary databases like social graph & search indexes?

YZF 13y ago

Twitter seems like the poster child for Scala. I've played with Scala a little and it definitely has a lot of things going for it yet I wonder: - Is Scala used exclusively for systems programming in Twitter or are there other languages in the mix? - The majority of other big software players seem to be not using it for various reasons. Is that a concern? - How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)? - Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters? - Is it easier to hire people for Scala positions? Harder? The same? - How do engineers in Twitter with a lot of experience in other platforms and languages feel about Scala after having used it for a while? E.g. vs. C++, Go, Java, C# just to throw a few names into the hat there. Do people tend to just write C++ in Scala or Java in Scala or whatever or do they adopt to new paradigms with ease? I know we always say that a good engineer can pick up any new language and to some extent that's true but I think expertise in a specific language builds up over time. Maybe an analogy would be a musical instrument. It seems some of the other languages have larger pools of talent, more mature tools, etc. I guess it's hard to be a new language in this world...

meowface 13y ago
I've heard many complaints about Scala being too much of a "kitchen sink" language; too many paradigms and too many features from each paradigm. It still seems like the best language that uses the JVM, though.
- RussianCow 13y ago
  Clojure seems to be picking up some steam, though it's a dynamic language and thus appeals to a different crowd.
- danieldk 13y ago
  There's also Kotlin, which is largely a better Java. And Java 8 is an enormous improvement too, since it supports lambdas, default methods, streams, etc.
yummyfajitas 13y ago
Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters? Actually no - having a good concurrent garbage collector actually helps a lot. Consider any message passing system. Messages come into the system, you create an immutable object, and send a reference to that object to other subsystems. Eventually, when each subsystem has processed that object, it is garbage collected. In C++ you need to manually handle tracking the object and deleting it when it's no longer needed. This dramatically hinders the sort of systems you can reliably build. Further thoughts on this here: http://www.chrisstucchio.com/blog/2013/why_not_python.html http://www.chrisstucchio.com/blog/2013/why_not_python.html As for low level performance, the JVM is actually very fast - i.e., nearly as fast as C. The main places where it becomes an issue is that you can't always control the cache locality of objects. class Foo { val bar: Bar val y: Int } In C/C++ you can demand that foo.bar be stored in the same page. On the JVM you can't.
virtualwhys 13y ago
May I ask what you think Scala has going for it? All of your questions undermine rather than point out the strengths of the language and the platform on which it runs. > The majority of other big software players seem to be not using it for various reasons. Is that a concern? Twitter, LinkedIn, Sony, FourSquare, Tumblr, Amazon, UBS, NASA, The Guardian, etc., is that a concern? On the flipside it's not likely that Facebook uses F# as its primary language, and that Apple does not use Haskell; in neither case is this a cause for concern ;-) > How is the learning curve for new hires? How long before they become productive? How long before they write great code in Scala (as opposed to still figuring things out but maybe being productive)? So, your asking 2 questions in one: how long before new hires become productive in Scala, and how long before they develop mastery of the language. The former, very quickly, write Java without semicolons, and the latter, depends on the abilities of the new employee, could be in as little as 3 months to learn the ropes, corner cases, etc., but for average programmers I'd say at the very least 6 months, and more likely a year or more. The biggest hindrance in learning Scala, IMO, is not the language itself, but the build system, SBT. God help you, there are some hard yards to wade through at first, especially if you have trouble getting the build *.scala files recognized by the IDE (i.e. blind coding plain text files with no idea what <+= is vs. <<=, or % vs. %%, or that these types of symbolic method pointers are even available to begin with). > Isn't the JVM too big of a performance compromise for large scale software where every bit of performance matters? What? The JVM is a performance beast, perhaps if you drop down to straight C or go with C++ you'll get better performance, but you'll do so at a price (i.e. reduced maintainability and flexability). > Is it easier to hire people for Scala positions? Harder? The same? Compared to what? > Twitter seems like the poster child for Scala sounds like a backhanded complement...

readme 13y ago

I don't understand how this is about systems programming. Would someone care to clarify?

dbrower 13y ago
I take it to imply the 'system' they are programming is the distributed/concurrent complex they are running. To other people, 'system' is below that, and this is application programming, or "middleware" It seems like at any level, you write the system, and applications are above. -dB
- seanmcdirmid 13y ago
  What else does the systems community do these days besides concurrent distributed systems?
  - weland 13y ago
    Power management and flash/SSD-aware filesystems are also quite the rage.
    - seanmcdirmid 13y ago
      Ya: distributed computing, storage, and power are hot. Where did all the kernel research go?
      - weland 13y ago
        Much of it happens with different objectives, or at different levels, in the context of computing, storage and power management. See Akaros for example: http://akaros.cs.berkeley.edu/akaros-web/overview.php http://akaros.cs.berkeley.edu/akaros-web/overview.php .
snarfy 13y ago
I'm also confused. I have a book on my bookshelf titled "Systems Programming", and it's about writing kernels and runtimes in C. Can someone explain to me when the definition went from kernel level to server duct tape? I'm guessing it's about the same time the term 'full stack developer' started showing up.

noname123 13y ago

Hi, I'm currently in the researching & prototyping phase of what's the best and will-be-the-most-community supported distributed real time distributed framework based on the reactor pattern? So far, I've got: Queuing Framework (rolling your own streams): 1) ZeroMQ (language agnostic, so have to roll your own serialization and services, seems like Google Protocol Buffer is the best way to go) 2) RabbitMQ (relies on a central message broker, with lots of built-in bindings) Event-Based Frameworks: 2) Akka (JVM) 3) RxExtensions (.net) but have RxCpp and RxJava ports 4) Reactor (JVM) Stream-Based Frameworks 4) Twitter Storm (Java-ish, but suppose to be language agnostic?) 5) S4 (Java-ish, but suppose to have bindings?) Can any peeps help me out on your opinion what is the best or give me a better one? Currently, I'm having fun multiplexing sockets, dereferencing C pointers and seg-faulting, so any suggestion is welcome.

noelwelsh 13y ago
IF you're looking at RabbitMQ you should take a look at Kafka. Overall, I think looking for one framework to-rule-them-all is a mistake. Different components demand different tradeoffs.
tegeek 13y ago
You would like Storm http://storm.incubator.apache.org/ http://storm.incubator.apache.org/
eweise 13y ago
Check out Amazon Kinesis for messaging.

wging 13y ago

Is anyone else having trouble viewing this on a mobile device?

jeffreyrogers 13y ago

The content here is interesting, but I don't like the way the information is displayed. Most of the time when I read HN I skim the articles I'm interested since it's not worth my time to dive into all the details. This layout makes it pretty difficult to do that efficiently.