8 ms·

Based on my current connections, and experience, I can't fully see myself ever getting the chance to do so, but I'd love to have the opportunity to work on simi

by Skywing 13y ago

Based on my current connections, and experience, I can't fully see myself ever getting the chance to do so, but I'd love to have the opportunity to work on similar problems - at Facebook or anywhere. I've always loved working on lower level problems such as this. I learned to program by writing computer game hacks, reverse engineering games and coding in C / ASM. Currently, I find myself writing C# every day for a small company at which nobody understands a single word I say about programming.

rdl 13y ago
Yeah, it's interesting how Facebook does something which on its face seems trivial and unimportant, but due to scale, has some really amazing engineering and infrastructure challenges -- and they solve them by building tech to make things scale from commodity systems (like Google, etc have done), vs. buying third-party big iron solutions (which is what eBay did when they had similar challenges, and what most big companies do).
- salemh 13y ago
  Do you think its a cost/control to not buy big iron (outsource to a degree) vs build/scale themselves? I'd like to hear some HN thoughts if some don't mind sharing.
  - applecore 13y ago
    If you're a pure technology company, like Google and Facebook, you're not going to outsource your core competency. It's not an issue of costs; buying "big iron", even if they could, would be akin to style drift for an investment manager.
    - rdl 13y ago
      You could definitely still do a lot less in-house than FB does, and be successful. FB seems to delight in building tools and infrastructure. Bloomberg is probably a better example of a company which builds "optional" technology in house, just to be awesome, though -- they're not at the scale of FB (where "traditional" solutions break down), but from what I've seen, they do a lot of interesting work in-house because their staff want to do it, and because it lets them have really top-quality staff in a highly competitive market.
      - deleted 13y ago
        [deleted]
      - morgante 13y ago
        > they're not at the scale of FB (where "traditional" solutions break down) I'm not so sure about that. Bloomberg processes an incredible amount of data, and they have strict latency requirements. In many cases, traditional solutions would in fact break down under those requirements.
        apaprocki 13y ago
        I work on the infrastructure team at Bloomberg. There are lots of problems solved by OSS, but there are also lots of pieces of infrastructure we have to build ourselves to scale the things we need to. Latency is killer, indeed. (Low-latency data aggregation/generation/distribution is only one part of the business, though.)
  - apaprocki 13y ago
    "Big iron" isn't all it's cracked up to be. Everything is a trade-off. Very few people are doing pure computation and that is where those machines excel (in addition to lots of aggregate I/O). The government research labs and the like get a lot of use from these machines. If you trying to scale an Internet-style app on one of these machines, you might need to expand past one machine after a while. By staying on one machine, you're avoiding all the complexity needed in your software to coordinate between multiple machines. If you lose the ability to fit on a single box, you'll need to add that complexity in anyway. So what does 10 beefy boxes buy you as opposed to 1000 smaller ones? There is of course an operational/DC/power cost involved with more boxes, but I think most shops consider that an easily solvable problem. For example, a maxxed out POWER7 box from IBM will give you 256 processors and all the memory and I/O trimmings you need. If you need more than 256 processors or the local amount of RAM, you'll pay the software complexity cost anyway.
    - jbangert 13y ago
      Well, the 10 beefy boxes will be much, much faster if your problem is not very distributable. Say, Facebook as an application shards very easily, because most users don't interact much with each other. Other applications, might have much more interactions. What you're really paying for when buying a 256 processor POWER7 box is the fact that the interconnect (and therefore the time to acquire a lock/update data from another node) is much faster and more reliable than commodity networks/kernels/stack.
      - apaprocki 13y ago
        Interconnect may be faster but as a whole system it is hard to compete with the raw speed of an x64 box with all the latest/greatest chipsets. You usually wind up having to write non-portable code to eke full performance out of the massive box and in the end your apps will probably still be faster on x64. They're best suited for massive parallel computation that isn't afraid of getting down to the metal and taking advantage of lots of the special chip instructions in asm. (Or alternatively you want POWER specifically because it has hardware dfp support.) The total gain from running on x64 will most likely exceed any loss from a network hop in a case where both have to go off to SAN for their data.
  - qq66 13y ago
    Yes. Facebook's "cost of revenue" (which they state is mostly infrastructure) was $1.875 billion in 2013, a year when they made $1.5 billion in net income. For comparison, research and development was $1.4 billion. Facebook's business model involves getting 1 billion people to post a ton of stuff inside Facebook, costing them about $2/user/year in infrastructure, $3.50/user/year in other costs, and making about $7/user/year in advertising revenue, yielding about $1.50 in profit. So cutting costs on that $2 makes them significantly more profitable.
- batbomb 13y ago
  Most big iron solutions are effectively commodity systems anymore anyways, occasionally with fancy interconnects and coprocessors.
  - nasalgoat 13y ago
    Yes, I laughed when I got a console prompt on an $80,000 EMC Isilon disk array and it was FreeBSD.
- rhizome 13y ago
  Facebook does something which on its face seems trivial and unimportant, but due to scale, has some really amazing engineering and infrastructure challenges Is this really true? Seems to me that these challenges are completely due to collecting a whole bunch of information that people would rather they didn't, and routinely rebuke them for. It's like working on drone targeting problems that are made more difficult because children move more unpredictably (or more quickly, etc.) than adults. "Yeah, but the math is insane!" You may dislike my analogy, but it only appears "trivial and unimportant" because it's the most mundane aspect of a larger unsavory project.
choult 13y ago
If you were in the UK I'd invite you to have a think about working with us (DataSift)... Anyone else feeling this way, drop me a line chris.hoult at datasift dot com.
deleted 13y ago
[deleted]
gibrown 13y ago
Can I suggest another company dealing with large amounts of data at a similar scale: http://automattic.com/work-with-us/data-wrangler/ http://automattic.com/work-with-us/data-wrangler/ Yes this is a shameless plug for the company I love working for, but I think it addresses Skywing's point. We are one of the few companies at this scale that are completely location agnostic, and we hire by trial (can you do the job), not by credentials.

by Skywing 13y ago

rdl 13y ago

Yeah, it's interesting how Facebook does something which on its face seems trivial and unimportant, but due to scale, has some really amazing engineering and infrastructure challenges -- and they solve them by building tech to make things scale from commodity systems (like Google, etc have done), vs. buying third-party big iron solutions (which is what eBay did when they had similar challenges, and what most big companies do).

salemh 13y ago
Do you think its a cost/control to not buy big iron (outsource to a degree) vs build/scale themselves? I'd like to hear some HN thoughts if some don't mind sharing.
- applecore 13y ago
  If you're a pure technology company, like Google and Facebook, you're not going to outsource your core competency. It's not an issue of costs; buying "big iron", even if they could, would be akin to style drift for an investment manager.
  - rdl 13y ago
    You could definitely still do a lot less in-house than FB does, and be successful. FB seems to delight in building tools and infrastructure. Bloomberg is probably a better example of a company which builds "optional" technology in house, just to be awesome, though -- they're not at the scale of FB (where "traditional" solutions break down), but from what I've seen, they do a lot of interesting work in-house because their staff want to do it, and because it lets them have really top-quality staff in a highly competitive market.
    - deleted 13y ago
      [deleted]
    - morgante 13y ago
      > they're not at the scale of FB (where "traditional" solutions break down) I'm not so sure about that. Bloomberg processes an incredible amount of data, and they have strict latency requirements. In many cases, traditional solutions would in fact break down under those requirements.
      - apaprocki 13y ago
        I work on the infrastructure team at Bloomberg. There are lots of problems solved by OSS, but there are also lots of pieces of infrastructure we have to build ourselves to scale the things we need to. Latency is killer, indeed. (Low-latency data aggregation/generation/distribution is only one part of the business, though.)
- apaprocki 13y ago
  "Big iron" isn't all it's cracked up to be. Everything is a trade-off. Very few people are doing pure computation and that is where those machines excel (in addition to lots of aggregate I/O). The government research labs and the like get a lot of use from these machines. If you trying to scale an Internet-style app on one of these machines, you might need to expand past one machine after a while. By staying on one machine, you're avoiding all the complexity needed in your software to coordinate between multiple machines. If you lose the ability to fit on a single box, you'll need to add that complexity in anyway. So what does 10 beefy boxes buy you as opposed to 1000 smaller ones? There is of course an operational/DC/power cost involved with more boxes, but I think most shops consider that an easily solvable problem. For example, a maxxed out POWER7 box from IBM will give you 256 processors and all the memory and I/O trimmings you need. If you need more than 256 processors or the local amount of RAM, you'll pay the software complexity cost anyway.
  - jbangert 13y ago
    Well, the 10 beefy boxes will be much, much faster if your problem is not very distributable. Say, Facebook as an application shards very easily, because most users don't interact much with each other. Other applications, might have much more interactions. What you're really paying for when buying a 256 processor POWER7 box is the fact that the interconnect (and therefore the time to acquire a lock/update data from another node) is much faster and more reliable than commodity networks/kernels/stack.
    - apaprocki 13y ago
      Interconnect may be faster but as a whole system it is hard to compete with the raw speed of an x64 box with all the latest/greatest chipsets. You usually wind up having to write non-portable code to eke full performance out of the massive box and in the end your apps will probably still be faster on x64. They're best suited for massive parallel computation that isn't afraid of getting down to the metal and taking advantage of lots of the special chip instructions in asm. (Or alternatively you want POWER specifically because it has hardware dfp support.) The total gain from running on x64 will most likely exceed any loss from a network hop in a case where both have to go off to SAN for their data.
- qq66 13y ago
  Yes. Facebook's "cost of revenue" (which they state is mostly infrastructure) was $1.875 billion in 2013, a year when they made $1.5 billion in net income. For comparison, research and development was $1.4 billion. Facebook's business model involves getting 1 billion people to post a ton of stuff inside Facebook, costing them about $2/user/year in infrastructure, $3.50/user/year in other costs, and making about $7/user/year in advertising revenue, yielding about $1.50 in profit. So cutting costs on that $2 makes them significantly more profitable.
batbomb 13y ago
Most big iron solutions are effectively commodity systems anymore anyways, occasionally with fancy interconnects and coprocessors.
- nasalgoat 13y ago
  Yes, I laughed when I got a console prompt on an $80,000 EMC Isilon disk array and it was FreeBSD.
rhizome 13y ago
Facebook does something which on its face seems trivial and unimportant, but due to scale, has some really amazing engineering and infrastructure challenges Is this really true? Seems to me that these challenges are completely due to collecting a whole bunch of information that people would rather they didn't, and routinely rebuke them for. It's like working on drone targeting problems that are made more difficult because children move more unpredictably (or more quickly, etc.) than adults. "Yeah, but the math is insane!" You may dislike my analogy, but it only appears "trivial and unimportant" because it's the most mundane aspect of a larger unsavory project.

choult 13y ago

If you were in the UK I'd invite you to have a think about working with us (DataSift)... Anyone else feeling this way, drop me a line chris.hoult at datasift dot com.

deleted 13y ago

[deleted]

gibrown 13y ago

Can I suggest another company dealing with large amounts of data at a similar scale: http://automattic.com/work-with-us/data-wrangler/ http://automattic.com/work-with-us/data-wrangler/ Yes this is a shameless plug for the company I love working for, but I think it addresses Skywing's point. We are one of the few companies at this scale that are completely location agnostic, and we hire by trial (can you do the job), not by credentials.