5 ms·

I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a

I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.

raimue 13y ago
It's 2014 already!
oneeyedpigeon 13y ago
HN isn't a mission-critical business. It's a fun, experimental distraction (albeit one that we all rate very highly, of course).
- lazyjones 13y ago
  > It's a fun, experimental distraction It seems to incur maintenance costs and reputation hits that are avoidable though, isn't that reason enough to invest some time in building a reliable solution?
  - user24 13y ago
    No-one's going to say "Yeah I was going to apply to YC but HN keeps going down so I don't think I will after all".
    - lmm 13y ago
      Speak for yourself
      - randallsquared 13y ago
        Could you elaborate on why that's a factor at all? Given that YC's core business has nothing to do with HN, I wonder if the idiosyncrasies of HN aren't actually contributing something like intermittent reinforcement training to community members.
        lmm 13y ago
        Why's it a factor at all? Because I wouldn't want to work with an organization that half-asses things. I don't understand the second half of your comment.
        sp332 13y ago
        "Positive reinforcement" would be a bar which gives some food to a rat who pushes it three times. If a rat is hungry, he pushes the bar enough times to get as much food as he needs. "Intermittent reinforcement" is a bar that gives some food after a random number of presses each time. A rat will push the bar frantically until he's exhausted, even if he's not very hungry. The technique works pretty well on humans too. https://en.wikipedia.org/wiki/Reinforcement#Intermittent_reinforcements https://en.wikipedia.org/wiki/Reinforcement#Intermittent_rei...
- pitchups 13y ago
  And also because that is the hallmark of a true hacker - roll your own - rather than rely on a ready made solution!
huhtenberg 13y ago
Kids these days. Next thing you say you can't whistle at 2400 baud. Sheesh.
- wlesieutre 13y ago
  2400 Hz, if I'm not mistaken. Baud indicates how many signals you can send in a second. Hz is the wave frequency, which you have to mimic for the signaling tones.
  - gvb 13y ago
    No, 2400 baud - phase shifting is used encode the bits, so the actual frequencies used are complex. 300 baud used two tones for Tx and two tones for Rx so it is easy to whistle (albeit the characters coming out are garbage). Higher baud rates are probably impossible to whistle because the encoding is much more complex. I never tried, acoustic couplers didn't work above 300 baud. ;-) Ref: http://en.wikipedia.org/wiki/Modem#1200_and_2400_bps http://en.wikipedia.org/wiki/Modem#1200_and_2400_bps
    - yebyen 13y ago
      I am pretty sure he meant 2600Hz anyway Besides everyone knows, nobody whistles 2600Hz, they just get the toy out of the Cap'n Crunch box to do it for them. http://en.wikipedia.org/wiki/John_Draper#Phreaking http://en.wikipedia.org/wiki/John_Draper#Phreaking
      - sp332 13y ago
        Or rehabilitate an old payphone with an in-house PBX, and then phreak it with a modded Radio Shack tone dialer. http://www.citizenengineer.com/ http://www.citizenengineer.com/
      - deleted 13y ago
        [deleted]
      - huhtenberg 13y ago
        Well, no. It was a joke, so it was 2600 baud, not Hz.
TelmoMenezes 13y ago
Maybe because it's fun? What's the point of getting rich if not to do what you really want? Also, I suspect that if a startup came to them with a loyal following like HN enjoys, they wouldn't be laughed out of the room at all...
matthewmacleod 13y ago
this sort of legacy thinking That's the kind of facile statement that makes people riotously mock the entire startup community, like "MongoDB is webscale" but even less valid. Cloud services are not a panacea, and there are myriad situations in which running one's own infrastructure can be a good idea. What matters is that the issues and benefits are taken into account; if one can show research demonstrating that a custom infrastructure is cheaper, or more reliable, or less prone to legal issues, for example, then there's nothing to laugh at. And remember that PaaS in particular can cost a buttload of money - I'm certain it's contributed to the downfall of more than one otherwise promising startup.
- uccrost 13y ago
  it's definitintely not more reliable in this case...
- eloff 13y ago
  The advantage of cloud servers is if one experiences corruption or goes down you just kill it and start a new one. If your cloudy EBS equivalent experiences corruption, you restore from snapshot and off you go again. Either way it involves less downtime than HN seems to have. The downside is it costs more (usually, depends on how high your server management and data center overheads are.) I'd like to point out that despite several high profile down time incidents in the aws, never was there a case that I'm aware of where you couldn't just restore from your last snapshot to another availability zone or region.
  - shiftpgdn 13y ago
    You realize cloud services are vulnerable to data loss as well?[1] The cloud isn't some magic machine off in a datacenter somewhere. Its a bunch of servers and SANs just like what you or I would roll out if we needed bare metal infrastructure. The only difference is the extraordinary markup that you're paying amazon to use their servers. [1]http://blogs.computerworld.com/18198/oops_amazon_web_services_ec2_cloud_lost_data http://blogs.computerworld.com/18198/oops_amazon_web_service...
gizmo 13y ago
Bespoke hardware is much cheaper, has much more consistent performance, and a beefy dedicated machine is much faster than a large AWS slice. If you can run your entire web app on a single server you have much less infrastructure to worry about, which is a big plus. All this vertical vs horizontal scaling nonsense makes your software so much more complex, and with a tiny bit of optimization it's often not needed. Besides, HN also uses cloudflare as CDN and DDOS protection, so it's not like they're stuck in 1999.
acomjean 13y ago
Periodic HN downtime helps software businesses be more productive. Repeated Disk corruption maybe indicates a hardware problem unless the file system isn't up to snuff. Generally those file systems are pretty reliable.
logicallee 13y ago
if a startup came to YC with on the order of 10k uniques within minutes of posting a top link, and the kind of user accounts and engagement that HN has, they wouldn't care if it was written in Perl 2.0 running on a repurposed old 747's partial fly-by-wire computer. Really, Nobody Cares. Dropbox was an rsync executed every two minutes.[1] AngelList was done via email. [1] can't find this reference atm, could be apocryphal
nullspace 13y ago
If you can plan your hardware capacity, which is not that hard for a simple website like HN, it becomes far more cost effective to run on real hardware, even with a large overhead. This is quite hard for startups to do, because their core function is to keep adding and removing features, experimenting and scaling, and the initial cost of buying hardware to support these functions is too large. This plus all the services provided by AWS et al. saves a lot of time and effort. Nothing to do with "cloud" vs "non-cloud".
rdl 13y ago
hn is mostly a single process application, so dedicated hardware is really the only way they can scale performance (and adding stuff in front like a proxy, cloudflare, etc.) They even went out of their way to get the fastest single core xeon possible, which is a fairly mid-range CPU, singe they care about process performance vs. overall.
KaiserPro 13y ago
because it's cheap and works my dear fellow. Yes it may seem archaic, but it a real server has real disk IO, something which on amazon and the like doesn't come cheap
trout 13y ago
If HN was on AWS, where would we go to discuss AWS outages?
- pconf 13y ago
  There are downside to AWS other than reliability (a la Netflix). One of those is privacy. If you host with AWS Amazon has access to all of your user data, and with Amazon likely the NSA. If you want to maintain a free and open forum for people to express support for people like Snowden and Assange and question the NSA or military / industrial complex' percent of the federal budget (>50) then you have to run your own servers in a skilled, ethical, small to mid-sized ISP's facility. This doesn't explain the penny-wise (IMO) allocation of hardware or dev/sysadmin resources of course.
- vacri 13y ago
  Just don't put it in AWS US East, and you can talk about AWS outages just fine.
deleted 13y ago
[deleted]
jrmenon 13y ago
FWIW, our start-up began on AWS during early days, but finally due to cost constraints, migrated most of our servers to our own h/w setup. We had substantial number of servers of varying types on AWS (live servers, full hadoop cluseters etc..), on east and west regions, lots of EBS storage etc.., so not some trivial setup. We really liked AWS, and the support we got for most of the times was good, but as we had 24x7 live traffic, even with reserved instances, the cost finally caught up to justify the move to our own hosting solution. One thing that the made the migration easier for us was that from day 1, we decided to treat AWS as a co-location, hence we set things up with the usual open-source s/w stack (i.e. avoided proprietary Amazon solutions like dynamo, messaging solutions etc.. maybe just used S3 for offline archiving of logs. It was tempting to build on their components, but ended up building it ourselves.), including own own hadoop setup. When the time came, we could easily migrate things out. Hope this gives some perspective. Still a fan of AWS, and if I were do another start-up, would follow the same script all over again.
IgorPartola 13y ago
Because it is actually 2014 :) Kidding aside, running a bare metal server you own or rent is always cheaper, assuming you cannot save money by turning things on and off and know you capacity needs. Sure HN grows but not nearly as fast as FB, Twitter, etc. the service is big enough where it would require expensive virtual servers. This is the best case scenario for a physical hardware box run by people who are familiar with such things.
corresation 13y ago
If a startup came to you with this sort of legacy thinking you'd laugh them out of the room. In many circles, you would be laughed out of the room for cargo culting like this, and for trying to draw a bias against a legitimate deployment choice by declaring it "legacy". There are many scenarios where hosting on your own hardware is a superior option for a variety of reasons: Financially, security, performance, flexibility. In the case of HN it seems like it's on some pretty meager hardware, making compromises like software RAID. If this were a critical system for YC, they would have it on redundant machines with redundant, flash-based, hardware-RAID equipped platforms, clustered with redundant 10Gb cross connects, etc. Criticizing a deployment strategy because of the peculiar issues they have faced is like writing off AWS because someone's unbacked up small instance got killed and they had no strategy for it.
deleted 13y ago
[deleted]

raimue 13y ago

It's 2014 already!

oneeyedpigeon 13y ago

HN isn't a mission-critical business. It's a fun, experimental distraction (albeit one that we all rate very highly, of course).

lazyjones 13y ago
> It's a fun, experimental distraction It seems to incur maintenance costs and reputation hits that are avoidable though, isn't that reason enough to invest some time in building a reliable solution?
- user24 13y ago
  No-one's going to say "Yeah I was going to apply to YC but HN keeps going down so I don't think I will after all".
  - lmm 13y ago
    Speak for yourself
    - randallsquared 13y ago
      Could you elaborate on why that's a factor at all? Given that YC's core business has nothing to do with HN, I wonder if the idiosyncrasies of HN aren't actually contributing something like intermittent reinforcement training to community members.
      - lmm 13y ago
        Why's it a factor at all? Because I wouldn't want to work with an organization that half-asses things. I don't understand the second half of your comment.
        sp332 13y ago
        "Positive reinforcement" would be a bar which gives some food to a rat who pushes it three times. If a rat is hungry, he pushes the bar enough times to get as much food as he needs. "Intermittent reinforcement" is a bar that gives some food after a random number of presses each time. A rat will push the bar frantically until he's exhausted, even if he's not very hungry. The technique works pretty well on humans too. https://en.wikipedia.org/wiki/Reinforcement#Intermittent_reinforcements https://en.wikipedia.org/wiki/Reinforcement#Intermittent_rei...
pitchups 13y ago
And also because that is the hallmark of a true hacker - roll your own - rather than rely on a ready made solution!

huhtenberg 13y ago

Kids these days. Next thing you say you can't whistle at 2400 baud. Sheesh.

wlesieutre 13y ago
2400 Hz, if I'm not mistaken. Baud indicates how many signals you can send in a second. Hz is the wave frequency, which you have to mimic for the signaling tones.
- gvb 13y ago
  No, 2400 baud - phase shifting is used encode the bits, so the actual frequencies used are complex. 300 baud used two tones for Tx and two tones for Rx so it is easy to whistle (albeit the characters coming out are garbage). Higher baud rates are probably impossible to whistle because the encoding is much more complex. I never tried, acoustic couplers didn't work above 300 baud. ;-) Ref: http://en.wikipedia.org/wiki/Modem#1200_and_2400_bps http://en.wikipedia.org/wiki/Modem#1200_and_2400_bps
  - yebyen 13y ago
    I am pretty sure he meant 2600Hz anyway Besides everyone knows, nobody whistles 2600Hz, they just get the toy out of the Cap'n Crunch box to do it for them. http://en.wikipedia.org/wiki/John_Draper#Phreaking http://en.wikipedia.org/wiki/John_Draper#Phreaking
    - sp332 13y ago
      Or rehabilitate an old payphone with an in-house PBX, and then phreak it with a modded Radio Shack tone dialer. http://www.citizenengineer.com/ http://www.citizenengineer.com/
    - deleted 13y ago
      [deleted]
    - huhtenberg 13y ago
      Well, no. It was a joke, so it was 2600 baud, not Hz.

TelmoMenezes 13y ago

Maybe because it's fun? What's the point of getting rich if not to do what you really want? Also, I suspect that if a startup came to them with a loyal following like HN enjoys, they wouldn't be laughed out of the room at all...

matthewmacleod 13y ago

this sort of legacy thinking That's the kind of facile statement that makes people riotously mock the entire startup community, like "MongoDB is webscale" but even less valid. Cloud services are not a panacea, and there are myriad situations in which running one's own infrastructure can be a good idea. What matters is that the issues and benefits are taken into account; if one can show research demonstrating that a custom infrastructure is cheaper, or more reliable, or less prone to legal issues, for example, then there's nothing to laugh at. And remember that PaaS in particular can cost a buttload of money - I'm certain it's contributed to the downfall of more than one otherwise promising startup.

uccrost 13y ago
it's definitintely not more reliable in this case...
eloff 13y ago
The advantage of cloud servers is if one experiences corruption or goes down you just kill it and start a new one. If your cloudy EBS equivalent experiences corruption, you restore from snapshot and off you go again. Either way it involves less downtime than HN seems to have. The downside is it costs more (usually, depends on how high your server management and data center overheads are.) I'd like to point out that despite several high profile down time incidents in the aws, never was there a case that I'm aware of where you couldn't just restore from your last snapshot to another availability zone or region.
- shiftpgdn 13y ago
  You realize cloud services are vulnerable to data loss as well?[1] The cloud isn't some magic machine off in a datacenter somewhere. Its a bunch of servers and SANs just like what you or I would roll out if we needed bare metal infrastructure. The only difference is the extraordinary markup that you're paying amazon to use their servers. [1]http://blogs.computerworld.com/18198/oops_amazon_web_services_ec2_cloud_lost_data http://blogs.computerworld.com/18198/oops_amazon_web_service...

gizmo 13y ago

Bespoke hardware is much cheaper, has much more consistent performance, and a beefy dedicated machine is much faster than a large AWS slice. If you can run your entire web app on a single server you have much less infrastructure to worry about, which is a big plus. All this vertical vs horizontal scaling nonsense makes your software so much more complex, and with a tiny bit of optimization it's often not needed. Besides, HN also uses cloudflare as CDN and DDOS protection, so it's not like they're stuck in 1999.

acomjean 13y ago

Periodic HN downtime helps software businesses be more productive. Repeated Disk corruption maybe indicates a hardware problem unless the file system isn't up to snuff. Generally those file systems are pretty reliable.

logicallee 13y ago

if a startup came to YC with on the order of 10k uniques within minutes of posting a top link, and the kind of user accounts and engagement that HN has, they wouldn't care if it was written in Perl 2.0 running on a repurposed old 747's partial fly-by-wire computer. Really, Nobody Cares. Dropbox was an rsync executed every two minutes.[1] AngelList was done via email. [1] can't find this reference atm, could be apocryphal

nullspace 13y ago

If you can plan your hardware capacity, which is not that hard for a simple website like HN, it becomes far more cost effective to run on real hardware, even with a large overhead. This is quite hard for startups to do, because their core function is to keep adding and removing features, experimenting and scaling, and the initial cost of buying hardware to support these functions is too large. This plus all the services provided by AWS et al. saves a lot of time and effort. Nothing to do with "cloud" vs "non-cloud".

rdl 13y ago

hn is mostly a single process application, so dedicated hardware is really the only way they can scale performance (and adding stuff in front like a proxy, cloudflare, etc.) They even went out of their way to get the fastest single core xeon possible, which is a fairly mid-range CPU, singe they care about process performance vs. overall.

KaiserPro 13y ago

because it's cheap and works my dear fellow. Yes it may seem archaic, but it a real server has real disk IO, something which on amazon and the like doesn't come cheap

trout 13y ago

If HN was on AWS, where would we go to discuss AWS outages?

pconf 13y ago
There are downside to AWS other than reliability (a la Netflix). One of those is privacy. If you host with AWS Amazon has access to all of your user data, and with Amazon likely the NSA. If you want to maintain a free and open forum for people to express support for people like Snowden and Assange and question the NSA or military / industrial complex' percent of the federal budget (>50) then you have to run your own servers in a skilled, ethical, small to mid-sized ISP's facility. This doesn't explain the penny-wise (IMO) allocation of hardware or dev/sysadmin resources of course.
vacri 13y ago
Just don't put it in AWS US East, and you can talk about AWS outages just fine.

deleted 13y ago

[deleted]

jrmenon 13y ago

FWIW, our start-up began on AWS during early days, but finally due to cost constraints, migrated most of our servers to our own h/w setup. We had substantial number of servers of varying types on AWS (live servers, full hadoop cluseters etc..), on east and west regions, lots of EBS storage etc.., so not some trivial setup. We really liked AWS, and the support we got for most of the times was good, but as we had 24x7 live traffic, even with reserved instances, the cost finally caught up to justify the move to our own hosting solution. One thing that the made the migration easier for us was that from day 1, we decided to treat AWS as a co-location, hence we set things up with the usual open-source s/w stack (i.e. avoided proprietary Amazon solutions like dynamo, messaging solutions etc.. maybe just used S3 for offline archiving of logs. It was tempting to build on their components, but ended up building it ourselves.), including own own hadoop setup. When the time came, we could easily migrate things out. Hope this gives some perspective. Still a fan of AWS, and if I were do another start-up, would follow the same script all over again.

IgorPartola 13y ago

Because it is actually 2014 :) Kidding aside, running a bare metal server you own or rent is always cheaper, assuming you cannot save money by turning things on and off and know you capacity needs. Sure HN grows but not nearly as fast as FB, Twitter, etc. the service is big enough where it would require expensive virtual servers. This is the best case scenario for a physical hardware box run by people who are familiar with such things.

corresation 13y ago

If a startup came to you with this sort of legacy thinking you'd laugh them out of the room. In many circles, you would be laughed out of the room for cargo culting like this, and for trying to draw a bias against a legitimate deployment choice by declaring it "legacy". There are many scenarios where hosting on your own hardware is a superior option for a variety of reasons: Financially, security, performance, flexibility. In the case of HN it seems like it's on some pretty meager hardware, making compromises like software RAID. If this were a critical system for YC, they would have it on redundant machines with redundant, flash-based, hardware-RAID equipped platforms, clustered with redundant 10Gb cross connects, etc. Criticizing a deployment strategy because of the peculiar issues they have faced is like writing off AWS because someone's unbacked up small instance got killed and they had no strategy for it.

deleted 13y ago

[deleted]