3 ms·

Attention Internet start-ups: operations is your core competency. You can't just expect to push your application code to "the cloud" and have somebody else make

by strlen 17y ago

Attention Internet start-ups: operations is your core competency. You can't just expect to push your application code to "the cloud" and have somebody else make it scalable and fault tolerant. It's fine to use a managed hosting service provider if you're just getting started and paying month-to-month based on credit cards and can't afford networking gear (hardware itself can be leased). However, it shocks me to see venture funded, post-series A start-ups exclusively relying on others to do their operations (including systems administration). Problem with outsourcing your operations to somebody else is that you're outsourcing it to somebody who has zero knowledge of your application and is also responsible for at the very least dozens of other customers. Essentially, rather than developing your own vertical technology team, you're relying on a on horizontal technology team whose resources multiple other companies (including your competitors) are fighting for. That's exactly how poorly run big companies function (multiple engineering teams competing for attention of monothlic operations, SCM, release, QA etc... teams). Well run Internet giants, however, function much differently. If you look at Google's job openings, you can note that they don't hire IT/Systems Administrators (aside from data-center technicians and corporate IT). Instead, they have Site Reliability Engineers (SREs) assigned to specific properties. The SREs are actually engineers who are able to write and debug code and deeply understand the application-specific stack they operate rather than treat it as a black box. Sure, there are great business and technical reasons (edit: it said financial, which I felt was an unfair strawman) reasons to use a managed/"cloud" provider (EC2 and Rackspace cloud are very attractive due to the ability to add/remove nodes as traffic goes up and down, as well as provision machines ad-hoc for analytics tasks/MapReduce) -- but even then, you're not off the hook for operations. It's still your responsibility to ensure that at the very least, there's a "hot standby" disaster recovery site. Yes, it's not easy - but running a successful business isn't supposed to be.

vaksel 17y ago
you yourself are at the same risk of getting the same exact problem...who do you think is better at fixing it, a startup guy who is no expert, or the 100 people at Rackspace who know what they are doing?
- strlen 17y ago
  If your start-up has nobody that is competent at operations (which really is a lot more engineering than IT/systems administration), it deserves to fail. Incidentally, I can tell from my personal knowledge that the "big successes" amongst Internet companies (i.e. those who have achieved a product/market fit on competitive, multi-billion dollar markets) have had a very strong operations engineer amongst either the founders of the first five employees. If you use Rackspace to host your machine, that's an acceptable way to save money on hardware and networking gear (while you're still in the early stages). However, you can't and shouldn't expect Rackspace's engineers do your operations for you. It is your responsibility to understand how to tune your operating system, application server, database et al. specifically to your application. If you expect others to turn your application into an Internet service, you're analogous to a "business guy" on Craigslist posting about his "next big idea" that he wants somebody else to build. The big point I meant to make is that if your site is down because your hosting provider is, it's still your fault (for not having a DR site).
  - vaksel 17y ago
    a hacker who dabbles in IT, will never be as competent as an expert. And bringing in an IT founder early on, is a waste of equity, since you are giving away a farm when the guy has nothing to do. + all that tuning etc, only matters when you are bringing in a ton of traffic, until then it doesn't matter if your server is 94% efficient or 97%. Worst case scenario, you throw another server into the mix.
    - strlen 17y ago
      Best software engineers are also strong systems administrators. It's also commonly understood that strong systems people must be strong software engineers. An operation department at a serious Internet company wouldn't even invite somebody who can't write production quality code for an in-person interview (they wouldn't pass a phone screen). Strong operations people would also resent being called "IT". IT sets up corporate desktops/mail/calendaring, the VPN and mounts machines in a rack. Indeed, this could be outsourced (mail to Google Apps, rack mounting to Rackspace, OpenVPN in lieu of an expensive Cisco product). Production operations involves something entirely different and requires an entirely different skillset. Think being able to create application specific automated provisioning, configuration, deployment, software load balancing, monitoring, fault tolerance and high availability. Monitoring is especially tricky: nearly all the enterprise and open source NMS products, for example, "get it wrong" (e.g. hyperic using a single database to store events and allowing you to only poll every few seconds). Monitoring an application (as opposed to a machine) requires writing custom code (by definition), otherwise you simply can't capture the essential vital statistics (and not just whether or not somebody listening on port 80). I hate doing an appeal to authority, but I should add I've done operations at a very large scale (10,000+ machines) in a big name Internet company, as well as on a smaller scale at startups. I've since moved on to software engineering (since that's what I love doing the most), but it bugs me to see misconceptions about the nature of running an Internet service. Unfortunately these misconceptions run deep and often disadvantage entire application stacks by making them unfriendly to operations (e.g. the flaws of JMX).
bretpiatt 17y ago
This is good advice to a point, everyone should do some basic things to have the proper level of availablity for their needs. Referencing a blog post I made about this during the BitBucket issue a couple months back: http://www.bretpiatt.com/blog/2009/10/03/availability-is-a-fundamental-design-concept/ http://www.bretpiatt.com/blog/2009/10/03/availability-is-a-f... I'm not a big fan of the DIY and this isn't because I work for a service provider (Rackspace). I fundamentally don't believe businesses should utilize resources with business specific knowledge on commodity tasks. A number of providers are capable of setting up a multi-DC configuration for customers. A number of consultancies are capable of setting up a multi-DC/multi-provider configuration. The provider or consultant can provide clear instructions and/or support through the commodity layers. Businesses will ultimately need to know the specific details of their application and how to troubleshoot it unless you're paying the provider/support team to do so.
wrs 17y ago
Comparing Techcrunch to Google is a bit lopsided, don't you think? Operations for most websites is like vehicle maintenance for a pizza delivery company: essential for success, but hardly their "core competency". This is especially true if the website is something relatively standard like a blog or news site. It takes quite a bit of scale and customization before it makes sense to hire full-time dedicated sysadmins.
- revicon 17y ago
  > Operations for most websites is like vehicle maintenance for a pizza delivery company: essential for success, but hardly their "core competency". Very interesting analogy.
  - strlen 17y ago
    Would you fly on an airline that treated their aircraft maintenance the same way? So why would you expect advertisers to run their ads on sites that don't have a person who understands the application carrying a pager?
- strlen 17y ago
  If you don't need a full time operations engineer, you should become competent as a part-time one. Both Google and Techcrunch share the fact that they have paying advertisers who expect their site to be up and delivering their content.

by strlen 17y ago

vaksel 17y ago

you yourself are at the same risk of getting the same exact problem...who do you think is better at fixing it, a startup guy who is no expert, or the 100 people at Rackspace who know what they are doing?

strlen 17y ago
If your start-up has nobody that is competent at operations (which really is a lot more engineering than IT/systems administration), it deserves to fail. Incidentally, I can tell from my personal knowledge that the "big successes" amongst Internet companies (i.e. those who have achieved a product/market fit on competitive, multi-billion dollar markets) have had a very strong operations engineer amongst either the founders of the first five employees. If you use Rackspace to host your machine, that's an acceptable way to save money on hardware and networking gear (while you're still in the early stages). However, you can't and shouldn't expect Rackspace's engineers do your operations for you. It is your responsibility to understand how to tune your operating system, application server, database et al. specifically to your application. If you expect others to turn your application into an Internet service, you're analogous to a "business guy" on Craigslist posting about his "next big idea" that he wants somebody else to build. The big point I meant to make is that if your site is down because your hosting provider is, it's still your fault (for not having a DR site).
- vaksel 17y ago
  a hacker who dabbles in IT, will never be as competent as an expert. And bringing in an IT founder early on, is a waste of equity, since you are giving away a farm when the guy has nothing to do. + all that tuning etc, only matters when you are bringing in a ton of traffic, until then it doesn't matter if your server is 94% efficient or 97%. Worst case scenario, you throw another server into the mix.
  - strlen 17y ago
    Best software engineers are also strong systems administrators. It's also commonly understood that strong systems people must be strong software engineers. An operation department at a serious Internet company wouldn't even invite somebody who can't write production quality code for an in-person interview (they wouldn't pass a phone screen). Strong operations people would also resent being called "IT". IT sets up corporate desktops/mail/calendaring, the VPN and mounts machines in a rack. Indeed, this could be outsourced (mail to Google Apps, rack mounting to Rackspace, OpenVPN in lieu of an expensive Cisco product). Production operations involves something entirely different and requires an entirely different skillset. Think being able to create application specific automated provisioning, configuration, deployment, software load balancing, monitoring, fault tolerance and high availability. Monitoring is especially tricky: nearly all the enterprise and open source NMS products, for example, "get it wrong" (e.g. hyperic using a single database to store events and allowing you to only poll every few seconds). Monitoring an application (as opposed to a machine) requires writing custom code (by definition), otherwise you simply can't capture the essential vital statistics (and not just whether or not somebody listening on port 80). I hate doing an appeal to authority, but I should add I've done operations at a very large scale (10,000+ machines) in a big name Internet company, as well as on a smaller scale at startups. I've since moved on to software engineering (since that's what I love doing the most), but it bugs me to see misconceptions about the nature of running an Internet service. Unfortunately these misconceptions run deep and often disadvantage entire application stacks by making them unfriendly to operations (e.g. the flaws of JMX).

bretpiatt 17y ago

This is good advice to a point, everyone should do some basic things to have the proper level of availablity for their needs. Referencing a blog post I made about this during the BitBucket issue a couple months back: http://www.bretpiatt.com/blog/2009/10/03/availability-is-a-fundamental-design-concept/ http://www.bretpiatt.com/blog/2009/10/03/availability-is-a-f... I'm not a big fan of the DIY and this isn't because I work for a service provider (Rackspace). I fundamentally don't believe businesses should utilize resources with business specific knowledge on commodity tasks. A number of providers are capable of setting up a multi-DC configuration for customers. A number of consultancies are capable of setting up a multi-DC/multi-provider configuration. The provider or consultant can provide clear instructions and/or support through the commodity layers. Businesses will ultimately need to know the specific details of their application and how to troubleshoot it unless you're paying the provider/support team to do so.

wrs 17y ago

Comparing Techcrunch to Google is a bit lopsided, don't you think? Operations for most websites is like vehicle maintenance for a pizza delivery company: essential for success, but hardly their "core competency". This is especially true if the website is something relatively standard like a blog or news site. It takes quite a bit of scale and customization before it makes sense to hire full-time dedicated sysadmins.

revicon 17y ago
> Operations for most websites is like vehicle maintenance for a pizza delivery company: essential for success, but hardly their "core competency". Very interesting analogy.
- strlen 17y ago
  Would you fly on an airline that treated their aircraft maintenance the same way? So why would you expect advertisers to run their ads on sites that don't have a person who understands the application carrying a pager?
strlen 17y ago
If you don't need a full time operations engineer, you should become competent as a part-time one. Both Google and Techcrunch share the fact that they have paying advertisers who expect their site to be up and delivering their content.