5 ms·

Do you think they were better off by having their own servers ? (Honest question we use AWS and Rackspace, and we feel the pain of Monthly bills as well).

autarch 13y ago
This should be fairly easy to price out, right? I suggest you start by simply comparing the cost of dedicated hosting-provided hardware. This is the middle ground between owning your own stuff and something like AWS. Pricing out the cost of owning your own servers can be tricky since you need to account for a number of factors: * Initial cost of hardware - don't forget to include things like network switches, remote console devices, etc. - you may also want to buy something like a remotely accessible power switch so you can forcibly reboot machines * Rack space costs * Bandwidth costs * Sysadmin time to manage the hardware (you already need a sysadmin to manage things like backups, logging, etc.) * On-site remote hands if your rack host is not physically near your sysadmin But regardless, I'm pretty sure that _anything_ ends up being cheaper than AWS once you get beyond a fairly small number of servers. The big advantages of AWS are integration with other services (storage, load balancing, etc.) and the ability to spin many machines up and down very quickly. However, if you can tolerate waiting a day or so to provision new servers, dedicated hosting-provided hardware is almost certainly going to be quite a bit cheaper. You could also compare this to something like Linode. Again, you lose the ability to provision new machines almost instantly, but they are pretty quick to bring up new machines (minutes, not hours or days). They also have a number of nice integrated services like backups. Finally, you can also do some sort of hybrid setup where you have core stable services on dedicated machines but still spin up AWS instances to deal with bursty computation needs.
- rpedela 13y ago
  If you just use EC2, no EBS, and reserved pricing then AWS is just as cheap as other providers if not cheaper. It is all the nice features that make it expensive. S3 is pretty damn cheap too. Of course, raw hosting costs are only part of the calculation. You have to include sysadmin time and salary.
  - autarch 13y ago
    You can't really compare EC2 sans EBS to something like a dedicated server. Most apps need permanently available storage. Of course, if you don't need permanent storage (and your needs are bursty) than using EC2 is probably a huge win.
    - rpedela 13y ago
      Dedicated servers don't really give you permanent storage either. Granted, it is more permanent than EC2 instance storage, but hard drives do get corrupted. You still need backup and replication for a robust solution.
      - rhizome 13y ago
        "Disk is cheap"
    - sameer_sundresh 13y ago
      Your second paragraph exactly describes Everpix's computational needs: bursty with no use for local persistent storage.
- dmourati 13y ago
  I did this math at Eye-Fi, a competitor of sorts to Everpix. What I learned, is the answer to whether to build or buy is the classic: "it depends". The driving factor? Data retention policy. The one move Amazon made during my analysis that had a huge shift in the calculus was to introduce Glacier. My takeaway, the economics of a lifetime storage offering are very difficult to achieve. Glacier changes the rules by altering the SLA for data access to make the money work out.
  - girvo 13y ago
    Eye-Fi is awesome. <3
- aespinoza 13y ago
  Thank you for your answer. I agree with you. And after I read this article I went on to do some research decided to move the servers in-house. But I am also researching a Hybrid cloud using AWS.
x0x0 13y ago
s3 seems great for scaling out storage, but I'm sure there are gotchas. I work in a very different area, but my experience with ec2 are you should drop what you're doing and immediately price out leaving. Roughly: for big hadoop tasks, ec2 is a horrid environment. The boxes are slow, flaky, and shitty; the network is slow and super shitty; and everything about the experience is ass. Oh, and the emr admins are assholes who push broken code; code that couldn't even have run once. These assholes, after issuing us a hotfix to a boot action, proceeded to change the path to the hotfix (without mentioning it to us), breaking our production clusters twice in in 30 hours. I guess I hold a grudge but I like sleeping. The point is emr and perhaps ec2 are complete amateur hour. Quoting myself earlier: from a former employer who wants to stay anonymous, ec2 bill per month: $97k peak, softlayer: $26k. 1/3 the cost, 2x the performance. And let's not mention emr; a $15k/mo cluster in softlayer did something like 5x the performance of an emr cluster that was costing (and running!) $2k/day. I'm not kidding about 1/3 the cost and twice the performance. And this wasn't even with rack/switch local boxes, which would improve performance even more.
- zaidf 13y ago
  I can vouch for your numbers. My music start-up was one of Softlayer's early customers. We stayed with them for a while before moving to an even cheaper host. In all, we were pushing over 100TB and paying under $3,000/mo for everything(web, storage, db servers). This was over 5 years ago. I am sure things would be even cheaper now.
- diziet 13y ago
  Soft Layer seems to be as expensive as EC2, if not more expensive, especially when compared to reserved instances. The 244gb cr1.8xlarge (16 cpus, 240gb ssd) amortized over 3 years costs $659 / month. Softlayer would charge quite a bit more for similar specs ( order of magnitude more).
keithwarren 13y ago
My startup is in the same space as them and I can promise you that running our own storage is much cheaper. Azure and AWS run around 7K per month for 100tb. When you factor redundacy I need to buy 200tb, factor in other server infrastructure with the cost of a 4TB drive running around 180 and I can spend 15K to buy 200TB of storage. Put another way, we can own twice the storage capacity (or 1x mirrored) for roughly 2 months of S3 storage.
- parasubvert 13y ago
  S3 is already redundant - it virtually guarantees no data loss and mirrors your storage across multiple data centres. You're also completely ignoring the costs of hosting this stuff, the computers they're in, redundant network switches, failed disks over time, and administration costs to rack and stack failed drives plus all the risk associated with your redundancy going wrong. You're also assuming your hard drives will be able to handle the performance needs. Not to mention this is one site - no DR. So, Amazon S3 at $7.5k a month for 100 TB is $90k annually with RIDICULOUSLY high redundancy and known performance. Reduce it to 99.99% availability (about what your setup with dual drives might be if you run it well) and you're at $60k annually. You really are going to host fast, reliable 100 TB for under $60k? Betting your business on it?
swisspol 13y ago
It's something that was absolutely on the table for our planned 4th generation infrastructure. It was unlikely to us we could stay on AWS. We already had 400 millions full-res photos and running 80+ EC2 / RDS instances for 50,000 users. At our growth rate, we would have been close to a billion photos by end of year. It's far from simple though: you need to find the people who can build such systems and also be able to afford the capex. And then there's the risks of running your own system and opportunity cost. It's really unclear we would have been able to go this far in the first place without AWS.
- keithwarren 13y ago
  Any idea what the average storage take per user was?
- rhizome 13y ago
  Not to pile on, but a storage startup should have had a systems person among the 7 employees. I would do it in the first 10, but the only reason it wouldn't be in the first 5 is that I have the skills myself. It's really unclear we would have been able to go this far in the first place without AWS. You could have drastically cut your burn on that last $550K by hiring a sysadmin and bringing it all out of AWS (maybe keeping S3). This would have likely also involved retooling on the app side, but in the long run it would have given you more time for positioning/promotion/etc. All in all, 4th Gen is way late for this. P.S. Number of photos is basically irrelevant, were you actually using market-fit metrics for infrastructural decision support?
asdasf 13y ago
Yes, when you have a large amount of storage, it is well worth doing it yourself. The cost of redundant storage devices with replication ends up being paid for in like 3 months of S3 bills.
beachstartup 13y ago
yes. they would have 100% been better off, provided they actually knew what they were doing, which i guess is questionable since the business failed. pricing out servers and researching / working with smaller infrastructure providers who actually care about you and your account (imagine that) is real work, which is why so many people avoid it these days. it's much easier to click a few buttons and "choose" machine type x in the offerings that amazon has designed to maximize their own profit. computers are hard, let's go shopping.
ye 13y ago
Your alternatives are 1) Switch to other cloud providers, I'm not aware of any very cheap ones. 2) Switch to other hostings, which can be 1/10 as cheap as AWS, but then you have to deal with the failures. 3) Colocate and build your own servers/storage. It's only cheaper if your bill is much more than a 24/7 sysadmin(s) salary. I suspect DropBox don't give a shit about server costs, they are swimming in money. But I bet having their own data centers will save them money.

autarch 13y ago

This should be fairly easy to price out, right? I suggest you start by simply comparing the cost of dedicated hosting-provided hardware. This is the middle ground between owning your own stuff and something like AWS. Pricing out the cost of owning your own servers can be tricky since you need to account for a number of factors: * Initial cost of hardware - don't forget to include things like network switches, remote console devices, etc. - you may also want to buy something like a remotely accessible power switch so you can forcibly reboot machines * Rack space costs * Bandwidth costs * Sysadmin time to manage the hardware (you already need a sysadmin to manage things like backups, logging, etc.) * On-site remote hands if your rack host is not physically near your sysadmin But regardless, I'm pretty sure that _anything_ ends up being cheaper than AWS once you get beyond a fairly small number of servers. The big advantages of AWS are integration with other services (storage, load balancing, etc.) and the ability to spin many machines up and down very quickly. However, if you can tolerate waiting a day or so to provision new servers, dedicated hosting-provided hardware is almost certainly going to be quite a bit cheaper. You could also compare this to something like Linode. Again, you lose the ability to provision new machines almost instantly, but they are pretty quick to bring up new machines (minutes, not hours or days). They also have a number of nice integrated services like backups. Finally, you can also do some sort of hybrid setup where you have core stable services on dedicated machines but still spin up AWS instances to deal with bursty computation needs.

rpedela 13y ago
If you just use EC2, no EBS, and reserved pricing then AWS is just as cheap as other providers if not cheaper. It is all the nice features that make it expensive. S3 is pretty damn cheap too. Of course, raw hosting costs are only part of the calculation. You have to include sysadmin time and salary.
- autarch 13y ago
  You can't really compare EC2 sans EBS to something like a dedicated server. Most apps need permanently available storage. Of course, if you don't need permanent storage (and your needs are bursty) than using EC2 is probably a huge win.
  - rpedela 13y ago
    Dedicated servers don't really give you permanent storage either. Granted, it is more permanent than EC2 instance storage, but hard drives do get corrupted. You still need backup and replication for a robust solution.
    - rhizome 13y ago
      "Disk is cheap"
  - sameer_sundresh 13y ago
    Your second paragraph exactly describes Everpix's computational needs: bursty with no use for local persistent storage.
dmourati 13y ago
I did this math at Eye-Fi, a competitor of sorts to Everpix. What I learned, is the answer to whether to build or buy is the classic: "it depends". The driving factor? Data retention policy. The one move Amazon made during my analysis that had a huge shift in the calculus was to introduce Glacier. My takeaway, the economics of a lifetime storage offering are very difficult to achieve. Glacier changes the rules by altering the SLA for data access to make the money work out.
- girvo 13y ago
  Eye-Fi is awesome. <3
aespinoza 13y ago
Thank you for your answer. I agree with you. And after I read this article I went on to do some research decided to move the servers in-house. But I am also researching a Hybrid cloud using AWS.

x0x0 13y ago

s3 seems great for scaling out storage, but I'm sure there are gotchas. I work in a very different area, but my experience with ec2 are you should drop what you're doing and immediately price out leaving. Roughly: for big hadoop tasks, ec2 is a horrid environment. The boxes are slow, flaky, and shitty; the network is slow and super shitty; and everything about the experience is ass. Oh, and the emr admins are assholes who push broken code; code that couldn't even have run once. These assholes, after issuing us a hotfix to a boot action, proceeded to change the path to the hotfix (without mentioning it to us), breaking our production clusters twice in in 30 hours. I guess I hold a grudge but I like sleeping. The point is emr and perhaps ec2 are complete amateur hour. Quoting myself earlier: from a former employer who wants to stay anonymous, ec2 bill per month: $97k peak, softlayer: $26k. 1/3 the cost, 2x the performance. And let's not mention emr; a $15k/mo cluster in softlayer did something like 5x the performance of an emr cluster that was costing (and running!) $2k/day. I'm not kidding about 1/3 the cost and twice the performance. And this wasn't even with rack/switch local boxes, which would improve performance even more.

zaidf 13y ago
I can vouch for your numbers. My music start-up was one of Softlayer's early customers. We stayed with them for a while before moving to an even cheaper host. In all, we were pushing over 100TB and paying under $3,000/mo for everything(web, storage, db servers). This was over 5 years ago. I am sure things would be even cheaper now.
diziet 13y ago
Soft Layer seems to be as expensive as EC2, if not more expensive, especially when compared to reserved instances. The 244gb cr1.8xlarge (16 cpus, 240gb ssd) amortized over 3 years costs $659 / month. Softlayer would charge quite a bit more for similar specs ( order of magnitude more).

keithwarren 13y ago

My startup is in the same space as them and I can promise you that running our own storage is much cheaper. Azure and AWS run around 7K per month for 100tb. When you factor redundacy I need to buy 200tb, factor in other server infrastructure with the cost of a 4TB drive running around 180 and I can spend 15K to buy 200TB of storage. Put another way, we can own twice the storage capacity (or 1x mirrored) for roughly 2 months of S3 storage.

parasubvert 13y ago
S3 is already redundant - it virtually guarantees no data loss and mirrors your storage across multiple data centres. You're also completely ignoring the costs of hosting this stuff, the computers they're in, redundant network switches, failed disks over time, and administration costs to rack and stack failed drives plus all the risk associated with your redundancy going wrong. You're also assuming your hard drives will be able to handle the performance needs. Not to mention this is one site - no DR. So, Amazon S3 at $7.5k a month for 100 TB is $90k annually with RIDICULOUSLY high redundancy and known performance. Reduce it to 99.99% availability (about what your setup with dual drives might be if you run it well) and you're at $60k annually. You really are going to host fast, reliable 100 TB for under $60k? Betting your business on it?

swisspol 13y ago

It's something that was absolutely on the table for our planned 4th generation infrastructure. It was unlikely to us we could stay on AWS. We already had 400 millions full-res photos and running 80+ EC2 / RDS instances for 50,000 users. At our growth rate, we would have been close to a billion photos by end of year. It's far from simple though: you need to find the people who can build such systems and also be able to afford the capex. And then there's the risks of running your own system and opportunity cost. It's really unclear we would have been able to go this far in the first place without AWS.

keithwarren 13y ago
Any idea what the average storage take per user was?
rhizome 13y ago
Not to pile on, but a storage startup should have had a systems person among the 7 employees. I would do it in the first 10, but the only reason it wouldn't be in the first 5 is that I have the skills myself. It's really unclear we would have been able to go this far in the first place without AWS. You could have drastically cut your burn on that last $550K by hiring a sysadmin and bringing it all out of AWS (maybe keeping S3). This would have likely also involved retooling on the app side, but in the long run it would have given you more time for positioning/promotion/etc. All in all, 4th Gen is way late for this. P.S. Number of photos is basically irrelevant, were you actually using market-fit metrics for infrastructural decision support?

asdasf 13y ago

Yes, when you have a large amount of storage, it is well worth doing it yourself. The cost of redundant storage devices with replication ends up being paid for in like 3 months of S3 bills.

beachstartup 13y ago

yes. they would have 100% been better off, provided they actually knew what they were doing, which i guess is questionable since the business failed. pricing out servers and researching / working with smaller infrastructure providers who actually care about you and your account (imagine that) is real work, which is why so many people avoid it these days. it's much easier to click a few buttons and "choose" machine type x in the offerings that amazon has designed to maximize their own profit. computers are hard, let's go shopping.

ye 13y ago

Your alternatives are 1) Switch to other cloud providers, I'm not aware of any very cheap ones. 2) Switch to other hostings, which can be 1/10 as cheap as AWS, but then you have to deal with the failures. 3) Colocate and build your own servers/storage. It's only cheaper if your bill is much more than a 24/7 sysadmin(s) salary. I suspect DropBox don't give a shit about server costs, they are swimming in money. But I bet having their own data centers will save them money.