10 ms·

MVPs and $100k AWS Bills: Reflections on our launch

94 points by paulstovell 7y ago

teddyuk 7y ago
"To bring Octopus Cloud to market quickly, we did the simplest thing possible; we took our self-hosted Octopus Server product and bundled it into an EC2 instance for each customer that signed up. We had to make changes to the product, but mostly around permissions." I have used octopus and I could have told them that they can't lift-and-shift. The amount of requests the app makes is ridiculous, the way they wrote their database is appalling (actually crazily dumb https://octopus.com/blog/sql-as-document-store https://octopus.com/blog/sql-as-document-store). Everything about it would mean they can't scale in the cloud without wasting a ton of money. "Cloud stuff can be really expensive" - yes if you write dog shit, putting that in the cloud will be expensive.
- teddyuk 7y ago
  From the "sql-as-document-store" post, I re-read it every now and then to realise how actually terrible it is: "For a many-to-many, we do something a bit naughty: we store the values with | characters between each value:" Not naughty, stupid.
  - swiley 7y ago
    Why would you bother with a relational database at that point? That sounds like it’s not normalized. I haven’t read the post... maybe I’m missing something.
    - pcr0 7y ago
      For ACID guarantees?
      - jfindley 7y ago
        ACID guarantees have nothing to do with relational vs non-relational. FoundationDB is one example of a non-relational key value store that none the less provides ACID guarantees.
    - dragonwriter 7y ago
      > Why would you bother with a relational database at that point? That sounds like it’s not normalized. It's non-1NF, though they sometimes (based on expected cardinality) normalize that aspect via a trigger, using the application-accessed table as, in effect, something of a write-through denormalized materialized view. It's not an entirely unreasonablev way to use an RDBMS in and of itself.
      - teddyuk 7y ago
        except they used sql server and sql server performance with this is awful, oracle would have been able to handle it much better
        tfehring 7y ago
        It’s much better now since SQL Server 2017 added a new string split function. But I actually had to do basically the same thing in an earlier version of SQL Server (in my case, it was because I was interacting with a legacy application that spits out pipe-delimited strings) and the string parsing increased the execution time by several orders of magnitude.
    - teddyuk 7y ago
      because they didn't know what they were doing - I can say this as this is what the blog shows, they don't understand databases and customers (including me) have suffered this
      - jcadam 7y ago
        I've learned much too late in my career that competent software engineering is completely orthogonal to creating a successful(from a market perspective) software product.
        swiley 7y ago
        Well yeah most of it’s just CRUD and that’s pretty straightforward even at a large scale (ex stack overflow.)
  - brodouevencode 7y ago
    Agreed. Lifting and shifting is hard. It's harder when you do it in a careless way.
  - phamilton 7y ago
    The approach seemed reasonable enough until I hit that point.
    - time0ut 7y ago
      I was like, ya, I do that sometimes too... Wait wtf?
  - LeonM 7y ago
    > Not naughty, stupid. Why didn't they just make 1 table with 1 column and put a JSON object in it with all customer data? This would fit their design objective "Avoid the need for joins as much as possible"! It's brilliant! We'll call it: noSQL, because we don't even have to write that pesky SQL! Can't wait for the next blog post where they discover another naughty trick called primary keys to increase their NoSQL performance even more!
- eightysixfour 7y ago
  While I agree with you that it seems like they make some serious engineering mistakes, it was a successful MVP in that it told there there was sufficient demand to justify a rewrite for the cloud.
- paulstovell 7y ago
  None of this had anything to do with how the database is used.
- omouse 7y ago
  The best part is the "simplest thing possible". They actually didn't do that at all. I hate to dog-pile on this team, but damn, 100k spend in a month isn't simple at all!
notyourday 7y ago
Pardon me for asking a really dumb question: but do you have a positive item charge - CGS? Because I'm sure one can get a great demand by selling a $1.00 for $0.90 but that's not a business model.
- tapatio 7y ago
  That is Amazon's business model.
  - notyourday 7y ago
    Amazon business model: $cost_of_a_thing < $charges_of_customer. $cost_of_a_thing + $additional_expenses_allocated_to_ thing could be > $charges_of_customer Described business model: $cost_of_a_thing > $charges_of_customer. hence $cost_of_a_thing + $other_expenses_allocated_to_thing is > $charges_of_customer First one can be profitable via scale. Second cannot.
- paulstovell 7y ago
  The post didn't make this very clear. $10 was the starting price - at the time it was for 5 users. Similar to Confluence or Jira with a very cheap (in this case loss-making) starting tier. Once a customer got to ~10 users it became profitable. What we didn't know if whether there'd be enough customers for us to care about the loss on those $10 tiers. E.g., if only 100 customers signed up a month, the costs would have been around $10K/month - enough to write off as a marketing expense. Once it got to $100K it became something we needed to fix.
  - notyourday 7y ago
    > The post didn't make this very clear. $10 was the starting price - at the time it was for 5 users. Similar to Confluence or Jira with a very cheap (in this case loss-making) starting tier. Once a customer got to ~10 users it became profitable. Err.. That's not a correct way of looking at a long tail. CGS on a long tail must still be lower than the unit charge, otherwise you are paying $1.00 for $0.90 which gives you an incorrect signal about the demand. Say you make apple juice. And each juice uses 4 apples and only apples. 4 apples cost you $3. Setting a price of juice to $1.00 does not actually tell you if there's a demand for the juice because it is below the CGS, which you cannot make up in volume. The long tail test is to set the juice price at (CGS + x) where x is some number and see if you get the demand even if x is less than the other costs attributed to juice ( rent/electric/juicer/salaries). If you do, then you can make up your loses in volume because the higher the volume you are going to have the smaller process cost attributed to every unit will be so there's going to be a number at which CGS + y < CGS + x where y is the process cost attributed to a unit.
scarface74 7y ago
The lesson to be learned here, and the one they admittedly already knew in advance, is if you do a lift and shift to AWS without changing your processes to be cloud native, you will always spend more. Unfortunately, I’ve met way too many old school netops guys who got one AWS certification and all they knew was how to log into the web console and duplicate their colo infrastructure, charge their client a lot of money, and call it a day. That being said, I’ve used a lot of CI/CD tools and OctopusDeploy is by far my favorite. Even though my company is very much a “pay someone to do the undifferentiated heavy lifting” type of company, OctopusDeploy is so easy to use and maintain, I am not sure we would migrate.
- teddyuk 7y ago
  Going to the cloud won't save you money. Going to the cloud lets you scale beyond your own data centre Going to the cloud can be cheaper if you use the elasticity and make good use of PaaS services (servless,paas, etc)
  - scarface74 7y ago
    I agree, but it also lets you automate and probably lets you get rid of or not hire as many infrastructure people between automation and outsourcing the simple stuff to a managed service provider.
echelon 7y ago
I have a streaming multimedia app that I want to launch. It ingests from the client, does some transformations, then sends it back to them (or to other parties) in realtime. It's pretty novel though - Snapchat adjacent. I can't do it clientside. Nature of the problem. I'm writing it in Rust, so it's relatively fast. But throwing thousands of people at it will still require extreme scale. I'm going to use K8S horizontal autoscaling. This will be obscenely expensive. I'm not sure I can afford this. But I think a month of successful demonstration will get me funded (or bought out). So in theory my costs could wind up getting paid. Maybe. Do I risk launching this and potentially having $100k multiples of AWS bills? I can't afford that. I don't want to deplete my life savings or potentially go bankrupt. Could I put it on a corporate card as an LLC and do my best to make it work? Then fold without harm or fowl if it doesn't? Is that unethical? Could I get sued? All that said, I'm almost certain that a successful demo at scale with the Internet thrown at it will get me money. It's a really cool and novel product. People will be talking about it. What should I do? Advice will be very appreciated.
- teddyuk 7y ago
  onboard customers slowly and charge them - as you scale, make sure your charging as well and get paid. you don't need to go from 0 - 100k a month in a single month, octopus could because they have backing.
  - jermaustin1 7y ago
    octopus is a bootstrapped company
  - echelon 7y ago
    This is a social media app, so I can't charge money for it. My hope would be for funding or a buyout. Otherwise it will be shut down.
    - bananocurrency 7y ago
      >My hope would be for funding or a buyout ...you're going to get bought by a company that knows it can't make money from it?
rubbingalcohol 7y ago
This is insane. It's the engineering equivalent of smashing one of your hands with a hammer and then being surprised it hurts. Well no shit. At that level, at the very least, it would make sense to run some amount of physical infrastructure.
plasma 7y ago
Octopus is a fantastic deployment product for .NET (happy user here).
- teddyuk 7y ago
  what's it like working for paul? :)
  - plasma 7y ago
    I’m a long term (5+ years) customer of Octopus, I know Paul is the CEO, I don’t work for him, I’m a startup founder with my own company.
- remon 7y ago
  Should I reply here or to sales@octopus.com?
- redleader9345 7y ago
  I agree. I've used Octopus to automate deployments of internal LOB crud apps in the insurance space for a few years now. My questions is, is it still aimed at .NET? I know it leans on Nuget as the package repo. But have they broadened to embrace deployment of other tech stacks? Is Octopus considered best in class for what it does? Who are the primary competitors? Does Azure DevOps spell the end for Octopus?
pjc50 7y ago
> Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up. > We also didn’t have our pricing quite right. We initially launched Octopus Cloud at $10/month, with a different pricing model from the one we currently use. Unfortunately, this was one of the most painful lessons we learned because the deficit between what we were charging and spending was magnified by the sheer number of people using Octopus Cloud; continued growth would further amplify the problem. "Every person who fills in a web form gets to burn a $100 bill of ours" is the real disaster here. I know it's trendy to think that software isn't like regular business, and you can lose money but make it up on volume, but going to a cloud vendor gives you a real cost of goods sold, and that needs to be accounted for. (also, the irony of a company whose product appears to be about cloud deployment having a disaster by deploying to the cloud should not be lost on us)
- LeonM 7y ago
  Even without the free trial, how on earth can you burn $100 in cloud resources in a month if you plan to resell it for $10?
  - ignoramous 7y ago
    That's the lesson: The most obvious advice isn't obvious. If one goes through https://startupschool.org/library https://startupschool.org/library one realises how much of the advise ought to be common knowledge and common sense, but that's the thing, folks ignore it all the time. Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't.
    - LeonM 7y ago
      > Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't. Yes, I have seen that happen a few times in SuS (I'm a 2018 and 2019 batch alumni). Founders are obsessive about their pricing and as a result they usually undercharge. My usual advice to pre-market-fit stage founders is that you don't know yet what the right price is, so choose $random_price and move on, you can always adjust later. And for post-market-fit stage: raise your prices. Of course, there is no single 'right' answer on pricing. It depends on many factors, but for B2B it usually holds true that pricing isn't that much of a differentiator.
mothsonasloth 7y ago
Can't wait for this Cloud meme to die out and we can go back to On Prem but instead of VMs managed by VMWare, Cloudstack, Vagrant, LXC etc, we will have nodes managed by Kubernetes.
- bronco21016 7y ago
  Why would cloud go away? If you’re a team of 3-4 people and have a startup idea what’s easier? Create an AWS account and smartly spin up required servers? Or find a location (rent$), get fiber($$) to that location($$), find hardware($$), install said hardware. Now you’re back to where you were with AWS. For larger already established businesses on prem is probably a setup you should explore but on day 0 of a new business, cloud will probably have a large cost and flexibility advantage as long as you smartly set it up.
  - shiftpgdn 7y ago
    I can purchase and then colocate 2x Dell R640s (8c/16t, 64gb of ram, as much NVMe storage as I can stomach) with tier 1 bandwidth billed at the 95th percentile and 24/7 remote hands support for about $150/month. The purchase price on the Dells is about $3500/ea. Total first year cost on this setup: $8,800 Total three year cost on this setup: $12,400 2x m5d.4xlarge EC2 instances (this isn't counting ingress/egress rates, or really anything else) First year cost: $15,621 . Total three year cost for the EC2 instance: $46,863.36 For the cost of using EC2 you could hire an extra person to help grow your company. Yes, you can get discounts in AWS for agreeing to a three year term. Yes AWS offers you flexibility to walk away at any time. If you have a base line load (which most companies do) you should really look at colocating your owned equipment. You also get the bonus of being out of the watchful eye of Amazon's profitability team looking to see if they can copy what you're doing.
    - mping 7y ago
      I'm pretty sure colocation is cheaper if you know what you're doing (what to buy, where to colocate, what kind of bandwidth required, etc). I'm also pretty sure a good portion of cloud customers don't know or don't want to risk it.
blantonl 7y ago
The architectural decision to deploy each customer on their own segmented components to make sure that "no one user’s data to mingle with another" seems to me like a solution in search of a problem. First, the deployment was on AWS (EC2 and RDS among other services) which was probably already co-mingled with other AWS customers. Second, when you are selling a SAAS offering, the assumption is already there that you'll build security and multi-tenancy into the app and the underlying infrastructure shouldn't matter. And 100K/month for a MVP SAAS offering. I'm speechless. I just, I can't even understand how this burn rate passed muster with the leadership. But Kudos to the team for owning up to it and sharing. Lots of lessons learned here.
- Spooky23 7y ago
  It could make alot of sense for alot of government, healthcare and financial use cases.
  - blantonl 7y ago
    I agree. In those cases, sell those customer separate products and expectations and price it accordingly.
- paulstovell 7y ago
  The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum. Cloud CI platforms typically solve this by timesharing build agents, but that takes engineering effort, which is a waste of effort if there's not a lot of demand in the first place. That's why we started with the MVP.
DVassallo 7y ago
> We quickly learned that everything in AWS has some kind of service limit Except S3. Unlimited objects, and unlimited Put and Get requests.
- jcims 7y ago
  Just keep in mind if you're using SSE KMS to encrypt you are subject to limits of that service when using S3 (there are a few ways to mitigate this).
- johnnymangos 7y ago
  Your comment is not true in all cases. S3 is limited to 5,000 get/put requests per prefix. So while it's possible to achieve what you've said, it's also possible to stumble upon this limit in a fashion similar to the original comment.
  - solidasparagus 7y ago
    It looks like the number is slightly higher now - https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3... Do you know what exactly a 'prefix' means in this context? The first N characters? What is N?
deleted 7y ago
[deleted]
Havoc 7y ago
Well glad they had the capacity to recover. Even accounting for hindsight being 20/20...every customer gets their own VM is a rather ambitious approach to scaling. Would love to know what made them shift to Azure. Especially while embracing linux at same time.
- paulstovell 7y ago
  Compute is one part of it but not much different across clouds. Databases are the other big part (each customer needs their own DB). The SQL Server databases in Azure for our usage scenario turns out a lot cheaper. We’ll explain more in a future post.
omouse 7y ago
Ha, I'm re-reading The Lean Startup and this paragraph seems like a contradiction in terms of what an MVP is: We decided to build an MVP based on our best estimates and test the market that way. The goal was to launch something in 6 months and test if the demand was there; and if it wasn’t, we’d only wasted 6 months. We chose to optimize for getting to market quickly rather than worrying about how much it would cost. Contrast it to what the Lean Startup principles say: The Lean Startup methodology has as a premise that every startup is a grand experiment that attempts to answer a question. The question is not "Can this product be built?" Instead, the questions are "Should this product be built?" and "Can we build a sustainable business around this set of products and services?" So instead of doing what a lean startup would do, they did the opposite. Let's see what they could have done instead... "How many customers will actually use it?" - Lean startup: create a signup page with a mock up or landing page and a few buttons that work and every potential behind a "coming soon" screen - Their approach: actually build the whole product and then launch it and see who's interested "What is it all going to cost?" - Lean startup: As little as possible to discover the insights we need to iterate and evolve the product - Their approach: well we already paid the salary for our engineers and we've given them six months, so what's half the salary+benefits of X number of engineers? "What should we charge for it? Is it going to cover the infrastructure costs?" - Lean startup: experiment on costing and pricing as you go along - Their approach: give away the product for a 30-day trial at a high cost to ourselves! Unbelievably expensive customer acquisition costs. Reference: http://theleanstartup.com/principles http://theleanstartup.com/principles And now I guess I've truly learned why product managers exist. Someone let this crazy experiment run for six months. You're saying in six months you couldn't spend one month finding the simplest/cheapest ways to test these various hypothesis? $100k customer acquisition spend is insane. The whole post is just a lesson in contradicting what an MVP minimal viable product is. Even when we were building v1, the team knew it wasn’t the ideal architecture. Before v1 even launched, there were plenty of conversations in our Slack about whether we should port Octopus to Linux and run it on Kubernetes, or see if we could run it on Windows within Kubernetes or use Nomad by Hashicorp? Lol, engineers always want something fun to work on and sometimes it turns out to be a great idea. But they porting software for an MVP? Why not evolve the new product so that it generates more revenue or reduce the costs in some other way instead of talking about porting and rewriting? If you're at an established company that has money to burn, go ahead, read the whole blog series. If you're trying to create a new startup, whether it's a product or service, avoid this article or read it as a warning. When you've spent as much as a decent engineer costs in one month...something's gone off the rails.
TedLePoireau 7y ago
Rent dedicated server from OVH, install ESXi, hire a sysadmin. You will save huge amount of money. Cloud can be great, but sometime it looks like people forget there are other solutions.
- megaremote 7y ago
  Dedicated servers are often so much cheaper these days, but it is not the new cloud, so a lot of startups look the other way.
remon 7y ago
I'm sorry but this was a bit of a painful read. Hindsight is 20/20 but this was pretty poor engineering and business modelling from the start. "To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database, and a large number of security configurations to prevent any funny business.". AWS has a myriad of security and isolation tools to do that in significantly better and cost efficient ways. "Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up." So you're essentially allowing anyone to press a "Submit" button to spend $100 for you. This sort of below-cost methods to gain marketshare works for Amazon, Google, Microsoft and son because they have the deep pockets but it's a questionable strategy for smaller companies.
poxrud 7y ago
To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database No this will not protect you. Understanding the proper way to configure IAM roles with principle of least privilege will. The author mentions that The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum. If you're using the same IAM role for your EC2 instances, and you're allowing your customers to run arbitrary code, well now at the very least they have access to everyone's databases.

94 points by paulstovell 7y ago

"To bring Octopus Cloud to market quickly, we did the simplest thing possible; we took our self-hosted Octopus Server product and bundled it into an EC2 instance for each customer that signed up. We had to make changes to the product, but mostly around permissions." I have used octopus and I could have told them that they can't lift-and-shift. The amount of requests the app makes is ridiculous, the way they wrote their database is appalling (actually crazily dumb https://octopus.com/blog/sql-as-document-store https://octopus.com/blog/sql-as-document-store). Everything about it would mean they can't scale in the cloud without wasting a ton of money. "Cloud stuff can be really expensive" - yes if you write dog shit, putting that in the cloud will be expensive.

teddyuk 7y ago
From the "sql-as-document-store" post, I re-read it every now and then to realise how actually terrible it is: "For a many-to-many, we do something a bit naughty: we store the values with | characters between each value:" Not naughty, stupid.
- swiley 7y ago
  Why would you bother with a relational database at that point? That sounds like it’s not normalized. I haven’t read the post... maybe I’m missing something.
  - pcr0 7y ago
    For ACID guarantees?
    - jfindley 7y ago
      ACID guarantees have nothing to do with relational vs non-relational. FoundationDB is one example of a non-relational key value store that none the less provides ACID guarantees.
  - dragonwriter 7y ago
    > Why would you bother with a relational database at that point? That sounds like it’s not normalized. It's non-1NF, though they sometimes (based on expected cardinality) normalize that aspect via a trigger, using the application-accessed table as, in effect, something of a write-through denormalized materialized view. It's not an entirely unreasonablev way to use an RDBMS in and of itself.
    - teddyuk 7y ago
      except they used sql server and sql server performance with this is awful, oracle would have been able to handle it much better
      - tfehring 7y ago
        It’s much better now since SQL Server 2017 added a new string split function. But I actually had to do basically the same thing in an earlier version of SQL Server (in my case, it was because I was interacting with a legacy application that spits out pipe-delimited strings) and the string parsing increased the execution time by several orders of magnitude.
  - teddyuk 7y ago
    because they didn't know what they were doing - I can say this as this is what the blog shows, they don't understand databases and customers (including me) have suffered this
    - jcadam 7y ago
      I've learned much too late in my career that competent software engineering is completely orthogonal to creating a successful(from a market perspective) software product.
      - swiley 7y ago
        Well yeah most of it’s just CRUD and that’s pretty straightforward even at a large scale (ex stack overflow.)
- brodouevencode 7y ago
  Agreed. Lifting and shifting is hard. It's harder when you do it in a careless way.
- phamilton 7y ago
  The approach seemed reasonable enough until I hit that point.
  - time0ut 7y ago
    I was like, ya, I do that sometimes too... Wait wtf?
- LeonM 7y ago
  > Not naughty, stupid. Why didn't they just make 1 table with 1 column and put a JSON object in it with all customer data? This would fit their design objective "Avoid the need for joins as much as possible"! It's brilliant! We'll call it: noSQL, because we don't even have to write that pesky SQL! Can't wait for the next blog post where they discover another naughty trick called primary keys to increase their NoSQL performance even more!
eightysixfour 7y ago
While I agree with you that it seems like they make some serious engineering mistakes, it was a successful MVP in that it told there there was sufficient demand to justify a rewrite for the cloud.
paulstovell 7y ago
None of this had anything to do with how the database is used.
omouse 7y ago
The best part is the "simplest thing possible". They actually didn't do that at all. I hate to dog-pile on this team, but damn, 100k spend in a month isn't simple at all!

notyourday 7y ago

Pardon me for asking a really dumb question: but do you have a positive item charge - CGS? Because I'm sure one can get a great demand by selling a $1.00 for $0.90 but that's not a business model.

tapatio 7y ago
That is Amazon's business model.
- notyourday 7y ago
  Amazon business model: $cost_of_a_thing < $charges_of_customer. $cost_of_a_thing + $additional_expenses_allocated_to_ thing could be > $charges_of_customer Described business model: $cost_of_a_thing > $charges_of_customer. hence $cost_of_a_thing + $other_expenses_allocated_to_thing is > $charges_of_customer First one can be profitable via scale. Second cannot.
paulstovell 7y ago
The post didn't make this very clear. $10 was the starting price - at the time it was for 5 users. Similar to Confluence or Jira with a very cheap (in this case loss-making) starting tier. Once a customer got to ~10 users it became profitable. What we didn't know if whether there'd be enough customers for us to care about the loss on those $10 tiers. E.g., if only 100 customers signed up a month, the costs would have been around $10K/month - enough to write off as a marketing expense. Once it got to $100K it became something we needed to fix.
- notyourday 7y ago
  > The post didn't make this very clear. $10 was the starting price - at the time it was for 5 users. Similar to Confluence or Jira with a very cheap (in this case loss-making) starting tier. Once a customer got to ~10 users it became profitable. Err.. That's not a correct way of looking at a long tail. CGS on a long tail must still be lower than the unit charge, otherwise you are paying $1.00 for $0.90 which gives you an incorrect signal about the demand. Say you make apple juice. And each juice uses 4 apples and only apples. 4 apples cost you $3. Setting a price of juice to $1.00 does not actually tell you if there's a demand for the juice because it is below the CGS, which you cannot make up in volume. The long tail test is to set the juice price at (CGS + x) where x is some number and see if you get the demand even if x is less than the other costs attributed to juice ( rent/electric/juicer/salaries). If you do, then you can make up your loses in volume because the higher the volume you are going to have the smaller process cost attributed to every unit will be so there's going to be a number at which CGS + y < CGS + x where y is the process cost attributed to a unit.

scarface74 7y ago

The lesson to be learned here, and the one they admittedly already knew in advance, is if you do a lift and shift to AWS without changing your processes to be cloud native, you will always spend more. Unfortunately, I’ve met way too many old school netops guys who got one AWS certification and all they knew was how to log into the web console and duplicate their colo infrastructure, charge their client a lot of money, and call it a day. That being said, I’ve used a lot of CI/CD tools and OctopusDeploy is by far my favorite. Even though my company is very much a “pay someone to do the undifferentiated heavy lifting” type of company, OctopusDeploy is so easy to use and maintain, I am not sure we would migrate.

teddyuk 7y ago
Going to the cloud won't save you money. Going to the cloud lets you scale beyond your own data centre Going to the cloud can be cheaper if you use the elasticity and make good use of PaaS services (servless,paas, etc)
- scarface74 7y ago
  I agree, but it also lets you automate and probably lets you get rid of or not hire as many infrastructure people between automation and outsourcing the simple stuff to a managed service provider.

echelon 7y ago

I have a streaming multimedia app that I want to launch. It ingests from the client, does some transformations, then sends it back to them (or to other parties) in realtime. It's pretty novel though - Snapchat adjacent. I can't do it clientside. Nature of the problem. I'm writing it in Rust, so it's relatively fast. But throwing thousands of people at it will still require extreme scale. I'm going to use K8S horizontal autoscaling. This will be obscenely expensive. I'm not sure I can afford this. But I think a month of successful demonstration will get me funded (or bought out). So in theory my costs could wind up getting paid. Maybe. Do I risk launching this and potentially having $100k multiples of AWS bills? I can't afford that. I don't want to deplete my life savings or potentially go bankrupt. Could I put it on a corporate card as an LLC and do my best to make it work? Then fold without harm or fowl if it doesn't? Is that unethical? Could I get sued? All that said, I'm almost certain that a successful demo at scale with the Internet thrown at it will get me money. It's a really cool and novel product. People will be talking about it. What should I do? Advice will be very appreciated.

teddyuk 7y ago
onboard customers slowly and charge them - as you scale, make sure your charging as well and get paid. you don't need to go from 0 - 100k a month in a single month, octopus could because they have backing.
- jermaustin1 7y ago
  octopus is a bootstrapped company
- echelon 7y ago
  This is a social media app, so I can't charge money for it. My hope would be for funding or a buyout. Otherwise it will be shut down.
  - bananocurrency 7y ago
    >My hope would be for funding or a buyout ...you're going to get bought by a company that knows it can't make money from it?

rubbingalcohol 7y ago

This is insane. It's the engineering equivalent of smashing one of your hands with a hammer and then being surprised it hurts. Well no shit. At that level, at the very least, it would make sense to run some amount of physical infrastructure.

plasma 7y ago

Octopus is a fantastic deployment product for .NET (happy user here).

teddyuk 7y ago
what's it like working for paul? :)
- plasma 7y ago
  I’m a long term (5+ years) customer of Octopus, I know Paul is the CEO, I don’t work for him, I’m a startup founder with my own company.
remon 7y ago
Should I reply here or to sales@octopus.com?
redleader9345 7y ago
I agree. I've used Octopus to automate deployments of internal LOB crud apps in the insurance space for a few years now. My questions is, is it still aimed at .NET? I know it leans on Nuget as the package repo. But have they broadened to embrace deployment of other tech stacks? Is Octopus considered best in class for what it does? Who are the primary competitors? Does Azure DevOps spell the end for Octopus?

pjc50 7y ago

> Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up. > We also didn’t have our pricing quite right. We initially launched Octopus Cloud at $10/month, with a different pricing model from the one we currently use. Unfortunately, this was one of the most painful lessons we learned because the deficit between what we were charging and spending was magnified by the sheer number of people using Octopus Cloud; continued growth would further amplify the problem. "Every person who fills in a web form gets to burn a $100 bill of ours" is the real disaster here. I know it's trendy to think that software isn't like regular business, and you can lose money but make it up on volume, but going to a cloud vendor gives you a real cost of goods sold, and that needs to be accounted for. (also, the irony of a company whose product appears to be about cloud deployment having a disaster by deploying to the cloud should not be lost on us)

LeonM 7y ago
Even without the free trial, how on earth can you burn $100 in cloud resources in a month if you plan to resell it for $10?
- ignoramous 7y ago
  That's the lesson: The most obvious advice isn't obvious. If one goes through https://startupschool.org/library https://startupschool.org/library one realises how much of the advise ought to be common knowledge and common sense, but that's the thing, folks ignore it all the time. Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't.
  - LeonM 7y ago
    > Another mistake, it was pointed out by Kevin Hale, iirc, this year at startupschool, that founders make is they think their pricing model (or their UX) is a key differentiator, when in most cases it isn't. Yes, I have seen that happen a few times in SuS (I'm a 2018 and 2019 batch alumni). Founders are obsessive about their pricing and as a result they usually undercharge. My usual advice to pre-market-fit stage founders is that you don't know yet what the right price is, so choose $random_price and move on, you can always adjust later. And for post-market-fit stage: raise your prices. Of course, there is no single 'right' answer on pricing. It depends on many factors, but for B2B it usually holds true that pricing isn't that much of a differentiator.

mothsonasloth 7y ago

Can't wait for this Cloud meme to die out and we can go back to On Prem but instead of VMs managed by VMWare, Cloudstack, Vagrant, LXC etc, we will have nodes managed by Kubernetes.

bronco21016 7y ago
Why would cloud go away? If you’re a team of 3-4 people and have a startup idea what’s easier? Create an AWS account and smartly spin up required servers? Or find a location (rent$), get fiber($$) to that location($$), find hardware($$), install said hardware. Now you’re back to where you were with AWS. For larger already established businesses on prem is probably a setup you should explore but on day 0 of a new business, cloud will probably have a large cost and flexibility advantage as long as you smartly set it up.
- shiftpgdn 7y ago
  I can purchase and then colocate 2x Dell R640s (8c/16t, 64gb of ram, as much NVMe storage as I can stomach) with tier 1 bandwidth billed at the 95th percentile and 24/7 remote hands support for about $150/month. The purchase price on the Dells is about $3500/ea. Total first year cost on this setup: $8,800 Total three year cost on this setup: $12,400 2x m5d.4xlarge EC2 instances (this isn't counting ingress/egress rates, or really anything else) First year cost: $15,621 . Total three year cost for the EC2 instance: $46,863.36 For the cost of using EC2 you could hire an extra person to help grow your company. Yes, you can get discounts in AWS for agreeing to a three year term. Yes AWS offers you flexibility to walk away at any time. If you have a base line load (which most companies do) you should really look at colocating your owned equipment. You also get the bonus of being out of the watchful eye of Amazon's profitability team looking to see if they can copy what you're doing.
  - mping 7y ago
    I'm pretty sure colocation is cheaper if you know what you're doing (what to buy, where to colocate, what kind of bandwidth required, etc). I'm also pretty sure a good portion of cloud customers don't know or don't want to risk it.

blantonl 7y ago

The architectural decision to deploy each customer on their own segmented components to make sure that "no one user’s data to mingle with another" seems to me like a solution in search of a problem. First, the deployment was on AWS (EC2 and RDS among other services) which was probably already co-mingled with other AWS customers. Second, when you are selling a SAAS offering, the assumption is already there that you'll build security and multi-tenancy into the app and the underlying infrastructure shouldn't matter. And 100K/month for a MVP SAAS offering. I'm speechless. I just, I can't even understand how this burn rate passed muster with the leadership. But Kudos to the team for owning up to it and sharing. Lots of lessons learned here.

Spooky23 7y ago
It could make alot of sense for alot of government, healthcare and financial use cases.
- blantonl 7y ago
  I agree. In those cases, sell those customer separate products and expectations and price it accordingly.
paulstovell 7y ago
The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum. Cloud CI platforms typically solve this by timesharing build agents, but that takes engineering effort, which is a waste of effort if there's not a lot of demand in the first place. That's why we started with the MVP.

DVassallo 7y ago

> We quickly learned that everything in AWS has some kind of service limit Except S3. Unlimited objects, and unlimited Put and Get requests.

jcims 7y ago
Just keep in mind if you're using SSE KMS to encrypt you are subject to limits of that service when using S3 (there are a few ways to mitigate this).
johnnymangos 7y ago
Your comment is not true in all cases. S3 is limited to 5,000 get/put requests per prefix. So while it's possible to achieve what you've said, it's also possible to stumble upon this limit in a fashion similar to the original comment.
- solidasparagus 7y ago
  It looks like the number is slightly higher now - https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3... Do you know what exactly a 'prefix' means in this context? The first N characters? What is N?

deleted 7y ago

[deleted]

Havoc 7y ago

Well glad they had the capacity to recover. Even accounting for hindsight being 20/20...every customer gets their own VM is a rather ambitious approach to scaling. Would love to know what made them shift to Azure. Especially while embracing linux at same time.

paulstovell 7y ago
Compute is one part of it but not much different across clouds. Databases are the other big part (each customer needs their own DB). The SQL Server databases in Azure for our usage scenario turns out a lot cheaper. We’ll explain more in a future post.

omouse 7y ago

Ha, I'm re-reading The Lean Startup and this paragraph seems like a contradiction in terms of what an MVP is: We decided to build an MVP based on our best estimates and test the market that way. The goal was to launch something in 6 months and test if the demand was there; and if it wasn’t, we’d only wasted 6 months. We chose to optimize for getting to market quickly rather than worrying about how much it would cost. Contrast it to what the Lean Startup principles say: The Lean Startup methodology has as a premise that every startup is a grand experiment that attempts to answer a question. The question is not "Can this product be built?" Instead, the questions are "Should this product be built?" and "Can we build a sustainable business around this set of products and services?" So instead of doing what a lean startup would do, they did the opposite. Let's see what they could have done instead... "How many customers will actually use it?" - Lean startup: create a signup page with a mock up or landing page and a few buttons that work and every potential behind a "coming soon" screen - Their approach: actually build the whole product and then launch it and see who's interested "What is it all going to cost?" - Lean startup: As little as possible to discover the insights we need to iterate and evolve the product - Their approach: well we already paid the salary for our engineers and we've given them six months, so what's half the salary+benefits of X number of engineers? "What should we charge for it? Is it going to cover the infrastructure costs?" - Lean startup: experiment on costing and pricing as you go along - Their approach: give away the product for a 30-day trial at a high cost to ourselves! Unbelievably expensive customer acquisition costs. Reference: http://theleanstartup.com/principles http://theleanstartup.com/principles And now I guess I've truly learned why product managers exist. Someone let this crazy experiment run for six months. You're saying in six months you couldn't spend one month finding the simplest/cheapest ways to test these various hypothesis? $100k customer acquisition spend is insane. The whole post is just a lesson in contradicting what an MVP minimal viable product is. Even when we were building v1, the team knew it wasn’t the ideal architecture. Before v1 even launched, there were plenty of conversations in our Slack about whether we should port Octopus to Linux and run it on Kubernetes, or see if we could run it on Windows within Kubernetes or use Nomad by Hashicorp? Lol, engineers always want something fun to work on and sometimes it turns out to be a great idea. But they porting software for an MVP? Why not evolve the new product so that it generates more revenue or reduce the costs in some other way instead of talking about porting and rewriting? If you're at an established company that has money to burn, go ahead, read the whole blog series. If you're trying to create a new startup, whether it's a product or service, avoid this article or read it as a warning. When you've spent as much as a decent engineer costs in one month...something's gone off the rails.

TedLePoireau 7y ago

Rent dedicated server from OVH, install ESXi, hire a sysadmin. You will save huge amount of money. Cloud can be great, but sometime it looks like people forget there are other solutions.

megaremote 7y ago
Dedicated servers are often so much cheaper these days, but it is not the new cloud, so a lot of startups look the other way.

remon 7y ago

I'm sorry but this was a bit of a painful read. Hindsight is 20/20 but this was pretty poor engineering and business modelling from the start. "To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database, and a large number of security configurations to prevent any funny business.". AWS has a myriad of security and isolation tools to do that in significantly better and cost efficient ways. "Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up." So you're essentially allowing anyone to press a "Submit" button to spend $100 for you. This sort of below-cost methods to gain marketshare works for Amazon, Google, Microsoft and son because they have the deep pockets but it's a questionable strategy for smaller companies.

poxrud 7y ago

To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database No this will not protect you. Understanding the proper way to configure IAM roles with principle of least privilege will. The author mentions that The issue here is Octopus is like a CI server - customers can run arbitrary code in the instance (not a problem most SaaS apps have). So a VM per customer was almost the bare minimum. If you're using the same IAM role for your EC2 instances, and you're allowing your customers to run arbitrary code, well now at the very least they have access to everyone's databases.