19 ms·
Principles for Building New SaaS Products on AWS
- indymike 6y agoSome really good advice here. Especially the "build as if you are going to sell at any moment" and "build as if you are going to open source at any moment".
- jackgill 6y agoI'm interested in hearing opinions about this principle in the context of data engineering: > This also means leaning heavily into all the service offerings and orchestration tooling that is afforded to you by your platform. I've built a data lake and several ETL pipelines using AWS native services (Kinesis, Lambda, Athena). It works but it's a bit...fiddly. I spend a lot of time configuring these services and handling various failure modes. I've been wondering if I should be looking at third party vendors like Fivetran or Matillion for ETL. Does anyone who's worked with AWS data engineering services have thoughts on the trade-off between AWS native services and third party vendors in this area?
- ElFitz 6y agoRegarding the configuration and failure modes, I think something like CDK could be a great way to set all this up in a more familiar and readable way https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
- ramraj07 6y agoI can strongly attest to snowflake. I regret AWS doesn't offer the same features without making us jump through the maze of services with which we can emulate the same concept.
- jackgill 6y agoThanks for sharing, I've heard many good things about Snowflake. In the past I've seen them as more of a Redshift competitor (data warehouse, as opposed to a data lake) but if they can simplify data ingest then I am definitely interested.
- ramraj07 6y agoThey're fundamentally different only in the model of decoupling storage and compute completely, but in a far more simpler way than redshift spectrum I feel like. Some of their features like zero-copy-clone are just not possible in AWS and make it extremely simple to do pipeline management in way that (at least to me) makes the most sense. It's also the most democratizable model I have seen - anyone who knows the slightest amount of SQL can be set up to explore the data in minutes. The elephant in the room is that you need to use SQL. Their spark connecters are as of now useless, so you either have to go with DBT, some homebrew SQL stringing mess or something like sqlalchemy. We're currently developing some wrappers around sqlalchemy to make this a bit less painful, but it's still so worth it.
- taylorwc 6y agoI found myself nodding agreement until I reached the Cloud9 part... I think I can count on one hand the number of professional devs I know that use Cloud9 as their primary editor/IDE. Curious if I'm in a bubble on that front?
- k__ 6y agoI use it quite often, but as a tech-blogger, I'm an outlier. I write about AWS products and it's quite nice to get an IDE preinstalled with AWS CLI tools with the push of a button. For JS it's mostly an okay-ish IDE, not as good as VSCode, but okay. For things like Rust or Reason it sucks quite much.
- underbluewaters 6y agoYes! Slowing down all software development seems like an overreaction to the risk of dev/prod environment differences. Good architecture design, modular components, and isolated unit tests should really minimize the number of times this will become a problem. I can't imagine giving up the productivity improvements of insert-fav-ide-here to address unforeseen or even hypothetical bugs.
- indymike 6y agoYou probably aren't in a bubble, but the whole edit through the browser, cloud-native editors are really becoming interesting. The interesting part is more the integration of dev, testing and deployment into the editor in a way you just can't do without a lot of devops work with a traditional IDE.
- Icer5k 6y agoI don't think you're in a bubble, but we've recently started evaluating Coder and have found the switch to a cloud-based IDE (especially when you add Progressive Web Apps to the mix for native keyboard shortcuts) has been extremely attractive. I'm finding myself more and more drawn to hosted IDEs where I don't need to worry about network performance or how the Docker VM on my Mac is eating up my battery life...
- fenwick67 6y agoIME Cloud 9 is helpful for building applications where you need public endpoints (like if you're using OAuth) or in cases where you want a Linux environment but are on windows/OSX/chromeOS. But I would pass for everyday use.
- eugenekolo 6y agoNot sure how much I can trust "AWS Gurus" about actually building SaaS products in a real world at a real company. It's all good advice, but please provide me with the money, and resources necessary to do it all.
- ComodoHacker 6y agoThis sounds like an excellent guide to a perfect vendor lock-in. It's just missing the final principle: #4 Operate as if you may be bought or cloned by Amazon at any time.
- sl1ck731 6y agoUnless your entirely Kubernetes (and even then there are some sticky points), using a cloud of any kind already has you "locked in" with some of the most basic things like IAM. If you are going into a cloud you may as well reap the benefits of the premium you're paying.
- mrkurt 6y agoThat's true to varying degrees. Sure you're locked in when you use ALB, but it's not too hard to replace that with HAProxy. Same with RDS to Postgres or Fargate to just running an app on your own server. In general, if you build apps with open source runtimes, use open source dbs, and avoid the proprietary data services cloud providers really stick you to, you can move around pretty easily. And you still reap most of the benefits. The real problem with the special AWS services is you end up having to hire AWS ops people or expensive consultants to architect around them and run them for you. So it's proprietary AND eating up salaries.
- scarface74 6y agoWhat benefit do you get out of going to a cloud provider instead of a cheap VPS solution or a colo if you’re not using any of their managed services?
- mrkurt 6y agoThey have managed services that are relatively easy to replace and managed services that are entirely proprietary. Many of them map to reasonable OSS tools. ALB/RDS/Fargate and even just ec2 + vpcs are super powerful, and replacing them is a known quantity. But stuff like SQS and (to a lesser extent) Lambda is really hard to replace because it's thoroughly baked into an application architecture. As an example, we (fly.io) have a tool that'll hoist a Fargate app into our infrastructure and let you run it all over the world. We even have people tunneling back into their VPCs to access other AWS services. But that only works because we're both somewhat standard, Fargate takes a Docker image and runs it, Fly takes a Docker image and runs it, the app inside doesn't care about either of us.
- Mizza 6y agoThis is blogspam.
- unethical_ban 6y agoIt isn't blogspam just because it doesn't interest you.
- Mizza 6y agoIt's a "3 tips" promotional listicle article on a company's website. It's the very definition of blogspam. I am interested in the top which is why I clicked. It's blogspam.
- gfodor 6y agoWhat’s the right granularity to shard aws accounts? I haven’t gone down this road. Is it madness to consider this as a multi tenant mechanism vs the usual foreign keys in the database approach? Is applying cloudformation across thousands of accounts feasible?
- mrkurt 6y agoI feel like the harder part of that would be migrations on databases and application code deploys. CloudFormation to configure VPCs would probably work fine. If you're going to do that, though, you may as well do CloudFormation into customer VPCs and let them do all the "paying Amazon".
- shortj 6y agoProbably the most useful mechanism I have for determining this is “if this AWS account disappears, how screwed am I / can I recover.” I tend to separate all of my projects/services, and each of those to environments. A cold storage AWS account, audit and security (ship logs, config changes, etc), shared services to another account. If dev account gets hacked, that sucks, but we can clear it out. It prod gets hacked (and deleted!) that super sucks. But hopefully cold storage and audit accounts can help us out. If some other services/projects account gets hacked, I don’t want to be worried about impact to unrelated projects.
- gfodor 6y agoNice approach - for cold storage what do you mean exactly? Manually rsynced backups or something? Most aws services I’ve used that have backups built in I don’t recall having cross account writability.
- shortj 6y agoRDS snapshot copying, EBS snapshot copies, S3 cross account bucket replication, etc. Write only with no entry points into that account from your other accounts. (Preferably its own locked down IAM role with MFA required)
- wiradikusuma 6y ago"I'd ask most developers to start their day in Cloud9" -- hmm, no. Most developers I know have powerful computers (many of them are gamers), so it's wasteful not to maximize the ROI. Also, not everyone has fast/reliable internet all the time :)
- shortj 6y agoAuthor here. Similar reply to one I did below, but a significant amount of developers I work with in enterprise or corporate contexts don’t have this similar situation. Cloud9 can be fairly liberating for them short term, especially while learning the ropes of AWS. The majority of HN readership I’d encourage to continue using their own tooling, you’ve got fast internet, unrestricted access and powerful equipment. That all said, I default to Cloud9 these days just so I can bounce around machines and have a consistent dev environment when I need it. A lot of my daily job is meeting teams where they are and helping them be productive fast as possible so I need to stay semi-fluent in most operating systems.
- cj 6y agoHave you tried Linux Workspaces? (Compared to Cloud9, I greatly prefer Workspaces, but still use Cloud9 on occasion for a few niche use cases)
- shortj 6y agoYep! They work great in many situations. However, Cloud9 is quite a bit more usable and stable on something like shaky/inconsistent airplane wifi. It’s also way less friction to setup and tear down 3 or 4 Cloud9 instances in a day compared to workspaces. I treat Cloud9 like any other ephemeral editor process. Need a new editor window? Cloud9 project. Done for the day? Commit everything I care about. Tear it down. That said, I frequently spin up Windows workspaces to test software or workflows if I’m writing a guide or content.
- ignoramous 6y ago> For instance, AWS doesn't have anything quite as tuned to fast frontend search experiences like Algolia. https://aws.amazon.com/kendra/ https://aws.amazon.com/kendra/ ?
- the_resistence 6y agoThanks for this perspective. It really helps the noobs trying to build more than single user stuff.
- jcims 6y agoThese get a little boutique, but if you want to attract and close large enterprise/regulated customers, I'd extend a few of yours: - Build your application to suit many customers or one. Large customers love to have their instance run in a dedicated account. - Open source your operations. Large customers love to see logs/operational activity from their environments (e.g. cc: cloudtrail/config/cloudwatch logs to customer, this assumes dedicated account) - Open source your security. Be prepared to ship guardduty, config rules, etc to your customer (this assumes dedicated account). Also, since we're focusing on AWS here - Build your application to support hybrid cloud customers. Expose it through private link, VPC connections, transit gw, firehose, api gateways, VPC lambdas, whatever is appropriate for your architecture. - Leverage IAM as much as possible for authentication/authorization. Not AWS-specific but implement SSO and assume customers will require numerous instances of your service when developing your user account model. - Leverage KMS as much as possible to protect data at rest, and support customer CMKs. There are more but I'll stop here.
- simonebrunozzi 6y agoWhat a lame article. I don't find any utility in any of the principles expressed here. Also, in my experience (at AWS 2008-2014, before/after in the same industry), I can't recall instances of companies that either mentioned, or followed, these principles. Specific critiques: > #1 Build as if you may sell at any time > ... it forces you to build with best practices and isolation. The opposite. It gives you an incentive to postpone technical debt, you want to grow and be acquired at the expense of whoever is going to integrate your startup into $bigco later. Side note: "AWS Organizations" to me is simply a way for AWS to try to cover for the poor design choices of the organizational structure of an AWS account, and the unnecessary complications related to billing and metering. Try to understand the AWS bill of a sufficiently large company - you won't. The AWS rep won't. The AWS Solutions Architect won't. Also, never heard of a company being acquired at a higher price because they had a "proper" AWS setup. Ah, forgot this: if your acquirer is using MS Azure, good luck telling them that you are using Cloud9 or other AWS-specific stuff. Let's continue... > #2 Build as if you may open-source at any time Yes. In an ideal world. In practice, almost nobody follows best practices. Because there's always some urgency that takes precedence. This is why companies like Accenture, PwC, Deloitte, etc, keep billing monstrous amount of money to help large companies "migrate" or "evolve" or "adapt" or whatever buzzword they use. > #3 Build with a cloud-native mindset > ... going outside of the platform should be an exception and something you do only when truly needed > My thinking on serverless these days in order of consideration. > - If the platform has it, use it > - If the market has it, buy it Serverless, really? A promising, cutting-edge technology, sure. You want to bet your startup on Lambda? Go ahead. Lambda has been around for ~6 years now (I even tried a super early version internally before it was released), and I still haven't seen a large company doing A LOT of development on Lambda. Most project using Lambda are small, confined, isolated projects and/or teams. Cloud native is great, WHEN it makes sense. I don't like religion too much, and I don't like religious people either; the author seem to have taken his faith in AWS too seriously.
- bubba1236 6y agoIt seems you left AWS in 2014 and so your impression is everything is like it was then. Lambda is used by everyone and it's great when it fits for a startup. don't understand the bet your startup part, are you insinuating lambda will shut down
- peterwwillis 6y agoImmutable Infrastructure is more important than Infrastructure as Code, fwiw. The latter just means "it's in [version controlled] code". This has a variety of use cases, and it might mean you end up in a quagmire of complexity. It's become a cargo cult thing where I've seen people adopt horribly complex, fragile solutions over practical ones "because IaC". The former is a principle that basically has no downside, and only improves operational integrity. Even if you're literally deploying everything by clicking in the Console, it's still massively more reliable, repeatable, and recoverable as immutable artifacts. The next thing I'd recommend before investing heavily in IaC is auto-recovery. The most obvious example is Autoscaling Groups, but any health check combined with an automatic action such as restart or re-deploy can work. This works best with Immutable Infrastructure, and typically does not require IaC.