6 ms·

At this point AWS has gotten so complex that one has to jump through a dozen hoops to get the simplest of things done. To scale up our reporting I thought it’s

At this point AWS has gotten so complex that one has to jump through a dozen hoops to get the simplest of things done. To scale up our reporting I thought it’s a good idea to offload report generation to a lambda function. Read from MySQL, dump into S3 and mail a link to that S3 file, make the link expire after couple of hours. Reasonably simple enough. But by God was it painful!! IAM is a rabbit hole, now later VPC restrictions on top and couple that with terribly organized document it makes no fun at all. I get the need for all those layers of security but without well organized document everyone has to go through the ordeal. What we need now is plethora of how-tos for these common architectural patterns.

fakedang 5y ago
There really ought to be sandbox "default" settings or templates that Firebase has for noobs, instead of configuring everything change by change. I still dread configuring IAM and now just resort to using one big EC2 instead of multiple services.
fridif 5y ago
A good starting point is elastic beanstalk, at least, but yes it is a big tangled web and one wonders why one is trying to copy netflix for a simple web app most of the time if yoyr workload isnt simple though, then its time to pay the price until a better abstraction comes along ^.^
noir_lord 5y ago
This mirrors my experience working with Lambda's - I went the "deploy docker images as lambda's" route and that was mostly OK (I mean it's just node/sharp etc) but the actual setup up - good god, I swear some of that documentation if read aloud would summon Cthulu.
- Aeolun 5y ago
  I recently did this with API gateway v2. The documentation basically stops at ‘there is this field, and it is a string’. I mean, fantastic, but what does this resource even represent, and what do I put in the string? It’s clearly an enum.
yepthatsreality 5y ago
Lock-in 101, build a system just as complex but tell the end user it’s easier.
whimsicalism 5y ago
I think avoiding the console, getting a solid understanding of IAM and cloudformation are a good basis to build on. AWS can definitely be poorly documented and buggy at times.
- Lutger 5y ago
  It can be if you haven't tried any other of the big cloud vendors. In my experience, it's one of the best documented services I have ever used, especially considering it's so vast.
sackerhews 5y ago
I felt your pain when I read that. If it's of any help, I can promise you that creating a Terraform configuration to set this up will improve your life, happiness and general well being in the future. Yes, Terraform has a learning curve, but it's also a black magic effery for the cloud.
- misiti3780 5y ago
  second this, terraform is a game changer if you are using aws (or any of the other clouds, it supports multiple backends)
  - Aeolun 5y ago
    And if you plan to use Terraform, you might as well skip that step and go directly to Pulumi.
    - sackerhews 5y ago
      Hey, hadn't that a look before, thanks! Yes, Terraform/HCL can sometimes feel like the instructions for the holy hand grenade.
jrockway 5y ago
Sometimes I wonder if the old world where you had to buy computers and drive to the datacenter at 3 in the morning to replace dead hard drives was really so bad. I guess it was really so bad. Everything you have to build for yourself in the cloud is something you probably needed in your self-hosted stack -- a directory of users, permissions (and escalation), firewall rules, SSH key material, etc. I feel like no matter what production stack you choose, you're always in that "6 of one, half-dozen of another" state. Lots of stuff to interfere with getting your code out there and keeping it running. (My real question is why the cloud providers don't offer these things for you -- they give you the tools to build solutions, not solutions. But I just want to work on my app. I wonder why things like App Engine never really caught on. I guess being opinionated drives more people away than it brings in with the promise of more simplicity, though to some extent App Engine was too simple, which is always bad. Meanwhile, what cloud providers offer certainly lets you bring your own opinion and build it out yourself. I wish there was a comfortable middle ground.)
- bigyikes 5y ago
  Haven’t used it in years, but Heroku was really great in this respect. It was very easy to just focus on making a good app instead of trying to configure everything myself. I guess there’s probably not much demand for this kind of thing at large scales.
  - hnzix 5y ago
    Heroku can be a hard sell to mgmt because of the cost, but if it's framed in terms of "this means you don't need to pay for two AWS cloud architects to be on call" then it's a much easier pitch.
- jfoster 5y ago
  I think the primary concerns with what App Engine started out as were flexibility and vendor lock-in. Feels like it is a very solvable problem, though. Only a matter of time.
  - topkeks 5y ago
shard 5y ago
Not knowing AWS, this sub-thread reads like a real-life version of the Microservices sketch: https://youtu.be/y8OnoxKotPQ https://youtu.be/y8OnoxKotPQ
- corpMaverick 5y ago
  LOL. That is not a sketch. I swear it is real life in the back end world. The good news is that now we have Micro front ends for the enjoyment of the UI engineers.
- GSGBen 5y ago
  For anyone skimming the comments and considering skipping this video: do yourself a favor and don't. Krazam's tech comedy is unparalleled.
  - arolihas 5y ago
    The day in the life one was amazing as well.
    - celim307 5y ago
      It hit too close to home. It described me and everyone I knew age 22 - 30 in tech to a T. We really aren’t as original as we’d like to believe
      - arolihas 5y ago
        The lack of originality hurts I guess. What was more biting was the faux productivity, tech addiction, and social isolation and alienation of such a mundane daily routine.
dzonga 5y ago
I ran into the same problem. kicked out s3 and ended up using seaweedfs. couldn't be happier. coding instead of reading aws documentation.
riffic 5y ago
I always tell people "stay off AWS unless you or someone you're paying very well on your team holds an Associate-level certification at the very minimum". It's easy to get in over your head over there.
- MonaroVXR 5y ago
  > you or someone you're paying very well on your team holds an Associate-level certification at the very minimum". It's easy to get in over your head over there. I will never forget the story about the expert Amazon engineer that lost 80K USD.
  - drevil-v2 5y ago
    Sounds like an interesting read. Link please.
    - MonaroVXR 5y ago
      https://news.ycombinator.com/item?id=22719573 https://news.ycombinator.com/item?id=22719573
properdine 5y ago
Totally get what you mean. I'll pitch for either terraform or AWS CDK as good ways to make some of this plumbing less painful (they both have you describe the end state you want and then generate resources to get there).
- jachee 5y ago
  Terraform is—as a colleague described it—a combination very-sharp-knife and fully-loaded foot-seeking gun. Very powerful, but also requires a sufficient amount of respect and care. e.g. You’ve defined a nice, fully-functional end-state, which was generated n time ago, but now you need to make a change to that state, so you run your terraform plan …and it wants to destroy your entire infrastructure and recreate it all from scratch just to e.g. add a new hostname to the certs generated by certmanager. That may be okay, but you should probably measure at least 3 times before you cut.
KptMarchewa 5y ago
>IAM is a rabbit hole, now later VPC restrictions on top My idea is to build simpler system on top of cloud networking and IAM/permissions. Why would I want to have networking between components that are permissioned to interact with each other, and the other way, why some app/lambda's service account have permission to interact with particular database, but be blacklisted on firewall? Have GUI on top to drag and drop "routes" between apps. Of course, make it optional etc, but I think it would save tons of devops time for smaller organisations without audit requirements unlike fintechs, medical stuff etc.
igotroot 5y ago
I totally agree with you. A part of my duties is setting up logging from AWS to Splunk and there's so many gotchas and stumbling blocks that's it's infuriating to even attempt and follow best practices. A great example is Cloudtrail. The best practice is to send your OrgTrail (an org-wide Cloudtrail) to a separate account for security reasons. Cool. Okay, sounds easy enough. The Splunk docs are useless for AWS, so consulting with YouTube, Reddit, etc is the go-to for this. It's so much easier to just leave the OrgTrail logging into the management account it's not even funny.

fakedang 5y ago

There really ought to be sandbox "default" settings or templates that Firebase has for noobs, instead of configuring everything change by change. I still dread configuring IAM and now just resort to using one big EC2 instead of multiple services.

fridif 5y ago

A good starting point is elastic beanstalk, at least, but yes it is a big tangled web and one wonders why one is trying to copy netflix for a simple web app most of the time if yoyr workload isnt simple though, then its time to pay the price until a better abstraction comes along ^.^

noir_lord 5y ago

This mirrors my experience working with Lambda's - I went the "deploy docker images as lambda's" route and that was mostly OK (I mean it's just node/sharp etc) but the actual setup up - good god, I swear some of that documentation if read aloud would summon Cthulu.

Aeolun 5y ago
I recently did this with API gateway v2. The documentation basically stops at ‘there is this field, and it is a string’. I mean, fantastic, but what does this resource even represent, and what do I put in the string? It’s clearly an enum.

yepthatsreality 5y ago

Lock-in 101, build a system just as complex but tell the end user it’s easier.

whimsicalism 5y ago

I think avoiding the console, getting a solid understanding of IAM and cloudformation are a good basis to build on. AWS can definitely be poorly documented and buggy at times.

Lutger 5y ago
It can be if you haven't tried any other of the big cloud vendors. In my experience, it's one of the best documented services I have ever used, especially considering it's so vast.

sackerhews 5y ago

I felt your pain when I read that. If it's of any help, I can promise you that creating a Terraform configuration to set this up will improve your life, happiness and general well being in the future. Yes, Terraform has a learning curve, but it's also a black magic effery for the cloud.

misiti3780 5y ago
second this, terraform is a game changer if you are using aws (or any of the other clouds, it supports multiple backends)
- Aeolun 5y ago
  And if you plan to use Terraform, you might as well skip that step and go directly to Pulumi.
  - sackerhews 5y ago
    Hey, hadn't that a look before, thanks! Yes, Terraform/HCL can sometimes feel like the instructions for the holy hand grenade.

jrockway 5y ago

Sometimes I wonder if the old world where you had to buy computers and drive to the datacenter at 3 in the morning to replace dead hard drives was really so bad. I guess it was really so bad. Everything you have to build for yourself in the cloud is something you probably needed in your self-hosted stack -- a directory of users, permissions (and escalation), firewall rules, SSH key material, etc. I feel like no matter what production stack you choose, you're always in that "6 of one, half-dozen of another" state. Lots of stuff to interfere with getting your code out there and keeping it running. (My real question is why the cloud providers don't offer these things for you -- they give you the tools to build solutions, not solutions. But I just want to work on my app. I wonder why things like App Engine never really caught on. I guess being opinionated drives more people away than it brings in with the promise of more simplicity, though to some extent App Engine was too simple, which is always bad. Meanwhile, what cloud providers offer certainly lets you bring your own opinion and build it out yourself. I wish there was a comfortable middle ground.)

bigyikes 5y ago
Haven’t used it in years, but Heroku was really great in this respect. It was very easy to just focus on making a good app instead of trying to configure everything myself. I guess there’s probably not much demand for this kind of thing at large scales.
- hnzix 5y ago
  Heroku can be a hard sell to mgmt because of the cost, but if it's framed in terms of "this means you don't need to pay for two AWS cloud architects to be on call" then it's a much easier pitch.
jfoster 5y ago
I think the primary concerns with what App Engine started out as were flexibility and vendor lock-in. Feels like it is a very solvable problem, though. Only a matter of time.
- topkeks 5y ago

shard 5y ago

Not knowing AWS, this sub-thread reads like a real-life version of the Microservices sketch: https://youtu.be/y8OnoxKotPQ https://youtu.be/y8OnoxKotPQ

corpMaverick 5y ago
LOL. That is not a sketch. I swear it is real life in the back end world. The good news is that now we have Micro front ends for the enjoyment of the UI engineers.
GSGBen 5y ago
For anyone skimming the comments and considering skipping this video: do yourself a favor and don't. Krazam's tech comedy is unparalleled.
- arolihas 5y ago
  The day in the life one was amazing as well.
  - celim307 5y ago
    It hit too close to home. It described me and everyone I knew age 22 - 30 in tech to a T. We really aren’t as original as we’d like to believe
    - arolihas 5y ago
      The lack of originality hurts I guess. What was more biting was the faux productivity, tech addiction, and social isolation and alienation of such a mundane daily routine.

dzonga 5y ago

I ran into the same problem. kicked out s3 and ended up using seaweedfs. couldn't be happier. coding instead of reading aws documentation.

riffic 5y ago

I always tell people "stay off AWS unless you or someone you're paying very well on your team holds an Associate-level certification at the very minimum". It's easy to get in over your head over there.

MonaroVXR 5y ago
> you or someone you're paying very well on your team holds an Associate-level certification at the very minimum". It's easy to get in over your head over there. I will never forget the story about the expert Amazon engineer that lost 80K USD.
- drevil-v2 5y ago
  Sounds like an interesting read. Link please.
  - MonaroVXR 5y ago
    https://news.ycombinator.com/item?id=22719573 https://news.ycombinator.com/item?id=22719573

properdine 5y ago

Totally get what you mean. I'll pitch for either terraform or AWS CDK as good ways to make some of this plumbing less painful (they both have you describe the end state you want and then generate resources to get there).

jachee 5y ago
Terraform is—as a colleague described it—a combination very-sharp-knife and fully-loaded foot-seeking gun. Very powerful, but also requires a sufficient amount of respect and care. e.g. You’ve defined a nice, fully-functional end-state, which was generated n time ago, but now you need to make a change to that state, so you run your terraform plan …and it wants to destroy your entire infrastructure and recreate it all from scratch just to e.g. add a new hostname to the certs generated by certmanager. That may be okay, but you should probably measure at least 3 times before you cut.

KptMarchewa 5y ago

>IAM is a rabbit hole, now later VPC restrictions on top My idea is to build simpler system on top of cloud networking and IAM/permissions. Why would I want to have networking between components that are permissioned to interact with each other, and the other way, why some app/lambda's service account have permission to interact with particular database, but be blacklisted on firewall? Have GUI on top to drag and drop "routes" between apps. Of course, make it optional etc, but I think it would save tons of devops time for smaller organisations without audit requirements unlike fintechs, medical stuff etc.

igotroot 5y ago

I totally agree with you. A part of my duties is setting up logging from AWS to Splunk and there's so many gotchas and stumbling blocks that's it's infuriating to even attempt and follow best practices. A great example is Cloudtrail. The best practice is to send your OrgTrail (an org-wide Cloudtrail) to a separate account for security reasons. Cool. Okay, sounds easy enough. The Splunk docs are useless for AWS, so consulting with YouTube, Reddit, etc is the go-to for this. It's so much easier to just leave the OrgTrail logging into the management account it's not even funny.