9 ms·

Cloud, why so difficult? (2022)

96 points by fagnerbrack 3y ago

xixixao 3y ago
If you're looking for something that fits the motivation (including "give me the developer experience of the 90s"), but uses a language you probably know already (JavaScript), check out https://www.convex.dev/ https://www.convex.dev/.
- foobarbecue 3y ago
  Is this like meteor?
  - xixixao 3y ago
    It is a modern alternative to Meteor, Firebase, Supabase and similar, but it should be much easier to work with, especially from React. Unlike Meteor it has a relational DB and data is accessed via server-side functions. It's got some cool properties like automatic realtime updates and ACID transactionality for any server write.
- xet7 3y ago
  Is Convex backend Open Source? I found only frontend related code https://github.com/get-convex https://github.com/get-convex , is backend somewhere there? Meteor has all of frontend and backend Open Source.
  - xixixao 3y ago
    Not yet.
- danjac 3y ago
  I remember when early 2000s enterprise web development had J2EE, a convoluted mess of over-abstraction oversold by Sun, presumably so they could sell more servers. I recall a Sun consultant telling us that this was the "one true way". Meanwhile, we got the job done with PHP (albeit the kind of PHP that got written in the early 2000s could be its own convoluted mess). Now you hear consultants telling you Kubernetes on top of whatever cloud service they are shilling is the one true way, or some other convoluted mess built on top of a Jenga tower of Cloud Formation and Terraform configuration files. JSON has replaced XML but otherwise, same old.
  - Exoristos 3y ago
    The one true way was actually Perl CGIs.
deathanatos 3y ago
> Or that in order to be able to publish a message to SNS, my IAM policy must have a statement that allows the sns:Publish action on the topic's ARN. Like, sure … it's "annoying", in a sense, to need to understand what permissions your program requires. > Can we get rid of it? This is the thesis the article ought to have dug into. It seems innately obvious that there is no removing the need for understanding that your program requires sns:Publish, and then writing an IAM policy to that effect. (Well … short of just removing authorization outright, but good luck convincing the security team that IAM is silly?) AFAICT glacing at the repo, it derives the policy from the code, and proposes that as "PoLP". In a sense … "yes", it's PoLP as to what the code literally requires … but it ignores the number of times an IAM policy has been proposed to me that is trivially not PoLP in what the code should require, and you ask "what do you need these permissions for?" and first round trip is through "well, the code does $thing that requires that permission" and well, okay, but what possibly for? since you tend to know the rough design of the thing being built, and the implementing the design shouldn't require anywhere near as many perms as what are being requested … and then you suss out the bugs in the code for that. S3 is notorious in my book for this: devs will merrily call "GetBucket" on a bucket that by design you know exists, and boto's API encourages this. (And then you have legions of devs who think S3 is slow, and … well, yes, when you make network API calls whose result is entirely discarded!) I've seen "Publish" on a topic requested by the service that's the consumer. The bucket thing about. Requesting a permission against all resources, not a specific one.
- baq 3y ago
  Let me declare permissions inside the service and not in some completely unrelated cdk repo. OpenBSD pledge style. Yes privileges should be the least necessary but effort to actually declare them should also be the least necessary.
- eladbenisrael 3y ago
  Elad here (the author). That's a very good point. If a developer decides to use `bucket.list().filter(x => x == myFile)` instead of `bucket.get(myFile)` the result will be fundamentally different both from a security perspective (e.g. `ListObjects` instead of `GetObject` on AWS), but also from a performance perspective (`list` is O(n) but `get` is O(1)). If you think about this is primarily an issue of software engineering quality. The fact that today we are forcing the developer to go their devops team to request these permissions creates an organizational workflow that increases quality. That's why most of the teams I worked at requires code reviews. I believe the right approach is to enable devops teams to put up boundaries and rules that will get enforced downstream (build => deployment => production). This will allow both developers and devops teams to be be more independent but still be in control. Curious what people think about winglang's compiler plugin system as a way to help streamline these boundaries (https://www.winglang.io/docs/tools/compiler-plugins https://www.winglang.io/docs/tools/compiler-plugins).
mbroecheler 3y ago
Totally agree with the motivation - it is too cumbersome to stitch all these cloud services together by hand. Another project that's similar in motivation but focused on cloud data infrastructure is https://www.datasqrl.com/ https://www.datasqrl.com/
user6723 3y ago
Developers want something that is like Heroku
- danjac 3y ago
  Pretty much, yes. An application developer should be able to focus on data model, business logic and user interface, and everything else at a minimum. If you are burning your available hours working on issues with your infrastructure - whether it's cloud or hosted or on-prem - that's not delivering business value.
  - gravlaks 3y ago
    As long as it's part of the required steps to delivering business value, it doesn't matter that much if it's application code or infrastructure (as code). Both are required, hence both are delivering business value.
    - danjac 3y ago
      OK, but ultimately the customer wants a product built, features added and bugs fixed within whatever timescale and budget. With all things being equal, they would prefer that you spend the time on those things, and not on infra. Now of course, you will have to spend some time on infra, but that should be minimised as much as possible. Whether that is using a home-grown Kubernetes based platform or a PAAS or cloud solution depends on your requirements, skill-set, budget and time, but your aim should be to have the simplest solution that works.
- mdekkers 3y ago
  Developers don’t want to ops.
  - sam_lowry_ 3y ago
    You'll surprised.
    - 3y ago
vbezhenar 3y ago
Cloud is difficult because it stiched together from fragments of old applications and approaches. Cloud will be easy with consistent approach. One or very few languages for everything, from configuring resources, to configuring policies and writing code. Cloud will be easy with opinionated tooling. No Dockerfile and supporting of miriads languages. Just file with dependencies and folder with source code. Cloud will be easy with opinionated approach to configuration. No need to support PASSWORD env, PASSWORD_FILE env, some secret access service. Only one secure and simple way. Basically rebuild cloud from the ground up. Remove Linux from equation, we don't need it. Remove filesystem from equation. You have a source code which executex somewhere. You can declare that your service needs database and it'll get it. You can declare that your service needs to access another service and you'll be able to use API to access another service, without thinking about URLS, auth tokens and all this nonsense. Just RPC. Lambdas are good idea, but very convoluted. Lots of abstractions over abstractions. Terraform which calls AWS API. Things could be so much simpler and faster. There's nothing that prevents you from redeploying your service in microsecond, just as you hit "Save" button in vscode.
- drdaeman 3y ago
  > and supporting of miriads languages It might have better chances with some common language runtime, than declaring One True Language. No matter what language you pick or invent, it's gonna be hated by many. Intermediate representations? Sure, but much less so. But you are absolutely right that clouds need a fundamentally different runtime, built on different ideas and principles - not virtual machines running containers pretending to be functions, all completely separate from resource declarations (which are in turn all individual nuanced messes).
- afandian 3y ago
  That tight, opinionated integration of hardware and software sounds a lot like 'mainframe'.
  - drdaeman 3y ago
    Lisp machines strike back? ;)
emptysongglass 3y ago
Another project in this space is Klotho. I think it's the only one with broad language support so you can write in Go or Python or JavaScript or Java or C#. https://klo.dev/ https://klo.dev/
eladbenisrael 3y ago
Elad here (I am the author). Happy to answer any questions and dive deeper. Feel free to ask me anything. We also have a bunch of weird videos on YouTube with tons of geeking out and sharing our progress on YouTube https://youtube.com/@winglangio https://youtube.com/@winglangio
devjab 3y ago
> Here's how an iteration looks like in the cloud today: I make a change to my code; then I need to compile it; deploy it to my test account; find my way around the management console to actually trigger it; I’m not an Ops specialist, but once I got tired of doing that (which was the second time), I spent time figuring out how to automate it. Don’t get me wrong, I naively think that the “network” part of working with the cloud ought to be easier, but aside from networking, basically everything we do in azure is now automated. And the networking part is slowly getting there as I learn more and more about it. I do agree that it sort of makes no sense that everyone has to spend time setting the automation up. Like why doesn’t Azure come with CI/CD templates? I’m sure a lot of the automation we’ve written to make an accepted pull request deploy into different app deployment slots (and the first time spin up an app with TSL on a VNET with Private Endpoints on the correct subnets) looks almost exactly like yours does. So why did we both have to write it from scratch? But beyond that, I’m not so sure what the author is talking about. I recall when we used Windows servers instead, and maybe the author didn’t have to work with their settings or making sure they had the correct components in place, but as I recall the experience wasn’t too dissimilar to what working in Azure is, expect for the fact that we developers are also working into the networking part or it, which we weren’t when that was solely an Ops thing. The actual setup of our Networking is still very much an Ops thing, but I still have to “chose” the VNETs and Subnets my Opes people tell me to use, which wasn’t really part of the process when we were self-hosting. Anyway, I guess the only thing I really agree with is the speed. It can be very slow, depending on your setup, but isn’r that why we use local containers and temporary “dev” deployment slots? When I push code to a dev branch, it automatically gets build and deployed to a branch slot, it does take time, but I rarely have to actually wait for it. Partly because I can just up my own container and see the same thing, but also because the actual manual testing is typically done by the people who use the applications, and they’ll never see the “downtime” because it spins up automatically and replaces what they load when it’s ready and the old deployment dies. Notifying them can be as simple as having your CI pipeline send them an email with “app is updated, in relation to work item please go see if you’re satisfied” or whatever. It is difficult, but then it frankly always was, and I do think it’s maybe better in the hands of actual Ops people, but since those can be sort of hard to find, it’s not too uncommon to expect “you build it, you run it” to be part of the modern developer role.
chrismarlow9 3y ago
Because selling a wide ecosystem with certificates and consultants is more lucrative.
systemvoltage 3y ago
I feel like everytime a new abstraction layer is put together, it goes through the whole Gartner cycle, and somewhere along the plateau of productivity we realize that may be it should be rebuilt from scratch a few times. The answer is probably not on-prem, but a better clould that tames the abstraction layers and keeps things relatively flat. The whole infra-as-code, terraform, kubernetes, multicloud infra abstraction today, it's all too much.
nerdile 3y ago
> I disliked C++ because abstractions are leaky. So I am building a new abstraction on the cloud. Oh boy, buddy. Let me know how that works out for you. I've worked on app code, OS code, on-prem services, datacenter services, cloud on Azure and AWS. No matter what your platform, someone has to do the hard work to make the inner loop quick and the deployment process reliable. But no matter what, the abstractions are leaky, and the engineers who grok the underlying platform have the fast cycle times, and the ones who don't, are frequently blocked and need help from those who do. Building on CDK now, I can build my code and run all my unit tests in under a minute, and deploy via CDK and run all my integration tests in under five. Since I'm focusing on microservice components with independent delivery, that's ok. Obviously, the monoliths are more complex and tougher. I'm not sure how a new language and abstraction are going to help here. (Sorry, that's what I would say to a new hire on my team to be gentle. What I mean is, sorry this won't help.) (Also, CDK is a mess because the underlying CFN techs are inconsistent in their adoption. SNS being one of the worst offenders.)
- deleted 3y ago
  [deleted]
- cocochanel 3y ago
  Also from the author: >I've spent the last five years building the AWS CDK
- drdaeman 3y ago
  Unless the foundations themselves are changing (that is, AWS bringing something different), I suspect it's going to make it worse. It just hides the ugly truth behind yet another layer of abstraction, hoping this layer will make things work. I don't remember when building extra abstraction layer had ever helped runtime performance. Faster and simpler coding, less learning? Sure - and that's great. But I suspect it's going to be as bad as my first large Terraform project (and oh man it sucked so bad). Or it could be decently fast, but it'll make a number of assumptions and $deity help you if you step in a wrong direction.
  - TuringTest 3y ago
djleni 3y ago
Heh, I didn’t know who this was, read the intro, thought “have you tried CDK?” only to find the author worked on it.
skybrian 3y ago
Platform-as-a-service providers try to do this, but there are a lot of pieces to build and there isn’t one sufficiently opinionated standard. If no other vendor picks the same standard, it results in fear of lock in. Deno seems to be making a good attempt, but it’s still early and not enough pieces are there yet. App Engine had a good run, but seems to have lost its way. Supporting many languages seems worse than picking one and sticking with it?
zsoltkacsandi 3y ago
Sorry to say this, but some developers tend to think about themselves as the center of the world. “Cloud” is nothing but a bunch of virtualized resources (computer, network, storage, etc) that you can manage on demand through an API. You could write a post about “Computer, Why So Difficult?”. By the way all of the complaints in this post can be solved if OP uses a PaaS solution.
- alex_lav 3y ago
  > By the way all of the complaints in this post can be solved if OP uses a PaaS solution. This statement is untrue. PaaS offerings just trade one set of problems for a maybe slightly smaller but mostly the same set of problems. I actually can't find a single complaint in the article that is solved by PaaS solutions.
mlhpdx 3y ago
Yet another sincere effort to make the easy things easy.
cookiengineer 3y ago
Honestly I never understood why Docker stopped at where it's at. Imagine a world where a "docker push aws.io production" leads to an automatically updated docker remote, and where the endpoint is automatically NATed, scaled and shutdown if not needed anymore. Could have been so convenient. Instead there's this pile of networking glue code that's so custom for each platform that nobody wants to deal with load balancers. In my opinion the biggest "innovation" in the cloud space was heroku back then, where code deployment was fully integrated to the dev workflow via git.
- siquick 3y ago
  They kind of have it[0] with ECS deployments but it’s being retired in November. [0] https://docs.docker.com/cloud/ecs-integration/ https://docs.docker.com/cloud/ecs-integration/
  - cookiengineer 3y ago
    The issue I had with the EC2 deployment workflow was that it needed the AWS CLI configured, so it relied totally on AWS'es locked in workflow. I was thinking more of the lines as a standard API/set of protocols that works on the NAT-scale automatically. Humans are bad at guessing resources, and having people tweak all their VMs all the time with crappy code and crappy APIs is kinda the weakness of all dyno based workflows nowadays. All the "web scale" jokes have a core truth in them, and are actually not so funny for a reason :-/ But relying on a standard set of NAT rules and relationships, which the Dockerfile already provides, could make this an advantage instead.
- re-thc 3y ago
  > Honestly I never understood why Docker stopped at where it's at. They originally wanted to build docker swarm, which was to compete with kubernetes. Direction 360-ed after that fizzed.
  - fomine3 3y ago
    They originally dotCloud...
olup 3y ago
Closer to pulumi than CDk IMO. Pulumi tried this infra and code mixed together approach. I still keep the code separated but it allows that.
ChoHag 3y ago
[dead]
gourabmi 3y ago
I saw something similar presented at NorCal Database Day this year in the Bloom project from Berkeley http://bloom-lang.net/features/ http://bloom-lang.net/features/
zacksmocha 3y ago
Wow,brave move that could make a difference. The question is how you keep up with the never ending changes in AWS, Azure and GCP..
- nathants 3y ago
  you rss subscribe to commits to the aws go sdk on github.
adontz 3y ago
>> It's the DevOps person with an endless ticket queue; it's the IAM policy that needs to be updated; it's the deployment failure that only the external part-time consultant knows how to debug; It's the endless backlog of missing knobs and APIs in the internal platform that we hoped will change everything. Resonates with https://news.ycombinator.com/item?id=36354049 https://news.ycombinator.com/item?id=36354049
samsquire 3y ago
Thanks for this blog post. Glad to see innovation in this space. I have tried to be innovative in this space, I wrote a tool that creates new environments in-order from beginning to end from Graphviz files that coordinate other tools. ( https://devops-pipeline.com https://devops-pipeline.com ) Not ready for other people to use but hopefully inspires people. There is also dark lang and unison lang. My dream: I want a CRUD dashboard like AWS Console that is also a REPL. So I can use a mixture of point and click and code to generate and wire up objects that become resources. I needed to generate a table of load balancer data for microservices in multiple load balancers for GTM and then wire it up with code. So we had one script to generate microservice data and another tool to configure load balancers from that data.
Ofer1000 3y ago
This is exactly what I was looking for: Accelerates my dev time by working locally, and integrating with cloud services. Finally a dev environment that gives me the freedom to migrate to other clouds with very little cost
presto138 3y ago
Is this open source ? What's this license here: https://github.com/winglang/wing/blob/main/apps/wing-console/LICENSE.md https://github.com/winglang/wing/blob/main/apps/wing-console...
- shaiber 3y ago
  The console is "source available". All the rest is fully open-source with MIT license.
Roark66 3y ago
I'm not buying it(yet). Yes, cloud (especially if you want to use serverless and not pay an arm and a leg)is relatively difficult for devs used to traditional model or just new devs period. But that's what devops is for. Not just to write infrastructure as code, test and scale, but to help devs get to know cloud services. I found the source of this perceived difficulty is the vastness of documentation and the fact lots of stuff is not documented. The services themselves are fairly easy to understand in a conversation with someone who knows them well. However, then the author complains about how long it takes to redeploy cloud infrastructure. So I'm thinking, sure. I too had cloudformation stacks that took 45min to run, but that can be resolved by not tearing all resources down and recreating, but by intelligent updates, container use and so on. So then, we learn a solution is a new provider agnostic programming language for the cloud... But we already have terraform for python that is provider agnostic, but don't imagine you'll have the same infrastructure as code run on aws one say and gcloud the next. There will always be tweaks required per provider. Still,personally I prefer to either use tools like ansible, terraform(for future readability) or straight python with boto(an aws specific framework). I see no need for an entire new programming language.
dang 3y ago
This had a big discussion just a month ago: Cloud, Why So Difficult? - https://news.ycombinator.com/item?id=36089084 https://news.ycombinator.com/item?id=36089084 - May 2023 (151 comments) It's a pity because a lot of the subthreads here are pretty interesting, but a month is super recent.
lelanthran 3y ago
> It doesn't make sense that every time I want to execute code inside an AWS Lambda function, I have to understand that it needs to be bundled with tree-shaken dependencies, AWS Lambda only executes Javascript code? > Cloud applications today are simply a patchwork of disconnected pieces. Agreed. > I have a compiler for my infrastructure, another for my functions, another for my containers, another for my CI/CD pipelines. Each one takes its job super seriously, and keeps me safe and happy inside each of these machines, but my application is not running on a single machine anymore, my application is running on the cloud. While I agree with the general sentiment of the article as summarised by this problem statement, I dunno if the best path forward is to create a new programming language. Replace "standards" with "tech stacks" in https://xkcd.com/927/ https://xkcd.com/927/ and you'll have exactly what is going to happen here. A better approach is tying all those bits and pieces into a single library (like the author says he did with CDK). The big problem is that all these bits and pieces that are tied together are temporarily tied together and you can usually swap out one part of the stack (like the CI/CD stuff, or the source code control) for a competing technology. This means that the new language will be in constant flux. To me, that's worse than simply having an existing library be in constant flux. TBH, if I was going to do something like this (build a new language for Cloud-As-A-Target-Platform), I'd use s-expressions as the language, and let the developer define everything that needs to be tied together using s-expressions, then my new "language" would be a limited subset of Lisp (no defmacro, CLOS, etc) It's compact, readable, expressive and powerful enough to represent any glue that you may need to for a cloud-native application.
choeger 3y ago
As someone with more than a little bit of PLs: That's the wrong approach. Doing PLs is hard it's the ultimate naysayer's profession. Modern programming languages are built around a dozen or so major design decisions, or dimensions for the machine learning crowd, (that's why there are so many) - and many combinations, if not most of them, are simply incompatible. Like, you-end-up-with-undecidable-problems incompatible. So doing a PL should focus solely on expressing computations. A different platform, even a very different platform like the cloud, does hardly justify the effort necessary to design and implement a language. In the end you won't do neither a good language nor good platform support. Instead, develop a dialect using as much of the parent language as possible and only restrict the parts that are indeed incompatible with your goal.
gravlaks 3y ago
https://fly.io/ https://fly.io/
nathants 3y ago
like linux, cloud is a lot to learn, but worth it. it adds capabilities you didn’t have before. like linux, cloud is best kept simple, or it can become brittle and confusing. like linux, cloud has a lot of cool things, think zfs, that should be appreciated but not used. like linux, using go makes your life easier. the aws go sdk is the documentation. like linux, you have to learn a lot and then find the core utility you actually care about. for me it is: https://github.com/nathants/libaws https://github.com/nathants/libaws like linux, cloud is very stable. focus on core utility. would it be better if cloud wasn't the massive union of all customers needs with a bias toward large customers? sure. the good news is some subset of those needs are actually core utility, well implemented, and stable.

96 points by fagnerbrack 3y ago

If you're looking for something that fits the motivation (including "give me the developer experience of the 90s"), but uses a language you probably know already (JavaScript), check out https://www.convex.dev/ https://www.convex.dev/.

foobarbecue 3y ago
Is this like meteor?
- xixixao 3y ago
  It is a modern alternative to Meteor, Firebase, Supabase and similar, but it should be much easier to work with, especially from React. Unlike Meteor it has a relational DB and data is accessed via server-side functions. It's got some cool properties like automatic realtime updates and ACID transactionality for any server write.
xet7 3y ago
Is Convex backend Open Source? I found only frontend related code https://github.com/get-convex https://github.com/get-convex , is backend somewhere there? Meteor has all of frontend and backend Open Source.
- xixixao 3y ago
  Not yet.
danjac 3y ago
I remember when early 2000s enterprise web development had J2EE, a convoluted mess of over-abstraction oversold by Sun, presumably so they could sell more servers. I recall a Sun consultant telling us that this was the "one true way". Meanwhile, we got the job done with PHP (albeit the kind of PHP that got written in the early 2000s could be its own convoluted mess). Now you hear consultants telling you Kubernetes on top of whatever cloud service they are shilling is the one true way, or some other convoluted mess built on top of a Jenga tower of Cloud Formation and Terraform configuration files. JSON has replaced XML but otherwise, same old.
- Exoristos 3y ago
  The one true way was actually Perl CGIs.

deathanatos 3y ago

> Or that in order to be able to publish a message to SNS, my IAM policy must have a statement that allows the sns:Publish action on the topic's ARN. Like, sure … it's "annoying", in a sense, to need to understand what permissions your program requires. > Can we get rid of it? This is the thesis the article ought to have dug into. It seems innately obvious that there is no removing the need for understanding that your program requires sns:Publish, and then writing an IAM policy to that effect. (Well … short of just removing authorization outright, but good luck convincing the security team that IAM is silly?) AFAICT glacing at the repo, it derives the policy from the code, and proposes that as "PoLP". In a sense … "yes", it's PoLP as to what the code literally requires … but it ignores the number of times an IAM policy has been proposed to me that is trivially not PoLP in what the code should require, and you ask "what do you need these permissions for?" and first round trip is through "well, the code does $thing that requires that permission" and well, okay, but what possibly for? since you tend to know the rough design of the thing being built, and the implementing the design shouldn't require anywhere near as many perms as what are being requested … and then you suss out the bugs in the code for that. S3 is notorious in my book for this: devs will merrily call "GetBucket" on a bucket that by design you know exists, and boto's API encourages this. (And then you have legions of devs who think S3 is slow, and … well, yes, when you make network API calls whose result is entirely discarded!) I've seen "Publish" on a topic requested by the service that's the consumer. The bucket thing about. Requesting a permission against all resources, not a specific one.

baq 3y ago
Let me declare permissions inside the service and not in some completely unrelated cdk repo. OpenBSD pledge style. Yes privileges should be the least necessary but effort to actually declare them should also be the least necessary.
eladbenisrael 3y ago
Elad here (the author). That's a very good point. If a developer decides to use `bucket.list().filter(x => x == myFile)` instead of `bucket.get(myFile)` the result will be fundamentally different both from a security perspective (e.g. `ListObjects` instead of `GetObject` on AWS), but also from a performance perspective (`list` is O(n) but `get` is O(1)). If you think about this is primarily an issue of software engineering quality. The fact that today we are forcing the developer to go their devops team to request these permissions creates an organizational workflow that increases quality. That's why most of the teams I worked at requires code reviews. I believe the right approach is to enable devops teams to put up boundaries and rules that will get enforced downstream (build => deployment => production). This will allow both developers and devops teams to be be more independent but still be in control. Curious what people think about winglang's compiler plugin system as a way to help streamline these boundaries (https://www.winglang.io/docs/tools/compiler-plugins https://www.winglang.io/docs/tools/compiler-plugins).

mbroecheler 3y ago

Totally agree with the motivation - it is too cumbersome to stitch all these cloud services together by hand. Another project that's similar in motivation but focused on cloud data infrastructure is https://www.datasqrl.com/ https://www.datasqrl.com/

user6723 3y ago

Developers want something that is like Heroku

danjac 3y ago
Pretty much, yes. An application developer should be able to focus on data model, business logic and user interface, and everything else at a minimum. If you are burning your available hours working on issues with your infrastructure - whether it's cloud or hosted or on-prem - that's not delivering business value.
- gravlaks 3y ago
  As long as it's part of the required steps to delivering business value, it doesn't matter that much if it's application code or infrastructure (as code). Both are required, hence both are delivering business value.
  - danjac 3y ago
    OK, but ultimately the customer wants a product built, features added and bugs fixed within whatever timescale and budget. With all things being equal, they would prefer that you spend the time on those things, and not on infra. Now of course, you will have to spend some time on infra, but that should be minimised as much as possible. Whether that is using a home-grown Kubernetes based platform or a PAAS or cloud solution depends on your requirements, skill-set, budget and time, but your aim should be to have the simplest solution that works.
mdekkers 3y ago
Developers don’t want to ops.
- sam_lowry_ 3y ago
  You'll surprised.
  - 3y ago

vbezhenar 3y ago

Cloud is difficult because it stiched together from fragments of old applications and approaches. Cloud will be easy with consistent approach. One or very few languages for everything, from configuring resources, to configuring policies and writing code. Cloud will be easy with opinionated tooling. No Dockerfile and supporting of miriads languages. Just file with dependencies and folder with source code. Cloud will be easy with opinionated approach to configuration. No need to support PASSWORD env, PASSWORD_FILE env, some secret access service. Only one secure and simple way. Basically rebuild cloud from the ground up. Remove Linux from equation, we don't need it. Remove filesystem from equation. You have a source code which executex somewhere. You can declare that your service needs database and it'll get it. You can declare that your service needs to access another service and you'll be able to use API to access another service, without thinking about URLS, auth tokens and all this nonsense. Just RPC. Lambdas are good idea, but very convoluted. Lots of abstractions over abstractions. Terraform which calls AWS API. Things could be so much simpler and faster. There's nothing that prevents you from redeploying your service in microsecond, just as you hit "Save" button in vscode.

drdaeman 3y ago
> and supporting of miriads languages It might have better chances with some common language runtime, than declaring One True Language. No matter what language you pick or invent, it's gonna be hated by many. Intermediate representations? Sure, but much less so. But you are absolutely right that clouds need a fundamentally different runtime, built on different ideas and principles - not virtual machines running containers pretending to be functions, all completely separate from resource declarations (which are in turn all individual nuanced messes).
afandian 3y ago
That tight, opinionated integration of hardware and software sounds a lot like 'mainframe'.
- drdaeman 3y ago
  Lisp machines strike back? ;)

emptysongglass 3y ago

Another project in this space is Klotho. I think it's the only one with broad language support so you can write in Go or Python or JavaScript or Java or C#. https://klo.dev/ https://klo.dev/

eladbenisrael 3y ago

Elad here (I am the author). Happy to answer any questions and dive deeper. Feel free to ask me anything. We also have a bunch of weird videos on YouTube with tons of geeking out and sharing our progress on YouTube https://youtube.com/@winglangio https://youtube.com/@winglangio

devjab 3y ago

> Here's how an iteration looks like in the cloud today: I make a change to my code; then I need to compile it; deploy it to my test account; find my way around the management console to actually trigger it; I’m not an Ops specialist, but once I got tired of doing that (which was the second time), I spent time figuring out how to automate it. Don’t get me wrong, I naively think that the “network” part of working with the cloud ought to be easier, but aside from networking, basically everything we do in azure is now automated. And the networking part is slowly getting there as I learn more and more about it. I do agree that it sort of makes no sense that everyone has to spend time setting the automation up. Like why doesn’t Azure come with CI/CD templates? I’m sure a lot of the automation we’ve written to make an accepted pull request deploy into different app deployment slots (and the first time spin up an app with TSL on a VNET with Private Endpoints on the correct subnets) looks almost exactly like yours does. So why did we both have to write it from scratch? But beyond that, I’m not so sure what the author is talking about. I recall when we used Windows servers instead, and maybe the author didn’t have to work with their settings or making sure they had the correct components in place, but as I recall the experience wasn’t too dissimilar to what working in Azure is, expect for the fact that we developers are also working into the networking part or it, which we weren’t when that was solely an Ops thing. The actual setup of our Networking is still very much an Ops thing, but I still have to “chose” the VNETs and Subnets my Opes people tell me to use, which wasn’t really part of the process when we were self-hosting. Anyway, I guess the only thing I really agree with is the speed. It can be very slow, depending on your setup, but isn’r that why we use local containers and temporary “dev” deployment slots? When I push code to a dev branch, it automatically gets build and deployed to a branch slot, it does take time, but I rarely have to actually wait for it. Partly because I can just up my own container and see the same thing, but also because the actual manual testing is typically done by the people who use the applications, and they’ll never see the “downtime” because it spins up automatically and replaces what they load when it’s ready and the old deployment dies. Notifying them can be as simple as having your CI pipeline send them an email with “app is updated, in relation to work item please go see if you’re satisfied” or whatever. It is difficult, but then it frankly always was, and I do think it’s maybe better in the hands of actual Ops people, but since those can be sort of hard to find, it’s not too uncommon to expect “you build it, you run it” to be part of the modern developer role.

chrismarlow9 3y ago

Because selling a wide ecosystem with certificates and consultants is more lucrative.

systemvoltage 3y ago

I feel like everytime a new abstraction layer is put together, it goes through the whole Gartner cycle, and somewhere along the plateau of productivity we realize that may be it should be rebuilt from scratch a few times. The answer is probably not on-prem, but a better clould that tames the abstraction layers and keeps things relatively flat. The whole infra-as-code, terraform, kubernetes, multicloud infra abstraction today, it's all too much.

nerdile 3y ago

> I disliked C++ because abstractions are leaky. So I am building a new abstraction on the cloud. Oh boy, buddy. Let me know how that works out for you. I've worked on app code, OS code, on-prem services, datacenter services, cloud on Azure and AWS. No matter what your platform, someone has to do the hard work to make the inner loop quick and the deployment process reliable. But no matter what, the abstractions are leaky, and the engineers who grok the underlying platform have the fast cycle times, and the ones who don't, are frequently blocked and need help from those who do. Building on CDK now, I can build my code and run all my unit tests in under a minute, and deploy via CDK and run all my integration tests in under five. Since I'm focusing on microservice components with independent delivery, that's ok. Obviously, the monoliths are more complex and tougher. I'm not sure how a new language and abstraction are going to help here. (Sorry, that's what I would say to a new hire on my team to be gentle. What I mean is, sorry this won't help.) (Also, CDK is a mess because the underlying CFN techs are inconsistent in their adoption. SNS being one of the worst offenders.)

deleted 3y ago
[deleted]
cocochanel 3y ago
Also from the author: >I've spent the last five years building the AWS CDK
drdaeman 3y ago
Unless the foundations themselves are changing (that is, AWS bringing something different), I suspect it's going to make it worse. It just hides the ugly truth behind yet another layer of abstraction, hoping this layer will make things work. I don't remember when building extra abstraction layer had ever helped runtime performance. Faster and simpler coding, less learning? Sure - and that's great. But I suspect it's going to be as bad as my first large Terraform project (and oh man it sucked so bad). Or it could be decently fast, but it'll make a number of assumptions and $deity help you if you step in a wrong direction.
- TuringTest 3y ago

djleni 3y ago

Heh, I didn’t know who this was, read the intro, thought “have you tried CDK?” only to find the author worked on it.

skybrian 3y ago

Platform-as-a-service providers try to do this, but there are a lot of pieces to build and there isn’t one sufficiently opinionated standard. If no other vendor picks the same standard, it results in fear of lock in. Deno seems to be making a good attempt, but it’s still early and not enough pieces are there yet. App Engine had a good run, but seems to have lost its way. Supporting many languages seems worse than picking one and sticking with it?

zsoltkacsandi 3y ago

Sorry to say this, but some developers tend to think about themselves as the center of the world. “Cloud” is nothing but a bunch of virtualized resources (computer, network, storage, etc) that you can manage on demand through an API. You could write a post about “Computer, Why So Difficult?”. By the way all of the complaints in this post can be solved if OP uses a PaaS solution.

alex_lav 3y ago
> By the way all of the complaints in this post can be solved if OP uses a PaaS solution. This statement is untrue. PaaS offerings just trade one set of problems for a maybe slightly smaller but mostly the same set of problems. I actually can't find a single complaint in the article that is solved by PaaS solutions.

mlhpdx 3y ago

Yet another sincere effort to make the easy things easy.

cookiengineer 3y ago

Honestly I never understood why Docker stopped at where it's at. Imagine a world where a "docker push aws.io production" leads to an automatically updated docker remote, and where the endpoint is automatically NATed, scaled and shutdown if not needed anymore. Could have been so convenient. Instead there's this pile of networking glue code that's so custom for each platform that nobody wants to deal with load balancers. In my opinion the biggest "innovation" in the cloud space was heroku back then, where code deployment was fully integrated to the dev workflow via git.

siquick 3y ago
They kind of have it[0] with ECS deployments but it’s being retired in November. [0] https://docs.docker.com/cloud/ecs-integration/ https://docs.docker.com/cloud/ecs-integration/
- cookiengineer 3y ago
  The issue I had with the EC2 deployment workflow was that it needed the AWS CLI configured, so it relied totally on AWS'es locked in workflow. I was thinking more of the lines as a standard API/set of protocols that works on the NAT-scale automatically. Humans are bad at guessing resources, and having people tweak all their VMs all the time with crappy code and crappy APIs is kinda the weakness of all dyno based workflows nowadays. All the "web scale" jokes have a core truth in them, and are actually not so funny for a reason :-/ But relying on a standard set of NAT rules and relationships, which the Dockerfile already provides, could make this an advantage instead.
re-thc 3y ago
> Honestly I never understood why Docker stopped at where it's at. They originally wanted to build docker swarm, which was to compete with kubernetes. Direction 360-ed after that fizzed.
- fomine3 3y ago
  They originally dotCloud...

olup 3y ago

Closer to pulumi than CDk IMO. Pulumi tried this infra and code mixed together approach. I still keep the code separated but it allows that.

ChoHag 3y ago

[dead]

gourabmi 3y ago

I saw something similar presented at NorCal Database Day this year in the Bloom project from Berkeley http://bloom-lang.net/features/ http://bloom-lang.net/features/

zacksmocha 3y ago

Wow,brave move that could make a difference. The question is how you keep up with the never ending changes in AWS, Azure and GCP..

nathants 3y ago
you rss subscribe to commits to the aws go sdk on github.

adontz 3y ago

>> It's the DevOps person with an endless ticket queue; it's the IAM policy that needs to be updated; it's the deployment failure that only the external part-time consultant knows how to debug; It's the endless backlog of missing knobs and APIs in the internal platform that we hoped will change everything. Resonates with https://news.ycombinator.com/item?id=36354049 https://news.ycombinator.com/item?id=36354049

samsquire 3y ago

Thanks for this blog post. Glad to see innovation in this space. I have tried to be innovative in this space, I wrote a tool that creates new environments in-order from beginning to end from Graphviz files that coordinate other tools. ( https://devops-pipeline.com https://devops-pipeline.com ) Not ready for other people to use but hopefully inspires people. There is also dark lang and unison lang. My dream: I want a CRUD dashboard like AWS Console that is also a REPL. So I can use a mixture of point and click and code to generate and wire up objects that become resources. I needed to generate a table of load balancer data for microservices in multiple load balancers for GTM and then wire it up with code. So we had one script to generate microservice data and another tool to configure load balancers from that data.

Ofer1000 3y ago

This is exactly what I was looking for: Accelerates my dev time by working locally, and integrating with cloud services. Finally a dev environment that gives me the freedom to migrate to other clouds with very little cost

presto138 3y ago

Is this open source ? What's this license here: https://github.com/winglang/wing/blob/main/apps/wing-console/LICENSE.md https://github.com/winglang/wing/blob/main/apps/wing-console...

shaiber 3y ago
The console is "source available". All the rest is fully open-source with MIT license.

Roark66 3y ago

I'm not buying it(yet). Yes, cloud (especially if you want to use serverless and not pay an arm and a leg)is relatively difficult for devs used to traditional model or just new devs period. But that's what devops is for. Not just to write infrastructure as code, test and scale, but to help devs get to know cloud services. I found the source of this perceived difficulty is the vastness of documentation and the fact lots of stuff is not documented. The services themselves are fairly easy to understand in a conversation with someone who knows them well. However, then the author complains about how long it takes to redeploy cloud infrastructure. So I'm thinking, sure. I too had cloudformation stacks that took 45min to run, but that can be resolved by not tearing all resources down and recreating, but by intelligent updates, container use and so on. So then, we learn a solution is a new provider agnostic programming language for the cloud... But we already have terraform for python that is provider agnostic, but don't imagine you'll have the same infrastructure as code run on aws one say and gcloud the next. There will always be tweaks required per provider. Still,personally I prefer to either use tools like ansible, terraform(for future readability) or straight python with boto(an aws specific framework). I see no need for an entire new programming language.

dang 3y ago

This had a big discussion just a month ago: Cloud, Why So Difficult? - https://news.ycombinator.com/item?id=36089084 https://news.ycombinator.com/item?id=36089084 - May 2023 (151 comments) It's a pity because a lot of the subthreads here are pretty interesting, but a month is super recent.

lelanthran 3y ago

> It doesn't make sense that every time I want to execute code inside an AWS Lambda function, I have to understand that it needs to be bundled with tree-shaken dependencies, AWS Lambda only executes Javascript code? > Cloud applications today are simply a patchwork of disconnected pieces. Agreed. > I have a compiler for my infrastructure, another for my functions, another for my containers, another for my CI/CD pipelines. Each one takes its job super seriously, and keeps me safe and happy inside each of these machines, but my application is not running on a single machine anymore, my application is running on the cloud. While I agree with the general sentiment of the article as summarised by this problem statement, I dunno if the best path forward is to create a new programming language. Replace "standards" with "tech stacks" in https://xkcd.com/927/ https://xkcd.com/927/ and you'll have exactly what is going to happen here. A better approach is tying all those bits and pieces into a single library (like the author says he did with CDK). The big problem is that all these bits and pieces that are tied together are temporarily tied together and you can usually swap out one part of the stack (like the CI/CD stuff, or the source code control) for a competing technology. This means that the new language will be in constant flux. To me, that's worse than simply having an existing library be in constant flux. TBH, if I was going to do something like this (build a new language for Cloud-As-A-Target-Platform), I'd use s-expressions as the language, and let the developer define everything that needs to be tied together using s-expressions, then my new "language" would be a limited subset of Lisp (no defmacro, CLOS, etc) It's compact, readable, expressive and powerful enough to represent any glue that you may need to for a cloud-native application.

choeger 3y ago

As someone with more than a little bit of PLs: That's the wrong approach. Doing PLs is hard it's the ultimate naysayer's profession. Modern programming languages are built around a dozen or so major design decisions, or dimensions for the machine learning crowd, (that's why there are so many) - and many combinations, if not most of them, are simply incompatible. Like, you-end-up-with-undecidable-problems incompatible. So doing a PL should focus solely on expressing computations. A different platform, even a very different platform like the cloud, does hardly justify the effort necessary to design and implement a language. In the end you won't do neither a good language nor good platform support. Instead, develop a dialect using as much of the parent language as possible and only restrict the parts that are indeed incompatible with your goal.

gravlaks 3y ago

https://fly.io/ https://fly.io/

nathants 3y ago

like linux, cloud is a lot to learn, but worth it. it adds capabilities you didn’t have before. like linux, cloud is best kept simple, or it can become brittle and confusing. like linux, cloud has a lot of cool things, think zfs, that should be appreciated but not used. like linux, using go makes your life easier. the aws go sdk is the documentation. like linux, you have to learn a lot and then find the core utility you actually care about. for me it is: https://github.com/nathants/libaws https://github.com/nathants/libaws like linux, cloud is very stable. focus on core utility. would it be better if cloud wasn't the massive union of all customers needs with a bias toward large customers? sure. the good news is some subset of those needs are actually core utility, well implemented, and stable.