7 ms·
You've discovered what many other people have: The cloud is the new time-share mainframe. Programming in the 1960s to 80s was like this too. You'd develop some
by jiggawatts 6y ago
You've discovered what many other people have: The cloud is the new time-share mainframe.
Programming in the 1960s to 80s was like this too. You'd develop some program in isolation, unable to properly run it. You "submit" it to the system, and it would be scheduled to run along with other workloads. You'd get a printout of the results back hours later, or even tomorrow. Rinse and repeat.
This work loop is incredibly inefficient, and was replaced by development that happened entirely locally on a workstation. This dramatically tightened the edit-compile-debug loop, down to seconds or at most minutes. Productivity skyrocketed, and most enterprises shifted the majority of their workload away from mainframes.
Now, in the 2020s, mainframes are back! They're just called "the cloud" now, but not much of their essential nature has changed other than the vendor name.
The cloud, just like mainframes:
- Does not provide all-local workstations. The only full-fidelity platform is the shared server.
- Is closed source. Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. You can't peer into their source code, it is all proprietary and even secret.
- Has a poor debugging experience. Shared platforms can't generally allow "invasive" debugging for security reasons. Their sheer size and complexity will mean that your visibility will always be limited. You'll never been able to get a stack trace that crosses into the internal calls of the platform services like S3 or Lambda. Contrast this with typical debugging where you can even trace into the OS kernel if you so choose.
- Are generally based on the "print the logs out" feedback mechanism, with all the usual issues of mainframes such as hours-long delays.
- ShakataGaNai 6y agoThis weeks episode of DevOps paradox talked ... a lot about this exact "we're back to the mainframe" thought process. https://www.devopsparadox.com/episodes/does-history-repeat-itself-105/ https://www.devopsparadox.com/episodes/does-history-repeat-i...
- bombcar 6y agoPeople sometimes don’t understand the “why” of something like Literate Programming - it wasn’t really for documentation purposes. Part of it was to reduce as much as possible what we would call “compiler errors” as much as possible. Arguably that’s a more dignified form of programming for a more elegant time but it doesn’t really mesh with the reality of “try fail print retry”.
- bserge 6y agoWow, you put into words what I was feeling about modern development. And software in general. God forbid you make a completely offline program and (gasp) never update it, that's backwards, unacceptable, a terrible security risk, etc.
- xwolfi 6y agoAnd it's so hard to convince management, all rewarded on innovative solution like firing the data center team to replace it with an enormous Microsoft Hybrid Cloud contract, that they're only buying something way inferior for a much higher price. I envy the geniuses at Microsoft who can convince 60k+ software companies like mine where we're all engineer or mathematicians, led by incredibly clueless IT decision makers :D I've talked face to face with people from an other continent in charge of our switch to Azure, and he was neither aware nor involved in even the decision of Azure vs AWS vs Google Cloud. But he was in charge... I came to learn that the main reason we chose Azure (but I think the biggest problem was to choose Cloud) was because we already had teams doing Windows maintenance, server management, etc: we had way more relationship with MS already than with Amazon or Google. Now we have it, it's beautiful, nobody uses it, people grudgingly do trainings, and make big presentation about Cloud strategies, while waiting for legal and compliance to allow us to connect to it. Since we figured after the contract was signed that it would be mostly illegal for us to give Microsoft any of our client data... Whatever, happy to see sometimes other people struggling with the bewildering fact that we somehow regressed :D
- oceanplexian 6y agoLook on the bright side, when it comes crashing down, there will be a whole lot of money to be made in undoing the damage done by frameworks upon frameworks inside of containers/serverless locked up in proprietary cloud services and APIs. And there won't be a lot of competition, either, since the old timers are retiring and many of the younger engineers couldn't tell you how a Kernel Module or a POSIX Signal works if their lives depended on it.
- vineyardmike 6y ago
- Olreich 6y agoWhile that’s generally true, the inputs and outputs and execution environment that you can count on is well-documented on AWS Lambda. That means that recreation of execution environment for the function in question is pretty reasonable locally (why is this not a thing then Amazon?!?). For the rest of AWS, you can either get credentials in place for access or mock those out somehow, but that’s not officially supported either. I feel like the service is leaving a bunch of easy work on the table that would let people have a local development process that mirrors the remote development reality.
- scrollaway 6y agoIt is a thing; unfortunately it's not officially supported but I also find it crazy that so many developers are fine doing dev directly on "the cloud", as if that's a reasonable use of their time. https://github.com/lambci/docker-lambda https://github.com/lambci/docker-lambda https://github.com/localstack/localstack https://github.com/localstack/localstack
- zerubeus 6y agoLack of understanding, I think every new dev to serverless will have this stage of frustration, then eventually will find a way to do better with creative ideas, I myself never run lambda in the cloud I write a UT for my handler and assume I know the input and the expected output, I mock everything except step functions I use the official jar from AWS.
- pbourke 6y ago> You'd get a printout of the results back hours later, or even tomorrow. See also “big data” and machine learning.
- penguin_linux 6y agoWhile I understand this viewpoint, and the comparison makes _some_ sense.. Kubernetes seems to solve all of these problems to me. > Does not provide all-local workstations. - Solutions like minikube fix this. > Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. Kubernetes has the same developer experience everywhere. As far as OP's serverless trouble, why not just use knative?
- danmaz74 6y agoAs long you only use k8s and no (other) managed services in the mix.
- mk81 6y agoIn addition to K8s, the big clouds (and even DO) all support S3-compatible storage, SQL and Redis/Memstore.
- jokethrowaway 6y agoKyma would be the FaaS replacement for k8s/knative (I haven't tried it yet as the whole functions fad still hasn't hit me)
- mk81 6y agoHaven't tried Kyma but I like OpenFaaS.
- usr1106 6y ago> The cloud is the new time-share mainframe. > You'd get a printout of the results back hours later. Not sure what you are trying to say: What you describe is not time-share, but batch processing.
- junon 6y agoEh? No, the "hours later" bit was because other people were using the mainframe. Not because it was "big data".
- chrisweekly 6y agoIMHO leveraging serverless properly generally means writing tiny, stateless, ephemeral, single-responsibility, "pure" functions. Cloudwatch might be underwhelming, but complicated debugging scenarios, with hours-long delays, isn't necessarily representative.
- Silhouette 6y agoIMHO leveraging serverless properly generally means writing tiny, stateless, ephemeral, single-responsibility, "pure" functions. Unfortunately, what it most often seems to be used for is triggering side effects in response to some event, which is exactly the opposite of a pure function. Local testing of that logic is probably reliant on mocking all the other services it integrates with. Beyond that, the next step tends to be remote integration testing on the actual cloud hosting service, using some sort of separate staging infrastructure that will possibly be shared among many developers. Some of the ideas and flexibility that should be offered by cloud hosting are appealing, but in practice, it often feels like we've regressed decades in our software development practices, discarding successful ideas like shorter feedback loops and unit testing and one-step build/deploy processes.
- meekins 6y ago> the next step tends to be remote integration testing on the actual cloud hosting service, using some sort of separate staging infrastructure that will possibly be shared among many developers One of the results of the pay-per-use pricing model of services utilized in serverless architectures is that you can easily provision dirt-cheap developer and feature-branch specific environments for integration-level testing. While you still can't attach a debugger in your function, the isolation from changes others are working on makes testing and exploration much more pleasant than when working in a shared dev environment. Things only get complicated when your architecture includes more expensive components with different pricing schemes that you probably want to share with others (e.g. relational databases, data streams).
- Silhouette 6y ago
- pantulis 6y agoThis is a great insight.
- Bombthecat 6y agoIbm cloud is especially bad. Tried api connect. There is no way! To get any logs, debug information or anything! Nothing! Zero! You only get what the api endpoint tells the client... It's insane..
- tdfirth 6y agoThat's really interesting, thanks for sharing. I did not have the pleasure(?) of working in the industry back then, as is true of most developers these days. Do you hold out hope for a similar transition to a better local development workflow? What do you think it might look like?
- jiggawatts 6y agoThe major cloud vendors have a vested interest to the tune of trillions of dollars for ensuring lock-in. All the cloud vendors want to become the next IBM, except this time with recurring revenue instead of one-time sales, something IBM never succeeded at despite trying very hard. Openness, standardisation, commoditisation, and low-tier equivalent offerings are an anathema to the vision of these mega-corps. They will fight tooth and nail to prevent it. Look at what Microsoft is doing: Instead of moving Azure features down to their IDEs, they are moving their IDE into the cloud! VS Code is Electron-based, which is to say it is just a local web server plus a web browser shell. It's not rocket science to move the server component into the cloud, and have the standard customer web browser be the front-end. I've noticed recently that this has started to happen. There's already a SQL query editor based on VS Code, and there's editors popping up for App Service and I think Functions also (equivalent to AWS Lambda). This is the ultimate lock-in, where you develop "on" the platform directly, and have no hope of ever reproducing any part of it locally: the build system, the debugger, source control, tracing tools, or the IDE itself. Other people in this thread mentioned local Kubernetes development flows. They're too busy drinking the Kool-aid. The cloud vendors are not interested in ever making this a long-term viable option and will find creative ways of making such open development practices non-tenable.
- zerubeus 6y agoThere's already good projects doing it check the https://serverless-stack.com/ https://serverless-stack.com/
- the_arun 6y agoProblem is aws or gcp or azure do not follow any standard spec for their services. Ideally, they should first come up with a spec - and provide reference implementation. Every provider will implement solutions for these standard specs. That way customers can use services for the spec. Customers can move between providers or work with multiple providers at the same time. Level playing field & let the best provider win the customer loyalty. But, all providers want to lock in customers. So, we are at the wrong end of the stick.
- robertlagrant 6y agoGCP -> Kubernetes. Pretty portable.
- the_arun 6y agoI should have been bit more specific. I meant managed services like s3, dynamodb etc
- roguas 6y agos3 is pretty much defacto standardized api for object store... tons of others follow minio, digital ocean etc. dynamodb despite being a heavily promoted product at aws is not really a staple, but a crutch since rdbms doesn't scale that well
- roguas 6y agoTo some extent. I have seen tons of scenarios istio/loadbalancing/other managed services that cannot be easily reproducible without 'the cloud'. I would say it is much more portable.
- mraza007 6y agoWow I’m just amazed how you summed up the modern day development by giving an example from the past. I’m just curious we have three big cloud players at the moment Azure,AWS and GCP but do we have open source alternatives to their services that these cloud provides provide which you can host your self on large servers
- zerubeus 6y agok8s
- smbv 6y ago> Is closed source. Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. You can't peer into their source code, it is all proprietary and even secret. I'm a big fan of [OpenStack](https://www.openstack.org/ https://www.openstack.org/) as it is open source and definitely enterprise-grade.
- jxf 6y agoA fantastic summary. My bread and butter is helping companies get to the cloud, but my number one guideline is that the local developer experience must be supported as much as the infrastructure improvements are. Otherwise, you get a mishmash of fantastic visibility into your workloads and better provisioning cycle times, which then immediately get cancelled out by developers who have to work 3x as hard to do the same thing they were doing before.