4 ms·

Sounds like its working for you, which is great, I guess. However in general what you're doing isn't devops, its 'devs creating a big old mess for someone else

Sounds like its working for you, which is great, I guess. However in general what you're doing isn't devops, its 'devs creating a big old mess for someone else to cleanup later'. All you're doing is trading long term maintainability and quality for 'getting it done right now'. Sure; it works. I get it. ...but I've also been on the receiving end when that 2-person team has 1 person leave and gets scaled up to a 5-person team, and you have to a) bin everything, and b) write the entire thing from scratch. I personally consider it a very selfish way to build things. Me first. Me now. ...someone else's problem later. There are scalable solutions in this space (eg. pipelines, airflow, azure data factory, glue); doing it using lambdas is... not. ideal. That's not a problem with severless in general; but what you've described isn't any of the things which is good about serverless either: what you've described is the problem with the low barrier to entry with serverless, resulting in production serverless code that skips past 'quality control' to 'let's deploy straight to production!'.

justinram11 6y ago
While I think that there is definitely a worry of it turning into a big old mess (especially if they start depending on and talking to each other), I don't really know that I'd consider cron-job style independent one-off tasks any more maintainable in a monolith. I also don't know that I'd consider that specific to serverless (it's easy to create a ball of mud in anything). All of our serverless projects are located in a single repository, and each little service gets its own little google doc that explains what it does, what functions it has, and any resources that it uses. As mentioned below, as soon as an ETL gets multiple steps I agree that pure-lambda is not the right tool for the job (Airflow would be my preference, though we've also successfully used Step Functions)
nafey 6y ago
This assumes that survival is guaranteed if you do it the right way. Long term thinking can work in firms that are larger. For smaller firms with limited resources, motivation and technical expertise getting velocity is crucial. This is one of the reasons startups run into issues when they have to scale. I think it has to be a well thought out balance of speed and quality that is required in such environments.
- wokwokwok 6y ago
  That's such a straw man, it pains me to even respond to it. Doing it 'quick and dirty right now with the tools I happen to know' instead of 'doing some research, then doing it in a way that is maintainable using standard tools and taking a bit longer' isn't 'endangering the survival of the company', it's professional level practice. Failing to do so is, a) negligent, and b) incompetent. I'm certainly not suggesting the OP isn't competent; they seem on the ball, and maybe serverless is the best choice for them because (insert reasons here)...but I think its fair to say, if you're using your one tool (ie. serverless) for all your tasks, you probably haven't done your due diligence. Specifically in this case, using a standard ETL tool to do ETL type tasks... would probably be suitable. Anyhow; no. Long term thinking isn't what this is about; it's about behaving like a professional when you do a professional job. You pick the label you want to wear as an engineer, because, your practices define you.
  - cheeze 6y ago
    You are on such a high horse, it pains me to even respond to you. You can call it devops, a mess, whatever you want... but nothing about serverless frameworks is creating that mess. A service is a service, whether it's running Lambda, Spring Boot, or whatever the hottest new framework is... Nothing prevents you from documenting a service. Nothing prevents you from creating basic alarms (it's literally easier for me to create these alarms because of CDK than it is to alarm on the standard service framework I use). We get it, you don't like serverless. But you're going to rag on OP and question their competence, and then drop some "you pick the label you want to wear"? Come on dude, seriously? You're strawmanning too here... Can you show me where OP said they are using one tool for all of their tasks? they literally said "for one-off, little processes." I cant imagine what it's like to be this arrogant.
    - wokwokwok 6y ago
      If you read my comment, you’ll realise that I’m not saying serveless is bad. I’m saying; if that’s your only tool, you’re not doing your job. “Velocity” is not an excuse for not doing your job to a professional degree. Its a strawman people use to justify doing substandard work (or, often, pressuring engineers who know better, into doing substandard work).
gouggoug 6y ago
As someone in charge of cleaning up the mess in a company that has 800+ employees and 300+ developers, I can only agree. My company's infrastructure was never tended to and was apparently a free for all for a few years. A year or two ago, it was finally decided the free for all needed to end and all the developers saw their SSH root access removed. My team of 5 SREs is now facing a constant backlash from developers frustrated at the rotten infrastructure, constant bugs, and impossibility to ssh into machines. We're slowly fixing the mess, migrating to Kubernetes and things are starting to get better, but god, how painful it is. So yea, in its infancy a startup's infrastructure can't be perfect, but every single shortcut you're taking now, you'll pay double down the line. Choose your path wisely.

justinram11 6y ago

While I think that there is definitely a worry of it turning into a big old mess (especially if they start depending on and talking to each other), I don't really know that I'd consider cron-job style independent one-off tasks any more maintainable in a monolith. I also don't know that I'd consider that specific to serverless (it's easy to create a ball of mud in anything). All of our serverless projects are located in a single repository, and each little service gets its own little google doc that explains what it does, what functions it has, and any resources that it uses. As mentioned below, as soon as an ETL gets multiple steps I agree that pure-lambda is not the right tool for the job (Airflow would be my preference, though we've also successfully used Step Functions)

nafey 6y ago

This assumes that survival is guaranteed if you do it the right way. Long term thinking can work in firms that are larger. For smaller firms with limited resources, motivation and technical expertise getting velocity is crucial. This is one of the reasons startups run into issues when they have to scale. I think it has to be a well thought out balance of speed and quality that is required in such environments.

wokwokwok 6y ago
That's such a straw man, it pains me to even respond to it. Doing it 'quick and dirty right now with the tools I happen to know' instead of 'doing some research, then doing it in a way that is maintainable using standard tools and taking a bit longer' isn't 'endangering the survival of the company', it's professional level practice. Failing to do so is, a) negligent, and b) incompetent. I'm certainly not suggesting the OP isn't competent; they seem on the ball, and maybe serverless is the best choice for them because (insert reasons here)...but I think its fair to say, if you're using your one tool (ie. serverless) for all your tasks, you probably haven't done your due diligence. Specifically in this case, using a standard ETL tool to do ETL type tasks... would probably be suitable. Anyhow; no. Long term thinking isn't what this is about; it's about behaving like a professional when you do a professional job. You pick the label you want to wear as an engineer, because, your practices define you.
- cheeze 6y ago
  You are on such a high horse, it pains me to even respond to you. You can call it devops, a mess, whatever you want... but nothing about serverless frameworks is creating that mess. A service is a service, whether it's running Lambda, Spring Boot, or whatever the hottest new framework is... Nothing prevents you from documenting a service. Nothing prevents you from creating basic alarms (it's literally easier for me to create these alarms because of CDK than it is to alarm on the standard service framework I use). We get it, you don't like serverless. But you're going to rag on OP and question their competence, and then drop some "you pick the label you want to wear"? Come on dude, seriously? You're strawmanning too here... Can you show me where OP said they are using one tool for all of their tasks? they literally said "for one-off, little processes." I cant imagine what it's like to be this arrogant.
  - wokwokwok 6y ago
    If you read my comment, you’ll realise that I’m not saying serveless is bad. I’m saying; if that’s your only tool, you’re not doing your job. “Velocity” is not an excuse for not doing your job to a professional degree. Its a strawman people use to justify doing substandard work (or, often, pressuring engineers who know better, into doing substandard work).

gouggoug 6y ago

As someone in charge of cleaning up the mess in a company that has 800+ employees and 300+ developers, I can only agree. My company's infrastructure was never tended to and was apparently a free for all for a few years. A year or two ago, it was finally decided the free for all needed to end and all the developers saw their SSH root access removed. My team of 5 SREs is now facing a constant backlash from developers frustrated at the rotten infrastructure, constant bugs, and impossibility to ssh into machines. We're slowly fixing the mess, migrating to Kubernetes and things are starting to get better, but god, how painful it is. So yea, in its infancy a startup's infrastructure can't be perfect, but every single shortcut you're taking now, you'll pay double down the line. Choose your path wisely.