7 ms·
I’m working as a single solo developer of a tiny video game. I’m writing it in C with raylib. No coding assistants, no agents, not even a language server. I on
by agentultra 6mo ago
I’m working as a single solo developer of a tiny video game. I’m writing it in C with raylib. No coding assistants, no agents, not even a language server.
I only work on it for a few hours during the week. And it’s progressing at a reasonable pace that I’m happy with. I got cross-compilation from Linux to Windows going early on in a couple of hours. Wasn’t that hard.
I’ve had to rework parts of the code as I’ve progressed. I’ve had to live with decisions I made early on. It’s code. It’s fine.
I don’t really understand the, “more, better, faster,” cachet to be honest. Writing the code hasn’t been the bottle neck to developing software for a long time. It’s usually the thinking that takes most of the time and if that goes away well… I dunno, that’s weird. I will understand it even less.
Agree with writing less code though. The economics of throwing out 37k lines of code a week is… stupid in the extreme. If we get paid by the line we could’ve optimized for this long before LLM’s were invented. It’s not like more lines of code means more inventory to sell. It’s usually the opposite: the more bugs to fix, the more frustrated customers, the higher churn of exhausted developers.
- anitil 6mo agoI had a look at your github and blog but couldn't find the game, is it public? Or do I need to watch your streams to see it?
- bdangubic 6mo ago> Writing the code hasn’t been the bottle neck to developing software for a long time. I see this on HN just so much and I am not sure what this is, almost seems like a political slogan that followers keep repeating. I had to do some rough math in my head but in the last 5 years I have been involved with hiring roughly 40 SWEs. Every single one of them was hired because writing the code was THE bottleneck (the only one) and we needed more people to write the code
- ahefner 6mo agoBut how much time per week does an SWE actually spend writing code?
- bdangubic 6mo agoanother one, this is 2nd most frequent thing people write here, not sure how to even approach answering :) so I’ll do what I was thought in first grade to never do and answer a question with a question - how much time per week does a brick layer spend laying bricks? they are looking at these new “robots” laying bricks automatically and talking on BrickLayerNews “man, the brick laying has not been a bottleneck for a long time.” But to answer your question directly, a lot of time if other people do their job well. Last week I had about 7 hours of meetings, the rest of the time I was coding (so say 35 hours) minus breaks I had to take to stretch and rest my eyes
- paulhebert 6mo agoInteresting! I guess it really varies between jobs, roles, and companies. Thats never been my experience but I have an odd skill set that mixes design and dev. I’ve always spent a lot of time planning, designing, thinking, etc. How detailed are the tickets if you spend all your time coding? You never have to think through architecture, follow up on edge cases the ticket writers didn’t anticipate, help coworkers with their tasks, review code, etc.?
- deleted 6mo ago[deleted]
- apsurd 6mo agoi think this is it. you're a bricklayer. No, the bottleneck for erecting buildings is not bricklaying. Without taking all the time to write a dissertation to try to convince you, because why; how about we just start with even zoning laws and demographic analysis preclude the laying of the bricks. is it so unreasonable to think it is not about the laying of the bricks?
- slopinthebag 6mo agoI think you comparing software development to brick laying says all anyone needs to hear about your approach to software development. It's like saying the bottleneck in mathematics is arithmetic.
- jordwest 6mo agoI've worked in two different types of environments - one where what you said is absolutely true (most of my jobs), and another where it's not true and the quote holds up. The difference, I think is: - Code factories where everything is moving fast - there's no time to think about how to simplify a problem, just gotta get it done. These companies tended to hire their way out of slowness, which led to more code, more complexity, and more code needed to deal with and resolve edge cases introduced by the complexity. I can count many times I was handed directives to implement something that I knew was far more complex than it had to be, but because of the pressure to move forward it was virtually impossible to push back. Maybe it's the only way they can make the business case work, but IMO it undoubtedly led to far, far more code than would've been necessary if it were possible to consider problems more carefully and if engineers had more autonomy. In these companies also a lot of time was consumed by meetings trying to "sync up" with the 100 other people moving in one direction. - Smaller shops, open source projects, or indie development where there isn't a rush to get something out the door. Here, it's possible to think through a problem and come up with a solution that reduces code surface area. This was about solving the largest number of problems with the least amount of complexity. Most of my time at this company was spent thinking through how to solve the problem and considering edge cases and exploratory coding, the actual implementation was really quick to write. It really helped that I had a boss who understood and encouraged this, and we were working on safety critical systems. My boss liked to say "you can't birth a baby in less than 9 months just by adding another woman". I think most of the difference is in team size. A larger team inherently results in more code to do less, because of the n*(n-1)/2 communication overhead [1]. Recently I learned the Navy SEALs saying "Slow is smooth, smooth is fast" which I feel sums up my experience well. [1] https://en.wikipedia.org/wiki/The_Mythical_Man-Month https://en.wikipedia.org/wiki/The_Mythical_Man-Month
- sublinear 6mo agoI think your mind might be blown when you discover a third type of environment. It's neither a small shop of yak-shaving idealists, nor a desperate code factory. The third environment is a large business maintaining services long term. These services do not change in fundamental ways for well over a decade and they make a shit ton of money, yet the requirements never stop changing in subtle ways for the clients. Bugs pop up constantly, but there's more than enough time to fix them the right way as outlined by their contract where expectations have been corrected over the years. There's no choice to do it any other way. The requirements and deadlines are firm. Reliability is the priority. These are the stable businesses of the broader working world and they're probably what will remain after AI has driven the tech industry into the ground.
- agentultra 6mo agoIf you’ve never read Fred Brooks, I’d recommend it. The aphorism is a bit dated but rings true: you can’t add another developer and make the process go faster. It usually slows teams down. I’ve seen it time and again: startups move from their market-fit phase into an operational excellence phase on the backing of VC funding and they start hiring a ton of people. Most of those developers are highly educated, specialized people with deep technical skills and they’re often put to work making the boxes more blue or sitting in meetings with PMs for hours. Teams slow down output when you add more people. You don’t have a quota. It’s not like you’ll have fewer units to sell if you don’t add that 30k lines of code by Friday. This is knowledge work. The work is understanding problems and knowing how to develop solutions to them. You add more people and you end up adding overhead. Communication, co-ordination, operations overhead. The real bottle necks are people and releasing systems into production. Every line of code change is a liability. There’s risk tolerance to manage in order to achieve five-nines. A well-sized team that has worked together a long time can outperform a massive team any day in my experience.
- jordwest 6mo ago> they’re often put to work making the boxes more blue or sitting in meetings with PMs for hours Haha, this is exactly my experience. I'll never forget the best candidate I ever interviewed - my feedback was to absolutely hire him and put him on the most interesting and challenging problems. They put him in a marketing team tweaking upsell popups. He left after 2 months.
- bdangubic 6mo ago> The aphorism is a bit dated but rings true: you can’t add another developer and make the process go faster. It usually slows teams down. I've been doing this for 30-years and this is another political slogan of sorts. this is true in every single imaginable job - new people slow you down, until they do not and become part of the well-oiled machine that is hopefully your team. not sure why people insist on saying this, it is like saying "read this book, says that that Sun will rise tomorrow morning" > I’ve seen it time and again: startups move from their market-fit phase into an operational excellence phase on the backing of VC funding and they start hiring a ton of people. Most of those developers are highly educated, specialized people with deep technical skills and they’re often put to work making the boxes more blue or sitting in meetings with PMs for hours. Teams slow down output when you add more people. I wasn't talking about startups or developers making boxes more blue, I was talking about personal experience. The bottleneck, if you are doing amazing shit and not burning some billionaries money on some silly "startup" is always the code which is why we hire developers to write the code. Everything else is just coming up with some silly unrelated examples - of course there are people (at every job again) doing nothing or menial tasks - this isn't what I was talking about. > You don’t have a quota. It’s not like you’ll have fewer units to sell if you don’t add that 30k lines of code by Friday. I do have customers that want features that would make their lives easier and are willing to handsomely pay for it, that good enough? > This is knowledge work. The work is understanding problems and knowing how to develop solutions to them. You add more people and you end up adding overhead. Communication, co-ordination, operations overhead. This is only on super shitty teams with super shitty co-workers (especially senior ones) and super shitty managers. I feel for the careers in this industry where this is/was the case. A whole lot of people are terrible at their jobs in places like this - a whole lot of people... > A well-sized team that has worked together a long time can outperform a massive team any day in my experience. a well-sized team was at one point (well-sized team - 1) and (well-sized team - 2) and (well-sized team - 3) and in the future if it is right team will be even more amazing as well (well-sized team + 1), (well-sized team + 2)
- apsurd 6mo agoEven if the entire totem pole of decision makers in a company thinks writing code is the bottleneck doesn't make it true that writing code is the bottleneck. On the extreme end to prove the point, the suits intentionally abstract out reality into neat forecasts and spreadsheet cells. It's hard for me to think of something concrete that will convince you. Does code map directly to business outcomes in your experience? Because it's overwhelmingly not even remotely true in my experience. even just "all lines of code are not created equal" tells me there's no direct correlation with business value.
- Jach 6mo agoI agree the slogan isn't very true. It's similar to another line of commentary that would suggest soft skills are more important than the hard skills of actually being able to program, i.e. the primary service being paid for. There is some truth to it, like Brooks' Law (https://en.wikipedia.org/wiki/Brooks's_law https://en.wikipedia.org/wiki/Brooks's_law) about how adding people to an already late project will just make it later. There are many factors in how long a software engineering task takes beyond pure typing speed, which suggests there are factors beyond code produced per day as well. But some typing has to be done, and some code has to be produced, and those can absolutely be bottlenecks. Another way of looking at it that I like is Hickey's hierarchy of the problems of programming and their relative costs, from slide 22: https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/EffectivePrograms.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi... If you have inherent domain complexity, or a misconception on how to apply programming to a domain, those are 10x worse costs than any day-to-day practice of programming concerns ("the code"), and there's a 10x further reduction for trivialisms like typos. I think some of it must be cope since so many are in organizations where the more they get promoted the less they program, trending towards (and sometimes reaching) 0. In such an organization sure, code isn't the bottleneck per se, it's a symptom of an underlying cause. The bottleneck would be the bad incentives that get people to schedule incessant unnecessary meetings with as many people as they can to show leadership of stakeholders for promotion doc material, and other questionable things shoved on the best engineers that take them away from engineering. Remove those, and suddenly productivity can go way up, and code produced will go up as well. I've also always been amused by estimates of what constitutes "good" productivity if you try to quantify it in lines of code. There's a paper from 1994 by Jim Coplien, "Borland Software Craftsmanship: A New Look at Process, Quality, and Productivity". It's summarized in the free book by Richard Gabriel, "Patterns of Software". (https://www.dreamsongs.com/Files/PatternsOfSoftware.pdf https://www.dreamsongs.com/Files/PatternsOfSoftware.pdf pg 135) They were making new spreadsheet software for Windows, and had a group of "very high caliber" professionals, with a core group of 4 people (2 with important prior domain expertise) and then 4 more team members added after a year. "The QPW group, consisting of about eight people, took 31 months to produce a commercial product containing 1 million lines of code. This elapsed time includes the prototypes, but the line count does not. That is, each member of this group produced 1,000 lines of final code per week." Later on, Coplien was asked "what he thought was a good average for US software productivity", and the answer was "1000 to 2000 non-commentary source lines per programmer per year". Also: "this number was constant for a large in-house program over its 15-year lifetime -- so that original development and maintenance moved at the same pace: slowly". An average of 1k lines a year is 19 lines a week, or about 4 lines a day for a work-week. This was considered acceptable for an average, whereas for an exceptional team you could get 200 a day. Might not there be ways to boost the average from 4 to something like 12 or 20? If your organization is at 4, there is clearly a bottleneck. (For extra context, the QPW group was in C++, and Gabriel notes he had personal experience with several groups demonstrating similar productivity levels. "I watched Lisp programmers produce 1000 lines of Lisp/CLOS code per month per person, which is roughly equivalent to 350 to 1000 lines of C++ code per week." Of course language matters in lines of code comparisons.)
- slopinthebag 6mo agoIt sounds like you just aren't very good at managing teams of programmers tbh. If your bottleneck is producing code, very rarely does hiring more programmers actually help.
- bdangubic 6mo agothis makes just so much sense, I'll manage my programmers to work 37 hours per day, imma try this next week and will let you know how it goes
- slopinthebag 6mo agoTry having your programmers work 30 hours a week instead of 40 and measure their output. You might be surprised.
- brianwmunz 6mo agoHonestly I think you can tell pretty quickly if a company or person is approaching AI from the viewpoint of accelerating development and innovation or just looking to do the same amount of work with less people. The space has been flooded by mean-spirited people who love the idea of developers becoming obsolete, which is a viewpoint that isn't working out for a lot of companies right now...many are already scrambling to rehire. Approaching the situation practically, integrating AI as a tool and accelerator is the much smarter way and if done right will pay for itself anyway.
- shimman 6mo agoThose mean spirited people are actually capitalists and they've been chasing the dream of perpetual labor since the 1800s.
- 20k 6mo ago>I don’t really understand the, “more, better, faster,” cachet to be honest. Writing the code hasn’t been the bottle neck to developing software for a long time. It’s usually the thinking that takes most of the time and if that goes away well… I dunno, that’s weird. I will understand it even less. This is what I've always found confusing as well about this push for AI. The act of typing isn't the hard part - its understanding what's going on, and why you're doing it. Using AI to generate code is only faster if you try and skip that step - which leads to an inevitable disaster
- koolba 6mo ago> The act of typing isn't the hard part - its understanding what's going on, and why you're doing it. Using AI to generate code is only faster if you try and skip that step - which leads to an inevitable disaster It’s more than just typing though. A simple example remembering the exact incantation of CSS classes to style something that you can easily describe in plain English. Yes, you could look them up or maybe even memorize them. But there’s no way you can make wholesale changes to a layout faster than a machine. It lowers the cost for experimentation. A whole series of “what if this was…” can be answered with an implementation in minutes. Not a whole afternoon on one idea that you feel a sunk cost to keep.
- sublinear 6mo agoThis is not correct. CSS is the style rules for all rendering situations of that HTML, not just your single requirement that it "looks about right" in your narrow set of test cases. Nobody writing production CSS for a serious web page can avoid rewriting it. Nobody is memorizing anything. It's deeply intertwined with the requirements as they change. You will eventually be forced to review every line of it carefully as each new test is added or when the HTML is changed. No AI is doing that level of testing or has the training data to provide those answers. It sounds like you're better off not using a web page at all if this bothers you. This isn't a deficiency of CSS. It's the main feature. It's designed to provide tools that can cover all cases. If you only have one rendering case, you want an image. If you want to skip the code, you can just not write code. Create a mockup of images and hand it off to your web devs.
- colechristensen 6mo ago>Writing the code hasn’t been the bottle neck to developing software for a long time. Then we're doing different things. I didn't like GitHub so I wrote my own. 60k lines of code later... yes writing code was the bottleneck which has been eliminated. The bottleneck is now design, review, and quality assessments that can't be done trivially. This isn't even the project I wanted to be doing, the tools that were available were holding me back so I wrote my own. It also consumes a few hours a week. If you think writing code isn't the bottleneck then you aren't thinking big enough. If you don't WANT to think big enough, that's fine, I also do things for the joy of doing them.
- ozim 6mo agoWe do different things, I do code for other people to use. Once we tried shipping features and updates every week, because we could ideate, code, test and deploy that fast. No user wanted that - product owners and business wanted that or they thought they wanted, until users came with torches and pitchforks. Don’t forget there is user adoption and education. Churning out features no one will use because they don’t know about is useless.
- bitexploder 6mo agoYou can use LLM to write less code too. Just takes more intention. Which is kind of the whole point.
- dpark 6mo ago> Writing the code hasn’t been the bottle neck to developing software for a long time. It’s usually the thinking that takes most of the time Does your coding not involve thinking? And if not, why are you not delighted to have AI take that over? Writing unthinking boilerplate is tedious garbage work. Today I wanted to address a bug I found on a product I work on. At the intersection of platform migration and backwards compatibility I found some customers getting neither. I used an LLM to research the code paths and ensure that my understanding of the break was correct and what the potential side effects of my proposed fix would be. AI saved me stepping through code for hours to understand the side effects. I asked it for a nice description of the flow and it gave it to me, including the pieces I didn’t really know because I’d never even touched that code before. I could have done this. Would it have been a better use of my time than moving on to the next thing? Probably not. Stepping through function calls in an IDE is not my idea of good “thinking” work. Tracing through glue to understand how a magical property gets injected is a great job for a machine.
- lelanthran 6mo ago>> Writing the code hasn’t been the bottle neck to developing software for a long time. It’s usually the thinking that takes most of the time > I used an LLM to research the code paths and ensure that my understanding of the break was correct and what the potential side effects of my proposed fix would be. Using the LLM for understanding is very different to using the LLM for codegen. You are not really disagreeing with the author here; it's just that for the specific project he is talking about, he already understands it just fine so the advantages of LLM help in understanding is tiny.
- dpark 6mo agoMy point is that these are not separate activities. They are drawing a false distinction between thinking and coding and then asserting that code speed doesn’t matter and implying that AI only helps with the coding bit. None of this is actually true, though. Coding and thinking are often tightly intertwined, as rarely is the coding piece so straightforward that it requires no interesting thought. Coding speed does matter, even if it’s not the primary bottleneck for many things. And AI can be very helpful outside the context of pure coding.
- throwaw12 6mo ago> Writing the code hasn’t been the bottle neck to developing software for a long time It was! pre-2022 people needed developers to build software for them, now with platforms like Replit, Lovable - people are creating their own tiny software projects, which wasn't easily accessible in the past. If you say coding wasn't a bottleneck, then indirectly you could also say, you don't need developers. If you need developers, outcome of their other type of work (thinking, designing based on existing tools and so on) is actually CODE.
- changyou 6mo ago[dead]
- ignoramous 6mo ago> Writing the code hasn’t been the bottle neck to developing software for a long time. For who? There's no lack of professional programmers who couldn't clear FizzBuzz now coding up company-sized systems using Agents. This is all good as long as agents can stick to the spec/req & code it all up with decent enough abstractions... as the professional approving it is in no position to clue it on code organization or bugs or edge cases. I think, we (as a society) are looking at something akin to "reproducibility crisis" (software full of Heisenbugs) as such "vibe coded" systems get widely sold & deployed, 'cause the "pros" who excel at this are also good at... selling.
- agentultra 6mo agoValid concern and one I share. If you’re going to vibe code up an operating system you will still need the experience and understanding of operating system fundamentals to have a chance at producing anything useful. The one-shot vibe-coded C compiler is a good example. Sure it created a compiler that could pass the basic tests. But it was no where near a plausible or useful compiler you’d use in a production system. Someone who knows compilers reviewed it and was able to prompt Claude or Gemini to fix the issues. But still… you’re not going to be able to do that unless you know what to look for. On an enterprise development team doing boring, Line of Business software? Might have a chance at rolling the dice and trusting the agents and tests and processes to catch stuff for you but I’d still be worried about people who don’t know what questions to ask or have deep expertise to know what is “good,” etc.
- daotoad 6mo agoThe gap between quality work and baseline LLM output is precisely the understanding. If it can be validated by automation, the bot will do it. But no automation suite is complete or perfect. What concerns me is that building software using the LLMs gives a distance that inhibits the formation of the sort of understanding I need to "just know" a code base intuitively. So when product asks for a feature, my ability to be sufficiently pedantic about the 6 different non-obvious things this impacts is less effective. And when I need to choose abstractions and try to form an effective ontology, my intuition is less effective. I believe I can still grind out an effective solution, but I start farther from the finish line. Does the LLM's ability to "answer questions" about the codebase make up for my lack of intuition? Does my apparent ability to run faster make up for the fact that I am starting farther from the end of the race? I don't know yet.
- hnthrowaway0315 6mo ago“more, better, faster,” I have heard these words, almost verbatim, from manager-yes-men coming from a FAANG background, and surprisingly concentrated in a certain demography (if someone find this offensive, I'll remove this part). My CTO wants us to "deliver as fast as possible", and my VP wants us to "go much faster, and more ownership". "Better" or anything related with quality was definitely mentioned, too, but always at a second place. To this day, I consider these yes-men to be a major red flag, so I always tried to probe for such information during interviews.
- drekipus 6mo agoI'm interested in how you're probing for them? I haven't needed to change jobs but I'm worried that one day I'll have to
- hnthrowaway0315 6mo agoJust ask along the line "Do you have someone from FAANG as a director/VP? Because I have worked with a few and really loved the learning process" and do a bit of research once you know the name of the hiring manager.
- YesBox 6mo agoI've been developing a moderately popular (for an indie) game for over 4 years at this point (full time). C++, SFML, SQLite. Same as you: no coding assistants, no agents, etc. I also don't use git. [1] One of the largest speedups is from how much of the codebase I can keep in my head. Because I started from an empty C++ file, the engine reflects how I reason and organize concepts (lossless compression). Thus most of the codebase is in my brains RAM. I don't see how LLM agents are going to improve my productivity in the long run. The less a person understands their code (organized logic), the more abstracted the conversation is going to become when directing an agent. The higher up the abstraction ladder you go, the less distinct your product becomes. [1] And very, very rarely have I wished I had it for a moment. Not using git simplifies abstracted parts of development. No branches, no ballooning of conceptual tangents, etc. Focus on one thing at a time. Daily backups and a log of what I worked on for the day suffices should I need to revisit/remember earlier changes. I've never been in a situation where I change I made over a week ago interfered with todays work.
- agentultra 6mo agoI definitely feel like understanding the system is a big part of what makes it relatively easy to maintain/understand. My game is just in my spare time while I'm looking for work and the scope of the project is small so that I can finish it, release it, and start working on the next one. I'm not trying to build an engine or anything. Just a game. Not even the best game I can make. I can iterate on it fast because I know the structures. I can refactor it fast because I've built an intuition for a process over time that keeps code amenable to changes over time. I know I'm not going to make the right decisions at the start so I avoid committing to generalizations, etc. Editing code is pretty fast for me. Again, years working with a particular setup. I still have expandable snippets, multi-cursor editing and a host of macros for common editing motions. Checking changes... pretty fast. I'm getting to the point where I might invest in using dynamic reloading for my in-development builds. I suspect it will take a few hours to do at most. Not a big deal. For now I have a basic system that just watches for file changes and recompiles/re-runs the program. In a different context, working on a team in a large multi-million line codebase... I dunno what other people find it's like but I've never found it terribly slow to write/edit code or ship features. I can usually knock most tasks out at a reasonable pace especially when my familiarity with the area of the code I'm working in increases. I usually find my priorities shift with the demands of users, the business, etc. Some times I work on shipping new features quick. Other times it's making sure what we ship the right things and done well so that we don't leak PII. Either way... actually writing the code isn't the slowest part, in my experience. It's all the other stuff: the meetings, the design, maintenance, documentation, understanding the problem domain, actually shipping the code to production, etc that takes up the most time for me.
- pdimitar 6mo agoThis: > I don’t really understand the, “more, better, faster,” cachet to be honest And this: > I’m working as a single solo developer ...I believe explain it all here. You likely are not beholden to PMs, CEOs and the like. Of course you can go at your own pace. I am actually puzzled that you don't understand that aspect yourself. > The economics of throwing out 37k lines of code a week is… stupid in the extreme Again, bosses. CEOs have 14 calls a week with potential prospects and sometimes want demos, sometimes they sign quickly and want a prototype, and sometimes they arrange a collab with a friend or family. Then 3 weeks later the whole thing falls apart and you have to throw it away because it's getting in the way of delivering what actually still pays the bills. I am not the CEO. I try making his visions come true. I don't get to make the calls on whether 37k of lines will be quickly churned out and then deleted some weeks later. I think your comment is overly focused only on the coding/programming aspect of things. We don't exist in a vacuum. May I ask how do you make your living? That might shed extra light on your trouble understanding the inevitable churn when writing code for money. --- All of this does not even mention the fact that I 100% agree that less coding lines == less trouble. Code is generally a liability, I believe every mature dev understands that. But often we are not given a choice so we have to produce more code and periodically compress it / re-architect it (while never making the mistake of asking to be given time to do so because we never will).
- quacker 6mo agoWriting the code hasn’t been the bottle neck to developing software for a long time. Code may not be the bottleneck, but writing it absolutely does consume time. Especially with solo game dev, I can prototype ideas, try them out, and then refine or scrap them at a rate I could never do without AI. This type of experimentation is a perfect use-case for AI. It’s actually super fun, and if I pay attention and give the AI decent instructions, I don’t really lose out on code quality.
- h4kunamata 6mo ago>I don’t really understand the, “more, better, faster,” cachet to be honest Numbers, it is all about numbers. I have worked in a company where one release a week would drive the managers insane, by them we should have one release every minute. Things break all the time because in order to deploy faster and faster, corners were cut and now one of the most visited online pet store in the country is down :)