7 ms·
In recent years, I have learned how to build sufficiently robust systems fast. Here are some things I have learned: * Learn one tool well. It is often better
by kukkeliskuu 1y ago
In recent years, I have learned how to build sufficiently robust systems fast.
Here are some things I have learned:
* Learn one tool well. It is often better to use a tool that you know really well than something that on the surface seems to be more appropriate for the problem. For extremely large number of real-life problems, Django hits the sweet spot.
Several times I have started a project thinking that maybe Django is too heavy, but soon the project outgrew the initial idea. For example, I just created a status page app. It started as a single file Django app, but luckily realized soon that it makes no sense to go around Djangos limitations.
* In most applications that fit the Django model, data model is at the center of everything. Even if making a rought prototype, never postpone data model refactoring. It just becomes more and more expensive and difficult to change over time.
* Most applications don't need to be single-page apps nor require heavy frontend frameworks. Even for those that can benefit from it, traditional Django views is just fine for 80% of the pages. For the rest, consider AlpineHJS/HTMX
* Most of the time, it is easier to build the stuff yourself. Need to store and edit customers? With Django, you can develop simple a CRM app inside your app in just few hours. Integrating commercial CRM takes much more time. This applies to everything: status page, CRM, support system, sales processes, etc. as well as most Django apps/libraries.
* Always choose extremely boring technology. Just use python/Django/Postgres for everything. Forget Kubernetes, Redis, RabbitMQ, Celery, etc. Alpine/HTMX is an exception, because you can avoid much of the Javascript stack.
- physicsguy 1y agoI also like Django a lot. I can get a working project up and running trivially fast. In my day job I work with Go and while it's fine, I end up writing 10x more code for simple API endpoints and as soon as you add query parameters for filtering, pagination, etc. etc. it gets even longer. Adding a permissions model on top does similar. Of course there's a big performance difference but largely the DB queries dominate performance, even in Python, at least for most of the things I do.
- aatd86 1y agooh that's interesting. Is that due to missing libraries in Go? That could be a nice open source project if so.
- physicsguy 1y agoThere's an almost pathological resistance to using anything that might be described as a 'framework' in the Go community in the name of 'simplicity'. I find such a blanket opinion to be unhelpful, what's fine for writing microservices is less good for bootstrapping a whole SaaS app and I think that people get in a bit too much of an ideological tizz about it all.
- aatd86 1y agoI don't think anyone would advise to do everything from scratch all the time. It's mostly about libraries vs opinionated frameworks. No one in their right mind would say: just use the standard library but I've seen it online. That discourse is not helping. I think people get this miscontrued on both sides. A set of reusable, composable libraries would be the right balance in Go. So not really a "framework" either. I think that reflects better the actual preferred stance.
- physicsguy 1y agoIt's always going to be more work with composable libraries since they don't 'flow'. Just picking one of the examples I gave, pagination - that requires (a) query param handling (b) passing the info down into your database query (c) returning the pagination info in the response. In Django (DRF), that's all built in, you can even set the default pagination for every endpoint with a single line in your settings.py and write no more code. In Go your equivalent would be wrangling something (either manually or using something like ShouldBindQuery in Gin) to decode the specific pagination query params and then wrangling that into your database calling code, and then wrangling the results + the pagination results info back. Composable components therefore always leave you with more boilerplate
- 1y ago
- TechDebtDevin 1y agoWhats so hard about writing an SQL query and some json struct tags? I really dont like LLMs but all the pain points of Go you described above are relatively mute now if you use them for these little things
- physicsguy 1y agoThere's nothing hard about it, it's just that the sum total of code you need to write is much larger, and that takes more time.
- TechDebtDevin 1y agoedit, my karma for this comment has been dealing with type assertion problems in my repo functions all day.
- physicsguy 1y agoHaha :D
- bbkane 1y agoYes I really wish for something like Django for a statically typed language (maybe Spring? Haven't tried it). I'm writing a CRUD CLI ( https://github.com/bbkane/enventory/ https://github.com/bbkane/enventory/ ) partly to practice doing it in Go, and while I'm mostly happy with the resulting code, theres just a whole lot of it. At least it's simple enough that I can trust LLMs to ok jobs with detailed prompts (example: https://github.com/bbkane/enventory/blob/master/.github/prompts/2025-07-03-var-ref-update-cmd.prompt.md https://github.com/bbkane/enventory/blob/master/.github/prom... )
- t312227 1y agohello, as always: imho. (!) hmmm ... i think spring-boot combined with either java or kotlin is a very good alternative to django. even so i wouldn't compare them directly, but static typing avoids a lot of problems. idk ... for me personally one of djangos great features is its custom db-model and relative (!) painless db schema-migration. for spring-boot i often went with the tool flyway for handling db-migration ... just my 0.02€
- oblio 1y agoYou mention Django, but these days, are you using the full Django experience or are you mostly writing REST APIs? In Java land a very nice and much lighter weight framework is Dropwizard: https://github.com/dropwizard/dropwizard https://github.com/dropwizard/dropwizard It's basically a sort of All-Stars collection of Java libraries, nicely packaged and with some nice conventions. Towards the more servless route route there's Micronaut: https://micronaut.io/ https://micronaut.io/
- dingi 1y agoI haven't tried Dropwizard. Is it a batteries included as Spring Boot? Eg: How is authentication support? Do we need a lot of boilerplate to implement let's say an OAuth2 resource server?
- brabel 1y agoI used it a long time ago. As far as I know, it's an abandoned project now, very few people use it in the Java community. Go for one of the more popular ones: Spring, Micronaut, Helidon or Quarkus.
- GaryNumanVevo 1y agoI feel like Django has the largest RoI of any framework out there
- physicsguy 1y agoI think Rails is stiff competition, it's just I prefer Python.
- omnimus 1y agoLaravel/PHP world is also not bad. It gets bad rep but for CRUD apps the PHP model of every request being isolated is pretty great.
- physicsguy 1y agoI last touched PHP when you sprinkled it in between HTML (and deployment was FTP-ing it to a server) so I'm well out of date with that though I've heard people say that it's quite nice nowadays.
- runlaszlorun 1y agoCompletely OT and apologies if rude, but you're Gary Numan the musician? I love finding out when celebrities have talents elsewhere. And Wikipedia says you've had quite a bit of aviation experience as well. Kinda makes my morning... lol p.s. The inner city cynical kid in me is now required to throw in that I found Django disappointing even though every else seems to love it. Ok... identity restored... can now go back to shaking my fist at the sky as per usual...
- kisamoto 1y agoFully agree. I would also say it's easy enough to use Django for (almost) everything for a self contained SaaS startup. Marketing can be done via Wagtail. Support is managed by a reusable app that is a simple static element on every page (similar to Intercom) that redirects to a standard Django page, collects some info about the issue including the user who made it (if authenticated) etc. I try to simplify the stack further and use SQLite with Borg for backups. Caching leverages Diskcache. Deployment is slightly more complicated. I use containers and podman with systemd but could easily be a git pull & gunicorn restart. My frontend practices have gone through some cycles. I found Alpine & HTMX too restrictive to my liking and instead prefer to use Typescript with django-vite integration. Yes it means using some of the frontend tooling but it means I can use TailwindCSS, React, Typescript etc if I want.
- creshal 1y ago> Always choose extremely boring technology. Just use python/Django/Postgres for everything. Hell, think twice before you consider postgres. Sqlite scales further than most people would expect it to, especially for local development / spinning up isolated CI instances. And for small apps it tends to be good enough for production too.
- Leherenn 1y agoI have had some bad experiences with Sqlite for local desktop apps with regards to memory usage, especially on MacOs. Insert and delete a few thousands rows per hour, and over a few days your memory usage has ballooned. It seems to cause a lot of fragmentation.
- dkga 1y agoCurious to hear more about your experience, since my impression from hacking around in native Apple software is that pretty much a bunch of it is based on top of sqlite3. The Photos app is a case in point I remember off the cuff.
- Leherenn 1y agoI think we might be using it in a slightly unusual way: collect data for a while, do one big query once in a while to aggregate/transform the data and clean everything. Rinse and repeat as it's a background app. So lots of allocations/deallocations. If you're only storing a few key/value pairs long term, you won't have any issues.
- echelon 1y agoWhat kind of memory usage are you talking about? That sounds wild. Are you keeping the entire database in memory, or are you flushing to disk? It seems wild that such a critical and pervasive piece of software would behave like that.
- Leherenn 1y ago
- benterix 1y ago> Forget Kubernetes, Redis While I agree with you, these two are the boring tech of 2025 for me. They work extremely reliably, they have well-defined use cases where they work perfectly and we know very well where they shouldn't be used, we know their gotchas, the interest around them seems to slowly wane. Personally, I'm a huge fan of these, just because they're very stable and they do what they are supposed to do.
- osigurdson 1y agoAgree. Maybe it should be more like "forget learning Kubernetes, Redis or anything you don't already" know since you don't want to bifurcate time between learning curve and product.
- acedTrex 1y agoA lot of the conversation about "simplicity" with kubernetes ignores that there are MANY different distributions of k8s. I would argue that k3s on a vm is not too much harder to setup than a normal reverse proxy setup. And the configs are a bit easier to find on a whim when coming back after months. Obviously a full self hosted setup of k8s is pretty large task but in other forms its a competitive option in the "simple" area.
- marcyb5st 1y agoLast time I tried what stumped me was getting "ReadWriteMany" volumes working. My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them. I tried Longhorn, but even if I manage to get a volume showing up in the UI, it was never detected as healthy and after a while I refactored the workflow to use Apache Beam so that I could drop databases and volumes and run everything from within a single beefy VM. Is it still an issue (it was a while ago TBH)?
- JeffMcCune 1y ago
- fhd2 1y agoAgree with almost everything, but Celery is pretty common in my Django projects. I don't like the complexity cost, but especially when using some PaaS for hosting, it's usually the least painful option. I kinda always start out thinking this time I'll manage without, and then I have a bunch of jobs triggered via HTTP calls running into timeouts. At that point it's either threads, cron jobs (tricky with PaaS) or Celery. What's your approach?
- chistev 1y agoWhat Paas do you use? Pythonanywhere has an always on task feature or scheduled tasks where you can run command scripts.
- fhd2 1y agoHeroku, AWS Elastic Beanstalk (not a fan), Fly.io. With these, it always boils down to using their Celery/RabbitMQ offering. I also used Ofelia in the past, a simple Docker image for cron jobs, but quickly outgrew it. Maybe I'm missing it, but I don't think any of these has a nice simple built-in feature.
- smashed 1y agoThis. I use celery with the same code base/docker image. Just a different entry point to start a celery worker instead of a wsgi (web) worker. Too many http requests? Add web worker instances. Background jobs piling up? Add celery workers. Clearly separate read endpoints from write/transactional endpoints and you can hit a slave postgres db or the master db depending on the http call. This creates a very robust system that can scale easily from a single code base.
- physicsguy 1y agoI do the same, it's easy enough and doesn't require a ton of hosting logic. Out of interest, how do you run your migrations in production, deploy the service then run an ad-hoc job with the same container again? That was one thing I was never super happy with.
- gl-prod 1y agoIn Django, how do you create components well and handle user interactions?
- kaptainscarlet 1y agoI think that's where htmx comes in
- deleted 1y ago[deleted]
- Izkata 1y agoIt's been a long time (we've gone all React), but I remember doing reusable components with templatetags.
- jollyllama 1y agore:componenets Just use jinja templates.
- kukkeliskuu 1y agoI use django-components.
- lmm 1y ago> Most applications don't need to be single-page apps nor require heavy frontend frameworks. Even for those that can benefit from it, traditional Django views is just fine for 80% of the pages. For the rest, consider AlpineHJS/HTMX Doesn't that contradict "learn one tool well"? I write every webpage in React, not because I think everything needs to be an SPA, but because enough things end up needing client-side state that I need something that can manage it well, and at that point it's easier to just do everything in React even if it initially seems like it would be too heavy, just like your example.
- kukkeliskuu 1y agoYes, I contradicted myself in that sense. I am not going to argument against your point. But let me elaborate my approach which explains this apparent contradiction. Backend is needed in any case, and for me that is almost always Django. It is extremely fast to build traditional (non-SPA) CRUD apps with Django if you have only one model / one page. I can make a new one less than an hour. If I need more interactivity or more data on the page, I use the same approach (models, views, forms with validation) but I create partial forms and return HTML fragments, instead of full pages. AlpineJS/HTMX allows me to implement 80% with Django/python, and I can avoid implementing a REST API and JSON serialization/deserialization, all the frontend data management, and there is usually almost no need for Javascript.
- lmm 1y ago> Backend is needed in any case Well, sort of. Much as I hate the idea, maybe Next.js for everything is the way to go, bashing out a CRUD app there is very quick. > create partial forms and return HTML fragments, instead of full pages I loved that approach with Wicket and made systems that way for many years, but it just isn't good enough these days, at least for a website that's going to be used in more than one location. When it takes a network roundtrip to update your UI it's always going to be noticeable.
- kukkeliskuu 1y agoI assume you are talking about geographical distribution of the user base. I don't have experience implementing apps that require heavy geographical distribution, but why does REST/JSON work better for this? I don't have experience with Wicket, but with HTMX, I have tried to make components very small. For example one switch button is one component. Loading that is very fast in my experience, as compared to reloading and rerendering all data with React etc.
- nakovet 1y ago> Forget Celery How do you do background job processing without Celery? Every app/website I ever developed required some sort of background job processing, for Python I use Celery, Rails I use Sidekiq, Node.js I use Faktory. One of the biggest drawbacks (at least until a while ago) was that setting this up had to be a hacky webhook call to your own app that allowed up to N seconds request/response.
- tempay 1y agoI've had success with a boring event loop which looks at the state of the system and does all the background work. It's simple to implement and much easier to manage failing tasks that need manual intervention to stop failing. It also makes it easy to implement batching of work as you always know everything that needs to be done in each loop. I also have multiple celery applications, but I wouldn't recommend it for smaller or simplier apps.
- porridgeraisin 1y ago> boring event loop ... state of the system ... background work Can you explain what you mean?
- marcosdumay 1y agoYou write the demand to the database, and have a worker process execute it. Exactly like you'd do with celery. Celery is a solution to scalability, not to any business problem in particular.
- hyllos 1y agoYears ago I ditched Celery in favour of using BLPOP with redis. It was multiple times faster than Celery. Is this still the case today? So with vanilla Redis you need less workers for same throughput; if you need more throughput, spawn multiple workers.
- williamdclt 1y agoDB-based queues are pretty common (see outbox pattern). You actually don't have much choice but an outbox in your relational DB anyway, if you want the job to be enqueued transactionally.
- jimbokun 1y agoThis is great, but it's for a very narrow set of programming problems. You clearly work on web applications of moderate scale. For example, Kubernetes and Redis can suddenly become almost necessities for a back end service once it reaches a certain scale.
- kukkeliskuu 1y agoIn my experience it works for a large variety of use cases. As one of my side projects, I am running a website with 2M page views per month that has been implemented on Django, deployed to fly.io. Another side-project built-in tenant support, and several independent deployments as well. Yes, those are moderate scale (for modern hardware), but in my experience very few sites really require "webscale". It is far more common to build prematurely for scalability than the opposite. If you have "webscale" problems, then you also have have a custom platform and team working on those. The context of this discussion is how to build robust systems as fast as possible. My approach fits surprisingly large number of use cases.
- jollyllama 1y agoYes, my default strategies are quite similar. I would also add: pre-populate data where you can rather than relying on robust data entry by users.
- adamtaylor_13 1y agoYour first point resonates. I had an idea I wanted to explore and decided to use Make.com and google sheets. After two hours I said, screw this, and spun up my entire idea in a Rails app in 30 minutes. Knowing a good Swiss army tool very well is a super power.
- whizzter 1y agoI think HTMX as "simple" solutions comes into the same class as you mention "django being too heavy". Sure, Django,Rails,Asp MVC,etc with static pages could be fully functional for something that is 99% CRUD. But I myself foolishly tried using static view+HTMX for something that should have been built with something more dynamic like Vue/React(perhaps Alpine?) from the start due to actual client-side complexity. In general imho one should classify what your main interaction points are and use the correct tools for each. - Is it static pages for masses of consumers with little to no interaction? (Ie a news-site,etc) then fast SEO-friendly pages is your nr 1 priority. - Is it mainly massive CRUD updates that needs persistence, go with server built pages with Django, Rails, etc. - Is it an actual client/data heavy UI (games but also scenarios where you want fast client data slicing or processes/sources that complete at different paces), use an SPA. Ie, right tool for the right job.
- kukkeliskuu 1y agoWho could argue with the "right tool for the job"? I am not saying that there are no use cases for other approaches. But when you say you had a "static view" (in singular, not plural) with HTMX, what do you mean? I am very interested in hearing what kind of complex interaction you had where that approach failed? With HTMX, I have found the most useful approach is to build your pages from lots of very small components. Initially, I don't worry about reuse of those components, it is more about decomposition than reuse. Of course, if you want to do "fast client data slicing", then HTMX makes no sense, because it pushes logic to the backend and by definition your solution wants to process data in the fronted. So if you start from that premise, HTMX is obviously the wrong tool. If you want to use HTMX, you need to rethink your solution, i.e. question whether you really need "fast client data slicing". So why bring all the data to the frontend, and not just the slice you want to visualize? One reasonable use case is reducing server load. I have one app where data is loaded from static generated pages (periodically updated, for performance, stability and reduced server load), and sorting is done on the frontend using Javascript, based on data attributes on the DOM.
- qwertox 1y ago> Learn one tool well. That works well until that tool stops being maintained, or is evolving in such an odd direction that it breaks your code. Django is really an exception to something one can rely on, but reality is more like Angular -> breaking, Vue -> breaking, React -> breaking.