4 ms·
You don't need a database, a queue, a distributed system: Go is enough
- sgundo 3y agoThis article is a reminder to me and you all that you probably don’t need to make things harder for yourself.
- ComputerGuru 3y agoSounds like it’s actually “SQLite is enough.” With which I’d agree.
- osigurdson 3y agoThis is only used to store questions. They could have just hard coded them I think.
- rapsey 3y agoIf you are working on this specific silly use case. What is the point of this article?
- sgundo 3y agoI agree it's silly, but sometimes ideas or POCs are silly but worth shipping. Don't you agree?
- gwd 3y agoWorth shipping, but not worth investing massive amounts of time in. On the other hand, suppose you actually hoped to grow and make enough money on something to pay the salary of a few dozen people. Let's say you start with golang and sqlite3, ready to handle 1000s of people an hour, everything is going great, and then someone posts your thing to Hacker News (or it goes viral elsewhere). Suddenly you have millions of eyeballs an hour, of whom your non-scalable infrastructure turns away 99.9%. What's the chance you get another shot after you've re-architected everything to be more scalable? I don't have a good idea how likely that is; at some point I seem to recall hearing that Stack Overflow was run on just two servers, made possible because it was just efficient. But it's certainly something to take into consideration if you're hoping to grow large; and only something to ignore if you don't really care.
- sgundo 3y agothose are great points. I have read the article about SO and it is inspiring
- phyrex 3y agoI don’t disagree with the premise for very small applications, but > The quiz state machine is full in memory > The queue system for the matchmaking is in memory Means those will be nuked as soon as a new version gets uploaded. it really wouldn’t have been much extra work to put this into SQLite as well, and would lead to a much better user experience during updates
- sgundo 3y agoYou are right, but i can roll updates during the night without causing any issue. Since after you finished playing you don't have a way to access your old games anymore
- jstanley 3y ago> i can roll updates during the night without causing any issue This isn't universally true. I work on a telephony platform, some components of which need to be upgraded very carefully, because if you just restart the daemon without first transferring load elsewhere then you drop people's calls. OK, there are less calls at night time, but there are calls all of the time and people don't expect their calls to randomly drop just because it's night time.
- 9dev 3y agoThat works for the specific case presented here. As soon as you have users from other timezones, ”during the night“ becomes a really complicated concept quickly.
- sgundo 3y agoyou are definetely right. :)
- deleted 3y ago[deleted]
- Capricorn2481 3y agoCan't you just serialize those things and then migrate them?
- masfoobar 3y agoI am a fan of "distributed computing" - but I do not make it complicated. I have created a generic broker daemon, which reads messages (in) and places them in the appropriate service (queue) which, at this time, is using sqlite. Then, a bunch of daemons (workers) connect to the broker service, and processes anything in the queue (process) and tells the broker when the message is OK or errored (output) I also have commands to check a service queue, active workers, etc. The broker is generic, simple, and fast. The workers do the actual, specific job. The communication between the client<-->broker and broker<-->worker is ZeroMQ. There is also a "publish" if you choose to join, which some client programs do.. so they can see any errors on-the-fly. Installation/deployment is on a linux server using SystemD. Nothing fancy or complicated. Purpose is that this broker could serve useful for many projects I do in the future.
- vishnugupta 3y agoTitle: You don't need a database. Half way down the article: The questions are stored in a sqlite database
- pluc 3y agoFAQ Does it scale? No. oh good
- sgundo 3y agoI am gonna be honest, I was laughing writing that part! :)
- deleted 3y ago[deleted]
- moribvndvs 3y ago> It means that scalability would become an issue if and when Binary Duel is used by more than a few thousands of people an hour. In this fortunate case, I will be very happy to host the infrastructure on a K8 cluster with autoscaling, self-healing, a distributed database, a Redis server and so on.
- queil 3y agoIt's a bit click-bait-ish. You don't need computers, here is my tic-tac-toe on a piece of paper + pen. Just use the right tool for the job.
- sgundo 3y agoYeah, it is kind of provocatory. Of course you can't build a business out of an in-memory game, but it is fun to ship something as it is. Without thinking too much about scalability.
- yanis_t 3y agoWe developers take proud for being able to get our head around a complex solution. And unfortunately sometimes that becomes a driver for building complex solutions for problems, that otherwise could have been solved much simpler. But in that case we loose interest. Me, personally, I often find myself in situations like this. Where if I clearly understand the solution in my head, I might be a bit reluctant to actually code it and get the job done. Kinda like, you know the solution exists, and then you move on to something more interesting.
- osigurdson 3y agoIf the data in your app doesn't matter at all, and you have no availability requirement, sure keep everything in memory. I don't see what Go has to do with it however, any language should be fine.
- sgundo 3y agogolang is very efficient memory wise. And it is easy to develop with. (Rust gods please dont see this)
- osigurdson 3y agoThe author states that scalability doesn't matter here. Any language would have been fine. The title should be "do everything in memory in a single process until you can't ".
- boutell 3y agoCan attest - for similar use cases. I did much the same (shameless plug, just like the original): https://friendsyeights.com/ https://friendsyeights.com/ Game state is kept in memory, the server is single process (Node.js as it happens), connections use websockets, it just works. Does it scale? No. Could it scale with a similar approach? Sure, with some additional work: separate games could be distributed over many instances (a sharding strategy). There's no redundancy there, but it's a game, not a credit card processing system. However, I do snapshot game state in JSON files on disk so that games are not terminated by every new deployment. In a single-process model, that's safe when using synchronous filesystem calls. Just don't try it in a multithreaded system. Though it may take many moons, you will eventually lose your "poor man's database" to a race condition. Still, would I use this technique to host people's taxes? Hard no. I wouldn't even use it to host their blog posts. Durability matters for those use cases.
- quantified 3y agoMissing from the title is "enough" for what? For any trivial personal project? Python, Ruby, PHP are each enough. There's not a need for literature about personal low-risk development or deployment. That said, Scalability? But at what COST? [http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf] was a good note that orienting around scaleable technologies can be very inefficient- scalable does not mean either fast or economical.
- Joker_vD 3y ago> We need to choose a database: so, let’s start with that. ...Wait, ten years ago the advice was that this should be one of your last decisions: your application should be able to work with minimal changes with any DBMS, and tying yourself to a specific vendor from the start was considered a bad idea. Did it change while I was not looking?
- Capricorn2481 3y agoI don't think these things are incompatible. You can choose a database and still use interfaces to make replacing it possible.
- w3news 3y agoI like the idea, think about what you really need, keep it simple. I also try to move centralized computing to decentralized computing more and more. And that can also be done to let the client do more computing. And also storage, is it needed to store in central, or can the user store it. Many times people thinking that we should have a central system, with all the truth. But let it be a part of the users. It makes also the central part simple, and because the central part is simple, it is easier to scale.
- donatj 3y agoYou can get pretty far with simple in memory solutions when your problem space fits entirely within the constraints. We have a Go microservice for linking accounts to their proper federated logins with some associated metadata. There’s a refresh process that runs every fifteen minutes. The entire set fits happily in memory and takes milliseconds to respond.
- sgundo 3y agogreat to hear!
- JodieBenitez 3y agoMildly related: I wanted to learn some Go and I thought "I will just reimplement my music library with it". So basically the task was to write a backend API that maintains a database of music files with associated ID3 tags, and so on. The original backend is Python over SQLite, which works fine enough. As I started to reimplement the functionalities, I though "why should I use a database anyway ?" and started just pushing the ID3 tags a list of structs. It turns out that having an in-memory list of structs for 50k music files is not a problem at all for a raspberry pie. It's not that musch memory and it's fast enough to create so that I don't have to persist it and it's fast enough to scan for a search. In the end, I may use SQLite anyway because FTS5 is just so good, but yes... sometimes you don't need a database.
- nunez 3y agoyeah...you need all of that if you're writing anything that will be abused by tons of users and handles data in any form.
- gwbas1c 3y agoThe article should really lead with: > Let the scalability be a consequence of the success, not something that holds you back. A large portion of my career was spent fighting the problems that come with "premature scalability."