5 ms·
What game engines know about data that databases forgot
- RandyRanderson 7mo agoOr in a more general sense: Get your code as close as you can to your data
- Garlef 7mo agoSerious question: If you take a BDD/TDD approach - do technologies like this still give you something? I've dabbled a bit into smth similar (SpacetimeDB) with the aim of creating a headless backend for a game. But then I realized that I'd still need to define the interface that I was testing in the traditional software layer and that all the nice DB magic seemed worthless since I'd still have to orchestrate traditionally. (In short: No matter how nice your DB magic is, you will still hide it away in the basement/engine room, right?)
- ok_dad 7mo agoSounds like we need IDE support for databases like we have for programming runtimes. Need to be able to lint, debug live, etc.
- jjordan 7mo agoI've always passively wondered why this wasn't more of a thing. Something like pgAdmin is fine I guess, but it's always felt like "just barely good enough" rather than an immersive power tool to get things done, and done well. Possibly just a skill issue, but that's been my impression.
- ok_dad 7mo agoNo I get it; and it’s not a skill issue because debugging with proper tools is a skill and the issue is that lack of those tools means you lack the ability to even use your skill. My last job used a lot of fancy internal pg stuff and we could never really reason about it properly. I wish I could debug it like I do with a Go app with delve, or in my IDE. Adding NOTIFY everywhere is print debugging which in my opinion is not a very good debugging strategy.
- mamcx 7mo agoCheck the dBase/FoxPro family. My nostalgia remember it fondly, and there was not disconnected in how everything worked (example: The forms were stored as tables, so you can do alike `SELECT * FROM form_name WHERE widget=`), and there was zero ORM. This is my aim https://tablam.org https://tablam.org. BTW I worked in SpacetimeDB and I proudly say is half-way :)
- spullara 7mo agointeresting! foundationdb was created after the team was going to build a massively multiplayer game and couldn't find a database that could support it...
- dpe82 7mo agoThis seems cool but man it's hard to get over the very, very obvious AI writing.
- hedgehog 7mo agoIt's also a bit odd they don't mention column-oriented databases at all.
- zffr 7mo agoWhy would column-oriented databases be mentioned? My understanding is that these are typically used for OLAP, but the article seems to talk only about OLTP.
- SigmundA 7mo agoBecause there is a whole section that describes column based storage without mentioning that some databases have column based storage as an option.
- goalieca 7mo agoThis is one of the main problems I have with LLMs. It finds patterns in words but not content. I see this in code reviews and eventually outages. Something looks reasonable at the micro scale but clearly didn’t understand something important (because they don’t understand) and it causes a major issue.
- jandrewrogers 7mo agoModern database engines tend to use PAX-style storage layouts, which are column structured, regardless of use case. There is a new type of row-oriented analytic storage layout that would be even better for OLTP but it is not widely known yet so I wouldn't expect to see it mentioned.
- dpe82 7mo agoYeah that was my second thought. ECS' favoring of structs-of-arrays over traditional arrays-of-structs for game entities boils down to the same motivations and resulting physical layout as column-stores vs row-stores.
- doctorpangloss 7mo agoI'm getting fatigued out by "I had a conversation with a chatbot, here's the output" blog posts.
- fireant 7mo agoAccording to the github readme the author has had the idea in mind since 2015. Perhaps you ought to be a little less dismissive based on, what I assume your issue is, the writing style.
- SigmundA 7mo ago>Cache locality by default. In a traditional row store, reading all player positions means loading entire rows — names, inventories, health, everything. Most of those bytes are wasted. Not one mention of column stores? This didn't come from ECS... https://en.wikipedia.org/wiki/Data_orientation https://en.wikipedia.org/wiki/Data_orientation
- bitwize 7mo agoWhen I learned about ECS, I realized that Tablizer (old Slashdot guy who insisted that OOP was a dead end and "table oriented programming" as was done in FoxPro was the way to go) was probably right. Using an ECS for my Android game was a bit more cumbersome, but paid for itself many times over with the ability to create new kinds of entities, implement familiar behaviors for them, and add new ones without code copypasta or inheritance tree entanglement.
- samiv 7mo agoNone of that requires ecs though it can be done simply with a composition. Traditionally called something like entity and attachments.
- bitwize 7mo agoI tried that too, I called it "agents". (This was long before AI-mania.) An agent was an object that handled some aspect of behavior (like gravity and collision physics) "on behalf of" some entity, hence the name. The word I was actually searching for was probably "delegate", but I was a stupid 20-something. ECS is to me still conceptually cleaner and easier to work with, if more tedious and boilerplate-y.
- jbonatakis 7mo agoCMU very recently had a seminar [1] with the founder of SpacetimeDB, which was very interesting. The recording should be up in the next few days I expect [1] https://db.cs.cmu.edu/events/pg-vs-world-spacetimedb-tyler-cloutier/ https://db.cs.cmu.edu/events/pg-vs-world-spacetimedb-tyler-c...
- scottcodie 7mo agoI'm not sure a game needs transactions. It seems to me that entity streaming concepts are a better fit: reading from an event queue and consistently reconciling the state over time. Like the sorted top-n in the example, if it was a game and the query is generally known, it would be better to just materialize it as a fast rank, which goes beyond the IVM-style materialization they suggested.
- hinkley 7mo agoEeeeeh. You can never trust the client, but the server is far away, so most games have speculative execution of actions and then the server can tell them to fuck off and revert/ignore the action. That smells a bit like a transaction to me.
- dbt00 7mo agoMany games have obvious inventory dup exploits that this is a very reasonable solution for.
- deleted 7mo ago[deleted]
- cmontella 7mo agoI feel like this is a good place to bring up the "Have You Tried Rubbing a Database on It?" conf, which deals with this sort of topic. If you'd like to read about the intersection between Databases and X, where X is games, robots, user interfaces, developer tools, etc. there are some good talks there, and they're only 10 minutes each so very easy to consume. Also unlike TFA they are not AI generated. https://www.hytradboi.com https://www.hytradboi.com
- Keyframe 7mo agoI know something about game engines, I know a lot more about real time graphics, and I do even more about databases and implementation of those. While it's nice to get yourself acquainted with all, especially easy to do with AI these days.. I do have to point out few things visible outside of myopia induced when looking from one perspective into another. In this case from gamedev to data world. In Cliff's notes since I don't have much time but I also don't want to give a drive-by snark since there are hidden values to this, imo, contrary to what I'm going to say. What gamedev perspective myopia kind of ignores is, in no particular order.. persistence vs throughput - goal of data management is not only about execution speed. ACID, WAL, etc all are core features, not an overhead per se. Sometimes you can forego these, of course. Sometimes, if you understand what you're giving way to. columnar fallacy, for the lack of better words - DoD, SoA are not secrets game engines use which others have forgotten. This in-particular ignores existence of OLAP. Clickhouse, Snowflake, HN's darling duckDB have been using SoA for _quite awhile_. AoS in OLTP is used because updating a single record _is_ faster that way. Why one over the other - see OLAP vs OLTP Game engines obsess over cache hierarchy (L1-L3), and in particular cache misses by avoiding those with cache line packing, prefetching. Databases operate in a different real, that of I/O boundaries; Disk and network primarily. Bottleneck of databases is often the speed of light (latency) or the bus (NVME for example). packing structs is going to give you marginal benefit if you're waiting for 10ms for a network packet or a disk read. Suggested are micro optimizations in the context of a global system. Different realms of execution flows. ECS are cool, everyone agrees on that. Relational databases are built on top of relational algebra though. So if you're up for running the same logic over many objects, ECS is going to be cool. If you want to allow for complex, arbitrary queries and set based logic you will need some guarantees of consistency. Complex, many-to-many joins across disparate ECS ssystems without predefined _system_ and you're foregoing the promised performance. What DBs are doing is general purpose, ECS ain't that. Finally, yes game engines manage thousands, millions, of entities in localized 3D space. Databases manage billions, trillions, of records across petabytes of distributed storage. So, what gives? Entity model does not scale to distributed systems because of CAP theorem. No such thing as instant consistency in a globally distributed system without violating physics or sacrificing availability. TBH, some middle ground, localized to a machine, might give way to the idea but at what cost? Don't let it shoot you down though. If there is still a kernel of idea tingling after that chat with AI, go ahead and drive through that CAP wall!
- kwillets 7mo agoThis has a lot in common with a columnstore, but there are a few interesting differences. The main one is subrow versioning. Column stores (in OLAP at least) have always used row-level versioning, which gets in the way of small updates. A single change to a row amplifies into deleting and re-inserting the whole thing, and operations that seem sensible like adding or dropping a column break previous versions. This scheme is the first I've seen that tries to fix that problem. One other difference is a lack of compression, as it's zero-copy, so the performance gains of operating on compressed data are lost.
- trgn 7mo agoThe web is a front-end environment, where users expect 60fps, but where developers violate pretty much all the rules. > Zero-copy is the default, not the optimization: the amount of fluffy mapping, destructuring, temp scope-creation, ... that is the norm now for JS/TS devs is excruciating. how did this become the norm? do it once it doesnt matter, but every single layer in the app doing this and things become jittery. you first take the hit at creation time, and then another time at GC. stop doing that! Pass references around to objects, not recreate them each time at some function boundary. > Entity as pure identity. Stop json.stringifiyng every thing! how many hashThing() implementations i have seen that are essentially Json.stringify. stop it! > Cache locality by default. a little less clear for web dev, much is missing in terms of primitives. but do think about it. if anything, it's good hygiene. fixed typed array does make sense, dont mix&match types in your array, ... Save the web, think like a videogame dev!
- Sharlin 7mo agoWell, it's certainly not going to get better given that most of new Web code is by now probably written by LLMs. Which definitely aren't trained to write performance-oriented JS/TS.
- vanderZwan 7mo agoYou forgot dynamic property creation/lookup instead of using a constant "shape" that JS engines can actually JIT optimize away. And then there's the more detailed version of that where people write {x: 0, y: 1} in one spot and {y: 1, x: 0} in another and do not seem to realize that this under the hood they just sabotaged the ability of every JS engine out there to fully optimize any code related to it. Which also extends to situations where functions take objects that happen to share some properties as parameters: if you can put the shared properties first and in the same order in the object creation, it will result in better optimized functions. (but tbh I think that as long as we don't fix the bloat that is tracking libraries first, all of this is optimizing the wrong thing)
- kelseyfrog 7mo agoThe connection between ECS and DBs wasn't lost on developers in the early days of ECS. I'm reminded of Scott Bilias's 2002 ECS presentation: A Data-Driven Game Object System[1] which clearly says, "This is a database." In my mind, there's a pedagogical gap in OOP where ontologies are formed using classes and this almost never works in real life applications (no in real life for that matter). We almost always construct ontological relations using predicates not rigid hierarchies. For this, ECS tends to be flexible, and expressive enough to be a pragmatic choice. 1. https://www.gamedevs.org/uploads/data-driven-game-object-system.pdf https://www.gamedevs.org/uploads/data-driven-game-object-sys...
- boricj 7mo agoI'm working on an embedded project where I'm actually thinking about using ECS on a STM32H5. It's not about cache-friendly locality (waitstates on internal SRAM for MCUs is basically a rounding error compared to the DRAM latency seen on desktop or server-class hardware), but the business is so complex that the traditional inheritance/composition is getting quite messy. When you end up using virtual and diamond inheritance in C++, you know you're in trouble... It's too bad that ECS isn't more widely known outside of gamedev. It's not just good for performance, it's also legitimately a useful architecture design on its own to solve problems.
- vmchale 7mo agokdb does two of those by default—columnar and it uses memory mapping (no serialization/deserialization) Not entirely forgotten in the database world!
- deleted 7mo ago[deleted]
- samus 7mo agoIsn't this pretty much what SQL databases with column stores provide? It's not a novel idea at all, but obvious to anyone who knows how caches work and how to optimize applications for it.
- mcphage 7mo agoThis is interesting, but it's a lot of talking around columnar databases without ever directly addressing them—making them seem like the proverbial elephant in the room. Even their table comparing DB concepts with ECS concepts doesn't make a lot of sense unless you're talking in the context of columnar databases.