7 ms·
Handles are the better pointers (2018)
- tel 6y agoThis appears to be more-or-less the entity-component system (ECS) architecture as well.
- shoo 6y agoI agree. Perhaps also known as "data-oriented design". Decompose entities into components by modelling the data in a normalised way, as is done when modelling for relational databases. Group together similar data as records in homogeneous arrays, and batch process it without need for OOP polymorphic branching etc. Further reading/viewing, somewhat gamedev oriented (as is the author of this article): Richard Fabian's Data-Oriented Design book https://www.dataorienteddesign.com/dodbook/ https://www.dataorienteddesign.com/dodbook/ ; Mike Acton's 2014 CppCon "data oriented design and c++" talk https://www.youtube.com/watch?v=rX0ItVEVjHc https://www.youtube.com/watch?v=rX0ItVEVjHc This way of structuring applications may not be a good fit for all software development contexts. It may be a good fit for game development with high performance requirements, or similar kinds of situations.
- tel 6y agoThat's helpful, thanks! I've been learning about it in a simulation environment, not a game dev one. I think some, but not all, of the constraints are similar. I'll take a look at those references.
- DecoPerson 6y agoNah there’s far far more to ECS than just allocating this way. Put it this way: You can do ECS without this strategy, and you can do this strategy without doing ECS. Therefore: they are not conceptually related. However, they go really well together. (My cress: introduced/advocated-for ECS at two different game studios, with varying levels of success. One already used the handle strategy heavily, for all kinds of resources, but definitely wasn’t doing ECS.)
- joe_the_user 6y agoHow would this work for a hierarchical structure like a B-tree? Also: If, say, 30% of your system is pointers and you have them "spread all over the place", wouldn't moving are your pointers to a central location result in pointers "spread all over" two different places. "Cache oblivious" systems organize B-trees and related so related objects are close together in memory. It seems like a system with bits of things relating to each other scattered about would have to just use some ad-hoc similarity measure to minimize cache hits.
- yellowapple 6y agoYou could encode the item's location in the tree into the handle itself, though this assumes that the tree will never be rebalanced while that handle remains active. This is basically (from what I understand) how hierarchal triangular meshes (HTM) works. The "coordinate" is a number describing the location of a coordinate's triangle within a larger triangle within a larger triangle within a larger triangle and so on until you get to an octahedron.
- stephc_int13 6y agoI don't like to use tree or linked-list, I prefer arrays and hash tables. I haven't found many problems that couldn't be solved with simple arrays or hash maps. Same thing about recursive functions, I never use them.
- chrisseaton 6y agoHow would you for example represent a program in memory with simple arrays and hash maps, where most people would use a tree or graph?
- stephc_int13 6y agoDo you mean an AST?
- chrisseaton 6y ago
- jtolmar 6y agoI like handles too. Are there any programming languages that natively support them? Like, something that's physically a tuple of (data structure pointer, index), but with the syntax allowing you to directly access it like any other reference. Bonus points for "get all handles to me" as an operation.
- Macha 6y agoIf I'm understanding your ask correctly, this is basically what implementing Deref in Rust or operator-> in C++ gives you.
- jtolmar 6y agoThat can give you the syntax! But what I'm curious about is more the implications of actually having this as a part of the language. Does it open up interesting space in optimization or memory safety?
- eslaught 6y agoThis is a bit like what Regent does. "Pointers" are really offsets into regions, which means that we can transparently move them around a distributed machine on the user's behalf. http://regent-lang.org/ http://regent-lang.org/
- derefr 6y agoFew things that expect a pointer do it by expecting an opaque dereferenceable thing. Most things that expect pointers actually expect a specific concrete pointer type (even if it's a pointer to an opaque thing.) IMHO, a language with "real suppport" for handles would use them everywhere, instead of pointers—i.e. make all places where pointers pass through "wide enough" for handles to be passed instead—with reduction to a pointer being a kind of WPO for cases where it's proven to be possible.
- Blikkentrekker 6y agoImplementing `Deref` in Rust implements deref coercions, meaning that it can be used in any function that can accept an appropriate reference. This is purely syntactic sugar, and no different the compiler inserting `x.deref()` instead of `x` for function arguments. — it's simple q.o.l..
- stephc_int13 6y agoI've used handles for memory management for more than 15 years and I'll never go back to pointers, smart or not. I'll publish an article about the memory management architecture of Crayfish, my new engine, when I find the time. Handles have many nice properties, for example you use them to easily implement promises.
- dataflow 6y ago> Handles have many nice properties, for example you use them to easily implement promises. Confused, (how) is this different from pointers?
- isaiahg 6y agoI haven't read the whole article yet but it says pretty early on that they're indexes to an array
- stephc_int13 6y agoHandles in the general way are an opaque reference that can be resolved into a pointer. Index into an array is only one of the multiples implementations. It can also be a hashed key or even contains directly a pointer Because handles are weak references they can be used as promises, returned immediately but resolvable only when the the actual work is done.
- dataflow 6y agoBut all of these are true for pointers too? Preallocate memory, return a pointer to it, lazily fill in parts later. You're already doing that with a handle anyway, whether it's indexing into array memory or something else.
- stephc_int13 6y agoYou can't always pre-allocate and the pointer is not giving you much information about the availability of the data behind it.
- amelius 6y agoHow is garbage collected?
- moonchild 6y agoIt's not. If you implement this strategy in a garbage collected language, you can end up with use-after-frees or double frees (though with the advantage of type safety—if you use a handle of type Foo after freeing it, you'll never stomp on the memory of an object of type Bar, for any Bar≠Foo).
- deleted 6y ago[deleted]
- omginternets 6y agoI believe this was C, so presumably it’s not?
- amelius 6y agoIn C garbage is collected "manually", by calling malloc and free. If you design your own heap, you also have to design the memory management around it.
- dataflow 6y agoNot mentioned: the collateral damage. You'll probably trash your debugger's object visualizer in doing this. And probably other facilities in your IDE that helps you deal with class members, really.
- stephc_int13 6y agoIn my opinion, when you're serious about debugging your code, you'll use your own debugging tools, and from experience I'd say that using handles is not an issue. But yeah, standard IDEs won't see anything behind the opaque reference.
- dataflow 6y ago> In my opinion, when you're serious about debugging your code, you'll use your own debugging tools So what does this mean about my past debugging experiences? And and those of everyone else who uses an IDE? They've been jokes all along?
- seibelj 6y agoI find print debugging to be fine. Unless we are talking Java - coding in that requires the IntelliJ command center.
- stephc_int13 6y agoI think this is highly context dependent. I've worked in fields where no debugger where available, not even printf. (embedded software, interrupt driven) I had to write ad-hoc debugging tools with difficult contraints. I grew the habit of debugging by reading the code first, then using traces, then writing custom tests or ad-hoc tools as last resort solution. (extremely rare) I used Visual Studio debugger and GDB a few times, but I've not been convinced by the productivity gains, quite the contrary, but I am probably not using them well. In practice, the most difficult bugs are often tied to concurrency issues, and I haven't found much help with those general purpose tools.
- 6y ago
- spaetzleesser 6y ago"There is no problem in computer science that can't be solved with an additional level of indirection" :-)
- worik 6y agoDrugs are for people who cannot handle reality Reality is for people who cannot handle drugs Handles are for people whom reality drugs
- hnedeotes 6y agoRoses are blue violets too who da hell dropped acid on my peek-a-boo
- The_rationalist 6y agoAre Java varHandles related? https://docs.oracle.com/javase/9/docs/api/java/lang/invoke/VarHandle.html https://docs.oracle.com/javase/9/docs/api/java/lang/invoke/V...
- bob1029 6y ago> and may even be useful in higher-level garbage-collected languages to reduce pressure on the garbage collector. > The approach described here works very well though with a data-oriented architecture, where central systems work on arrays of data items packed tightly in memory. I have been pushing us into a new data-oriented architecture for our software, but for reasons of domain modeling (managing complexity) rather than performance. This is a very interesting read from my perspective. Let me give an example - We took deep, circular object graphs like: Account.Customers[0].Accounts[1].Customers[2].FirstName And turned them into 2 flat arrays of Account and Customer. A 3rd relation type is also developed to arbitrarily relate these types together. The relation type also conveys all facts that pertain to the nature of the relationship, rather than forcing this knowledge onto Account or Customer. This ideology is carried to the logical extreme - We do not permit any nesting of complex types in our revised domain model object graph. There is a root type which contains all of the List<T>s of domain types (and applicable relation types). The domain model is composed such that we can directly map every single type & public property into SQL tables. And we do. Because SQL is amazing for executing business logic. Now that I am thinking this through, performance is certainly also seeing a benefit here. We are no longer serializing these absolutely horrific JSON object graphs. It's just a bunch of flat complex type arrays now, with everything related together using GUIDs. From a developer QOL perspective, one could argue that this is more difficult to work with because everything is separated by relations. I would say if you are fortunate enough to be working with something like .NET, you can abuse the shit out of LINQ and not really feel the added complexity.
- AlphaSite 6y agoIt sounds like you’re describing normalised data, is that a fair statement?
- bob1029 6y agoYes - Something around 3NF/BCNF.
- tehlike 6y agoI was going to implement exactly this, but out of laziness ended up using this: https://www.npmjs.com/package/json-serialize-refs https://www.npmjs.com/package/json-serialize-refs For my use case serialized json was 30+% smaller.
- dang 6y agoIf curious, past threads: Handles Are the Better Pointers (2018) - https://news.ycombinator.com/item?id=24141541 https://news.ycombinator.com/item?id=24141541 - Aug 2020 (106 comments) Handles are the better pointers - https://news.ycombinator.com/item?id=17332638 https://news.ycombinator.com/item?id=17332638 - June 2018 (80 comments)
- MaxBarraclough 6y agoRelated: the famous C++ FQA has an entry on handles: https://yosefk.com/c++fqa/ref.html#fqa-8.7 https://yosefk.com/c++fqa/ref.html#fqa-8.7
- jhgb 6y ago> move all memory management into centralized systems (like rendering, physics, animation, …), with the systems being the sole owner of their memory allocations > when creating an item, only return an ‘index-handle’ to the outside world, not a pointer to the item > in the index-handles, only use as many bits as needed for the array index > only convert a handle to a pointer when absolutely needed, and don’t store the pointer anywhere DataDraw, anyone? http://datadraw.sourceforge.net http://datadraw.sourceforge.net
- dkneeland 6y agoIs it just me that finds this font really hard to read? It makes it really hard to focus.
- delusional 6y agoHandles are the better handles? Pointers are just handles to heap memory. All i see advocated for here is a per "system" memory manager that also has custom code that gets called on deref. On one side you get a chance to do some logic before a deref, on the other you now no longer know how long a deref takes. I like this approach too and use it a lot, so it's good to see it advocated for. I just don't understand why they have to be "better than pointers" when they are just a different tradeoff.
- perl4ever 6y agoStrange to read if you were programming the Macintosh in the 80s. I kind of thought of handles as just what you used when you didn't have an OS that supported an MMU...