9 ms·
Zig: Pointer Stability for ArrayLists
- _bohm 2mo agoIt's a nice feature but I can't help feeling like, if you need a stable pointer to an item in a collection, ArrayList is the wrong data structure to use? Maybe someone can chime in and give me an example of when you'd do this instead of, e.g., just storing an index. Alternatively, you could use an Unrolled Linked List (FKA SegmentedList in Zig before it was removed in 0.16, not sure why).
- nvme0n1p1 2mo agoSegmentedList had a weird API, especially the way you control the list growth factor by the size of an inline array. And it hadn't kept up with stdlib norms in recent versions. I do hope it comes back eventually with an improved API. At least we got Deque in exchange. I use that far more often than I used SegmentedList.
- _bohm 2mo agoAha, gotcha. To tell you the truth, I don't think I ever used it. I have a custom implementation I wrote because I didn't realize at the time that SegmentedList was an unrolled linked list :p
- beepbooptheory 2mo agoMaybe one use case is if you are interfacing with external C library and you're stuck with pointers?
- the__alchemist 2mo agoMy mental model of Zig is that it is explicitly the language for developers who prefer using pointers in business logic (instead of just in MMIMO, and are looking for something with improvements over C); i.e. exactly this class of abstraction.
- _bohm 2mo agoHaving written a bunch of Zig, I wouldn't say that the language design or culture explicitly encourages the use of pointers over indices in such situations. I would say it's more a language which trusts the programmer to make correct decisions about which constructs are appropriate in any given circumstance.
- svachalek 2mo agoHasn't the past 30 years of the Internet age taught us that given such trust, programmers will make the incorrect decision with horrifying predictability? The most trivial level of software security requires that pointer safety needs to be mathematically proven not up to human (or LLM) judgment.
- yefol 2mo ago[flagged]
- insanitybit 2mo agoWho cares what mojo handles? This is about Zig and memory safety.
- insanitybit 2mo agoYes, undoubtedly. Anyone in denial of this should be legally barred from programming.
- the__alchemist 2mo agoI appreciate the explanation! I'm a noob at this sort of thing. (My experience with pointers is limited to MMIO and C FFI). Regarding trusting the programmer: is that in the context of pointers, or more broadly? I'm betting I'm missing more, e.g. maybe mutation control too across threads/cores etc? Or as a broad design principle? (I have written some Zig as a learning dive, but don't Grok it, as am waiting for some core functionality like HALs, GUI libs, 3D libs etc. Was also a bit miffed by the 'operator overloading will never be allowed' as the use cases where I use low level languages have a near total overlap with the ones where I use vectors, quaternions, and matrices. Stated another way: I learned with higher level langs first, which I believe biased my mental model. Example: Say I want to use a C library in my rust program/lib. (Example: CMSIS DSP). One of the actions I take in the wrapper is convert the pointer to an array ref; my mental model is the param is a list of items; it is divorced from memory. If I want to read/write a reg, or access FLASH, that's where I look to pointers. I e I think zig vs others is about if you want to conflate or divorce collections and memory.
- bvrmn 2mo agoArrayList is a very generic (pun not intended) structure and could be stretched quite freely in any direction with useful property of owning underlying slice. Like readonly preallocated ArrayList is a thing.
- _bohm 2mo agoThat's a good point. Using lockPointers would be a good way to enforce at runtime that your ArrayList is truly read-only.
- hansvm 2mo agoI use it in a lot of places where I know the max capacity ahead of time -- ensureCapacity() followed by a lot of *AssumeCapacity()-styled commands. It's convenient for all of the ... convenience ... methods (append() requires some bookkeeping somewhere, appendSlice() requires more, and so on). In those usages, it's basically syntactic sugar over a slice. That's not a perfect solution, but it's reasonably good often enough that I keep doing it. The proposed change doesn't do much for me personally (memory safety is ensured in other ways, and if it weren't I wouldn't be annoyed debugging the allocator-observed errors), but I could see myself using it at some other point in time for the same class of usages, or I could see other people relying on it when they choose that class of coding.
- dataflow 2mo agoI don't know Zig, but conceptually: a direct pointer is the fastest way to access an object. An arraylist is the fastest dynamic sequence of objects (fattest in access, not in growth). You use these when you need the performance. It's not often but it certainly happens. The most trivial example is a string that you append to but still need to pass to a C API in between that expects it to be contagious, but it's far more useful than just for storing characters.
- _bohm 2mo agoSure, in cases where you need elements to be contiguous in memory then certainly an unrolled linked list is not appropriate. There's usually not a meaningful performance difference between a pointer deref and an indexed array access, however.
- kllrnohj 2mo agoIndexing into an array is direct pointer access, there's just an addition in front of it but it's hard to imagine that showing up at all in even the tightest of benchmark loops
- dataflow 2mo agoI can't speak for your imagination, but this absolutely does come up if you're writing high-performance code. Also note that being able to access arbitrary objects (as opposed to objects in the same array) requires storing two pieces of information: an index and a pointer to the beginning of the array. So it can use twice as much memory, which affects your cache etc., though you don't even need that to see the effect.
- pocksuppet 1mo agoThe addition is much cheaper than the memory access. The register pressure is worse than the addition. The size reduction of the index (versus the pointer) and resulting cache pressure reduction can outweigh the register pressure and the addition if you are storing many indices.
- ivanjermakov 2mo agoI count this as a "rookie at system programming" mistake alongside returning a reference to a local variable. Rust is great at this because borrow checker can catch those at compile time and it can _teach_ devs to not do that.
- amluto 2mo agoThis seems weak. In a language like Rust, the compiler will “lock” the pointers for you, and you can’t forget. In a language like C++ (and presumably Zig), one could, in theory at least, have the iterators and slices that reference the storage of a dynamic array hold some sort of lock that pins the storage. But this API requires the programmer to remember to lock the pointers and also requires the programmer to keep the lock alive for the correct region of code. And it looks to me like even the example in the blog post has the lock taken completely outside the function that requires stability, so there is nothing whatsoever that gets the lock scoping right. Even the type system can’t help — the offending parse function can’t declare that it wants a pointer-locked ArrayList parameter.
- pron 2mo agoI reach for a low-level language only when I want low-level control over what operations happen and when, what memory is used and when etc.. At present, no language offers me this control and safety at the same time. With Rust, when I need such control (which is always, otherwise I would use a higher-level language), I need to give up safety, anyway, at which point I have no safety and the complexity of a language that offers safety. So right now, when we want control, we need to give up some safety, but weaker things are still helpful. Also, in low-level code, the problem of "I might forget to do something" sometimes clashes with the problem of "I need to see exactly what operations are done and where". Various kinds of implicitness help with the former at the expense of the latter. I'm not saying this is universally better than other approaches, but many people who do serious low-level programming would prefer this.
- Rendello 2mo agoAside: one Zig (syntax) feature that I really missed in Rust is shown in the second code block, namely prefixed multi-line string literals à la: const text = \\This is a long comment \\But I can split it among lines arbitrarily \\And keep my indentation. ; I've started using the Rust macro library `docstr` [1], which does the same thing: const TEXT: &'static str = docstr!( /// Now I can do it in Rust, too. /// I prefer this style a lot of the time /// for long texts. ); It even works with macros (example from the docs): let greeting: String = docstr!(format! /// Hello, my name is {name}. /// I am {} years old! age ); 1. https://docs.rs/docstr/latest/docstr/ https://docs.rs/docstr/latest/docstr/
- Gibbon1 2mo agoIn C you can do that. printf ("Things:\n" " thing1=%u\n" " thing2=%u\n" " thing3=%u\n", thing1, thing2, thing3);
- metaltyphoon 2mo agoC# solves this so elegantly string text = """ This is a long comment But I can split it And keep my indentation. """;
- Rendello 2mo agoI believe the prefixed literals are a direct response to this style of multiline string. The style you showed is pretty common in a programming languages, but language implementers are faces with a tradeoff: - Keep the initial indentation for each line in the string literal; or - Track the indentation level and attempt to remove the whitespace for each line. For a lot of strings, extra whitespace doesn't matter (eg. SQL), but when you don't want it, you end up removing the indentation in the string literal, and having a string like this: fn f() { text = "This is a decent way to format strings, but surely it could be a little nicer indentation-wise, right?"; print(text); } The prefixed lines have the disadvantage of being a pain to use if your editor doesn't have nice multi-line editing like Vim or Sublime. But I think it's a nice option when it's available.
- afdbcreid 2mo agoThey forgot to add (I believe this was not deliberate, maybe their users already infer that) that this only actually performs the check on Debug and ReleaseSafe modes, not on ReleaseFast mode. Which is reasonable I guess because this is a memory write/read/branch in a super hot code path, but undermines a large part of the guarantee in my opinion (doesn't Zig have a debug allocator that could catch the mistake in the example just as well?).
- ivanjermakov 2mo agoDebug allocator can't catch it because it's not an allocation bug. Debug allocator finds bugs by marking memory during alloc/free and inspects them upon deinit. Pointer to a memory location change is not something allocator has control over. Possible solutions: smart array list implementation (this article), move semantic analysis (Rust's borrow checker), runtime introspection (https://fil-c.org/ https://fil-c.org/).
- afdbcreid 2mo agoIt is an allocation bug, it's a use-after-free. It's only a UAF if you actually have a reallocation (something that happens in the given example), but debug allocators don't require you to annotate your code.
- duped 2mo agoThis is a gripe of mine, and I will admit it is weak. Changing a segfault to a panic with a stack trace is an improvement in developer experience. It does not make better software. The advantage of automatic strategies to mitigate memory safety mistakes either by using GC to make the program sound or static analysis to prevent the mistake by construction is plainly better. There is a direction in some systems programming circles away from this by eschewing "complexity" (in other words, fixing the damn problems) for programs that have better error messages when the programmer made a mistake. I don't see that as better software.
- boricj 2mo agoIt took me a minute to understand that this asserts on pointer change within the container, rather than lock/unlock the data structure like a SDL surface. I recently implemented a custom C++ container for a path whose components could be iterated, backed by a std::string. I just store indices and a reference to the string, such that my iterators are not invalidated if the std::string gets reallocated after being modified. Far less error prone for little added cost.
- portly 2mo agoThis makes a lot of sense if you consider that it is consistent with the rest of the language. It is one more way to set up tripwires in your code to to catch your own programming errors. Similar to using asserts in your functions to vet input and output. I use Array list a lot so excited to add this throughout the code to harden them. I can imagine this is not everyone's cup of tea, but then you probably also wouldn't enjoy any of the other explicitness.
- Prydown 2mo agoConstant headache in C++ with `std::vector` element references. Zig's explicit stability here is a welcome relief for data structures.
- afdbcreid 2mo agoIt seems you misunderstood. It's not an explicit stability guarantee (such things is not possible), it's a debugging helper to crash the program more easily when it happens, requiring you to annotate the code.
- stub_out 2mo agoC++ developers constantly wrestle with `vector` iterator invalidations. Good on Zig for making this a first-class concern.
- brcmthrowaway 2mo agoHow does Rust avoid this?
- trucks-refinish 2mo agoIn rust you cannot mutate anything that is being read basically. The borrow checker enforces this. So an iterator takes an immutable reference to the vector and mutation requires a mut ref, and you can't have both at the same time.
- Devin3162 2mo ago[dead]
- massimodeluisa 2mo ago[dead]