4 ms·
If you have a formatting operation that outputs some number of chunks of text to a stream, using some dynamically dispatched output operation which does somethi
by kragen 5y ago
If you have a formatting operation that outputs some number of chunks of text to a stream, using some dynamically dispatched output operation which does something similar to writing to a buffered output file, you can use it to produce a string fairly easily, as long as it fits in memory: you implement an output operation that appends to a string buffer. This is close to optimally efficient except when appending characters one or two at a time, in which case the extra dynamic dispatch overhead may be significant compared to whatever else you're doing to choose those characters.
If you have a formatting operation that produces an in-memory string, you cannot use it if the amount of text being produced is too large, or if you do not know if it is too large. But if it is not too large, you can then output the string to a buffered output file, although this is less efficient because it involves an extra copy and potentially dirties a lot more memory.
So these two ways of handling formatting are easily interconvertible when memory permits, but in some circumstances only the first one is applicable, and it is always more efficient.
Therefore, when reliability or efficiency is important, generating a formatted string should be implemented as a layer on top of sending formatted output to a stream, not vice versa.
There are cases where reliability and efficiency are not important, because your code is being used by end users and not as part of a library, and an error message is also an acceptable result of trying to run your program, and in those cases you should probably write your program in a language like Lua, Python, or JS.
It is probably true that if you are used to languages like those, a lot of design decisions that are necessary for reliability and efficiency will be counterintuitive to you.
- lifthrasiir 5y agoNote that this explains why string formatting is inherently tied with the I/O layer, but not necessarily why `println` (which is a very specific I/O operation) should be in the `fmt` module. That one is more subjective and probably only for the convenience.
- kragen 5y agoDo you mean why you can write (untested) fmt::println(x, y, z); instead of (also untested) fmt::fprintf(os::stdout, "{}{}{}\n", x, y, z); ? I mean I agree that presumably anything you could do in the first form could be done in the second form, less conveniently as you say, but if you omit println from fmt then all the Golang programmers learning Hare will be surprised about the missing stair they expect as the Hare counterpart to fmt.Println. And I think omitting the terminating newline in your format string is a common enough bug that it's probably worthwhile to include a separate function that adds it implicitly, especially in the case of fmt::print, which doesn't have a format string to add it to. Even without Golang experience, if you're going to have an fprintf and a println, I'd look for the println in the module that has the fprintf in it.
- lifthrasiir 5y agoWell, I just meant `println` should be probably in `io` rather than `fmt`---it is a great convenience function to have. Also not all streams are equal: stdio is heavier than file streams, which in turn are heavier than memory-backed streams (for example, stdio will probably need a built-in lock while string streams needn't). If you only need for example memory-backed streams you want to never see any code related to stdio, which might not be possible in some designs.
- kragen 5y agoYeah, it would be a reasonable design for fmt to contain no direct dependency on io, just on the stream interface. And that would enable you to put print and println (as opposed to fprintf) in io rather than fmt without creating a circular dependency between modules. And it might be a convenient way to remove unneeded dependencies on embedded targets (though I don't think they're targeting those). But it would be a significant departure from the Golang interface. I don't remember if Golang "stdio" has built-in locks. Hare doesn't seem to have threads or locks, o that may be a non-issue.
- ddevault 5y ago"io" cannot depend on "fmt", since "fmt" depends on "io" (technically, dependency cycles in Hare can be solved, but it's messy and strongly discouraged if avoidable). We did previously have a simple io::println which accepted strings but did not do formatting, but it's not super useful and fmt::* does it much better (and it was unbuffered, because bufio depends on io), so we ended up removing it. I thought about keeping it but making it use vectored writes to get around the lack of buffering, but in the end it's not really worth it.
- kragen 5y agoWhat do you mean by "vectored writes"? I thought you meant "invoking a function pointer" but io::stream already does that. Why is io::handle a separate type from io::stream? Is it just an efficiency hack to avoid an indirection through a function pointer for the common case where what you're writing to actually is a file? Can't io::handle eliminate the entanglement between buffered output and users of streams like the old io::println? I feel like there's something I'm not understanding here. (Is fd: int really the right way to handle a pointer to a structure representing how data is being transparently compressed into a zipfile, or a terminal emulator state that is being updated by writing bytes to it?)
- pierrebai 5y agoSo, the case is that the formatting string fits into memory, the data being formatted fits in memory but the output formatted text does not? This seems like an extremely rare case, which is itself a subset of the very rare case that you are formatting something that can be arbitrarily large. Oh, and the programmer must somehow have overlook this and not written it formatted output to output in reasonably-sized chunks. Meanwhile, having back-and-forth between the formatter and I/O means the formatter must correctly handles all the I/O problems and corner cases: disk full, disconnect, transient failures, ... not even going into that for some of them, you might want the caller to decide the outcome, which is easy when you split formatting and I/O, but hard when the formatting performs the I/O. I think this is a case that it may sounds superior to mix both, but it is actually buying next to nothing but both complicates matters and probably hides or ignores real issues.
- ddevault 5y ago> So, the case is that the formatting string fits into memory, the data being formatted fits in memory but the output formatted text does not? It takes a minimum of twice as much memory to store the original data and its formatted stream. Some systems don't have much memory, and some systems work with large amounts of data. Some systems work with more data than fits in RAM, or more data than even fits on disk, or even infinite streams of data, from sensors or generated data. All of these use-cases must be accommodated by Hare. That said, you can easily format into a dynamically allocated string in Hare if you so desire: let s = fmt::asprintf("Hello, {}!", user); defer free(s); This has the obvious downside of requiring an additional memory allocation, which can also fail just as easily as I/O can, and must be freed after use. However, this function does not require I/O, so all of the I/O failure cases are ruled out. There are trade-offs between these two approaches. Hare's design is meant to let the user evaluate these trade-offs for their use-case and make an explicit, planned decision regarding their needs and potential failure modes.
- kragen 5y agoIt might be worth putting a short note on this into the library documentation so people understand the library's design objectives. Among other things it will help them find other things in the library. Incidentally this is one of the things that the limited form of laziness found in Python generators could help with, if Hare had it; you can think of an input stream as a lazy string. Unlike full laziness, generators have very predictable memory usage.