2 ms·
I'm not convinced. Can the Rust function which is shown (that function alone) tokenize a number out of a string such that the number is overflowing the target t
by kazinator 5y ago
I'm not convinced. Can the Rust function which is shown (that function alone) tokenize a number out of a string such that the number is overflowing the target type? Yet indicate to the caller where that overflowing number ends, so that tokenization can continue with subsequent characters, if any?
E.g. suppose we have a string with this kind of syntax:
"12345 : 12345 , 12345"
We can
1. use strtol to get the first integer and a pointer to just after it.
2. use ptr += strspn(ptr, " ") to skip spaces
3. check for the colon and if we find it, skip with ptr++
4. use strtol to get the second integer (possibly preceded by space).
5. similarly to the colon handling, do the comma
6. strtol again to get the integer.
This is efficient: no splitting of the string into pieces requiring memory allocation, and extra list processing.
We can code this robustly: it can recognize valid syntax even if some of the numbers overflow. So for this kind of input:
"1234523442345345345234534545454545 : 12345 12345"
the code could diagnose the overflow, and the missing comma in one pass.
If you don't care about the details, just "is this number in range, with no trailing junk, or else is it bad", then raw strtol isn't convenient. But takes only a little code to wrap it.
- kaba0 5y agoRust can return string slices, so a tuple of a return type with a potential number and a slice for the still unprocessed string would be an option, which is much safer than your proposed one and arguably more readable.
- iovrthoughtthis 5y agobut this is not a appropriate place for that functionality.
- kazinator 5y agoIt is entirely appropriate for a function which lexically analyzes a buffer in memory in order to match an integer to be able to tell you where that integer ends.
- iovrthoughtthis 5y agoi think thats a reasonable opinion in the context of say, language implementation. for the relatively simple case of parsing cli arguments i would want an equally simple api. "is this string a valid representation of a number?" and "what number does this string represent?" should be separate apis. even in language implementation, i'd want the identification of number tokens to be separate from the parsing of those number tokens. i would then have another, separate api for "where does the first number end in this string?" which would probably more likely be "return to me the next substring from this string that represents a number"
- xigoi 5y agoHow often do you actually need this?
- tempodox 5y agoWhenever you do lexical analysis on syntax containing numbers. On today's hand-held supercomputers avoiding allocations and, generally, exercising memory-efficiency may not be a primary concern, but it very much was at the time when this stuff was built. And it's still relevant today on restricted systems, like microcontrollers, where C is still the primary language.
- lifthrasiir 5y ago> Whenever you do lexical analysis on syntax containing numbers. If that was an intention C should have a full set of lexical analysis functions, but it doesn't (scanf doesn't count). strtol being able to distinguish two error cases and thus being marginally useful for lexical analysis is most likely accidental.
- xigoi 5y agoIn my experience, reading a single number is a much much much more common operation than doing lexical analysis.