6 ms·

> Built-in strings: hopefully UTF-8! Hopefully just bytes, which trivially allows storage of UTF-8. > built-in dynamic arrays, yess. Dynamic arrays are only

> Built-in strings: hopefully UTF-8! Hopefully just bytes, which trivially allows storage of UTF-8. > built-in dynamic arrays, yess. Dynamic arrays are only few lines to implement. There are different ways to do them, and no matter how, there will always be some problematic aspects. Not sure why you would want to choose one particular implementation and elevate it to a higher status. > No preprocessor. Heck yes, it's a crutch. It's ugly and inexperienced users will write bugs using it, but it's also tremendously useful. You mentioned quasiquoting as an alternative, but I'm not positive that it works as a preprocessor replacement for a language that lacks the "homomiconity" of LISP. Are there examples that show it works?

FeepingCreature 5y ago
> Dynamic arrays are only few lines to implement. There are different ways to do them, and no matter how, there will always be some problematic aspects. Not sure why you would want to choose one particular implementation and elevate it to a higher status. Just having a default in the language is insanely useful. I can only appeal to experience here (with D, which has built-in arrays), but I never want to be without them again. This goes doubly for my language, where dynamic arrays are actually a bit involved due to the need for slices, refcounting and capacity tracking for the doubling strategy on append. Not something you want to reimplement everywhere. > It's ugly and inexperienced users will write bugs using it, but it's also tremendously useful. You mentioned quasiquoting as an alternative, but I'm not positive that it works as a preprocessor replacement for a language that lacks the "homomiconity" of LISP. Are there examples that show it works? As an example, something like #define SQUARE(X) ({ typeof(X) x = X; x * x; }) could in a language with macros (and a better function macro syntax than I have at the moment :p) be rewritten as macro SQUARE(X) ({ typeof($X) x = $X; x * x; }); Which has the exact same effect, but does not suffer from the C preprocessor problems caused by string/token interpolation. Also, errors can be easily and cleanly attributed to the actual location they occur, because SQUARE's nature is a parse tree, not a token list. As for macros like `#define BEGIN {`, I consider it an advantage that they don't work. :)
- jstimpfle 5y ago
  > refcounting and capacity tracking for the doubling strategy on append. Not something you want to reimplement everywhere. Not something I want everywhere, in the first place. Especially refcounting. > macro SQUARE(X) ({ typeof($X) x = $X; x * x; }); Yes, stuff like that works, but only if your replacement body is a fully formed syntactical expression. Such a "syntactical macro" system can be nice because it's safer, and it's applicable in _most_ cases. But not for all - there are many situations where the full generality of C preprocessor macros (which are lexical / "token list" macros as you say) are useful. X-macros might be the example that I use most, and that wouldn't work with syntactical macros. Another application are macros around for loops, like FOREACH_FOO(...) or SCOPED_LOCK(...) for example. All dirty hacks that I like to use from time to time and that I prefer immensely to systems built in to the programming language and lead to immense complications in the language. Another example would be partial lists of any kind, for example lists of compiler intrinsic attributes #define FOO_API __attribute__((what1)) __attribute__((what2)).. String interpolation is probably not in scope for a syntactical macro system either printf("Hello from " PROG_NAME " version: " PROG_VERSION "\n"); Another example, but you might sneeze at that one - my current project is a pile of macro hacks, it contains a lot of stuff like #define DEFINE_BUILTIN(bkind, bname, num_args) else if (is_identifier(t->t_name.buf, bname)) \ ... #define PREC(t, p) case t: prec = p; break; It's more "temporary" stuff that will need factoring into data tables where there are extensions, but a lot of it is just good enough and will never be touched again. -- Expecting code to be so clean, always from the beginning, that all macros one would need ever can always be defined by syntactically complete expression bodys, that is not going to work. Just like there have been many attempts at getting rid of text-based programming languages and moving to structured (syntactical) editors - that hasn't panned out either.
  - FeepingCreature 5y ago
    > Another application are macros around for loops, like FOREACH_FOO(...) or SCOPED_LOCK(...) for example. This is the exact case where full macros can deliver a lot more power, more cleanly, than preprocessor macros. In this case, in Neat, you'd probably use a full parser macro rather than a call macro, and you could recognize arbitrary syntax, without allowing you to violate parenthesis order like C does. Your macro can do whatever it wants, but it cannot conflict with other, preexisting syntax, such as (in C) defining half a loop or half a variable declaration. This way, it can fully own its syntax. > Expecting code to be so clean, always from the beginning, that all macros one would need ever can always be defined by syntactically complete expression bodys, that is not going to work. Correct, however the example syntax was a simplification of macros. Fundamentally, there's no reason a macro shouldn't be able to do anything that the compiler itself can do.
pjmlp 5y ago
> Not sure why you would want to choose one particular implementation and elevate it to a higher status. I don't know, maybe for interoperable type safe types to use across an ecosystem of libraries without wasting CPU cycles converting among them?
- jstimpfle 5y ago
  To make it "safe" as in "protect against out-of-bounds accesses", slices would be enough. My strong opinion is that "data shape" concerns should be separate from "storage allocation" concerns as far as possible. This is especially true for the "to use across libraries without wasting time on conversions". I've said it many times, plain C interfaces (pointer + length, or slices if you insist but I don't like them because they are a less normalized representation) ... are the best way to design interfaces optimizing for interoperability. No need for any pointless conversion, just tell the API where your data is located. The physical fact that is needed for communication is the memory (address + length), it's the necessary and sufficient information to carry out the task. Yes, nowadays "safety" is not just about Out-of-bounds accesses but people expect the system to even protect against resource leaks, double-free, user-after-free, and race conditions. But even when it is the goal to machine check this by introducing a system that requires thinking on the small scale in isolated mini-units ("classes"/"types") - is there a point in locking in on a specific implementation of dynamic arrays? (Not a rhetoric question)
  - pjmlp 5y ago
    So Oracle, Apple, ARM, Google and Microsoft (Intel bothched their design) are investing piles of money moving the industry into hardware memory tagging for nothing? Maybe we should tell them to stop if they are so good.
    - jstimpfle 5y ago
      "Oracle, Apple, ARM, Google, and Microsoft" are actually a LOT of programmers and non-programmers with a huge variety of opininons, and I'm sure opinions similar to mine can be found there as well. Also, they have loads and loads of money and their jobs come with prestige, so they have no problem attracting developers to jobs that are perceived by some programmers (such as me) as boring boilerplate jobs that make me miserable. That answer was more related to the dynamic arrays discussion. If you want to move to hardware memory tagging, is that even a big thing? In any case my understanding is that it would work with pointer + length just as well, because the hardware tags are created at buffer allocation time, not based on arguments passed to a function.
ordu 5y ago
>> Built-in strings: hopefully UTF-8! > Hopefully just bytes, which trivially allows storage of UTF-8. Yes, it trivially allows, but then you'd need to deal with utf8-errors all over the program at runtime. Any method for string would need to be written such a way, that deals with invalid utf8 byte sequences. Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. It gives a lot of edge cases which a programmer must bear in his/her mind constantly, because these edge cases would choose the least expected moment to jump on you. And then you'd need, for example, to invent a way to output an arbitrary byte sequence into a place where only UTF8 is allowed. I, personally, hate it. Like arbitrary byte sequences as a file names, even when I know that no one uses non-utf8 file names, I need to write programs working with file names in such a way, that allows arbitrary byte sequences, and to devise some syntax to output arbitrary byte-sequence into a terminal. Or into a web-page. I see no good reasons to replace strings with arbitrary byte-sequences. If you need arbitrary byte-sequences then you have another abstraction for your task: an array. Much more powerful, because it can be array of bytes, of uint16_t, or of int64_t, or of your own struct. Why to spoil an abstraction of string with an ability to deal with arbitrary byte sequences?
- flohofwoe 5y ago
  Detecting UTF-8 encoding errors is only needed on the input/output boundary though. This should be solved in a string processing library, not in the language (and all other string processing functions in said library should not produce invalid output strings).
  - ordu 5y ago
    Not so easy. String processing library might want to iterate over chars, to do it a code needs to decode UTF8 string, if string is invalid-UTF8, then you'll get UTF8-error while trying to find a substring in a string. Or when trying to get a slice with chars from 5 to 12. Such an error could jump on you unexpectedly in any place of your program. > This should be solved in a string processing library Then a type String also should be defined in a string processing library. Either String follows assumptions of a string processing library, or it doesn't. If it doesn't it makes the task of writing a good processing library much more difficult, and you'll get errors thrown from inside of a string processing library, and you'll need to deal with them.
lerno 5y ago
Regarding dynamic arrays. Right now it looks like I have a way to do nicely namespaced (macro based!) operator overloading for types (it was unclear whether it would be possible with the feature set or not), so with this actually being in the language the need for built-in dynamic arrays is smaller - since you can use the normal foreach on the dynamic type - and so it can be implemented as a library type. (I want to emphasize that operator overloading is not done by functions, but through macros in C3 - so it's different from C++)

FeepingCreature 5y ago

> Dynamic arrays are only few lines to implement. There are different ways to do them, and no matter how, there will always be some problematic aspects. Not sure why you would want to choose one particular implementation and elevate it to a higher status. Just having a default in the language is insanely useful. I can only appeal to experience here (with D, which has built-in arrays), but I never want to be without them again. This goes doubly for my language, where dynamic arrays are actually a bit involved due to the need for slices, refcounting and capacity tracking for the doubling strategy on append. Not something you want to reimplement everywhere. > It's ugly and inexperienced users will write bugs using it, but it's also tremendously useful. You mentioned quasiquoting as an alternative, but I'm not positive that it works as a preprocessor replacement for a language that lacks the "homomiconity" of LISP. Are there examples that show it works? As an example, something like #define SQUARE(X) ({ typeof(X) x = X; x * x; }) could in a language with macros (and a better function macro syntax than I have at the moment :p) be rewritten as macro SQUARE(X) ({ typeof($X) x = $X; x * x; }); Which has the exact same effect, but does not suffer from the C preprocessor problems caused by string/token interpolation. Also, errors can be easily and cleanly attributed to the actual location they occur, because SQUARE's nature is a parse tree, not a token list. As for macros like `#define BEGIN {`, I consider it an advantage that they don't work. :)

jstimpfle 5y ago
> refcounting and capacity tracking for the doubling strategy on append. Not something you want to reimplement everywhere. Not something I want everywhere, in the first place. Especially refcounting. > macro SQUARE(X) ({ typeof($X) x = $X; x * x; }); Yes, stuff like that works, but only if your replacement body is a fully formed syntactical expression. Such a "syntactical macro" system can be nice because it's safer, and it's applicable in _most_ cases. But not for all - there are many situations where the full generality of C preprocessor macros (which are lexical / "token list" macros as you say) are useful. X-macros might be the example that I use most, and that wouldn't work with syntactical macros. Another application are macros around for loops, like FOREACH_FOO(...) or SCOPED_LOCK(...) for example. All dirty hacks that I like to use from time to time and that I prefer immensely to systems built in to the programming language and lead to immense complications in the language. Another example would be partial lists of any kind, for example lists of compiler intrinsic attributes #define FOO_API __attribute__((what1)) __attribute__((what2)).. String interpolation is probably not in scope for a syntactical macro system either printf("Hello from " PROG_NAME " version: " PROG_VERSION "\n"); Another example, but you might sneeze at that one - my current project is a pile of macro hacks, it contains a lot of stuff like #define DEFINE_BUILTIN(bkind, bname, num_args) else if (is_identifier(t->t_name.buf, bname)) \ ... #define PREC(t, p) case t: prec = p; break; It's more "temporary" stuff that will need factoring into data tables where there are extensions, but a lot of it is just good enough and will never be touched again. -- Expecting code to be so clean, always from the beginning, that all macros one would need ever can always be defined by syntactically complete expression bodys, that is not going to work. Just like there have been many attempts at getting rid of text-based programming languages and moving to structured (syntactical) editors - that hasn't panned out either.
- FeepingCreature 5y ago
  > Another application are macros around for loops, like FOREACH_FOO(...) or SCOPED_LOCK(...) for example. This is the exact case where full macros can deliver a lot more power, more cleanly, than preprocessor macros. In this case, in Neat, you'd probably use a full parser macro rather than a call macro, and you could recognize arbitrary syntax, without allowing you to violate parenthesis order like C does. Your macro can do whatever it wants, but it cannot conflict with other, preexisting syntax, such as (in C) defining half a loop or half a variable declaration. This way, it can fully own its syntax. > Expecting code to be so clean, always from the beginning, that all macros one would need ever can always be defined by syntactically complete expression bodys, that is not going to work. Correct, however the example syntax was a simplification of macros. Fundamentally, there's no reason a macro shouldn't be able to do anything that the compiler itself can do.

pjmlp 5y ago

> Not sure why you would want to choose one particular implementation and elevate it to a higher status. I don't know, maybe for interoperable type safe types to use across an ecosystem of libraries without wasting CPU cycles converting among them?

jstimpfle 5y ago
To make it "safe" as in "protect against out-of-bounds accesses", slices would be enough. My strong opinion is that "data shape" concerns should be separate from "storage allocation" concerns as far as possible. This is especially true for the "to use across libraries without wasting time on conversions". I've said it many times, plain C interfaces (pointer + length, or slices if you insist but I don't like them because they are a less normalized representation) ... are the best way to design interfaces optimizing for interoperability. No need for any pointless conversion, just tell the API where your data is located. The physical fact that is needed for communication is the memory (address + length), it's the necessary and sufficient information to carry out the task. Yes, nowadays "safety" is not just about Out-of-bounds accesses but people expect the system to even protect against resource leaks, double-free, user-after-free, and race conditions. But even when it is the goal to machine check this by introducing a system that requires thinking on the small scale in isolated mini-units ("classes"/"types") - is there a point in locking in on a specific implementation of dynamic arrays? (Not a rhetoric question)
- pjmlp 5y ago
  So Oracle, Apple, ARM, Google and Microsoft (Intel bothched their design) are investing piles of money moving the industry into hardware memory tagging for nothing? Maybe we should tell them to stop if they are so good.
  - jstimpfle 5y ago
    "Oracle, Apple, ARM, Google, and Microsoft" are actually a LOT of programmers and non-programmers with a huge variety of opininons, and I'm sure opinions similar to mine can be found there as well. Also, they have loads and loads of money and their jobs come with prestige, so they have no problem attracting developers to jobs that are perceived by some programmers (such as me) as boring boilerplate jobs that make me miserable. That answer was more related to the dynamic arrays discussion. If you want to move to hardware memory tagging, is that even a big thing? In any case my understanding is that it would work with pointer + length just as well, because the hardware tags are created at buffer allocation time, not based on arguments passed to a function.

ordu 5y ago

>> Built-in strings: hopefully UTF-8! > Hopefully just bytes, which trivially allows storage of UTF-8. Yes, it trivially allows, but then you'd need to deal with utf8-errors all over the program at runtime. Any method for string would need to be written such a way, that deals with invalid utf8 byte sequences. Unix and C is a living example of what happens when you think of a string as of a arbitrary sequence of bytes. It gives a lot of edge cases which a programmer must bear in his/her mind constantly, because these edge cases would choose the least expected moment to jump on you. And then you'd need, for example, to invent a way to output an arbitrary byte sequence into a place where only UTF8 is allowed. I, personally, hate it. Like arbitrary byte sequences as a file names, even when I know that no one uses non-utf8 file names, I need to write programs working with file names in such a way, that allows arbitrary byte sequences, and to devise some syntax to output arbitrary byte-sequence into a terminal. Or into a web-page. I see no good reasons to replace strings with arbitrary byte-sequences. If you need arbitrary byte-sequences then you have another abstraction for your task: an array. Much more powerful, because it can be array of bytes, of uint16_t, or of int64_t, or of your own struct. Why to spoil an abstraction of string with an ability to deal with arbitrary byte sequences?

flohofwoe 5y ago
Detecting UTF-8 encoding errors is only needed on the input/output boundary though. This should be solved in a string processing library, not in the language (and all other string processing functions in said library should not produce invalid output strings).
- ordu 5y ago
  Not so easy. String processing library might want to iterate over chars, to do it a code needs to decode UTF8 string, if string is invalid-UTF8, then you'll get UTF8-error while trying to find a substring in a string. Or when trying to get a slice with chars from 5 to 12. Such an error could jump on you unexpectedly in any place of your program. > This should be solved in a string processing library Then a type String also should be defined in a string processing library. Either String follows assumptions of a string processing library, or it doesn't. If it doesn't it makes the task of writing a good processing library much more difficult, and you'll get errors thrown from inside of a string processing library, and you'll need to deal with them.

lerno 5y ago

Regarding dynamic arrays. Right now it looks like I have a way to do nicely namespaced (macro based!) operator overloading for types (it was unclear whether it would be possible with the feature set or not), so with this actually being in the language the need for built-in dynamic arrays is smaller - since you can use the normal foreach on the dynamic type - and so it can be implemented as a library type. (I want to emphasize that operator overloading is not done by functions, but through macros in C3 - so it's different from C++)