5 ms·

Towards improved C++ support for games, graphics, real-time, embedded systems [pdf]

70 points by JSnake 11y ago

harry8 11y ago
"We started by looking at a paper from Paul Pedriana on N2771 “EASTL -- Electronic Arts Standard Template Library”. While this paper was submitted in 2007, neither chairs of LWG recall reviewing this paper and no record of its review exists." I give you the C++ standardization committee. Someone submits a paper to them with their issues and, moreover with solutions fully implemented in a library and also details the whys and wherefores. I have nothing to say whether this was good, bad or middling work just that the C++ standardization committee "has no record" of actually having read and considered it 8 years later and the chairs don't recall doing so. But I'm sure they all ran around as "anointed smart people" claiming all the others on the committee are anointed smart people instead. Sorry if I'm sounding a bit bitter but they really need to be called out, this is really gross. It's contemptuous of people making the effort to make the language better is the best you can say of it. Arrogant, stupid, unkind, vicious and bitter might describe me and I am sorry for it if you feel that way (please note it and I will try to do better), but I think it's also a fair description of this act by the committee. Also on the front page today was Bruce Eckel's "thoughts on scala" which were utterly ridiculous for the first 10 minutes I watched. -He's a C++ stanardization committee alumni. "We feel that a holistic "here's a new STL and our problems" approach to any change seems a bit doomed. " WTF? You can't learn anything about what changes are desirable in the language from the implementation of how they solved their problems and assess whether they are more generally applicable? Pieces of them aren't worth assessing to be just adopted (or rejected)? You can't criticise what they tried and note the implementation and why the STL can do better? Seriously if you can't learn anything from this on the committee what the hell are you doing? Well NOW they are looking at it, hurrah! While justifying why it wasn't done before, boo! Mea culpa and a rocket up the committee members of the time seems vastly more appropriate if you actually care about this language. (Obviously if you don't, resign from the committee, immediately, anything else is immoral) If the members didn't care then, highlight the fact and state it is unacceptable going forward. If it's common or garden (hopefully isolated) incompetence like all of us display at times. "We're very sorry this happened." Would seem to be about the minimum standard here. It's the arrogance that goes along with the committee when they act like this that really gets me.
- acqq 11y ago
  I also remember Bloomberg submitted in 2005 some very nice template library changes that avoid some very real pain points of STL: www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1850.pdf "Using an existing implementation of the C++ standard library as a base, we made backward-compatible modifications to enable such per-instance allocators." I still don't know how I'd achieve per-instance allocator in modern "part-of-standard C++ STL." Is it possible? If yes, how? Does anybody know?
  - humanrebar 11y ago
    > I still don't know how I'd achieve per-instance allocator in modern "part-of-standard C++ STL." Is it possible? If yes, how? Does anybody know? Well, there are two things you can do. 1. (Accepted in C++11) Allow stateful allocators. This is especially useful for things like pool allocators. Your container would keep a pointer to its allocator and perform allocator operations through it. 2. (Proposed but not accepted yet) Allow polymorphic allocators. This would allow different data structures to use different allocators but still have the same type. Basically it would look like 1 above, but instead of keeping an allocator pointer, you would keep a pointer to an allocator interface and perform allocator operations through virtual method calls.
doctorpangloss 11y ago
It's actually kind of fascinating that the HN shortened title omits "games," which is really what this document focuses on. Specifically the document investigates "EASTL", i.e. the Electronic Arts STL, and how its learnings can be applied to the common STL. The actual title: "Towards improved support for games, graphics, real-time, low latency, embedded systems." These little pieces of censorship—that's not a sign that for some reason, we really don't like to talk about games. There's no pervasive second-class-citizenry attitude towards gaming. I don't really know how to interpret the omission, other than it's a little piece of little consequence. Gaming keeps surfacing in places big and small as a motivation behind some technical or business innovation. Here we get very real improvements to the STL. This certainly matters: we celebrate Quake's square-root estimation trick as a cultural artifact, evidence of the beauty of programming. Elsewhere, in recent news, we hear Stewart Butterfield's narrative, which was as much about Flickr and Slack as it is about his twice-failed game concept. Lots of ideas and characters in the tech world have touched gaming in one way or another. In my personal background at my startup, I keep running into folks pushed out of gaming. They start making "apps." Surrounded by app people, I might miss that a third of all new apps in the App Store are actually games[0]. Even then, games are disproportionately represented in Apple Design Awards. It's tough to get an accurate figure we'd all agree on, but I believe games are nonetheless also overrepresented in revenue (definitely from the looks of top grossing, if we're strictly speaking about mobile software). If we really want to talk about software, I'd expect 1/3 to 1/2 of the content on the front page to actually be about games. It's certainly what most software people are working on, and it is probably where the plurality of the money is made. I'm not going to speculate as to why (I'd leave that for certain ex-Google developers, who possibly wanted to make a card game all his life). Here, we're actually seeing gaming omitted from the title, as though it's verboten. [0] http://www.pocketgamer.biz/metrics/app-store/ http://www.pocketgamer.biz/metrics/app-store/
- JSnake 11y ago
  I certainly didn't truncate "games" from the title myself. Did HN do that on its own?
- shmerl 11y ago
  > It's certainly what most software people are working on, and it is probably where the plurality of the money is made. While gaming does drive innovation and it's a very interesting field, I doubt that's what most software people are working on and surely it's not the biggest sector of software industry in general, it's actually quite small in comparison with many others. That's not to undermine its value, just pointing out the broader perspective.
jwatte 11y ago
Still no support for static compile and runtime introspection. That could be easily and cheaply implemented, at least for POD and aggregates thereof. If I have to build one more system that requires either a custom parser or repeating the names of all members, I think I will cry. The compiler ALREADY KNOWS!
- plorkyeran 11y ago
  The latest mailing (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/#mailing2015-04 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/#mai...) includes several new reflection proposals. N4447 probably covers what you want (although I'm not a huge fan of it).
- kevinnk 11y ago
  There's actually an official "study group" looking at compile time refection proposals, so this is definitely something in the works. However, the proposals have ranged from mediocre to downright horrible so it's probably not going to be in the standard any time soon. Some light reading [1] (static if) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3613.pdf [2] (call for compile-time reflection) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3814.html [3] (enum properties queries) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3815.html [4] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3996.pdf
- saboot 11y ago
  I am very much in support of this. Not for games but serialization. Pretty surprised that there don't seem to be any compiled languages supporting static compile time reflection. I think D does, but that is it.
- _yosefk 11y ago
  One (longish) shortcut is getting the info from the debugger which already knows. That is, in the build system, compiling a file #including what you want, firing a debugger, printing out the necessary type information, generating code, building that code with the rest of your system.
- shin_lao 11y ago
yoklov 11y ago
The allocator support in C++11 and, from what I've seen, the proposed allocator support in C++1y/z/whatever still leaves a massive amount to be desired. Polymorphic allocators solve what's basically a non-problem for games (ABI boundary problems) while introducing huge other issues. Alignment is a PITA but not really a pressing IMO. The real issue is stuff like rebind and the requirements to call destructors (which destroys many practical implementations of pools and arenas, respectively). This probably falls under "STL puts an emphasis on correctness before practicality and performance". ... Honestly (and I am aware this isn't a popular opinion, although it's much more popular inside the game industry than outside of it), IMO the STL-style of C++ programming is fairly wrong for games in general, largely because of memory usage concerns. This style of programming emphasizes worrying about lifetimes of objects by way of RAII, in order to allow a uniform way of treating all types of resources, both memory and otherwise. This is convenient, but games need extremely tight control over non-memory resource management (need to reuse anything from the system, need to load lots of stuff in the background, etc). This means tying it to object and scope lifetime tends to be inflexible and cause future pain when you need to change the resource management scheme. And for memory, games tend not to care a lot about cleaning up each objects memory individually, preferring to do it in bulk (e.g. resetting an arena or pool). This means that using RAII (or using something that uses RAII) for memory management tends to fit poorly for games. So, IDK. I'm not holding my breath on the C++ standards committee making a decision that I care for, but it looks like they're making steps in the right direction nonetheless.
- Paul_S 11y ago
  There are no gamdevs on the c++ committee, probably because games companies are cheap bastards and don't want to expend resources on something that will affect them 10 years down the line (or even 2). Join the committee.
  - yoklov 11y ago
    Haha, even if that were something I was interested in at all, I'm honestly not sure what you could do to fix it at this point. I think that ship has sailed. Most of the issues are library issues (and are far deeper than my complaints about allocators). The language issues largely come from having too many features, and its not like any of them can be taken out. FWIW, I'm currently pinning my hopes on JAI, even though it's a long shot.
pavlov 11y ago
C++ has left the horrors of the '90s largely behind it, and I'm rather liking it these days... But every time I work on a C++ project, I soon get "array angst" -- a feeling of profound incompetence because I feel uncertain about basic array operations. Seems like every project has its own take on array classes and their ownership semantics. Std::vector appears to be largely useless (or at least it's not used anywhere in practice, so the effect is the same). Boost is just scary. Qt is almost like a different language by now. I wish the C++ standard had something like Cocoa's NSArray/CFArrayRef -- a general-purpose ordered container that is consistently used everywhere in APIs and does the job everywhere, even though it may not be quite the optimal data structure. But I guess it's 15 years too late for that.
- shin_lao 11y ago
  Could you explain why you think std::vector is useless?
  - pavlov 11y ago
    Like I wrote in my comment, it doesn't seem to be used in the real-world projects and SDKs that I deal with.
    - mreiland 11y ago
      Not if those projects/SDK's have to interface with C, but that's just the reality.
- humanrebar 11y ago
  > I wish the C++ standard had something like Cocoa's NSArray/CFArrayRef I think the proposed array_view is a much better choice in C++ land. The actual storage format of data (the data structure) should be an implementation detail, but there should be standard ways to share that data across various unrelated pieces of code. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4177.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n417...

70 points by JSnake 11y ago

"We started by looking at a paper from Paul Pedriana on N2771 “EASTL -- Electronic Arts Standard Template Library”. While this paper was submitted in 2007, neither chairs of LWG recall reviewing this paper and no record of its review exists." I give you the C++ standardization committee. Someone submits a paper to them with their issues and, moreover with solutions fully implemented in a library and also details the whys and wherefores. I have nothing to say whether this was good, bad or middling work just that the C++ standardization committee "has no record" of actually having read and considered it 8 years later and the chairs don't recall doing so. But I'm sure they all ran around as "anointed smart people" claiming all the others on the committee are anointed smart people instead. Sorry if I'm sounding a bit bitter but they really need to be called out, this is really gross. It's contemptuous of people making the effort to make the language better is the best you can say of it. Arrogant, stupid, unkind, vicious and bitter might describe me and I am sorry for it if you feel that way (please note it and I will try to do better), but I think it's also a fair description of this act by the committee. Also on the front page today was Bruce Eckel's "thoughts on scala" which were utterly ridiculous for the first 10 minutes I watched. -He's a C++ stanardization committee alumni. "We feel that a holistic "here's a new STL and our problems" approach to any change seems a bit doomed. " WTF? You can't learn anything about what changes are desirable in the language from the implementation of how they solved their problems and assess whether they are more generally applicable? Pieces of them aren't worth assessing to be just adopted (or rejected)? You can't criticise what they tried and note the implementation and why the STL can do better? Seriously if you can't learn anything from this on the committee what the hell are you doing? Well NOW they are looking at it, hurrah! While justifying why it wasn't done before, boo! Mea culpa and a rocket up the committee members of the time seems vastly more appropriate if you actually care about this language. (Obviously if you don't, resign from the committee, immediately, anything else is immoral) If the members didn't care then, highlight the fact and state it is unacceptable going forward. If it's common or garden (hopefully isolated) incompetence like all of us display at times. "We're very sorry this happened." Would seem to be about the minimum standard here. It's the arrogance that goes along with the committee when they act like this that really gets me.

acqq 11y ago
I also remember Bloomberg submitted in 2005 some very nice template library changes that avoid some very real pain points of STL: www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1850.pdf "Using an existing implementation of the C++ standard library as a base, we made backward-compatible modifications to enable such per-instance allocators." I still don't know how I'd achieve per-instance allocator in modern "part-of-standard C++ STL." Is it possible? If yes, how? Does anybody know?
- humanrebar 11y ago
  > I still don't know how I'd achieve per-instance allocator in modern "part-of-standard C++ STL." Is it possible? If yes, how? Does anybody know? Well, there are two things you can do. 1. (Accepted in C++11) Allow stateful allocators. This is especially useful for things like pool allocators. Your container would keep a pointer to its allocator and perform allocator operations through it. 2. (Proposed but not accepted yet) Allow polymorphic allocators. This would allow different data structures to use different allocators but still have the same type. Basically it would look like 1 above, but instead of keeping an allocator pointer, you would keep a pointer to an allocator interface and perform allocator operations through virtual method calls.

doctorpangloss 11y ago

It's actually kind of fascinating that the HN shortened title omits "games," which is really what this document focuses on. Specifically the document investigates "EASTL", i.e. the Electronic Arts STL, and how its learnings can be applied to the common STL. The actual title: "Towards improved support for games, graphics, real-time, low latency, embedded systems." These little pieces of censorship—that's not a sign that for some reason, we really don't like to talk about games. There's no pervasive second-class-citizenry attitude towards gaming. I don't really know how to interpret the omission, other than it's a little piece of little consequence. Gaming keeps surfacing in places big and small as a motivation behind some technical or business innovation. Here we get very real improvements to the STL. This certainly matters: we celebrate Quake's square-root estimation trick as a cultural artifact, evidence of the beauty of programming. Elsewhere, in recent news, we hear Stewart Butterfield's narrative, which was as much about Flickr and Slack as it is about his twice-failed game concept. Lots of ideas and characters in the tech world have touched gaming in one way or another. In my personal background at my startup, I keep running into folks pushed out of gaming. They start making "apps." Surrounded by app people, I might miss that a third of all new apps in the App Store are actually games[0]. Even then, games are disproportionately represented in Apple Design Awards. It's tough to get an accurate figure we'd all agree on, but I believe games are nonetheless also overrepresented in revenue (definitely from the looks of top grossing, if we're strictly speaking about mobile software). If we really want to talk about software, I'd expect 1/3 to 1/2 of the content on the front page to actually be about games. It's certainly what most software people are working on, and it is probably where the plurality of the money is made. I'm not going to speculate as to why (I'd leave that for certain ex-Google developers, who possibly wanted to make a card game all his life). Here, we're actually seeing gaming omitted from the title, as though it's verboten. [0] http://www.pocketgamer.biz/metrics/app-store/ http://www.pocketgamer.biz/metrics/app-store/

JSnake 11y ago
I certainly didn't truncate "games" from the title myself. Did HN do that on its own?
shmerl 11y ago
> It's certainly what most software people are working on, and it is probably where the plurality of the money is made. While gaming does drive innovation and it's a very interesting field, I doubt that's what most software people are working on and surely it's not the biggest sector of software industry in general, it's actually quite small in comparison with many others. That's not to undermine its value, just pointing out the broader perspective.

jwatte 11y ago

Still no support for static compile and runtime introspection. That could be easily and cheaply implemented, at least for POD and aggregates thereof. If I have to build one more system that requires either a custom parser or repeating the names of all members, I think I will cry. The compiler ALREADY KNOWS!

plorkyeran 11y ago
The latest mailing (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/#mailing2015-04 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/#mai...) includes several new reflection proposals. N4447 probably covers what you want (although I'm not a huge fan of it).
kevinnk 11y ago
There's actually an official "study group" looking at compile time refection proposals, so this is definitely something in the works. However, the proposals have ranged from mediocre to downright horrible so it's probably not going to be in the standard any time soon. Some light reading [1] (static if) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3613.pdf [2] (call for compile-time reflection) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3814.html [3] (enum properties queries) http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3815.html [4] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3996.pdf
saboot 11y ago
I am very much in support of this. Not for games but serialization. Pretty surprised that there don't seem to be any compiled languages supporting static compile time reflection. I think D does, but that is it.
_yosefk 11y ago
One (longish) shortcut is getting the info from the debugger which already knows. That is, in the build system, compiling a file #including what you want, firing a debugger, printing out the necessary type information, generating code, building that code with the rest of your system.
shin_lao 11y ago

yoklov 11y ago

The allocator support in C++11 and, from what I've seen, the proposed allocator support in C++1y/z/whatever still leaves a massive amount to be desired. Polymorphic allocators solve what's basically a non-problem for games (ABI boundary problems) while introducing huge other issues. Alignment is a PITA but not really a pressing IMO. The real issue is stuff like rebind and the requirements to call destructors (which destroys many practical implementations of pools and arenas, respectively). This probably falls under "STL puts an emphasis on correctness before practicality and performance". ... Honestly (and I am aware this isn't a popular opinion, although it's much more popular inside the game industry than outside of it), IMO the STL-style of C++ programming is fairly wrong for games in general, largely because of memory usage concerns. This style of programming emphasizes worrying about lifetimes of objects by way of RAII, in order to allow a uniform way of treating all types of resources, both memory and otherwise. This is convenient, but games need extremely tight control over non-memory resource management (need to reuse anything from the system, need to load lots of stuff in the background, etc). This means tying it to object and scope lifetime tends to be inflexible and cause future pain when you need to change the resource management scheme. And for memory, games tend not to care a lot about cleaning up each objects memory individually, preferring to do it in bulk (e.g. resetting an arena or pool). This means that using RAII (or using something that uses RAII) for memory management tends to fit poorly for games. So, IDK. I'm not holding my breath on the C++ standards committee making a decision that I care for, but it looks like they're making steps in the right direction nonetheless.

Paul_S 11y ago
There are no gamdevs on the c++ committee, probably because games companies are cheap bastards and don't want to expend resources on something that will affect them 10 years down the line (or even 2). Join the committee.
- yoklov 11y ago
  Haha, even if that were something I was interested in at all, I'm honestly not sure what you could do to fix it at this point. I think that ship has sailed. Most of the issues are library issues (and are far deeper than my complaints about allocators). The language issues largely come from having too many features, and its not like any of them can be taken out. FWIW, I'm currently pinning my hopes on JAI, even though it's a long shot.

pavlov 11y ago

C++ has left the horrors of the '90s largely behind it, and I'm rather liking it these days... But every time I work on a C++ project, I soon get "array angst" -- a feeling of profound incompetence because I feel uncertain about basic array operations. Seems like every project has its own take on array classes and their ownership semantics. Std::vector appears to be largely useless (or at least it's not used anywhere in practice, so the effect is the same). Boost is just scary. Qt is almost like a different language by now. I wish the C++ standard had something like Cocoa's NSArray/CFArrayRef -- a general-purpose ordered container that is consistently used everywhere in APIs and does the job everywhere, even though it may not be quite the optimal data structure. But I guess it's 15 years too late for that.

shin_lao 11y ago
Could you explain why you think std::vector is useless?
- pavlov 11y ago
  Like I wrote in my comment, it doesn't seem to be used in the real-world projects and SDKs that I deal with.
  - mreiland 11y ago
    Not if those projects/SDK's have to interface with C, but that's just the reality.
humanrebar 11y ago
> I wish the C++ standard had something like Cocoa's NSArray/CFArrayRef I think the proposed array_view is a much better choice in C++ land. The actual storage format of data (the data structure) should be an implementation detail, but there should be standard ways to share that data across various unrelated pieces of code. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4177.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n417...