5 ms·

A major feature of modern C++, and one of the most difficult to replace, is its expressiveness and optional strictness. You don't get this with C (or most other

A major feature of modern C++, and one of the most difficult to replace, is its expressiveness and optional strictness. You don't get this with C (or most other languages). The metaprogramming and compile-time facilities for example, while complex, are indispensable for some code bases. This becomes evident when you try to port rigorous modern C++ code bases to other programming languages du jour. A surprising amount of C++ is difficult to express with equivalent behavior and performance in a similar amount of code. I play around with new system languages by implementing parts of database engines (typically written in modern C++) in the new language. It terms of abstract code architecture, you can port a C-style database design with relative ease -- most new system languages aim to be an improved C with direct portability in mind -- but not modern C++-style database design, and the latter is unambiguously superior for performance and correctness when writing database engines. C++ will continue to see a lot of usage as long as it is difficult or impossible to write code in other languages that is functionally equivalent to C++.

bennofs 6y ago
Can you summarise how modern C++ database design differs from C-style design?
- jandrewrogers 6y ago
  C-style designs rely heavily on macros and aliasing, there is minimal type safety and most correctness properties can only be determined at runtime. Resource management is extremely manual which is both fiddly in the amount of code and error prone. Lots of function pointer passing instead of generics. Macros are used to hide just how tedious a lot of it is but they aren't safe. PostgreSQL is an exceptionally tidy example of a C-style database engine. In C++-style designs, you end up writing surprisingly little code, having metaprogramming scaffolding generate most of the code for you while doing fairly deep correctness and type safety checks at compile-time. Someone has to write the scaffolding libraries but they aren't that large, just tedious, and they get reused. Resource management, change detection, etc is automagic because C++ makes that easy to hide even in complex cases like DMA I/O. Every data structure and algorithm is highly optimized for the local use case and using the most highly compressed representation reasonable in context. It would be impractical to write all of this code and analyze it manually. I used to write databases in C99. It required several times the lines of code relative to C++17, with worse results, even if you include the scaffolding libraries. C++17 implementations, done well, is much closer to writing a specification for a subsystem design and behavior and having the compiler generate an optimized implementation for that specification and exporting types that hide the fiddly details that can be safely composed with other generated types, than writing code. The scaffolding is also generic and flexible: some can generate an OLTP database engine just as easily as an OLAP database engine largely by changing the subsystem specifications and composing things differently. This would not be possible without heavy use of the metaprogramming facilities to both generate the types and guarantee that type interactions will still be safe and reasonably optimal.
  - noahl 6y ago
    I'd be curious to know what the libraries are that allow this sort of database engine design. Are there any open source examples?
    - jandrewrogers 6y ago
      I’ve never seen one in open source but I’ve written a couple over the years and seen a few others. It forces you to solve many difficult metaprogramming challenges elegantly so it is a good excuse to achieve some mastery. I’ve burned more hours than I care to admit trying to figure out how to achieve seemingly simple results. Also, until C++17, this was an exercise in masochism due to the limitations of the expressiveness and type inference, which is why so few people tried. Basically, there were a lot of unpleasant rough edges that went away with C++17 because the language wasn’t smart enough previously. C++20 will also be a big step forward for this, whenever the compilers become usable. One thing to understand is that these libraries are highly opinionated about the abstract architectural model. It doesn’t make a lot of sense to mix components from user space and kernel space designs, for example, though both have advantages separately. It tends to be more along the lines of one abstract architectural model and enormous amounts of elasticity and flexibility within that model based on the data models, workloads, transaction semantics, and hardware you are targeting. You also still have to write a spec that makes sense from a database engineering standpoint. At least for me, there are still significant parts of a database engine for which I haven’t built a metaprogramming scaffold. That is largely a matter of time and effort. Other parts I haven’t had to write much code for years but still get state-of-the-art implementation to spec. Ultimately, I’m trying to automate away my job.
      - socialdemocrat 6y ago
        The problem with such fancy meta programming in C++, is that while it may be a thrill to program for a C++ wiz you may end up with making something that is completely unmaintainable, because nobody else can grasp what you wrote. Compare to writing a database system in something like Go. Sure you make end up with 50% more code, but you could have anybody up and running reading and understanding the code within 3 days. IBM have done studies of this and found that fancy code is not all that valuable. It ends up falling in disuse over time as people don't get it. I have seen my fair share of C++ code which simply had to be tossed because nobody at the company could understand what the previous whiz kid had written.
ncmncm 6y ago
This. For C++, the overarching criterion for almost any language feature is how useful it is for capturing semantics in an easy-to-use, performant, powerful, and generally usable library. The consequence is that using new core language features in non-library code often seems unnecessarily complicated, but a library constructed using the feature as intended is insanely powerful. The result is that you can write libraries in C++ you cannot write in other languages, and you can call into libraries that are so powerful only from C++. In most other languages, using a library means you give something up: usually performance, and often it comes with restrictions. The standard of usefulness for C++ libraries is very high, and always increasing.
- steveklabnik 6y ago
  While this is true in general, we are talking about a library feature that many people want to move into a language feature, specifically because of the drawbacks of it being a library.
  - ncmncm 6y ago
    That a library implementation of pattern-matching would be unsatisfactory exposes a weakness in the core language. Usually, it is better to strengthen the language to the point where a library component can do the job. Sometimes, though, that is too hard, and you give up and do it directly in the core language. It should always be noted as a failing. Something similar happened with coroutines, although what we did get has hooks that a library should be able to patch into, to achieve wonders not yet seen.
    - steveklabnik 6y ago
      Yeah, I do agree that this is the best way to go. We'll see!
socialdemocrat 6y ago
I have the complete opposite experience. Through my whole career writing C++, that language has been in the way due to all its idiosyncratic behavior. One spends so much time trying to figure out a way around some crazy boiler plate code requirement in C++ only to realize that no, it simply isn't possible to do it in a sane way in C++. Switching to Swift I felt almost everything worked as it should. It was almost a bit boring not having to deal with all the usual crap that C++ would give me. I am willing to be you can write any C++ system much better in Swift. C++ would probably have a performance edge, but in 95% of cases not enough to be worth dealing with an ugly language like C++. > C++ will continue to see a lot of usage as long as it is difficult or impossible to write code in other languages that is functionally equivalent to C++. I would challenge you to give me any language construct that is indispensable for a particular type of program which cannot be done more elegantly in another language and with less headache. I cannot think of a single feature in C++ which I have ever missed in any other of my preferred languages.
- a-nikolaev 6y ago
  Maybe the idiosyncratic BS of C++ is what's hard to express in a saner language? If someone only used hammers all their life, then they will say that a microscope is a bad hammer, which is kind of true =D
- jandrewrogers 6y ago
  You likely design very different software systems than I do. I don’t have the luxury of ignoring performance, efficiency, and scalability. I don’t have any particular love for C++, there are many things wrong with it, but sadly there isn’t a practical alternative that doesn’t have worse trade offs. Ironically, I largely refused to use C++ until C++11. I delivered systems in a long list of other languages before that time; my usage of C++ is relatively recent. My choice is pragmatic and based on experience. If you can’t think of reasons to use C++, that isn’t because reasons don’t exist.
- rualca 6y ago
  > One spends so much time trying to figure out a way around some crazy boiler plate code requirement in C++ only to realize that no, it simply isn't possible to do it in a sane way in C++. It seems you're complaining about the outcome of poor and ill-advised engineering practices instead of a programming language. And your description also covey's the idea that you had people who knew very little about C++ trying to figure out how to use it in ways that they have no idea was possible to use. I get the appeal of a good scapegoat. Yet, from your description it seems you're trying to dump the blame on a lot of engineering problems you're creating for yourself on a tool.

bennofs 6y ago

Can you summarise how modern C++ database design differs from C-style design?

jandrewrogers 6y ago
C-style designs rely heavily on macros and aliasing, there is minimal type safety and most correctness properties can only be determined at runtime. Resource management is extremely manual which is both fiddly in the amount of code and error prone. Lots of function pointer passing instead of generics. Macros are used to hide just how tedious a lot of it is but they aren't safe. PostgreSQL is an exceptionally tidy example of a C-style database engine. In C++-style designs, you end up writing surprisingly little code, having metaprogramming scaffolding generate most of the code for you while doing fairly deep correctness and type safety checks at compile-time. Someone has to write the scaffolding libraries but they aren't that large, just tedious, and they get reused. Resource management, change detection, etc is automagic because C++ makes that easy to hide even in complex cases like DMA I/O. Every data structure and algorithm is highly optimized for the local use case and using the most highly compressed representation reasonable in context. It would be impractical to write all of this code and analyze it manually. I used to write databases in C99. It required several times the lines of code relative to C++17, with worse results, even if you include the scaffolding libraries. C++17 implementations, done well, is much closer to writing a specification for a subsystem design and behavior and having the compiler generate an optimized implementation for that specification and exporting types that hide the fiddly details that can be safely composed with other generated types, than writing code. The scaffolding is also generic and flexible: some can generate an OLTP database engine just as easily as an OLAP database engine largely by changing the subsystem specifications and composing things differently. This would not be possible without heavy use of the metaprogramming facilities to both generate the types and guarantee that type interactions will still be safe and reasonably optimal.
- noahl 6y ago
  I'd be curious to know what the libraries are that allow this sort of database engine design. Are there any open source examples?
  - jandrewrogers 6y ago
    I’ve never seen one in open source but I’ve written a couple over the years and seen a few others. It forces you to solve many difficult metaprogramming challenges elegantly so it is a good excuse to achieve some mastery. I’ve burned more hours than I care to admit trying to figure out how to achieve seemingly simple results. Also, until C++17, this was an exercise in masochism due to the limitations of the expressiveness and type inference, which is why so few people tried. Basically, there were a lot of unpleasant rough edges that went away with C++17 because the language wasn’t smart enough previously. C++20 will also be a big step forward for this, whenever the compilers become usable. One thing to understand is that these libraries are highly opinionated about the abstract architectural model. It doesn’t make a lot of sense to mix components from user space and kernel space designs, for example, though both have advantages separately. It tends to be more along the lines of one abstract architectural model and enormous amounts of elasticity and flexibility within that model based on the data models, workloads, transaction semantics, and hardware you are targeting. You also still have to write a spec that makes sense from a database engineering standpoint. At least for me, there are still significant parts of a database engine for which I haven’t built a metaprogramming scaffold. That is largely a matter of time and effort. Other parts I haven’t had to write much code for years but still get state-of-the-art implementation to spec. Ultimately, I’m trying to automate away my job.
    - socialdemocrat 6y ago
      The problem with such fancy meta programming in C++, is that while it may be a thrill to program for a C++ wiz you may end up with making something that is completely unmaintainable, because nobody else can grasp what you wrote. Compare to writing a database system in something like Go. Sure you make end up with 50% more code, but you could have anybody up and running reading and understanding the code within 3 days. IBM have done studies of this and found that fancy code is not all that valuable. It ends up falling in disuse over time as people don't get it. I have seen my fair share of C++ code which simply had to be tossed because nobody at the company could understand what the previous whiz kid had written.

ncmncm 6y ago

This. For C++, the overarching criterion for almost any language feature is how useful it is for capturing semantics in an easy-to-use, performant, powerful, and generally usable library. The consequence is that using new core language features in non-library code often seems unnecessarily complicated, but a library constructed using the feature as intended is insanely powerful. The result is that you can write libraries in C++ you cannot write in other languages, and you can call into libraries that are so powerful only from C++. In most other languages, using a library means you give something up: usually performance, and often it comes with restrictions. The standard of usefulness for C++ libraries is very high, and always increasing.

steveklabnik 6y ago
While this is true in general, we are talking about a library feature that many people want to move into a language feature, specifically because of the drawbacks of it being a library.
- ncmncm 6y ago
  That a library implementation of pattern-matching would be unsatisfactory exposes a weakness in the core language. Usually, it is better to strengthen the language to the point where a library component can do the job. Sometimes, though, that is too hard, and you give up and do it directly in the core language. It should always be noted as a failing. Something similar happened with coroutines, although what we did get has hooks that a library should be able to patch into, to achieve wonders not yet seen.
  - steveklabnik 6y ago
    Yeah, I do agree that this is the best way to go. We'll see!

socialdemocrat 6y ago

I have the complete opposite experience. Through my whole career writing C++, that language has been in the way due to all its idiosyncratic behavior. One spends so much time trying to figure out a way around some crazy boiler plate code requirement in C++ only to realize that no, it simply isn't possible to do it in a sane way in C++. Switching to Swift I felt almost everything worked as it should. It was almost a bit boring not having to deal with all the usual crap that C++ would give me. I am willing to be you can write any C++ system much better in Swift. C++ would probably have a performance edge, but in 95% of cases not enough to be worth dealing with an ugly language like C++. > C++ will continue to see a lot of usage as long as it is difficult or impossible to write code in other languages that is functionally equivalent to C++. I would challenge you to give me any language construct that is indispensable for a particular type of program which cannot be done more elegantly in another language and with less headache. I cannot think of a single feature in C++ which I have ever missed in any other of my preferred languages.

a-nikolaev 6y ago
Maybe the idiosyncratic BS of C++ is what's hard to express in a saner language? If someone only used hammers all their life, then they will say that a microscope is a bad hammer, which is kind of true =D
jandrewrogers 6y ago
You likely design very different software systems than I do. I don’t have the luxury of ignoring performance, efficiency, and scalability. I don’t have any particular love for C++, there are many things wrong with it, but sadly there isn’t a practical alternative that doesn’t have worse trade offs. Ironically, I largely refused to use C++ until C++11. I delivered systems in a long list of other languages before that time; my usage of C++ is relatively recent. My choice is pragmatic and based on experience. If you can’t think of reasons to use C++, that isn’t because reasons don’t exist.
rualca 6y ago
> One spends so much time trying to figure out a way around some crazy boiler plate code requirement in C++ only to realize that no, it simply isn't possible to do it in a sane way in C++. It seems you're complaining about the outcome of poor and ill-advised engineering practices instead of a programming language. And your description also covey's the idea that you had people who knew very little about C++ trying to figure out how to use it in ways that they have no idea was possible to use. I get the appeal of a good scapegoat. Yet, from your description it seems you're trying to dump the blame on a lot of engineering problems you're creating for yourself on a tool.