3 ms·

Overall, I love how this looks -- it's a nice evolution of protobufs. I had a couple issues/questions while reading through the site, so I'll just throw them i

Overall, I love how this looks -- it's a nice evolution of protobufs. I had a couple issues/questions while reading through the site, so I'll just throw them in here: > struct Date { > year @0 :Int16; > month @1 :UInt8; > day @2 :UInt8; > } That's a fairly awkward syntax, even more than .proto files. I notice your compiler is written in Haskell, so it should be pretty easy to define a human-friendly syntax and parse it with Attoparsec and friends. In particular, the use of colons and semicolons here seems unnecessary. Are you set on that syntax, or would you accept patches to change it? I don't know how much breaking change you'd be willing to accept at this point. > Fields (and enum values, and interface methods) must > be numbered consecutively starting from zero in the > order in which they were added. I disagree with this decision. Being able to set tag numbers in arbitrary order makes reading the text format of a proto message (e.g. during debugging) much nicer, because fields can be grouped according to their purpose rather than whenever they happened to be added to a message. Speaking of which, I don't see any specification for a text-based serialization format on this site. If I were you, I'd just steal Protobuf's because it's simple and easy to read/write. > Unlike Protobufs, you cannot skip numbers when defining > fields – but there was never any reason to do so anyway. For me, the reason to skip field numbers is when some older version of the message defined some field, but new messages removed it and clients should not know about it. Typically, I indicate this by replacing the field definition with a comment to ensure the field number isn't accidentally re-used. This reduces the amount of legacy cruft in .proto files. > GCC 4.7 Needed > > If you are using GCC, you MUST use at least version > 4.7 as Cap’n Proto uses recently-implemented C++11 > features. If you are using some other compiler… > good luck. One of the best features of protobufs is that they are easily portable between many platforms and languages. If the C++ implementation requires advanced modern compiler features, that will severely reduce the number of developers who are able to use it. It would be better to write the reference implementation such that it's supported even by ancient compilers, so that users don't have to worry about upgrading lots of other packages just to try out CAPNP. > For example, exceptions are thrown on assertion > failures (indicating bugs in the code), network > failures, and invalid input. Exceptions thrown by > Cap’n Proto are never part of the interface and > never need to be caught in correct usage. I'm fine with making assertion failures throw exceptions that "never need to be caught", but network failures and invalid input are very much things that should be treated as common events. Error reporting (via return values or out-parameters) will make the interface slightly less elegant. This is fine. The benefit of having proper error handling by far outweighs minor verbosity of error checking. > Text: Like Data, but the content must be valid > UTF-8, the last byte of the content must be zero, > and no other byte of the content can be zero. If there's a length prefix, why bother with a terminating NUL? D-Bus does the same thing, and it serves no reason except to add another possible error condition. > So when Directory.open is called remotely, the > content of a FileInfo (including values for name > and size) is transmitted back, but for the file field, > only the address of some remote File object is sent. It's not obvious how such address-passing can be implemented, without falling into the black hole that is CORBA. I don't see anything on the encoding page about it, either. Is an address some sort of 64-bit cookie, a string, or something more complex? How does using addresses work when the server is written in a language that doesn't have the concept of addressable values? > The Cap’n Proto RPC protocol is not yet defined. > See the language spec’s section on interfaces for a > hint of what it will do. Roughly what is the timeline for defining the RPC protocol? I've been working on a Protobuf-based RPC protocol for a while (it'll replace my use of D-Bus for inter-machine and low-overhead inter-process communication), but it probably won't be in even a beta state for at least another two or three months. If you don't have any immediate plans, I'd love to merge my work into CAPNP.

barrkel 14y ago
All I want to comment on is the terminating null even in the presence of a length prefix: it means C and similar languages can treat it as a (read-only) string without the need to make a copy.
kentonv 14y ago
Thanks for the feedback! > Are you set on that syntax, or would you accept patches to change it? I'm not wedded to the syntax. Maybe start a thread on the discussion group and see what people think? > Being able to set tag numbers in arbitrary order makes reading the > text format of a proto message (e.g. during debugging) much nicer, > because fields can be grouped according to their purpose rather than > whenever they happened to be added to a message. I think a text format should write fields in the order in which they appear, not the order of the numbers. These don't have to be the same order. The purpose of the numbers is to allow you to insert a new field in an arbitrary position in your struct def without breaking wire-compatibility, rather than requiring you to add all new fields at the end. Maybe the wording in the docs is confusing? > Speaking of which, I don't see any specification for a text-based > serialization format on this site. If I were you, I'd just steal > Protobuf's because it's simple and easy to read/write. I intend to support JSON transcoding. The protobuf format is kind of quirky, although it is nice that it doesn't require as much quoting. Maybe the JSON transcoder could have a "less quoted" mode that breaks standards somewhat. > For me, the reason to skip field numbers is when some older version > of the message defined some field, but new messages removed it and > clients should not know about it. In practice, you actually shouldn't do that even with protobufs, because if you accidentally reuse that field number, you might misinterpret old data. Hence the convention with protobufs is usually to rename fields to "OBSOLETE_N". Cap'n Proto can't allow you to remove fields because that would change the offsets of subsequent fields. But you can certainly rename fields. > If the C++ implementation requires advanced modern compiler features, > that will severely reduce the number of developers who are able to > use it. This is true, but C++11 features are really useful, and the implementation (and even its interface) simply would not be as good without them. If this is a big problem, perhaps someone will start maintaining a back-port to C++98, as if it were another language. > I'm fine with making assertion failures throw exceptions that "never > need to be caught", but network failures and invalid input are very > much things that should be treated as common events. I think my wording in the docs may be a little off, in that it isn't completely clear what I mean by "correct usage". Yes, clearly you should catch network errors in a distributed environment. But typically the only things you really want to know about the error are: * Should I retry? * What error message should I log / display? Cap'n Proto exceptions can be caught and will carry this information. The point is more that these are logistical issues, not correctness issues. The "correct" output of your program never depends on what exceptions were thrown -- if any exception is thrown, either you don't produce output at all, or you solve the problem and produce the same output that you would have otherwise. Exceptions are somewhat necessary in Cap'n Proto because validation happens as you walk the message tree. Having every getter possibly return an error code would be pretty painful. You can, however, configure it so that getters flip an error flag and then return a default value on error. As long as you check that error flag before outputting garbage, you should be fine. > If there's a length prefix, why bother with a terminating NUL? It's helpful if the recipient intends to use the strings in C/C++ code -- they can avoid a copy. For example, if the message contains a file name and the recipient wants to open() it, they need a NUL-terminated string, and malloc()/copying one is kind of sad. Although I want to support many languages, I expect Cap'n Proto users to be biased towards C/C++ since those are the people who care deeply about speed. > It's not obvious how such address-passing can be implemented... This is certainly a complicated problem, but solvable. I wrote some comments elsewhere: https://news.ycombinator.com/item?id=5482782 https://news.ycombinator.com/item?id=5482782 I look to things like the E programming language and Tahoe-LAFS for inspiration here. > Roughly what is the timeline for defining the RPC protocol? I'm working full-time on Cap'n Proto right now. It's always hard to predict coding timelines, but I am hoping to have something within a month or two. > it'll replace my use of D-Bus for inter-machine and low-overhead > inter-process communication Hey, that sounds like exactly the main use case I'm interested in. I want to pass messages in shared memory, so making an IPC is basically no more expensive than posting to a semaphore. This makes it more plausible to have a sandboxed process which essentially makes all its system calls via IPC to the sandbox, without as much of a speed hit. Please do join the discussion group and tell us about your work! I'd love to have your help, or at the very least exchange notes. :)

barrkel 14y ago

All I want to comment on is the terminating null even in the presence of a length prefix: it means C and similar languages can treat it as a (read-only) string without the need to make a copy.

kentonv 14y ago

Thanks for the feedback! > Are you set on that syntax, or would you accept patches to change it? I'm not wedded to the syntax. Maybe start a thread on the discussion group and see what people think? > Being able to set tag numbers in arbitrary order makes reading the > text format of a proto message (e.g. during debugging) much nicer, > because fields can be grouped according to their purpose rather than > whenever they happened to be added to a message. I think a text format should write fields in the order in which they appear, not the order of the numbers. These don't have to be the same order. The purpose of the numbers is to allow you to insert a new field in an arbitrary position in your struct def without breaking wire-compatibility, rather than requiring you to add all new fields at the end. Maybe the wording in the docs is confusing? > Speaking of which, I don't see any specification for a text-based > serialization format on this site. If I were you, I'd just steal > Protobuf's because it's simple and easy to read/write. I intend to support JSON transcoding. The protobuf format is kind of quirky, although it is nice that it doesn't require as much quoting. Maybe the JSON transcoder could have a "less quoted" mode that breaks standards somewhat. > For me, the reason to skip field numbers is when some older version > of the message defined some field, but new messages removed it and > clients should not know about it. In practice, you actually shouldn't do that even with protobufs, because if you accidentally reuse that field number, you might misinterpret old data. Hence the convention with protobufs is usually to rename fields to "OBSOLETE_N". Cap'n Proto can't allow you to remove fields because that would change the offsets of subsequent fields. But you can certainly rename fields. > If the C++ implementation requires advanced modern compiler features, > that will severely reduce the number of developers who are able to > use it. This is true, but C++11 features are really useful, and the implementation (and even its interface) simply would not be as good without them. If this is a big problem, perhaps someone will start maintaining a back-port to C++98, as if it were another language. > I'm fine with making assertion failures throw exceptions that "never > need to be caught", but network failures and invalid input are very > much things that should be treated as common events. I think my wording in the docs may be a little off, in that it isn't completely clear what I mean by "correct usage". Yes, clearly you should catch network errors in a distributed environment. But typically the only things you really want to know about the error are: * Should I retry? * What error message should I log / display? Cap'n Proto exceptions can be caught and will carry this information. The point is more that these are logistical issues, not correctness issues. The "correct" output of your program never depends on what exceptions were thrown -- if any exception is thrown, either you don't produce output at all, or you solve the problem and produce the same output that you would have otherwise. Exceptions are somewhat necessary in Cap'n Proto because validation happens as you walk the message tree. Having every getter possibly return an error code would be pretty painful. You can, however, configure it so that getters flip an error flag and then return a default value on error. As long as you check that error flag before outputting garbage, you should be fine. > If there's a length prefix, why bother with a terminating NUL? It's helpful if the recipient intends to use the strings in C/C++ code -- they can avoid a copy. For example, if the message contains a file name and the recipient wants to open() it, they need a NUL-terminated string, and malloc()/copying one is kind of sad. Although I want to support many languages, I expect Cap'n Proto users to be biased towards C/C++ since those are the people who care deeply about speed. > It's not obvious how such address-passing can be implemented... This is certainly a complicated problem, but solvable. I wrote some comments elsewhere: https://news.ycombinator.com/item?id=5482782 https://news.ycombinator.com/item?id=5482782 I look to things like the E programming language and Tahoe-LAFS for inspiration here. > Roughly what is the timeline for defining the RPC protocol? I'm working full-time on Cap'n Proto right now. It's always hard to predict coding timelines, but I am hoping to have something within a month or two. > it'll replace my use of D-Bus for inter-machine and low-overhead > inter-process communication Hey, that sounds like exactly the main use case I'm interested in. I want to pass messages in shared memory, so making an IPC is basically no more expensive than posting to a semaphore. This makes it more plausible to have a sandboxed process which essentially makes all its system calls via IPC to the sandbox, without as much of a speed hit. Please do join the discussion group and tell us about your work! I'd love to have your help, or at the very least exchange notes. :)