8 ms·

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, deve

• kixpanganiban 10y ago
  Kenton, thank you for your work on this! I currently use ZeroRPC (which uses protobuf and msgpack) and I was blown away by Cap'n Proto. Really excited to try it out soon! Some questions: - Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own? - Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality? Thanks!
  • kentonv 10y ago
    > - Do you guys have an RPC library written in anything other than C++? If not, could you point me to protocol specs so I can start writing my own? People have written implementations in Rust, Go, and Erlang, and wrappers around the C++ library in Javascript and Python: https://capnproto.org/otherlang.html https://capnproto.org/otherlang.html Scroll down that page for some info on how to start writing an implementation in another language. The RPC protocol spec is here: https://github.com/sandstorm-io/capnproto/blob/master/c++/src/capnp/rpc.capnp https://github.com/sandstorm-io/capnproto/blob/master/c++/sr... > - Since it uses a streaming model to support random access, what encryption method do you think would work best with Cap ' n Proto that would keep it speedy and still retain all functionality? Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access. Regarding encryption, this is a very big question and there are a lot of different needs and use cases to consider. Mostly I don't think that use of Cap'n Proto affects encryption decisions much, but if you want to make sure you don't lose random access, you should of course use a cipher that supports random access, like chacha20 or AES-CTR.
    • kixpanganiban 10y ago
      > Hmm, I'm not clear on what you mean by "streaming model" -- I think of "streaming" as the opposite of random access. Right, sorry, I re-read my comment and confused myself too. Seems like it's bad for me to go on HN without a fresh cup of coffee (it's 10am now here in the Philippines). Thanks for your swift response, I'll experiment with AES-CTR (since I'm more familiar with it than chacha20). And thanks for pointing out that there are wrappers for Python/Go already, the programming language I use daily and was thinking of building libs for! Again, great work, and I'll stay posted.
      • aidenn0 10y ago
        Also as a note, both ciphers suggested are stream ciphers, so you will need some way of authenticating. Depending on your needs, you may consider using something like libsodium that takes care of all of that for you: https://download.libsodium.org/libsodium/content/secret-key_cryptography/authenticated_encryption.html https://download.libsodium.org/libsodium/content/secret-key_...
• lobster_johnson 10y ago
  I'd like to see Ruby bindings (not just serialization). We are using Go, Node.js and Ruby in production, and we have been looking to move from plain HTTP to gRPC. If Cap'n Proto is better, we might use it, but not without Ruby bindings.
  • kentonv 10y ago
    I don't think anyone is currently working on Ruby bindings, but if you're interested, feel free to take it on! (Note that Sandstorm is focused on making Cap'n Proto work well for Sandstorm. We welcome contributions, but we generally don't have resources available ourselves to work on third-party feature requests unrelated to Sandstorm. That is, unless you want to pay us a bunch of money, in which case, feel free to contact me. ;) )
    • bascule 10y ago
      Actually there was a Capnproto project for Ruby GSoC project this year! https://github.com/nemoNoboru/capnp-ruby https://github.com/nemoNoboru/capnp-ruby https://rubygems.org/gems/capn_proto-rpc https://rubygems.org/gems/capn_proto-rpc
      • nemoNoboru 10y ago
        Yep, that's my project. Please use it and contact me if somenthing doesn't work!
        • kentonv 10y ago
          Hmm, should this be linked on the "other languages" page? I don't remember you telling me it was ready -- sorry if I missed something!
      • lobster_johnson 10y ago
        Awesome, thank you!
• StephanTLavavej 10y ago
  As of MSVC 2015 Update 3, the support for C++11 is nearly complete. The missing pieces are that Expression SFINAE is partially supported (what's there is good enough for the STL and Boost, with certain workarounds that you must be aware of), and C99 preprocessor support is full of bugs. Aside from that, all features are present and generally usable. (Some are buggier than others; e.g. we're overhauling multithreading library support for the next major version of the libraries.)
  • Hydraulix989 10y ago
    There's still a bunch of minor differences/annoyances with the MS toolchain that require me to go in and make a bunch of small changes everywhere to my project, which compiles fine on clang/g++ (and I wouldn't say I'm doing anything _crazy_ like the aforementioned "Expression SFINAE"). Right now, my project's rather small 100 kLOC codebase compiles (with very minimal #ifdef hackery) on Android, iOS, Mac, and Linux; but Windows is still very much a WIP, even some of my third-party libraries don't compile on VC. I'm actually considering trying the MINGW toolchain at this point, and I'd be curious to hear anyone's thoughts? I'm sure part of it is that the compiler itself probably developed with the tight feedback loop from developing and maintaining large codebases in house at Microsoft (like Windows and Office). It's pretty hard when part of the internal pressure to support large codebases like that conflicts with the need to conform to outside third-party standards. I've heard great things about the compiler people at Microsoft, and I'm sure they have a technically strong team, but they are most likely caught in the middle of this organizational deadlock. I'm sure it doesn't help that I'm not mentioning any specifics, I'm going to be revisiting the Windows desktop port of my mobile app soon (which I gave up and haven't touched for a couple months), and everything will be fresh in my mind again. I do vaguely remember something about having to explicitly add more #include statements to pull in header files that were already getting pulled in by my other compilers.
    • david-given 10y ago
      I used to develop the Windows port of WordGrinder (my console word processor) using Mingw for Linux, testing with Wine and packaging using nsys for Linux. This actually worked pretty well and allowed a complete Windows development flow from Linux using the same rolling automation for both platforms. The big problem was Wine, which can behave quite differently to real Windows in some places. It's certainly not bug-for-bug compatible. So I kept finding myself having to boot into Windows anyway for debugging, so eventually I just installed an msys development system on Windows with gvim and used that. There was also something else wrong with mingw for Linux, but for the life of me I can't remember what. There was something about mingw being different to mingw32? Missing headers? It's been too long.
• oneloop 10y ago
  What a great pitch, that website is. I don't even understand what it is, but I'm stocked.
  • crossroads091 10y ago
    "Stoked". Sorry for the nitpick.
    • oneloop 10y ago
      I meant I'm in stock.
• zump 10y ago
  Why did you leave Google?
  • kentonv 10y ago
    Well, I could write a lot about this, but... In general, because I felt there was too much resistance to me pursuing ideas that I wanted to pursue. Google has become increasingly top-down, whereas it was fairly bottom-up when I started in 2005. That's not necessarily a bad thing, but I felt that I personally would be happier running a startup where I could call the shots. FWIW, if you just want to write code, be comfortable, and make a crapton of money, and you don't care if you're implementing someone else's ideas, I highly recommend working for Google. That's not meant to be sarcastic or disparaging -- I totally respect that approach and there are days when I wish that were me. But if you have ideas of your own and you won't be happy unless you see them implemented... it probably won't happen at Google. (To be fair, some people at Google would surely argue that my problem is that my ideas are crazy and bad, and I don't have any firm evidence -- yet -- that they are wrong.)
    • zump 10y ago
      Thanks for the candid reply!
    • DonHopkins 10y ago
      > and you don't care if you're implementing someone else's ideas and working for an advertising company FTFY
      • DonHopkins 10y ago
        I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company. It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising. Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work on, however explicable or inexplicable the reason might be. >A blow for mobile advertising: The next version of Safari will let users block ads on iPhones and iPads [1] : [...] An Apple realist might argue that its great rival Google makes more than 90 percent of its revenue from online advertising — a growing share of that on mobile, and a large share of that on iPhone. Indeed, Google alone makes about half of all global mobile advertising revenue. So anything that cuts back on mobile advertising revenue is primarily hurting its rival. (Google has been less friendly to adblockers than its “open” positioning would suggest.) [1] http://www.niemanlab.org/2015/06/a-blow-for-mobile-advertising-the-next-version-of-safari-will-let-users-block-ads-on-iphones-and-ipads/ http://www.niemanlab.org/2015/06/a-blow-for-mobile-advertisi...
• socmag 10y ago
  It sounds really well designed. Great job Cap'n Proto, I've been following your work for some time! I've also built another message format recently (yes I know, they are never ending). It can't do everything Cap'n Proto can do, although it shares a lot of the same values. One thing I chose to do is to have order preservation on types, which can be very useful. It does mean that the wire format is largely BE though. Anyway, that's an aside. I'm curious what you think of Amazon ION by the way? Good stuff!
  • kentonv 10y ago
    I haven't looked at Ion before, but it appears to be similar to BSON or Msgpack in that it's a binary format that encodes field names as textual identifiers, to be "self-describing" and avoid the need for an external schema. I generally like schemas, because I like static typing. Of course, static types vs. dynamic types are another ancient flamewar and I'm unlikely to cover any new ground by stating arguments here. :)
    • socmag 10y ago
      Hahaha very true let's not go there. Anyway yes I'm a big fan of richly typed schema-less formats as well, even better when they can be read easily by both JS stuff and C++. I'll definitely give it a whirl. Cheers!
    • aidenn0 10y ago
      Schemas are IMO the only way to go when you are transferring between heterogenous languages, even when all languages involved are untyped. Consider javascript talking to common lisp. Of course JSON has a canonical mapping to javascript, but it does not for common lisp. Should a JS array be a lisp list or vector? Should lisp's NIL be false or null? Should a JS object decode to an alist, plist, or hash-table? &ct.
      • nemo1618 10y ago
        Nitpick: I think you meant "&c". The t is superfluous.
• forrestthewoods 10y ago
  Treating Windows like a red headed step child is why I, with great disappointment, passed on Cap'n'Proto. Cross platform libraries that really only care about a single platform is an endless source of frustration. :(
  • Twirrim 10y ago
    Writing software that is safely and gracefully cross-platform is a pain in the neck :-/
    • vvanders 10y ago
      Except in this case you have flatbuffers which is a perfectly reasonable replacement and works across a wide array of platforms. We rejected Cap'n Proto for the same reasons. I don't see a need to use bleeding edge C++ features when the same(or better) results can be achieved with code generation. Kudos on the effort but if you're looking for wide adoption you've missed the mark.
      • kentonv 10y ago
        > if you're looking for wide adoption Well... We're not, actually. We're looking to support the needs of Sandstorm. It's nice if my code is useful to others, and if people want to contribute better Windows support I'll be totally happy to review and merge those changes! But wide adoption of Cap'n Proto (outside of the Sandstorm context) is not part of our business model, unfortunately. (Note: The poster you were replying to isn't associated with Cap'n Proto nor Sandstorm.)
      • nurettin 10y ago
        I heard the exact same thing said about templates 15 years ago.
• jcoffland 10y ago
  Recently I've started compiling more of my code for Windows using Mingw (or clang). Performance is very good and you can build in Linux for Windows inside a Docker container. You can even build Windows installers using NSIS. I've also tried to do this for OSX but with less success.
  • kentonv 10y ago
    Cap'n Proto supports MinGW as a target platform. (It has also supported Cygwin since very early on.) However, Cap'n Proto is a library, so it needs to support whatever compiler the users of that library are using, and for the vast majority of Windows developers that's MSVC.
    • jcoffland 10y ago
      Right, but you should be able to compile a library with either MinGW or clang from Linux that your users can use with MSVC.
      • kentonv 10y ago
        Not if it uses C++. MinGW uses the G++ ABI which is totally incompatible with the MSVC ABI. (Also, a lot of the issues we face affect headers, which need to be compiled into the client projects.)
        • jcoffland 10y ago
          Since the code is so small and in C++ why not make it a header only library?
• fpoling 10y ago
  I am curious, what is your take on Thrift? Is it really an improvement on protocol buffers?
  • kentonv 10y ago
    Thrift to me is a Protobuf clone. Very similar design. Early on it was not very well optimized compared to protobuf, but I imagine they've fixed that by now. The big advantage they had was that they included an RPC system in their first release -- although it was a FIFO RPC system which struck me as an odd design choice (I think they may have fixed this more recently?). But now GRPC exists, so I don't think there's much reason to choose Thrift over Protobuf/GRPC. However, I am obviously very biased. :)
• wvh 10y ago
  Slightly off-topic but it's good to see after all these years that companies like Microsoft (and Apple?) have to make a serious attempt to embrace the open-source community or miss out on a lot of cool new software and developers. Times have changed somewhat; the mantra used to be "doesn't work on Linux", while nowadays we've got a lot more heterogeneous environment.
• lux 10y ago
  I recently ran into flatbuffers https://google.github.io/flatbuffers https://google.github.io/flatbuffers), which claims the same no-serialization step. Curious how the two compare?
  • adrianratnapala 10y ago
    Does flatbuffers have an RPC mechanism? Or is it purely a serialisation tool? Cap'n Proto is both. I.e. you have the serialisation, which can be used on its own, but you can also define interfaces with methods that can pass those serialised structs as parameters, and which return asynchronous promises.
  • bchallenor 10y ago
    I recently evaluated the two and went with FlatBuffers, largely because its Java support appeared to be more mature. The FlatBuffers encoding is based on vtables and is relatively straightforward (the runtime library is tiny). This also means it's inefficient for small messages, but in my testing its vtable deduplication worked great for my use case (~100k messages of the same type per memory-mapped file), in that the vtable overhead tends quickly to zero. Cap'n Proto has a more complex encoding that is probably more efficient in terms of wire size, and particularly for small/standalone messages, but the runtime is larger as a result.
• lmm 10y ago
  If those platforms are causing issues would it be worth downgrading them to a second-class "LTS releases only" state or some such?
• DenisM 10y ago
  How about first-class map / dictionary support? That is, const or log time retrieval of key-value pairs. Right now I can only see a way to create lists, so access will be in linear time.
  • kentonv 10y ago
    The "lists" are actually arrays, so access by index is constant-time. You can build a hashtable on top of that. I'd like to add built-in support for maps at some point, but it's one of a very large number of wishlist items...
    • DenisM 10y ago
      The simplest implementation would be a sorted list with binary search for access. Segments create problems though. I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints. I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobile devices), but not yet large enough to have to use SQLite. I might be overthinking it, and could move to SQLite straight away. Just trying to keep my stuff as simple as possible. Conceptually simple, that is. Capnproto is conceptually simple: a tree dumped to disk, allowing on-demand memory mapped access to requested parts.
      • kentonv 10y ago
        The capnp structure itself could be a hashtable, and doesn't need to be (entirely) loaded into RAM. I'm saying you create a capnp list, and then you store elements into the list at position according to their hash -- i.e. how you'd build a hashtable, but the capnp list itself is the backing array. You would have to do this at write time, of course, and make sure the hashing is consistent between runs.
        • DenisM 10y ago
          Do'h. Of course. I'd still prefer the framework do it for me. It seems quite involved. Thanks for taking time to answer my questions!
• deleted 10y ago
  [deleted]
• anongoogler2 10y ago
  I could imagine implementing something like this as an alternate wire format and client library for protocol buffers. Can you outline the main reason you chose not to go this route? In particular, what aspects of the proto descriptor language don't fit Cap'n Proto? Or what aspects did you feel needed to be changed for some other reason?
  • kentonv 10y ago
    To be clear, by "alternate wire format and client library for protobuf" you mean "reuse the protobuf IDL but change everything else", right? There are a few major reasons I didn't go that route: * The .proto language has a lot of weird quirks that I don't like. Some of the quirks are specific to the protobuf encoding (e.g. int32 vs. sint32 vs. fixed32 being different types), while other quirks have no particular rationale behind them. I didn't want that baggage. * The .proto language does not treat interfaces (aka services) as a first-class type. That is, you cannot define a field whose type is an RPC interface type -- a reference to a remote object. The ability to do this is a critical part of Cap'n Proto's interface design. * It's highly unlikely that the protobuf team would be interested in accepting changes to the language which were not actually supported by protobuf. This means that if I shared the language, I would have my hands tied when it comes to new features -- or I'd also have to implement Protobuf equivalents to make them happy.
    • anongoogler2 10y ago
      Yeah that's what I meant. To me re-using the IDL would be a big plus because then all data can always be converted from one format to the other. The reasons you gave make sense, although I'm not super familiar with the RPC aspect of protocol buffers so that's new to me.
      • kentonv 10y ago
        For cases that don't use features specific to one format or the other, it should be easy to write a tool that automatically converts between schema formats, FWIW.
• easytiger 10y ago
  From reading the examples, it looks like one can only serialise to a file descriptor. What if i wan to serialise to a byte array? Am I perhaps missing something obvious?
  • kentonv 10y ago
    You can certainly serialize to a byte array. The example uses a file descriptor but in fact the code defines an abstract stream type which you can implement any way you want, and you can also obtain pointers to the message's underlying storage in order to extract the bytes directly (avoiding a copy).
    • easytiger 10y ago
      Great thanks. I'll give it a go again for a current project I'm kicking off.
      • easytiger 10y ago
        i've had a look in kj/io.h I see the classes: ArrayOutputStream/VectorOutputStream, i assumed these are the ones you are discussing.