4 ms·
I read through all those links, and even got your example working (on a phone - no curl etc - so had to write a litle java http client). Consequently, this comm
by hyperpallium 11y ago
I read through all those links, and even got your example working (on a phone - no curl etc - so had to write a litle java http client). Consequently, this comment is long. I hope it's useful to you!
BTW: I personally would like to see your idea usable on a phone - without a full local machine (iterating might be a pain, but your system sounds really fast). Not just serverless, but machineless! A currently underserved niche.
> deciding on just the right features to keep things manageable is incredibly difficult (more so than the tech itself I believe).
I agree implementation is the easier part, though we've been stuck so long, I think there must be a simpler way to look at the whole thing, involving some mathematical or algorithmic insight (as relational algebra did for databases).
Barrister: I sometimes get lost in special cases, and forget the main point that provides the fundamental help to people: Barrister seems feature-full, but from reading the first paragraph or so, it seems to only output docs - because that's all they say it does! If the reader already knows what Thift etc do (i.e. something of an expert, across the field), they could guess that maybe Barrister does more... but the users who just want the main thing you do are easier recruits. Perhaps this is partly why it's overlooked...
StackHut: I'm not sure you need a separate IDL, if it is generated automatically - why confuse the user with it? It seems like a more sophisticated customization tool, that you could leave aside for later? The IDL itself is pretty clear, and although I'd thought about (eg) java classes as defining a schema, I hadn't made the connection that IDL (as from CORBA) also do that.
(1). RPC format. I'm familar with OO serialization and schema languages (unfinshed PhD, book chapters, a library and business), but less so for RPC - so maybe I'm off-base here. And standards - even nascent standards - may be worth complying with. But why not omit the meta stuff, and make it even simpler:
{
"stackhut/web-tools":
{
"renderWebpage": ["https://stackhut.com", 711, 393]
}
}
I'm really just wondering if there's a strong reason for the meta data. It can be helpful to orient the reader, but here it is clear from context - the keys and how it's being used. Maybe there are other optional fields you sometimes need?
But... for your use-case, perhaps it doesn't matter that much, as the bindings hide it from users (but the point of text protocols is human readable, eg for debugging; so the simpler the better). You could use XML, or a binary protocol.
(2). JSON by example: This is my great idea for JSON schema, which I'm amazed no one has done yet: instead of another meta-format, do it by example:
Because JSON primitive values are typed, you can use a value to signify type. An object therefore also implicitly defines its type (like a java class). For example, the above JSON can also be used as a schema, because it indicates the two nested objects required, and the types of the primitives (string, number, number). Though I suggest a convention of using "", 0 and false as values.
Those zero values help convey that it's a type, not a value. It's temping to want to encode information in the value itself (as opposed to the type), such as a default value - but that maybe a mistake, because there is so much more than can be done with strings than with numbers. Keep it super simple.
[ Arrays are usually not fixed-length, enabling the next trick to encode optional values and polymorphism. ] The JSON spec allows duplicate keys: so you include the same key with different types for all the polymorphic types. Specifically, you include the null value to indicate it is optional. e.g. version is an optional string:
{
"version": "",
"version": null
}
This is a bit dodgey, because all JSON parsers simply return
one value if duplicate keys are found. You need to write your own parser. But it is valid JSON - and more importantly, it looks like valid JSON. That's the key idea of "by example" - it looks like what it represents; there's minimal cognitive leap from type to instance.
It's rare to want polymorphic primitive types (e.g. string and number), so this is more for polymorphic objects. Unlike the common trick of a "type" field for nominal polymorphism, this is structural polymorphism - where only the different fields distinguishes types. NB: there are some tricky cases here, when the fields overlap, and I'm not sure that client code would want to mess with it.
Finally those non-fixed length arrays: polymorphism is represented by the types of a set of values in the array. In other words, the values aren't ordered, but just represent the permissible types. eg:
[
{ "image_url":"", "width":0, "height":0 },
{ "text": "" },
{ "link_url":"", "link_text":""}
]
That's a schema for an arbitary-length list that can contain instances of those three types of objects, with those mandatory fields, of those primitive types.
NB: for a schema of the RPC "header" above, the fixed length array schema represents a fixed length array - a special case.
(3). primitive datatypes: This idea can be extended, in a second level, with explicit primitive datatypes - this is the next level of schema power that everyone wants. Every value is now a string, and looks something like: "url", "date", "email", and then gets closer to XS, with ranges like "int:1..31" and even those ridiculous regex defining valid values (great idea, awful in practice, like the regex for email). The key thing is it still is JSON and looks like JSON, since the datatype specification language is just a JSON primitive value (string), and the syntax and meaning is obvious and familar.
BTW minor typo on your website: s/intergrated/integrated/ (like integer)
- hyperpallium 11y agoRe: schema/RPC layer Barrister semantically seems like the java type system with minor syntactic differences like field/type order. It uses extends for choice/sum/polymorphism. In addition to schema definition, it also has interfaces for RPC definition. The only major limitation I noticed is it lacks recursion (cyclic type references). Important for completeness, but I don't think it's needed that often in practice - I'm interested if this is true. One great thing about an IDL approach to schema definition, for RPC, is its similarity to objects/classes - because that is the domain it is used from. It's familiar to users, and minimal cognitive burdern to switch between. (Similar argument for JSON - it's an "Object Notation"). In contrast, xml schema looks utterly different from objects/classes. I think there's even a hypothetical argument for making the IDL look exactly like java - not because it's great but because everyone knows it, even the haters (or, maybe borrow Python's type hint system - or whatever is familiar to your target users). XML schema has a rich type system, both for structure (eg minOccurs) and primitives. But these aren't needed to represent objects, because programming type systems are poorer and lack those features. --- I think your main problem is a very old one, of transferring primitive values between different languages (like different languages - even different C implementations - having 16 bit vs 32 bit ints). Text representations like XML and JSON mostly solve this, but of course they can still be too big. You might look at old solutions, like ASN.1, to see the problems that arise, but I think you're better off solving much problems when (and if) they occur - because maybe they just won't come up in practice today because of conventions unconsciously established, even though theoretically they should come up. Just make it as easy to use as possible, to get the thing done that the user wants to do, and you'll get all the feedback/guidance you need. Maybe you would like to have a type system subset that works for all languages - but this would just hamper those in the one language, or between two languages that are close. OTOH some enterprises like future proof options, in case they need to support other languages. Would love to hear your thoughts and experience on all these issues.
- mands 11y agoYes, we've been very happy with Barrister so far, in that it provides a simple syntax for RPC definitions that looks very similar to Java so is not hard to learn. And yes, keeping it similar to standard function/object calls lowers the cognitive load. Hmm, yep recursion not supported atm, although we are thinking of extending the basic system shortly - there are a lot of other features we'd like to add - true sum types, a dynamic operator, and more. Python annotations are nice, and a lot of users have asked for a way to annotate RPC functions in their code. Doing this all in a cross-language way will be hard and so we're thinking of keeping the external definition in the meantime. -- Thanks so much for your advice. We're certainly looking at past attempts, and the research in transferring data is vast - I think it boils down to a trade-off between expression and usability - and for now JSON seems to have found pretty good sweet spot. Perhaps with just a simple schema layer on top that will be suitable for most use-cases. I imagine Thrift, ProtoBufs, etc will always beat it on speed, XML (+XML Schema) on expressibility, etc. For us, as you have guessed, we are just trying to make it as easy as possible for the majority of users – particularly web and mobile developers. This does also mean we're trying to stay language agnostic and find a type subset that works across most languages - JSON primitives let us get some of the way there. Thank you so much for all your advice - it's been so helpful and we're really happy you took the time out to try it all and record your thoughts. Would love to chat further - please feel free to drop me an email at HNusername @stackhut.com.