6 ms·
Towards a Type System for Containers and AWS Lambda to Avoid Failures [pdf]
- gm-conspiracy 9y agoI am having SOAP-related WSDL flashbacks.
- cmeiklejohn 9y agoWith the exception that these systems wouldn't have identified the type errors we highlight.
- runT1ME 9y agoI would take SOAP/WSDL over barely documented restful APIs and JSON...
- cmeiklejohn 9y agoUnfortunately, popular opinion disagrees and that's part of the motivation for this work.
- cm2187 9y agoWas about to write "...and we shall call it SOAP".
- cmeiklejohn 9y agoMore like "...we call it the simply typed lambda calculus."
- bjz_ 9y agoIt's a bit like all the folks who recoil in horror when you say 'types' and all they can think of is Java.
- cm2187 9y agoI actually don’t think SOAP was that bad. You didn’t want to deal with the API by hand but it was easy to automate the consumption of the API, visual studio was taking care of all of the plumbing for instance and you would consume an API in just a few clicks.
- bjz_ 9y agoSounds pretty nice actually. I'm guessing there could be pain over time though when it came to maintaining it. Easy to add, hard to remove or change?
- cm2187 9y agoDepends what you mean by change. If the API changes, all you had to do is to right click on the reference of the API in visual studio and select update. It would fetch the latest WSDL and recreate the code to consume the API. So painless as long as the API is backward compatible. And on the server side, the WSDL was automatically generated from your code if you used .net, so every time you would update the API the WSDL was up to date. All that created a lot of plumbing and chatty XML traffic behind the scene, so the underlying exchange isn't as neat as a REST API, but as long as you didn't have to deal with it directly it didn't really matter.
- bjz_ 9y agoAh, thanks for the extra info. I'm too young to have dealt with this stuff myself, so first-hand experience is always really interesting! What I have had to deal with is schemaless JSON HTTP APIs - definitely know what a pain that is! GraphQL is a whole bunch nicer from what I hear (but will be interesting to see what folks say in several years time).
- deleted 9y ago[deleted]
- cbrozefsky 9y ago"CORBA, while successful" ....
- cmeiklejohn 9y agoAs much as I disagree with CORBA, as highlighted by my publication history, it's clear in many cases it was very successful in industry in limited applications.
- nickpsecurity 9y agoThere's even an implementation for high-assurance systems like INTEGRITY-178B supplied by Objective Interface: http://www.omg.org/news/meetings/workshops/SBC_2005/SBC_2005_Proceedings/03-2_Buesing.pdf http://www.omg.org/news/meetings/workshops/SBC_2005/SBC_2005... As I suspected, the slides I dug up indicated they built it with some rigor but not high assurance itself. Probably too complex: the very reason I object to CORBA. Like you said, though, CORBA did have successes with this one in very-demanding niche in terms of predictability and security.
- bitcrusher 9y agoWhat're they going to say? "CORBA was a giant shitpile and created tire fires for several years but since this is an academic paper, we have to present historical context for our ideas"...
- protomyth 9y agoI still think the saddest thing I ever saw in the wild was a Smalltalk shop using CORBA. Doing IDL for Smalltalk VM to Smalltalk VM communications is just wrong.
- cmeiklejohn 9y agoCan you extrapolate on this?
- 9y ago
- xinjo 9y agoI like the premise of the paper, but the motivating examples feel really weak to me.
- cmeiklejohn 9y agoCare to extrapolate? We tried hard to take industry use cases.
- timmytwotime 9y agoThose use cases weren't super-specific. IoT sensor data gathering can be accomplished in many, many ways using a Kafka platform. I've done enough operations work to see that this would mean nothing to me (as it stands). I well laid out use case explaining what it means to developers and ops would help your case. Good luck!
- xinjo 9y agokafka/zk bug: I just don’t think this is a compelling example of an underspecified interface being to blame. In my view, it’s squarely on kafka to correctly implement its replication policy, and that likely isn’t something I’d want to bake into an interface layer at the system/network boundary. Also, kafka is a fairly foundational infrastructure-y component from a modern “serverless” perspective, and its interface w/ zk is critical — it just doesn’t exemplify the type of pain you see from underspecified interfaces in large microservice deployments. Unlike between tons of small containers, the kafka/zk interface is worth scrutinizing and warrants a ton of manual testing and verification. lambdas/kafka: from my perspective, this one is kind of conflating the value of Options with the value of a typed IDL with some notion of generics baked in. It’s not clear to me that Option[Number] is what I’d want in this scenario, maybe I just want to reject anything that doesn’t validate to a Float — but I think that’s kind of tangential to whether or not generics in a more conventional sense would be useful. edit: I guess I would have liked to see examples that show how this would help with discoverability, code generation, distributed tracing, monitoring, verifying large systems of microservices. It's a nice position to start with though -- are you planning on a follow up or an implementation?
- stephen 9y agoI'm not sure what the point of this paper is? * It talks a lot about containers, but this is really just contracts across systems, whether they run in containers or not. So not sure why the word "container" is necessary. * It says "towards a type system", but then just muses about IDL/REST/Thrift, says "we need cross-system stuff ... so we should use better types", ...but...what does that look like? There are vague assertions that "we've done this", but I don't see any description of what that actually looks like. * The Zookeeper/Kafka example, while apt, I'm not sure if "having a cross-system type system" is exactly right, as Zookeeper is purposefully/validly ambivalent about what its clients encode within it's file system, so whether the replication data is F/F+1/whatever is meaningless to Zookeeper. So to me the solution is not "cross-system type system" where Zookeeper becomes aware of Kafka's invariants, it's just Kafka itself correctly/internally interpreting the Zookeeper data. If that means it's an Either[ValidCluster,InvalidCluster] within the Kafka "ClusterInfoApi" abstraction, that's fine, but that's not something the Zookeeper API/IDL is going to care/know about. * You're never going to get all networked services to agree on the One True networking format/One True type system/IDL, so why even muse about this as a possible approach? Disclaimer I don't read academic papers on a regular basis.
- jahewson 9y ago“Towards” papers are typically workshop papers which give a preview of ongoing work, usually by PhD students.
- keithwhor 9y agoThis is what we’ve done at StdLib [1] with FaaSlang [2]. FaaSlang uses static analysis and a superset of ESDoc comments to infer types for the API interface to serverless functions on our system. This allows for automatic type coercion from query parameters, automatic error checking, and more - all caked in at the gateway layer before a function is executed. Zero configuration; just write comments the way you normally would. It’s almost a healthy midpoint between TypeScript and JavaScript, operating above the runtime, but theoretically applicable to any language run within a function container. [1] https://stdlib.com https://stdlib.com [2] https://github.com/faaslang/faaslang/ https://github.com/faaslang/faaslang/
- cmeiklejohn 9y agoInteresting. I just went through this a bit and it doesn't appear that the specification allow for polymorphism in the type system. Am I missing something here? If that's true, it doesn't seem to address the issues highlighted by the paper, although does handle simple type checking across interfaces of primitive types.
- cmeiklejohn 9y agoTo be clear, it seems you have basic coercions (a la C, or ad-hoc polymorphism) but no general support for subtype polymorphism nor parametric polymorphism.
- cmeiklejohn 9y agoAlso, I have to sign up for stdlib? Do I have to pay for it? (Never got that far, and the fact I have to sign up is quite misleading for something called stdlib.)
- keithwhor 9y agoYou have to pay for APIs / Functions that the individual vendor charges for (if the functions are not free) - for example, MessageBird charges $0.005 per SMS sent [1] using their sms.create function. You also pay a nominal fee per ms of compute used when somebody uses an API you’ve built. But you can get started for free - $5.00 of credits included that should cover up to 100,000 requests, if you’re really experimenting feel free to message our team (my email is in my profile here, I think). [1] https://stdlib.com/@messagebird/lib/sms https://stdlib.com/@messagebird/lib/sms
- arcticbull 9y agoI've not nearly as much experience developing these systems, so getting started without types was a bit daunting. I wrapped my Lambda functions in protobufs and used a shared common definition repository. Then the lambda services support either the JSON rep of the protos or full-on binary protos, and the type checking happens on both ends. Curious what y'all think of this as a solution.
- cmeiklejohn 9y agoWe're trying to highlight that most of the work you're doing to make sure the interfaces are well defined and match up manually can easily be done by a type checker, as you would have if you wrote this as a single application.
- bjz_ 9y agoHow are you going to handle updates and migrations of types? Something I've been thinking about in terms of dependently typed APIs is versioning and migration - but more at the code ecosystem level than in terms of running systems that can't be turned off. But there is kind of a similarity there. You can think of the ecosystem of written programs as kind of a distributed system. Would be nice to give library authors the tools to more gracefully migrate their consumers code in a type directed way, and to highlight potential problems prior to deployment, but this could equally apply in a distributed computing context.
- geogriffin 9y agoWelcome to Erlang/OTP "releases" where the folklore is that Ericsson engineers spent as much time testing releases (read: state migration code) as they did application code.
- bjz_ 9y agoInteresting. Yeah, from what I understand the Erlang philosophy was to just throw out the idea of a type system and deal with faults dynamically, which is understandable given the time the language was created. But given what we've learned about type systems in the intervening years, it would be super nice to leverage a type system for this, and greatly reduce the testing overhead. I'd love to see this in an event sourcing context too. This paper, “The Dark Side of Event Sourcing: Managing Data Conversion”, seems to hint at there being some interesting algebraic foundations to splitting and merging streams, adding new fields, etc: http://files.movereem.nl/2017saner-eventsourcing.pdf http://files.movereem.nl/2017saner-eventsourcing.pdf
- bjz_ 9y agoI'm really happy more work is being done in this space! Also nice to see nods to various attempts from the past - it's important we learn from their failings, but also their successes. It's too easy to get stuck in the mindset of "Ugh CORBA" and "noes, SOAP", without being able to see an opportunity there. Let's be persistent and figure this stuff out!
- deleted 9y ago[deleted]