5 ms·

The issue I have with API design goes much deeper...if you have a structured database, a statically typed language, and want validation, you're writing 3 separa

The issue I have with API design goes much deeper...if you have a structured database, a statically typed language, and want validation, you're writing 3 separate schemas that all do slightly different but not really different things. Add protobufs onto that, you're now doing four. Maintaining four different schemas does not make refactoring easy, in fact I'd even say it's harder than just using a nosql database and a dynamic language where all you're maintaining is the validation schema. (Yes, I realize there are other benefits to static typing.) Sure, it might generate the code initially depending on what you use, but you still have to maintain it. And any attempt to combine them has been way too complex/verbose. I want to write a lot less code, not more. What I really want is a static language with a validation logic extensive enough that it can be used to generate CRUD APIs, validation, and still offer static typing. I don't know how this would work in practice, and I might hate it after I try it, but I'm tired of maintaining schemas. Just to add to this, things like authorization, login, registration, etc, seem tiresome to write every time for a new API. I've been trying to find something more along the lines of a Backend as a Service, which Parse is very interesting for...but I wish there were more options and easier to use libraries for crafting APIs. It actually feels like we're stuck in the past no matter what we do. I welcome suggestions for easier to write APIs.

kentonv 6y ago
This is exactly why a lot of people choose to use the protobuf-generated data types as their data model internally within their app in addition to for serialization, and many people even derive their database schemas and validation from their protobuf schemas. (When I left Google in 2012, Spanner table schemas were literally defined using protobuf IDL. I haven't followed it since so I have no idea if that's still the case.) Admittedly, protobuf wasn't explicitly designed for these purposes, it just of grew into them because they were convenient, and as a result there are certainly warts. But it's close enough that it works pretty well.
- finder83 6y ago
  Fascinating, I didn't know it could be used for all of that. Last time I tried it it felt more like static typing than true validation (e.g. validating email addresses, date formats, dependent fields, etc) I'll take a look at it again, thanks.
- p1necone 6y ago
  I've had a lot of fun hacking together graph/tree types in protobuf.
- fizx 6y ago
  You can take it all the way into the db with OSS if you like https://github.com/FoundationDB/fdb-record-layer/blob/master/docs/GettingStarted.md https://github.com/FoundationDB/fdb-record-layer/blob/master...
- hedora 6y ago
  The last time I checked, protobuf didn’t help with any sort of validation. In fact, the code I’ve had to maintain (written by others) is more verbose than a hand rolled, validating serializer would be in C++, even though it didn’t actually validate. My only explanation for its success is that people often cargo-cult Google technology, and that it can be used for cross language communication by people that can’t hand-serialize a struct.
  - gravypod 6y ago
    You can make use of features like annotations to magic away the validation of APIs [0]. > people that can’t hand-serialize a struct If you have 1 device type and 1 software version then this might work. If you have multiple CPU architectures and multiple versions of your software this will likely not work. [0] - https://github.com/envoyproxy/protoc-gen-validate https://github.com/envoyproxy/protoc-gen-validate
  - deleted 6y ago
    [deleted]
- gravypod 6y ago
  Very much this. We do that at my current job. We use protos to: 1. Define configuration options for all services/devices 2. Define data models 3. Persist them into our DB: https://github.com/CaperAi/pronto https://github.com/CaperAi/pronto 4. Define our APIs It's extremely convenient to have a single source of truth.
- bvrmn 6y ago
  Sounds like a bad idea. If your storage follows HTTP API 1:1 then why you need backend at all? Take firebase all other db-as-service solutions. It's like developing without public interfaces. Implementation changes will break clients.
kiwicopple 6y ago
In my previous startup I used my Postgres database to generate out JSON Schemas (detailed in [1]). It was great for validations. We even generated some of the UI from the Schemas. > Just to add to this, things like authorization, login, registration, etc, seem tiresome to write every time for a new API Check out Supabase [2], which is similar to Parse (disclosure: I'm a Supabase cofounder). We use Postgres as a backend and we hope to database as the source of truth (using Row Level Security + JSON Schema validation for the front end). [1] https://paul.copplest.one/blog/nimbus-tech-2019-04.html#declarative-data-structures https://paul.copplest.one/blog/nimbus-tech-2019-04.html#decl... [2] https://supabase.io https://supabase.io
- finder83 6y ago
  This looks great, look forward to trying it out. Love the direction I'm seeing though
danielbigham 6y ago
The project I'm working on does this -- we define high level semantic entity types, and from that you can: - Define APIs that take those entities as inputs, automatically validating API arguments for you, automatically generating documentation, and driving API consistency. - Can represent those entities in memory without writing any custom code beyond the schema. - As mentioned above, you don't have to write manual code to validate your entities, that's done for you from the schema. - Automatically generates database tables for those entity types without having to maintain a separate database schema, and can serialize/deserialize entities into or out of the database without having to write SQL. Acts like a document based database but still allows the power of relational SQL matching. - From git diffs, automatically produce expressions that specify how the entity types have evolved over time, and then have the ability to apply those entity type diffs to a production server & database. - Automatically maps natural language onto the schema allowing NLU queries like "orders today in Canada" to be turned into a semantic representation that the database can natively understand and execute, with control to override default NLU mappings if the default mapping isn't working right. - Automatically generates web UIs for viewing and editing entities. - Supports dynamic applicability to be defined so that whether a property is applicable, and which values are valid, can be defined as a dynamic function of other property values. - Supports inferred properties to be defined in either an inference rule approach, or a functional approach.
jayd16 6y ago
> you're writing 3 separate schemas that all do slightly different but not really different things This right here is the siren song of ORMs and many other code generation solutions. They are indeed different, with different concerns. Storage concerns and query concerns are different than your business/domain concepts and those are different than an external API. They're related and it would be nice if they were the same but fundamentally they are not.
ramraj07 6y ago
What are your thoughts on fastapi? We end up having to define schemas only twice, one for sqlalchemy models and another for the version of that model you want to deliver via the api. The experience has been that the code is almost completely just schema definitions with very little boiler plate, and even the slight duplication between sqlalchemy and pydantic actually end up being more useful than not in every case.
- finder83 6y ago
  I haven't given it a fair chance, I haven't done python in a few years at work, moving to typescript/node, Go, and Rust instead. I'll definitely check it out though. I was looking at it the other day and it reminded me a lot of flask or to a lesser degree Django with django rest framework, but I didn't make it far enough to understand what was going on with serialization. I actually just learned about fastapi two days ago, I guess I'm a lot more disconnected from python than I thought I was.
tylergetsay 6y ago
TypeScript + class-validators + reflect-metadata has let me build a GraphQL API w/ only one of these schemas. It's very intuitive/fast to develop for me.
Shorel 6y ago
A decade ago I wrote some Lisp code that generated C++ code with something similar. It was completely inadequate, I would use SQLite now for the same. But it was very fun to write.

kentonv 6y ago

This is exactly why a lot of people choose to use the protobuf-generated data types as their data model internally within their app in addition to for serialization, and many people even derive their database schemas and validation from their protobuf schemas. (When I left Google in 2012, Spanner table schemas were literally defined using protobuf IDL. I haven't followed it since so I have no idea if that's still the case.) Admittedly, protobuf wasn't explicitly designed for these purposes, it just of grew into them because they were convenient, and as a result there are certainly warts. But it's close enough that it works pretty well.

finder83 6y ago
Fascinating, I didn't know it could be used for all of that. Last time I tried it it felt more like static typing than true validation (e.g. validating email addresses, date formats, dependent fields, etc) I'll take a look at it again, thanks.
p1necone 6y ago
I've had a lot of fun hacking together graph/tree types in protobuf.
fizx 6y ago
You can take it all the way into the db with OSS if you like https://github.com/FoundationDB/fdb-record-layer/blob/master/docs/GettingStarted.md https://github.com/FoundationDB/fdb-record-layer/blob/master...
hedora 6y ago
The last time I checked, protobuf didn’t help with any sort of validation. In fact, the code I’ve had to maintain (written by others) is more verbose than a hand rolled, validating serializer would be in C++, even though it didn’t actually validate. My only explanation for its success is that people often cargo-cult Google technology, and that it can be used for cross language communication by people that can’t hand-serialize a struct.
- gravypod 6y ago
  You can make use of features like annotations to magic away the validation of APIs [0]. > people that can’t hand-serialize a struct If you have 1 device type and 1 software version then this might work. If you have multiple CPU architectures and multiple versions of your software this will likely not work. [0] - https://github.com/envoyproxy/protoc-gen-validate https://github.com/envoyproxy/protoc-gen-validate
- deleted 6y ago
  [deleted]
gravypod 6y ago
Very much this. We do that at my current job. We use protos to: 1. Define configuration options for all services/devices 2. Define data models 3. Persist them into our DB: https://github.com/CaperAi/pronto https://github.com/CaperAi/pronto 4. Define our APIs It's extremely convenient to have a single source of truth.
bvrmn 6y ago
Sounds like a bad idea. If your storage follows HTTP API 1:1 then why you need backend at all? Take firebase all other db-as-service solutions. It's like developing without public interfaces. Implementation changes will break clients.

kiwicopple 6y ago

In my previous startup I used my Postgres database to generate out JSON Schemas (detailed in [1]). It was great for validations. We even generated some of the UI from the Schemas. > Just to add to this, things like authorization, login, registration, etc, seem tiresome to write every time for a new API Check out Supabase [2], which is similar to Parse (disclosure: I'm a Supabase cofounder). We use Postgres as a backend and we hope to database as the source of truth (using Row Level Security + JSON Schema validation for the front end). [1] https://paul.copplest.one/blog/nimbus-tech-2019-04.html#declarative-data-structures https://paul.copplest.one/blog/nimbus-tech-2019-04.html#decl... [2] https://supabase.io https://supabase.io

finder83 6y ago
This looks great, look forward to trying it out. Love the direction I'm seeing though

danielbigham 6y ago

The project I'm working on does this -- we define high level semantic entity types, and from that you can: - Define APIs that take those entities as inputs, automatically validating API arguments for you, automatically generating documentation, and driving API consistency. - Can represent those entities in memory without writing any custom code beyond the schema. - As mentioned above, you don't have to write manual code to validate your entities, that's done for you from the schema. - Automatically generates database tables for those entity types without having to maintain a separate database schema, and can serialize/deserialize entities into or out of the database without having to write SQL. Acts like a document based database but still allows the power of relational SQL matching. - From git diffs, automatically produce expressions that specify how the entity types have evolved over time, and then have the ability to apply those entity type diffs to a production server & database. - Automatically maps natural language onto the schema allowing NLU queries like "orders today in Canada" to be turned into a semantic representation that the database can natively understand and execute, with control to override default NLU mappings if the default mapping isn't working right. - Automatically generates web UIs for viewing and editing entities. - Supports dynamic applicability to be defined so that whether a property is applicable, and which values are valid, can be defined as a dynamic function of other property values. - Supports inferred properties to be defined in either an inference rule approach, or a functional approach.

jayd16 6y ago

> you're writing 3 separate schemas that all do slightly different but not really different things This right here is the siren song of ORMs and many other code generation solutions. They are indeed different, with different concerns. Storage concerns and query concerns are different than your business/domain concepts and those are different than an external API. They're related and it would be nice if they were the same but fundamentally they are not.

ramraj07 6y ago

What are your thoughts on fastapi? We end up having to define schemas only twice, one for sqlalchemy models and another for the version of that model you want to deliver via the api. The experience has been that the code is almost completely just schema definitions with very little boiler plate, and even the slight duplication between sqlalchemy and pydantic actually end up being more useful than not in every case.

finder83 6y ago
I haven't given it a fair chance, I haven't done python in a few years at work, moving to typescript/node, Go, and Rust instead. I'll definitely check it out though. I was looking at it the other day and it reminded me a lot of flask or to a lesser degree Django with django rest framework, but I didn't make it far enough to understand what was going on with serialization. I actually just learned about fastapi two days ago, I guess I'm a lot more disconnected from python than I thought I was.

tylergetsay 6y ago

TypeScript + class-validators + reflect-metadata has let me build a GraphQL API w/ only one of these schemas. It's very intuitive/fast to develop for me.

Shorel 6y ago

A decade ago I wrote some Lisp code that generated C++ code with something similar. It was completely inadequate, I would use SQLite now for the same. But it was very fun to write.