3 ms·

The "v3/application/shops/{shop_id}/listings/{listing_id}/properties" structure is typically important for database partitioning when using e.g. DynamoDB. You h

by kennu 3y ago

The "v3/application/shops/{shop_id}/listings/{listing_id}/properties" structure is typically important for database partitioning when using e.g. DynamoDB. You have to include both the partition key and the sort key in the URL to access the database item. Otherwise the database doesn't scale out as intended. I also disagree with the advice to always use Arrays instead of Map objects. It is very difficult to partially update Arrays in an idempotent way. When you use a Map object, you can update individual items in the object by their keys. That is why I think you should avoid Arrays in API data structures as much as possible.

pringlecorn 3y ago
Regarding your point about the DB partitioning, isn't it the backend's job to figure out the shop ID based on the listing ID? I don't really see that as a valid argument
3cats-in-a-coat 3y ago
Why would you update your input. It's just input, you can locally have maps. Say for example when you read SQL results, it's a list of results, not a map of results. Then you make a map of whatever key you need locally. Or even several of them. let myCustomIndex = listOfResults.reduce((i, r) => { i[r.pk] = r; return i; }, {}); In general we call serialization "serialization" because we put things in serial order and send that sequence over the wire. You can't send maps over the wire. A map is a structure in memory optimized for direct access and modification. While JSON maps (objects) are just sequences of keys and vals, encoded as text, not an actual hashmap or a b-tree, as we surely understand. So you may as well save the redundancy and send a list of results, then format and index it however you please locally. The fact JSON has maps (objects) at all is a lot less useful than people realize. It's mostly useful for the purpose of denoting "that's a key" and "that's a value". But actually all the work is done after the JSON is being read. You can just as easily send a "map" like this: ["key1", "val1", "key2", "val2", ...]
- kennu 3y ago
  My point about arrays vs maps is based on the idea that you store the data in e.g. DynamoDB as the same structure as it is represented in the API. If you store an array in an attribute of the database item, you cannot update its contents in an idempotent way. Inserting or appending to the array multiple times (using the relevant DynamoDB functions) causes it to grow more and more each time. Whereas when you use a map object, updating a specific key in the object is an idempotent operation and has no effect when repeated. I realise this is pretty DynamoDB specific. But there are also other points to be made against arrays, such as being forced to maintain their order in any kind of database, which can be quite bad for performance. When using a key value map object, there is no guarantee of the order of the items and users of the API will reflect this in how they use the API, relying on the object keys instead of array indices or array order.
  - 3cats-in-a-coat 3y ago
    OK but transfer format and working representation shouldn't be the same. That is a bad goal to have. A working representation is sparse, indexed, you can jump to places and change parts. While input (and output) are streams of dense data. Adding redundancy to input/output so you don't have to make local decisions for your working representation may seem like a simplification, but you're basically chaining yourself from doing what you need to do in order to do work effectively, and burdening input/output with concerns that don't matter on the wire. We keep seeing this idea come back again and again where "you don't need services" or "you don't need controllers" or "you don't need mapping", so just, you know, grab the database and hose it out over HTTP and into clients as-is. But this always is one of those "immediate gratification" choices that ends up biting you in the ass not long after. It looks great in slides and demos though.
    - kennu 3y ago
      To me it's a good goal. I want to keep things simple and minimize my work. It works very well with a document database like DynamoDB. Even if often there is a small translation going on, removing a few attributes or adding a few to achieve backward compatibility, but mostly passing them 1:1.

by kennu 3y ago

pringlecorn 3y ago

Regarding your point about the DB partitioning, isn't it the backend's job to figure out the shop ID based on the listing ID? I don't really see that as a valid argument

3cats-in-a-coat 3y ago

Why would you update your input. It's just input, you can locally have maps. Say for example when you read SQL results, it's a list of results, not a map of results. Then you make a map of whatever key you need locally. Or even several of them. let myCustomIndex = listOfResults.reduce((i, r) => { i[r.pk] = r; return i; }, {}); In general we call serialization "serialization" because we put things in serial order and send that sequence over the wire. You can't send maps over the wire. A map is a structure in memory optimized for direct access and modification. While JSON maps (objects) are just sequences of keys and vals, encoded as text, not an actual hashmap or a b-tree, as we surely understand. So you may as well save the redundancy and send a list of results, then format and index it however you please locally. The fact JSON has maps (objects) at all is a lot less useful than people realize. It's mostly useful for the purpose of denoting "that's a key" and "that's a value". But actually all the work is done after the JSON is being read. You can just as easily send a "map" like this: ["key1", "val1", "key2", "val2", ...]

kennu 3y ago
My point about arrays vs maps is based on the idea that you store the data in e.g. DynamoDB as the same structure as it is represented in the API. If you store an array in an attribute of the database item, you cannot update its contents in an idempotent way. Inserting or appending to the array multiple times (using the relevant DynamoDB functions) causes it to grow more and more each time. Whereas when you use a map object, updating a specific key in the object is an idempotent operation and has no effect when repeated. I realise this is pretty DynamoDB specific. But there are also other points to be made against arrays, such as being forced to maintain their order in any kind of database, which can be quite bad for performance. When using a key value map object, there is no guarantee of the order of the items and users of the API will reflect this in how they use the API, relying on the object keys instead of array indices or array order.
- 3cats-in-a-coat 3y ago
  OK but transfer format and working representation shouldn't be the same. That is a bad goal to have. A working representation is sparse, indexed, you can jump to places and change parts. While input (and output) are streams of dense data. Adding redundancy to input/output so you don't have to make local decisions for your working representation may seem like a simplification, but you're basically chaining yourself from doing what you need to do in order to do work effectively, and burdening input/output with concerns that don't matter on the wire. We keep seeing this idea come back again and again where "you don't need services" or "you don't need controllers" or "you don't need mapping", so just, you know, grab the database and hose it out over HTTP and into clients as-is. But this always is one of those "immediate gratification" choices that ends up biting you in the ass not long after. It looks great in slides and demos though.
  - kennu 3y ago
    To me it's a good goal. I want to keep things simple and minimize my work. It works very well with a document database like DynamoDB. Even if often there is a small translation going on, removing a few attributes or adding a few to achieve backward compatibility, but mostly passing them 1:1.