6 ms·
Hyrumtoken: A Go package to encrypt pagination tokens
- njtransit 2y agoThis looks like a very simple wrapper around golang.org/x/crypto/nacl/secretbox What’s the point of this?
- maxbond 2y agoTo remove/obscure structure from a token so that the structure is not relied upon & can be changed in backwards incompatible ways without disrupting API consumers. If you think it's internal details are too simple to justify a dependency, you can vendor or reimplement it, but that's orthogonal to whether it's pointless. The README is pretty detailed & explicit about what the point is.
- njtransit 2y agoBut nothing about the API is specific to pagination. This library essentially is just two other API calls: marshal and seal. It can do this operation on any marshal-able type. By using this library, you lose control over marshaling, which seems like a high cost to pay for this very simple and basic functionality.
- maxbond 2y agoWould the library be better if it were more restrictive or more complicated?
- njtransit 2y agoNo, the library would be better if it provided some utility. As it stands, it provides negative utility.
- maxbond 2y agoYou're conflating how it works internally with whether it's useful. You've critiqued is internal details but you haven't engaged with the premise of why you might want to use it. A coin is just a metal disk. A dollar bill is just a piece of paper. Neither of them do anything. After adopting them, you lose control of how your cash is represented. And yet we find them useful. A simple implementation is a virtue, not an albatross.
- njtransit 2y agoNo, I'm sorry, you are not correct. I am not conflating internal implementation with whether or not it is useful. Rather, I am evaluating the opportunity cost, i.e. comparing this library to the "next best thing." For this library, the next best thing would be to make two simple API calls instead of one simple API call. As a cost, this is very low. However, the "next best thing" also has a number of desirable properties compared to this library: better support for custom serialization and a lower attack surface for supply-chain attacks. When you look at the costs vs. the benefits of this library, the utility is negative. Using your example of currency: a paper bill is not just a piece of paper. It's a piece of paper coupled with the vast machinations of a nation state that can enforce its currency via its monopoly on violence. You can't get all the benefits of a $100 bill just by having a green piece of paper.
- maxbond 2y agoIt was presumptive of me to tell you what you were thinking, and I apologize.
- tommiegannert 2y agoHuh. Anyone know why the nonce isn't baked into the box upon sealing? It's the same in the original: https://nacl.cr.yp.to/secretbox.html https://nacl.cr.yp.to/secretbox.html
- compressedgas 2y agoThe operation doesn't dictate how the nonce is to be conveyed to the recipient.
- tommiegannert 2y agoYes, but since you never want to reuse the nonce (at least not with the same key; and no one stores nonces for later use), they are 1:1 to the message, suggesting it would have been less error-prone to encode it in the box. I had the impression NaCL was about being highly opinionated, so this choice surprised me.
- EdSchouten 2y agoRelatedly, what's the advantage of that secretbox package over calling https://pkg.go.dev/crypto/cipher#NewGCM https://pkg.go.dev/crypto/cipher#NewGCM ?
- yencabulator 2y agoNaCl predates GCM by some 15 months. But that's an old concern, both are pretty old by now. NaCl tends to be faster on hardware without AES acceleration. Go's AES/GCM implementation is not constant time unless the hardware has AES acceleration.
- unsnap_biceps 2y agoLooks pretty cool, however, I'm curious about the usage of a pointer to the key slice. A slice is just two ints and a pointer, so the copy semantics is pretty cheap, and one less pointer for the GC to clean up later. Is there some reason why you choose to pass a pointer to the slice?
- silisili 2y agoYou're correct about slice semantics but key here is actually a Go array, not a slice. So it's passing a pointer instead of an entire array.
- unsnap_biceps 2y agoAhh, thank you. Still very new to go :)
- Xxfireman 2y ago“intellectual curiosity of your coworkers demands they base64-parse it.” This is crazy behavior. Creating your own pagination key, assuming it exists, and then putting that in production certainly proves “Hyrum’s law”.
- Arnavion 2y agoI have a similar story to OP's. I had made a service that provided access to cryptographic keys but did not reveal the key material directly. Instead it had an RPC API for requesting a key "handle" for the key you wanted to use, and API for performing operations like encrypt or sign that took that key handle, performed the operation inside the service and returned the result. The key handle was to be treated as opaque and implemented as a base64-encoded blob containing the key ID and a signature (for tamper-proofing). One day a coworker working on another project that would use my service contacted me to complain that the keys from my service were malformed. Turned out they had noticed that the return value was base64-encoded so they assumed it was a base64-encoded key, so they wrote code to base64-decode it and load the result into their language's crypto library to perform those operations directly. They figured that the service's API for doing those operations was just there to be convenient for callers that didn't have access to a crypto library.
- mst 2y agoThe author does make a point of giving an example of him perpetrating something equivalent wrt somebody else's API. In theory, yes, it's kinda crazy behaviour. In practice I suspect most of us have done something (im)morally equivalent at least once.
- taeric 2y agoPagination tokens have been a bane to my existence. I'm basically this backend engineer. Have done these exact steps. Between worry the frontend will decode and annoyance at explaining how tokens work, I am growing to hate working with frontend teams. Even worse, when I have to explain that they may get an empty result with a token meaning they need to call again. Unbounded service call means nothing, it seems. Kudos on this project!
- tossandthrow 2y agoI have been in this situation also. The biggest issue, that took several meetings, was getting the ux team to understand that they can not index the pages as we do not know their tokens.
- taeric 2y agoIf you have found good links to explain this to other teams, I'd love to see them.
- yencabulator 2y agohttp://use-the-index-luke.com/no-offset http://use-the-index-luke.com/no-offset
- xg15 2y agoMaybe the deeper message of this is: Pagination tokens are a shitty API and you should really offer something that gives the client more options. Pagination tokens feel great if you're providing the API, because they let you retain maximum control over API usage and let you do all kinds of optimizations in the back-end. But they are really limiting for the clients and - as we've seen here - often fail to address even basic use cases, such as going back a page or even reliably reloading the same page you're on. (Not even starting about arbitrary seeking.) This leads to dreaded infinite-scrolling or "click to load more" UIs, which overload my browser and will have me start again from the very beginning should I have to reload the page. I think it's also very easy to miss for non-technical stakeholders just how constraining this API is. E.g. in the OP's story, I could easily imagine some nontechnical PM assigning tasks like this: - back-end team: Implement data access API using pagination tokens (because the devs said this is the best way to do it) - front-end team: Implement the data view UI, including "next", " previous" and "go to page" buttons, linkability and page numbers in the URL. (Because that's what the product owners ordered) Without having some specific knowledge about how pagination tokens work, it's easy to miss that those requirements are contradictory - unless someone cheats, like in this case the front-end.
- alserio 2y agowhat's the solution space for these problems?
- mhuffman 2y agoIf the UI got you to a page that started with a search of some type, then you can look in your own experiences of how many times have you went to the second page on Amazon, Ebay, or Google search? Probably not often. So the answer is some combination of better search, related recommendations from the items on the remaining pages, and filtering.
- alserio 2y agoI don't agree. As an example a shop and a dashboard are different things. Sometimes you are not looking for an item, but for a pattern, an outlier, or something else. And there are many use cases for a list of something.
- rendall 2y agoI don't really agree with the entire premise. At first I read about this trying to understand what the security implications are of obscured pagination tokens. Why would one want to obfuscate pagination tokens from hackers? Turns out, it's to make things more opaque for one's own team ::scream emoji:: This is simply not a good way to work, sorry. Either implement paging the way the frontend team wants and expects it to work, or take the time to explain in a clear and friendly why you cannot (i.e. the data is not structured as pages, specs were clear and rigid, whatever). With healthy interteam communication, there won't be a need to obfuscate pagination tokens in this manner. It reads like OP built something as a backend engineer without reference to client or user needs and then threw it over the wall to the frontend team. Not good.
- mst 2y ago> or take the time to explain in a clear and friendly why you cannot The next person to work on the frontend paging code will likely look at the existing code and infer the contract from there. Doesn't matter how healthy your inteream communication is, humans just straight up don't work like that. The documentation already says you have to use the nextToken from the current request - adding more words to the effect of "no, really, you do *have* to use the nextToken from the current request" is sadly unlikely to help in practice. It isn't about making things more opaque for one's own team, it's about stopping people *accidentally* doing something unsupported, which is if anything a kindness to your colleagues since they won't end up with an apparently working feature that will break unexpectedly later. OP is aiming minimise unpredicted future pain for his team, his colleagues in the front-end team, and their users. Your point of view would be absolutely correct in a world where "everybody would just" ... but humans, as a species, don't "just" - so code accordingly.
- rendall 2y ago> The documentation already says you have to use the nextToken from the current request - adding more words to the effect of "no, really, you do have* to use the nextToken from the current request" is sadly unlikely to help in practice.* Agreed. That is not a good fix, either. Unfortunately, if I understood the situation, there was a disconnect between what the frontend team needed or expected (pagesize, offset), and what the backend engineer could or would deliver (next page). Rather than address this primary issue (lack of understanding), OP adds a technical fix to address what is fundamentally a human, "soft skills" problem. It is almost certainly not OP's fault, but a lack of managerial guidance of team dynamics. In my experience, separating teams into frontend and backend silos does not lead to good outcomes. Each team begins to see the other as a clueless adversary. But even if necessary, there must be strong efforts to unify their incentives and understanding. So, I disagree that this fix is going to fix what's wrong.
- nesarkvechnep 2y agoI really wish HATEOAS clients were the norm instead of concatenating strings like cavemen in order to obtain a URL. You could send the prev, next, first, and last relations of your collection resource. You could also send a templated link so the clients could still jump to whatever page they want.
- xg15 2y agoThe thing that always puzzled me about HATEOAS was that it basically treats the computer like a human - like yes, if I'm manually exploring an API via curl, the "relation" links can be immensely helpful (sometimes - often, the bare relation names can't be understood on their own). But if I'm having some script or app consume the API, then this script will have to "know in advance" how the API works anyway, and the "self-discoverability" of HATEOAS doesn't bring any benefit. (There is the idea, that we could at some point have a rich set of standardized relation types and a class of generic clients that could make use of them. But as with the Semantic Web, the incentives seem to be misaligned to make that dream a reality) But interestingly, with LLMs, the picture could change - because LLMs could actually explore the API "like a human" and even read plain-english apidocs of nonstandard relation types. So maybe some sort of LLM-augmented generic client could actually fulfill the role of the elusive "autonomous agent" that HATEOAS and Semantic Web people are designing for.
- Joker_vD 2y ago> then this script will have to "know in advance" how the API works anyway, and the "self-discoverability" of HATEOAS doesn't bring any benefit. Tentatively, it allows better flexibility in non-entrypoint URIs, by adding a layer of indirection: instead of hardcoding URLs itself, you hardcode the name of the fields in the response that will contain actual URIs to follow. But then there is a whole crowd of people who claim that URIs should never change anyhow, for any reason, in the first place.
- xg15 2y agoThat makes sense, but then it also trusts that every client will "play by the rules" - first do the "discovery" request, then use the discovered URL to do the real request - when they could just "cheat" and hardcode the URL. I think, again the incentives are misaligned, because hardcoding would be beneficial for the client devs in several ways: It's simpler to code, halves the number of requests at runtime and gets rid of several edge cases that you'd have to deal with - e.g. what happens if the discovered URL points to a different domain than the entrypoint? It makes sense from the server's POV, but because of this, I see a hard time to convince client devs to adopt it.
- crest 2y agoOr give your users a proper streaming API for large or partial reponses?
- nzach 2y agoIs this really a problem we need to solve with technology? I've never seen this happen in the real world. But by the motivation presented in the repo as well as some commentaries in this thread I got the impression this is a miscommunication issue. I my experience most mid-level engineers are fully aware about the differences between "pagination token" and "page/size" based pagination strategies. So when this kind of situation happens I get the impression that both teams were never in the same page. Maybe someone wrote "the api needs to have pagination" in a design doc and nobody bothered to ask for the details. To make things worse this problems only appear when we start to make some e2e tests, but by that point the deadline is generally pretty close. And this creates a big incentive for people to get creative in order to deliver on time.
- numbsafari 2y agoWhen you are presenting a public API… you are almost by definition never in the same room with the folks consuming your API. More to the point, you probably don’t have an agreed upon contract, beyond your terms of service page and internal SLOs.