4 ms·

> GitHub's migration guide tells developers to treat the new IDs as opaque strings and treat them as references. However it was clear that there was some underl

by agwa 9mo ago

> GitHub's migration guide tells developers to treat the new IDs as opaque strings and treat them as references. However it was clear that there was some underlying structure to these IDs as we just saw with the bitmasking Great, so now GitHub can't change the structure of their IDs without breaking this person's code. The lesson is that if you're designing an API and want an ID to be opaque you have to literally encrypt it. I find it really demoralizing as an API designer that I have to treat my API's consumers as adversaries who will knowingly and intentionally ignore guidance in the documentation like this.

maxbond 9mo ago
You could also say, if I tell you something is an opaque identifier, and you introspect it, it's your problem if your code breaks. I told you not to do that.
- lelandfe 9mo ago
  Once "you" becomes a big enough "them" it becomes a problem again.
  - vlovich123 9mo ago
    Exactly. When you owe the bank $10M it’s a you problem. When you owe the bank $100B it’s a them problem.
haileys 9mo ago
This is well understood - Hyrum's law. You don't need encryption, a global_id database column with a randomly generated ID will do.
- maxbond 9mo ago
  You could but you would lose the performance benefits you were seeking by encoding information into the ID. But you could also use a randomized, proprietary base64 alphabet rather than properly encrypting the ID.
  - haileys 9mo ago
    Encoding a type name into an ID is never really something I've viewed as being about performance. Think of it more like an area code, it's an essential part of the identifier that tells you how to interpret the rest of it.
    - maxbond 9mo ago
      That's fair, and you could definitely put a prefix and a UUID (or whatever), I failed to consider that.
  - pdpi 9mo ago
    XOR encryption is cheap and effective. Make the key the static string "IfYouCanReadThisYourCodeWillBreak" or something akin to that. That way, the key itself will serve as a final warning when (not if) the key gets cracked.
    - Retr0id 9mo ago
      Any symmetric encryption is ~free compared to the cost of a network request or db query. In this particular instance, Speck would be ideal since it supports a 96-bit block size https://en.wikipedia.org/wiki/Speck_(cipher) https://en.wikipedia.org/wiki/Speck_(cipher)
      - pdpi 9mo ago
        Symmetric encryption is computationally ~free, but most of them are conceptually complex. The purpose of encryption here isn't security, it's obfuscation in the service of dissuading people from depending on something they shouldn't, so using the absolutely simplest thing that could possibly work is a positive.
        Retr0id 9mo ago
        XOR with fixed key is trivially figure-out-able, defeating the purpose. Speck is simple enough that a working implementation is included within the wikipedia article, and most LLMs can oneshot it.
nwallin 9mo ago
Hyrum's law is a real sonuvabitch.
krisoft 9mo ago
> Great, so now GitHub can't change the structure of their IDs without breaking this person's code. And that is all the fault of the person who treated a documented opaque value as if it has some specific structure. > The lesson is that if you're designing an API and want an ID to be opaque you have to literally encrypt it. The lesson is that you should stop caring about breaking people’s code who go against the documentation this way. When it breaks you shrug. Their code was always buggy and it just happened to be working for them until then. You are not their dad. You are not responsible for their misfortune. > I find it really demoralizing as an API designer that I have to treat my API's consumers as adversaries who will knowingly and intentionally ignore guidance in the documentation like this. You don’t have to.
- vlovich123 9mo ago
  Sounds like you’ve maybe never actually run a service or API library at scale. There’s so many factors that go into a decision like that at a company that it’s never so simple. Is the person impacted influential? You’ve got a reputation hit if they negatively blog about how you screwed them after something was working for years. Is a customer who’s worth 10% of your annual revenue impacted? Bet your ass your management chain won’t let you do a breaking change / revert any you made by declaring an incident. Even in OSS land, you risk alienating the community you’ve built if they’re meaningfully impact. You only do this if the impact is minimal or you don’t care about alienating anyone using your software.
  - irjustin 9mo ago
    > Sounds like you’ve maybe never actually run a service or API library at scale. What was the saying? When your scale is big enough, even your bugs have users.
    - raincole 9mo ago
      Yeah, but when you are big enough you can afford to not care individual users. VScode once broke a very popular extension that used a private API. Microsoft (righteously) didn't bother to ask if the private API had users.
perfmode 9mo ago
The API contract doesn’t stipulate the behavior so GitHub is free to change as they please.
bigblind 9mo ago
I think more important than worrying about people treating an opaque value as structured data, is wondering _why_ they're doing so. In the case of this blog post, all they wanted to do was construct a URL, which required the integer database ID. Just make sure you expose what people need, so they don't need to go digging. Other than that, I agree with what others are saying. If people rely on some undocumented aspect of your IDs, it's on them if that breaks.
- plorkyeran 9mo ago
  Exposing what people need doesn’t guarantee that they won’t go digging. It is surprisingly common to discover that someone has come up with a hack that depends on implementation details to do something which you exposed directly and they just didn’t know about it.
- Macha 9mo ago
  GitHub actually do have both the database ID and URL available in the API: https://docs.github.com/en/graphql/reference/objects#pullrequest https://docs.github.com/en/graphql/reference/objects#pullreq... OP’s requirements changed and they hadn’t stored them during their crawl
lijok 9mo ago
Can GitHub change their API response rate? Can they increase it? If they do, they’ll break my code ‘cause it expects to receive responses at least after 1200ms. Any faster than that and I get race conditions. I selected the 1200ms number by measuring response rates. No, you would call me a moron and tell me to go pound sand. Weird systems were never supported to begin with.
kevin_thibedeau 9mo ago
> Great, so now GitHub can't change the structure of their IDs without breaking this person's code OP can put the decoded IDs into a new column and ignore the structure in the future. The problem was presumably mass querying the Github API to get those numbers needed for functional URLs.
cush 9mo ago
At a big enough scale, even your bugs have users
vlovich123 9mo ago
Literally how I designed all the public facing R2 tokens like multipart uploads. It’s also a security barrier because forging and stealing of said tokens is harder and any vulnerability has to be done with cooperation of your servers and can be quickly shut down if needed.
whateveracct 9mo ago
Who cares if their code is broken in this case? Stupid games stupid prizes.

by agwa 9mo ago

maxbond 9mo ago

You could also say, if I tell you something is an opaque identifier, and you introspect it, it's your problem if your code breaks. I told you not to do that.

lelandfe 9mo ago
Once "you" becomes a big enough "them" it becomes a problem again.
- vlovich123 9mo ago
  Exactly. When you owe the bank $10M it’s a you problem. When you owe the bank $100B it’s a them problem.

haileys 9mo ago

This is well understood - Hyrum's law. You don't need encryption, a global_id database column with a randomly generated ID will do.

maxbond 9mo ago
You could but you would lose the performance benefits you were seeking by encoding information into the ID. But you could also use a randomized, proprietary base64 alphabet rather than properly encrypting the ID.
- haileys 9mo ago
  Encoding a type name into an ID is never really something I've viewed as being about performance. Think of it more like an area code, it's an essential part of the identifier that tells you how to interpret the rest of it.
  - maxbond 9mo ago
    That's fair, and you could definitely put a prefix and a UUID (or whatever), I failed to consider that.
- pdpi 9mo ago
  XOR encryption is cheap and effective. Make the key the static string "IfYouCanReadThisYourCodeWillBreak" or something akin to that. That way, the key itself will serve as a final warning when (not if) the key gets cracked.
  - Retr0id 9mo ago
    Any symmetric encryption is ~free compared to the cost of a network request or db query. In this particular instance, Speck would be ideal since it supports a 96-bit block size https://en.wikipedia.org/wiki/Speck_(cipher) https://en.wikipedia.org/wiki/Speck_(cipher)
    - pdpi 9mo ago
      Symmetric encryption is computationally ~free, but most of them are conceptually complex. The purpose of encryption here isn't security, it's obfuscation in the service of dissuading people from depending on something they shouldn't, so using the absolutely simplest thing that could possibly work is a positive.
      - Retr0id 9mo ago
        XOR with fixed key is trivially figure-out-able, defeating the purpose. Speck is simple enough that a working implementation is included within the wikipedia article, and most LLMs can oneshot it.

nwallin 9mo ago

Hyrum's law is a real sonuvabitch.

krisoft 9mo ago

> Great, so now GitHub can't change the structure of their IDs without breaking this person's code. And that is all the fault of the person who treated a documented opaque value as if it has some specific structure. > The lesson is that if you're designing an API and want an ID to be opaque you have to literally encrypt it. The lesson is that you should stop caring about breaking people’s code who go against the documentation this way. When it breaks you shrug. Their code was always buggy and it just happened to be working for them until then. You are not their dad. You are not responsible for their misfortune. > I find it really demoralizing as an API designer that I have to treat my API's consumers as adversaries who will knowingly and intentionally ignore guidance in the documentation like this. You don’t have to.

vlovich123 9mo ago
Sounds like you’ve maybe never actually run a service or API library at scale. There’s so many factors that go into a decision like that at a company that it’s never so simple. Is the person impacted influential? You’ve got a reputation hit if they negatively blog about how you screwed them after something was working for years. Is a customer who’s worth 10% of your annual revenue impacted? Bet your ass your management chain won’t let you do a breaking change / revert any you made by declaring an incident. Even in OSS land, you risk alienating the community you’ve built if they’re meaningfully impact. You only do this if the impact is minimal or you don’t care about alienating anyone using your software.
- irjustin 9mo ago
  > Sounds like you’ve maybe never actually run a service or API library at scale. What was the saying? When your scale is big enough, even your bugs have users.
  - raincole 9mo ago
    Yeah, but when you are big enough you can afford to not care individual users. VScode once broke a very popular extension that used a private API. Microsoft (righteously) didn't bother to ask if the private API had users.

perfmode 9mo ago

The API contract doesn’t stipulate the behavior so GitHub is free to change as they please.

bigblind 9mo ago

I think more important than worrying about people treating an opaque value as structured data, is wondering _why_ they're doing so. In the case of this blog post, all they wanted to do was construct a URL, which required the integer database ID. Just make sure you expose what people need, so they don't need to go digging. Other than that, I agree with what others are saying. If people rely on some undocumented aspect of your IDs, it's on them if that breaks.

plorkyeran 9mo ago
Exposing what people need doesn’t guarantee that they won’t go digging. It is surprisingly common to discover that someone has come up with a hack that depends on implementation details to do something which you exposed directly and they just didn’t know about it.
Macha 9mo ago
GitHub actually do have both the database ID and URL available in the API: https://docs.github.com/en/graphql/reference/objects#pullrequest https://docs.github.com/en/graphql/reference/objects#pullreq... OP’s requirements changed and they hadn’t stored them during their crawl

lijok 9mo ago

Can GitHub change their API response rate? Can they increase it? If they do, they’ll break my code ‘cause it expects to receive responses at least after 1200ms. Any faster than that and I get race conditions. I selected the 1200ms number by measuring response rates. No, you would call me a moron and tell me to go pound sand. Weird systems were never supported to begin with.

kevin_thibedeau 9mo ago

> Great, so now GitHub can't change the structure of their IDs without breaking this person's code OP can put the decoded IDs into a new column and ignore the structure in the future. The problem was presumably mass querying the Github API to get those numbers needed for functional URLs.

cush 9mo ago

At a big enough scale, even your bugs have users

vlovich123 9mo ago

Literally how I designed all the public facing R2 tokens like multipart uploads. It’s also a security barrier because forging and stealing of said tokens is harder and any vulnerability has to be done with cooperation of your servers and can be quickly shut down if needed.

whateveracct 9mo ago

Who cares if their code is broken in this case? Stupid games stupid prizes.