14 ms·

Things you should do now (2011)

199 points by bshanks 6y ago

bryanrasmussen 6y ago
That start ids at a gigantic number idea is great, even though I've never encountered a bug caused by not doing it.
- joss82 6y ago
  Indeed, that and sql syntax errors are solved by using an orm or sql code generator like sql alchemy core.
- boatmansloth 6y ago
  I literally came here to comment that the very first bug I had to chase down in a production system in my first job was caused by this! There was a 'fuzzy' search field available to users and when one input a client's ID number (formatted as NNNNNNN) it was showing results for a different client because NNNNNNN was also the format of the unique ID for records in the database.
- tpetry 6y ago
  A better tip would be to use something like time-ordered uuids: You can‘t misuse them for something different and the added bonus is that no one can iterate your db records by just incrementing the url.
  - Cthulhu_ 6y ago
    My main gripe with UUIDs at the moment is that they look ugly in urls :/
    - pretzelhands 6y ago
      They're not the prettiest sight, yes, but I think anyone who isn't in some way a geek particularly cares about what's actually in URLs. There's a reason why Apple got away with only showing the hostname of a URL: It doesn't matter to ordinary people. :)
    - krab 6y ago
      You can encode it to a shorter string if you use larger alphabet. Still, it's 16 bytes of information, so not as short as sequential IDs. See for example https://pypi.org/project/shortuuid/ https://pypi.org/project/shortuuid/ >>> shortuuid.uuid() 'vytxeTZskVKR7C7WgdSP3d'
      - ivanbakel 6y ago
        Those shorter strings can come with their own caveats. Long-form UUIDs have the nice property of being URL-safe - but IIRC, path pieces in HTTP urls are not meant to be case-sensitive, and browsers have historically tried to "help" by re-casing URLs. For correct semantics, short UUIDs have to go in the request parameters instead.
        account42 6y ago
        > IRC, path pieces in HTTP urls are not meant to be case-sensitive, and browsers have historically tried to "help" by re-casing URLs. That would break with any webserver that is serving files from a case-sensitive filesystem. Which is most of them.
        Deadron 6y ago
        Windows....enough said.
        hansvm 6y ago
        Most of the url (including the path pieces) is supposed to be case sensitive unless otherwise specified for a given protocol (and treating e.g. %3A the same as %3a). This should be the most recent RFC on the topic: https://tools.ietf.org/html/rfc3986#section-6.2.2.1 https://tools.ietf.org/html/rfc3986#section-6.2.2.1
    - ozim 6y ago
      You should not use your database identifiers in urls anyway.
  - codingdave 6y ago
    If you are implying that having an un-guessable URL secures your data, you might want to reconsider that approach. Your data should be secured server-side based on the actual auth in your app.
    - derwiki 6y ago
      Not the OP but I read it as “much harder to scrape” than making it Secure. Which has some value.
- fumblebee 6y ago
  I once debugged a case where the id had left sided zero padding so the whole id length was a fixed number of digits, for example `238974` might be represented as `00238974`. In a process that joined two tables the match counts were off. Somehow, on one side of the join the id was being converted to an integer and then back to a string, which stripped the zero padding. Meaning that `00238974` was failing to match `238974`. Sigh.
- genezeta 6y ago
  Not bugs but I've encountered a consequence which is worse than a bug. In a recent project I arrived at, I started seeing 612 in random places in the code. It was, naturally, the ID of a very specific user. Having an easy ID to remember, it is a temptation for sloppy programmers to just hardcode certain checks against a particular ID instead of following proper procedures. It was a bad project and a bad team, and after some 10 years or so, the code was now flooded with 612 and a couple of other IDs. Sure, you could avoid such a thing with code revisions and such, but then again it's better to simply not put the temptation in front of the programmers, isn't it?
  - deleted 6y ago
    [deleted]
- acdha 6y ago
  I encountered a similar problem recently while migrating an old app. It modeled IDs as 64-bit BIGINT columns (a decision made in the 2000s) and populated them with MySQL’s UUID_SHORT function. This worked fine for over a decade until we migrated to a different MySQL system and started getting conflicts for IDs already existing or not being able to find a row which had just been inserted. They’d missed the part that UUID_SHORT() returns an unsigned integer and created the column as the default signed integer. MySQL uses an algorithm where the top n bits are based on the server ID, which worked on the old server which had id=1 and never returned a number where the first bit was 1. The new cluster fortunately always did so the problem was immediately identified, but it was confused by one of those bonus MySQL data-destruction features – the way it silently truncated data meant that it was silently truncating new IDs to the same value but the logged value wasn’t in the database at all.
- MaxBarraclough 6y ago
  I like that pattern too. I've tried using it in a rather ad-hoc way in C/C++ enums. I suppose the proper solution would be to have a header file, enum_starting_vals.h, to centralise them.
mobilemidget 6y ago
"As of 2011 Facebook is in the second group, and spends several milliseconds of CPU time sanitizing every display string on its way to the browser, which multiplies out to hundreds of servers worth of CPUs sitting in a datacenter paying the price for the invalid UTF-8 in the databases." I can imagine how companies will be taxed extra for this somewhere in the (probably not so near) future.
- corty 6y ago
  Why not sanitize it once when accessed, write the sanitized result back to the database and set a flag on the record. Use only presanitized strings if the flag is set. After some time, when enough of your strings are flagged, run a sanitizer over the rest that hasn't been touched. That way you don't waste those milliseconds. However, I can imagine that this would waste quite some storage bandwidth because such a read is now a write as well.
  - monsieurbanana 6y ago
    Doing that at the scale of facebook does not seem like a problem you can solve in a paragraph of explanations, which is the point of the blog: The longer you wait, the more difficult it is to solve.
teddyh 6y ago
I thought we weren’t supposed to call it “blacklist” or “whitelist” anymore?
- faeyanpiraat 6y ago
  Sigh..
  - teddyh 6y ago
    What? Did I miss a memo or something? I would have hoped to get an actual answer to my question instead of rhetorical sighs (and downvotes). Note: I am not a native English speaker.
    - adrianN 6y ago
      This topic is pretty close to flamebait so it gets downvoted would be my guess.
      - corty 6y ago
        Yes, but why? How? I'm a non-native-speaker as well, and these things might be obvious to you, but they aren't to me. Is it a "should not use it", "must not use it", "maybe" or "it's fashionable"? How bad would it be if I accidentially use it, more like a four-letter-word or more like "well, he's a foreigner, he doesn't know"?
        throwaway525142 6y ago
        Basic Googling: https://en.wikipedia.org/w/index.php?title=Blacklist_(computing)&oldid=1006094213#Controversy_over_use_of_the_term https://en.wikipedia.org/w/index.php?title=Blacklist_(comput...
        thomasahle 6y ago
        That's the problem with flamewars. You can't really recommend in any direction without baiting the flames. You'll just have to look for previous discussions and decide for yourself.
marcus_holmes 6y ago
Things you should never do: use integers as ID's. This is literally a solved problem, and the solution is UUIDs, which were invented for exactly this job.
- mherrmann 6y ago
  What's wrong with integer IDs?
  - dkarp 6y ago
    The main issue in my experience are so called "user enumeration attacks", especially with sequential ids that are normally used. This is where an attacker is able to leak information from your system just by guessing ids. If you used a sequential id then you can cycle from 1 to X and probably easily find which are valid users. You can then likely see how many valid users/ids there are and potentially pull their data if the authentication has been implemented incorrectly. By using a UUID, the key space is so large that you cannot reasonably guess user ids. So you can't use those same brute force techniques and it makes extricating data much harder.
    - marcus_holmes 6y ago
      this, also there's no possibility of collision. You can't have two processes generate the same ID (which means you can generate your IDs in code and send them to the database, which is really useful in some situations). Also, and this is the main one for me - if I mess up my code and accidentally use a document_id instead of a user_id, I'll just get "not found" instead of someone else's data.
      - mherrmann 6y ago
        Actually, afaik uuids are pseudo-random and can thus absolutely collide in theory. Collision is solved by every db backend out there.
        6y ago
continuational 6y ago
Don't use a raw integer as the ID type. Use different types for different IDs: UserId, OrganizationId, etc. It's a common bug to get different ID types mixed up, and a gigantic offset will do nothing to help you with that.
- MaxBarraclough 6y ago
  I agree, the 'microtype' pattern can be a good way of getting the type-checker to catch silly mistakes. In Ada it's standard practice. In C++ you really need a library to do it easily, but there's a good one out there ready to go: https://github.com/foonathan/type_safe/ https://github.com/foonathan/type_safe/
deleted 6y ago
[deleted]
boris 6y ago
For me #1 would be to add a version to any data format or communication protocol. If you want to know how hard not doing so can bite, don't look further than Git and it's tourcherous migration from sha1.
- foolinaround 6y ago
  are there good examples for this, and how it scaled across versions over time?
  - ctvo 6y ago
    This covers the wider problem of contract (protocol) changes in distributed systems: https://aws.amazon.com/builders-library/ensuring-rollback-safety-during-deployments/?did=ba_card&trk=ba_card https://aws.amazon.com/builders-library/ensuring-rollback-sa... > With each change, we explicitly assign a distinct version to serializers. > We do this independent of source code or build versioning. We also store the serializer version with the serialized data or in the metadata. Older serializer versions continue to function in the new software. We find it’s usually helpful to emit a metric for the version of data written or read. It provides operators with visibility and troubleshooting information if there are errors. All of this applies to RPC and API versions, too. Backwards compatibility and the ability to roll out new changes (that can co-exist with current data) are the primary drivers.
- ahaferburg 6y ago
  What happened with git? Could somebody elaborate?
  - scaladev 6y ago
    https://lwn.net/Articles/811068/ https://lwn.net/Articles/811068/ https://lwn.net/Articles/823352/ https://lwn.net/Articles/823352/
- cousin_it 6y ago
  Why? We use protocol buffers for communication between services, evolve them all the time, and don't use version numbers. Any new functionality is enabled by booleans or optional fields in the proto. The approach works fine, I haven't noticed any problems in years.
silvestrov 6y ago
> Only Store Valid UTF-8 Linux could learn something about filenames here. Personally I would also disallow anything below 32 to avoid having filenames contain escape sequences. I have absolutely no need for a filename that contains an escape character, and would see this as a major bug, like his description of SQL injection. Better to fail fast.
evanelias 6y ago
This is from 2011, according to the commit blame info [1]. The advice appears to be from the standpoint of LAMP development a decade ago. [1] https://secure.phabricator.com/source/phabricator/browse/master/src/docs/flavor/things_you_should_do_now.diviner$1 https://secure.phabricator.com/source/phabricator/browse/mas...
- dang 6y ago
  Year added above. Thanks!
simonblack 6y ago
I knew of a trucker once, who when he was starting his transport business, marked his first truck with "Number 8". It never hurts to make people believe you are bigger and better than you really are.

199 points by bshanks 6y ago

bryanrasmussen 6y ago

That start ids at a gigantic number idea is great, even though I've never encountered a bug caused by not doing it.

joss82 6y ago
Indeed, that and sql syntax errors are solved by using an orm or sql code generator like sql alchemy core.
boatmansloth 6y ago
I literally came here to comment that the very first bug I had to chase down in a production system in my first job was caused by this! There was a 'fuzzy' search field available to users and when one input a client's ID number (formatted as NNNNNNN) it was showing results for a different client because NNNNNNN was also the format of the unique ID for records in the database.
tpetry 6y ago
A better tip would be to use something like time-ordered uuids: You can‘t misuse them for something different and the added bonus is that no one can iterate your db records by just incrementing the url.
- Cthulhu_ 6y ago
  My main gripe with UUIDs at the moment is that they look ugly in urls :/
  - pretzelhands 6y ago
    They're not the prettiest sight, yes, but I think anyone who isn't in some way a geek particularly cares about what's actually in URLs. There's a reason why Apple got away with only showing the hostname of a URL: It doesn't matter to ordinary people. :)
  - krab 6y ago
    You can encode it to a shorter string if you use larger alphabet. Still, it's 16 bytes of information, so not as short as sequential IDs. See for example https://pypi.org/project/shortuuid/ https://pypi.org/project/shortuuid/ >>> shortuuid.uuid() 'vytxeTZskVKR7C7WgdSP3d'
    - ivanbakel 6y ago
      Those shorter strings can come with their own caveats. Long-form UUIDs have the nice property of being URL-safe - but IIRC, path pieces in HTTP urls are not meant to be case-sensitive, and browsers have historically tried to "help" by re-casing URLs. For correct semantics, short UUIDs have to go in the request parameters instead.
      - account42 6y ago
        > IRC, path pieces in HTTP urls are not meant to be case-sensitive, and browsers have historically tried to "help" by re-casing URLs. That would break with any webserver that is serving files from a case-sensitive filesystem. Which is most of them.
        Deadron 6y ago
        Windows....enough said.
      - hansvm 6y ago
        Most of the url (including the path pieces) is supposed to be case sensitive unless otherwise specified for a given protocol (and treating e.g. %3A the same as %3a). This should be the most recent RFC on the topic: https://tools.ietf.org/html/rfc3986#section-6.2.2.1 https://tools.ietf.org/html/rfc3986#section-6.2.2.1
  - ozim 6y ago
    You should not use your database identifiers in urls anyway.
- codingdave 6y ago
  If you are implying that having an un-guessable URL secures your data, you might want to reconsider that approach. Your data should be secured server-side based on the actual auth in your app.
  - derwiki 6y ago
    Not the OP but I read it as “much harder to scrape” than making it Secure. Which has some value.
fumblebee 6y ago
I once debugged a case where the id had left sided zero padding so the whole id length was a fixed number of digits, for example `238974` might be represented as `00238974`. In a process that joined two tables the match counts were off. Somehow, on one side of the join the id was being converted to an integer and then back to a string, which stripped the zero padding. Meaning that `00238974` was failing to match `238974`. Sigh.
genezeta 6y ago
Not bugs but I've encountered a consequence which is worse than a bug. In a recent project I arrived at, I started seeing 612 in random places in the code. It was, naturally, the ID of a very specific user. Having an easy ID to remember, it is a temptation for sloppy programmers to just hardcode certain checks against a particular ID instead of following proper procedures. It was a bad project and a bad team, and after some 10 years or so, the code was now flooded with 612 and a couple of other IDs. Sure, you could avoid such a thing with code revisions and such, but then again it's better to simply not put the temptation in front of the programmers, isn't it?
- deleted 6y ago
  [deleted]
acdha 6y ago
I encountered a similar problem recently while migrating an old app. It modeled IDs as 64-bit BIGINT columns (a decision made in the 2000s) and populated them with MySQL’s UUID_SHORT function. This worked fine for over a decade until we migrated to a different MySQL system and started getting conflicts for IDs already existing or not being able to find a row which had just been inserted. They’d missed the part that UUID_SHORT() returns an unsigned integer and created the column as the default signed integer. MySQL uses an algorithm where the top n bits are based on the server ID, which worked on the old server which had id=1 and never returned a number where the first bit was 1. The new cluster fortunately always did so the problem was immediately identified, but it was confused by one of those bonus MySQL data-destruction features – the way it silently truncated data meant that it was silently truncating new IDs to the same value but the logged value wasn’t in the database at all.
MaxBarraclough 6y ago
I like that pattern too. I've tried using it in a rather ad-hoc way in C/C++ enums. I suppose the proper solution would be to have a header file, enum_starting_vals.h, to centralise them.

mobilemidget 6y ago

"As of 2011 Facebook is in the second group, and spends several milliseconds of CPU time sanitizing every display string on its way to the browser, which multiplies out to hundreds of servers worth of CPUs sitting in a datacenter paying the price for the invalid UTF-8 in the databases." I can imagine how companies will be taxed extra for this somewhere in the (probably not so near) future.

corty 6y ago
Why not sanitize it once when accessed, write the sanitized result back to the database and set a flag on the record. Use only presanitized strings if the flag is set. After some time, when enough of your strings are flagged, run a sanitizer over the rest that hasn't been touched. That way you don't waste those milliseconds. However, I can imagine that this would waste quite some storage bandwidth because such a read is now a write as well.
- monsieurbanana 6y ago
  Doing that at the scale of facebook does not seem like a problem you can solve in a paragraph of explanations, which is the point of the blog: The longer you wait, the more difficult it is to solve.

teddyh 6y ago

I thought we weren’t supposed to call it “blacklist” or “whitelist” anymore?

faeyanpiraat 6y ago
Sigh..
- teddyh 6y ago
  What? Did I miss a memo or something? I would have hoped to get an actual answer to my question instead of rhetorical sighs (and downvotes). Note: I am not a native English speaker.
  - adrianN 6y ago
    This topic is pretty close to flamebait so it gets downvoted would be my guess.
    - corty 6y ago
      Yes, but why? How? I'm a non-native-speaker as well, and these things might be obvious to you, but they aren't to me. Is it a "should not use it", "must not use it", "maybe" or "it's fashionable"? How bad would it be if I accidentially use it, more like a four-letter-word or more like "well, he's a foreigner, he doesn't know"?
      - throwaway525142 6y ago
        Basic Googling: https://en.wikipedia.org/w/index.php?title=Blacklist_(computing)&oldid=1006094213#Controversy_over_use_of_the_term https://en.wikipedia.org/w/index.php?title=Blacklist_(comput...
      - thomasahle 6y ago
        That's the problem with flamewars. You can't really recommend in any direction without baiting the flames. You'll just have to look for previous discussions and decide for yourself.

marcus_holmes 6y ago

Things you should never do: use integers as ID's. This is literally a solved problem, and the solution is UUIDs, which were invented for exactly this job.

mherrmann 6y ago
What's wrong with integer IDs?
- dkarp 6y ago
  The main issue in my experience are so called "user enumeration attacks", especially with sequential ids that are normally used. This is where an attacker is able to leak information from your system just by guessing ids. If you used a sequential id then you can cycle from 1 to X and probably easily find which are valid users. You can then likely see how many valid users/ids there are and potentially pull their data if the authentication has been implemented incorrectly. By using a UUID, the key space is so large that you cannot reasonably guess user ids. So you can't use those same brute force techniques and it makes extricating data much harder.
  - marcus_holmes 6y ago
    this, also there's no possibility of collision. You can't have two processes generate the same ID (which means you can generate your IDs in code and send them to the database, which is really useful in some situations). Also, and this is the main one for me - if I mess up my code and accidentally use a document_id instead of a user_id, I'll just get "not found" instead of someone else's data.
    - mherrmann 6y ago
      Actually, afaik uuids are pseudo-random and can thus absolutely collide in theory. Collision is solved by every db backend out there.
      - 6y ago

continuational 6y ago

Don't use a raw integer as the ID type. Use different types for different IDs: UserId, OrganizationId, etc. It's a common bug to get different ID types mixed up, and a gigantic offset will do nothing to help you with that.

MaxBarraclough 6y ago
I agree, the 'microtype' pattern can be a good way of getting the type-checker to catch silly mistakes. In Ada it's standard practice. In C++ you really need a library to do it easily, but there's a good one out there ready to go: https://github.com/foonathan/type_safe/ https://github.com/foonathan/type_safe/

deleted 6y ago

[deleted]

boris 6y ago

For me #1 would be to add a version to any data format or communication protocol. If you want to know how hard not doing so can bite, don't look further than Git and it's tourcherous migration from sha1.

foolinaround 6y ago
are there good examples for this, and how it scaled across versions over time?
- ctvo 6y ago
  This covers the wider problem of contract (protocol) changes in distributed systems: https://aws.amazon.com/builders-library/ensuring-rollback-safety-during-deployments/?did=ba_card&trk=ba_card https://aws.amazon.com/builders-library/ensuring-rollback-sa... > With each change, we explicitly assign a distinct version to serializers. > We do this independent of source code or build versioning. We also store the serializer version with the serialized data or in the metadata. Older serializer versions continue to function in the new software. We find it’s usually helpful to emit a metric for the version of data written or read. It provides operators with visibility and troubleshooting information if there are errors. All of this applies to RPC and API versions, too. Backwards compatibility and the ability to roll out new changes (that can co-exist with current data) are the primary drivers.
ahaferburg 6y ago
What happened with git? Could somebody elaborate?
- scaladev 6y ago
  https://lwn.net/Articles/811068/ https://lwn.net/Articles/811068/ https://lwn.net/Articles/823352/ https://lwn.net/Articles/823352/
cousin_it 6y ago
Why? We use protocol buffers for communication between services, evolve them all the time, and don't use version numbers. Any new functionality is enabled by booleans or optional fields in the proto. The approach works fine, I haven't noticed any problems in years.

silvestrov 6y ago

> Only Store Valid UTF-8 Linux could learn something about filenames here. Personally I would also disallow anything below 32 to avoid having filenames contain escape sequences. I have absolutely no need for a filename that contains an escape character, and would see this as a major bug, like his description of SQL injection. Better to fail fast.

evanelias 6y ago

This is from 2011, according to the commit blame info [1]. The advice appears to be from the standpoint of LAMP development a decade ago. [1] https://secure.phabricator.com/source/phabricator/browse/master/src/docs/flavor/things_you_should_do_now.diviner$1 https://secure.phabricator.com/source/phabricator/browse/mas...

dang 6y ago
Year added above. Thanks!

simonblack 6y ago

I knew of a trucker once, who when he was starting his transport business, marked his first truck with "Number 8". It never hurts to make people believe you are bigger and better than you really are.