11 ms·
If you had asked me to make a wild guess as to how Cloudflare stores internal headers and then removes them, I would have come up with some options: - An entir
by amluto 2y ago
If you had asked me to make a wild guess as to how Cloudflare stores internal headers and then removes them, I would have come up with some options:
- An entire separate dictionary or other data structure.
- One single header containing all internal metadata.
- All headers have a prefix, and the internal ones start with I and the external ones start with E.
- All internal headers start with “CFInt”.
I would not have come up with a scheme in which headers in a particular list are internal. (What if someone else uses this name? What if something forgets to sanitize? What if different simultaneously running programs disagree in the list? What if the Connection header names a Cloudflare-internal header? What if the set-difference algorithm is annoyingly slow?)
The web is already full of obnoxiously ambiguous in-band signaling and header naming, and I find it bizarre that a company with Cloudflare’s scale uses such a tedious and error-prone mechanism internally.
- jgrahamc 2y agoYeah, me too, but systems grew over time and grew and grew and we were using HTTP headers for all sorts of stuff. This optimization makes the situation better, but the solution (which is underway) is to use a different mechanism for IPC and get rid of the use of HTTP headers completely.
- DamonHD 2y agoAnd when that happy day dawns, slightly sadly, this nice optimisation will evaporate!
- jgrahamc 2y agoMaybe the real treasure was the optimizations we made and discarded along the way.
- DamonHD 2y agoIndeed. I am very happy with some of the tricks that I developed (speeding up, and indeed allowing, Solaris C++ linking by prefiltering on MD5 hashes of canonicalised .o files -> nm, anyone?) even though they were lost without trace (yep, Lehman Brothers, where are my Imakefiles now?)...
- jkaptur 2y agoThey're probably as critical as ever at Barclays :)
- DamonHD 2y agoI really hope not, given that that very old Solaris compiler should have been laid to rest many many years ago. Yes, some of my stuff had kinda made it into BarCap (even onto my desktop dev machine!) when I turned up there later, and even some at Nomura when I rocked up there again IIRC!
- alohashirt99 2y agoI recall working at a broker/dealer on Broadway where a curious sysadmin noticed a power cord that underneath an internal wall that fenced off "dead space." Said space turned out to contain a "forgotten" minicomputer living just outside a machine room. I recall returning to a bank and discovering that router and load balancer admin passwords hadn't changed in seven years..
- Twirrim 2y agoMy likely overly cynical concern here is that this suggests trie-hard will soon end up abandoned, as you're making it sound like it is a stop-gap solution for you.
- throwaway77385 2y agoFor anyone not in the loop, the above poster (JGC) is Cloudflare's CTO himself :)
- amluto 2y agoI’m certainly familiar with systems growing over time and outgrowing their original schema. Is this the same phenomenon that resulted in Cloudflare Tunnels apparently being domain names? Or why domains that are proxied by Cloudflare show up as “CNAME” in the DNS panel? (The latter one seems extra strange — wasn’t the reverse proxy service the very first service that Cloudflare offered? There must be some history here.)
- hinkley 2y agoAll new products are full of scaffolding that has to be removed when robustness becomes a higher priority than existence. Where we fail is not calling it that. Instead we just call some code “good” while our thicker skinned coworkers call the rest “fine”. It’s not fine. I don’t want to be working overtime the week of thanksgiving because you can’t think past the horizon.
- deleted 2y ago[deleted]
- gmueckl 2y agoThere is a very fine line between "good enough" and "not good enough" in any product beyond a certain complexity. Finding the pieces that cross that line can be hard and improving "good enough" parts is (sadly) mostly a waste of time in commercial settings.
- hinkley 2y agoYou have to backdate starting a project so it happens before things stop being good enough. And take into account bad estimates and the five other deadlines between then and now. Often it’s better to clean as you go. If you have the time and the inclination, shore up one of the least good enough things. It’s not unheard of to open up new feature sets via refactoring. Something that previously would have taken an “impossible” amount of time now becomes profitable, not just possible.
- 2y ago
- alex_suzuki 2y agoYou chiming in on this post makes me slightly bitter at having gone with CloudFront (I have everything on AWS so it seemed the obvious choice) instead of Cloudflare.
- deleted 2y ago[deleted]
- efitz 2y agoIve worked at several huge corporations in IT security, where we care about headers a lot, and they all use headers in a manner similar to CloudFlare. Including using proxies at the edge to strip out internal headers bidirectionally- yes, inbound too.
- sitkack 2y agoThat doesn't make it better, it makes it worse.
- hinkley 2y agoAre you on a corporate network? Do you use a firewall at home? You’re on enclaves all the time. This is just a different one. Separate networks per class of traffic used to be de rigeur before Cloud. Now it’s all munged together.
- sitkack 2y agoI understand that security needs to be pragmatic, but because security does it, doesn't make it right. It isn't about being in the enclave, it is having to keep track of what headers you set vs external. It is fragile and error prone and it will absolutely break someone else when there is a name collision. All security exploits are a subset of bugs.
- hinkley 2y agoAnd what would you have them do instead? SSL to every service? Reauthenticate on every single call? What’s your clever solution? There’s probably a dozen things you can terminate at a boundary that would cost so much to run through the entire system that it would bankrupt a company. And then there’s tracing, which also uses headers.
- immibis 2y agoone of the listed options was to begin all the Cloudflare internal header names with "CFInt"
- 620gelato 2y agoOr perhaps even, insert yet another header with just the list of internal headers being added to the request, assuming this happens at a single place, otherwise a recipe for disaster. I have a slightly different example of this, where a rpc framework used in my company disallows the service owner from modifying certain headers (say request identifier), and will instead create a duplicate header with a suffix. In that scenario at least, I can see this as a fairly reasonable tradeoff, as the goal is to control certain headers from being modified, not because they are platform internal but because there are certain assumptions associated with those headers that are followed company-wide. I'll go check what mechanism is used for this matching.
- amluto 2y agoThis doesn’t work if the front end is older than an internal node and a header gets added, at least not without extra logic to patch everything up and handle conflicts. The HTTP Connection header works like this, and one should generally assume that almost nothing implements it correctly.
- adrianmonk 2y ago> yet another header with just the list of internal headers Or the same but with a list of headers which AREN'T internal. You'll probably have a custom header-adding function that people should always use instead of the regular one. And this way, if someone forgets to use it, their header will get stripped. You can think of a header escaping the internal network as something that needs to be authorized. This is a deny by default approach.
- 620gelato 2y agoAh yes - perfect.
- taeric 2y agoOther ideas: - Ensure all headers added internally that are not for export at the front of the header list - Preseed the hashtable of all requests with internal headers you plan to remove. In fact, if you preseed, you are basically combining these ideas but fixing how many internal headers are on each request. At that point, you can use a linked hash table that preserves creation order and just remove the first N from the final list that you send back to clients.
- amluto 2y ago> At that point, you can use a linked hash table that preserves creation order and just remove the first N from the final list that you send back to clients. While Python provides data structures like this out of the box, designing a big system to require a special, nonstandard data structure to identify private metadata seems like a mistake to me.
- taeric 2y agoI'm not sure I follow? Does rust not have the equivalent of a LinkedHashMap from Java? All I'm proposing is to start each map of headers off with a sentinel value for all of the headers you don't intend to send back to the client. Then, have a method that iterates over all values, for the places you need to send to internal servers, and another (probably the default?) that starts iteration after all of the internal items, which should just be a different pointer. At this point, there is no need to inspect each header by name, as that was effectively done at compile time when you put the protected headers at the front of the list. Is this somewhat special? I mean, sure. But it wasn't long ago that a hash table was already in the special territory of data structures. Edit: I should add that I'm not certain this idea would be better. Just more ideas on what you could do.
- amluto 2y agoThere’s a third party crate for this. C++’s STL also doesn’t have it by default. The creation-order-preserving hash map is basically two data structures in one, it’s more complex and slower than a normal hash map, and IMO it’s rather bizarre and I have never really understood why anyone wants one.
- giancarlostoro 2y ago> What if someone else uses this name? Couldn't you bypass this by pre-fixing the ones that aren't yours? Or prefixing your internal ones with something unique enough?
- e63f67dd-065b 2y agoFormer employee here; the interesting (horrifying?) fact is that you can set some of these internal headers (I remember a notable bug with cf-cache-status) in workers (the serverless offering) and cause all kinds of bad things to happen :)
- hn_throwaway_99 2y agoI feel like I can point out lots of similar problems to the other solutions you suggest (heck, I thing some of those problems you list even apply to those other solutions). The list approach has some downsides, but it also has a bunch of upsides. I feel like when people like to point out potential flaws of these approaches, they're ignoring the history and difficulties that comes with Cloudflare's scope. An enumerated list is the simplest and most flexible of the approaches, and it also doesn't require any a priori agreement on the structure of a header key - this is probably important when you think about the sheer number of teams at Cloudflare, potential technology acquisitions, etc.
- amluto 2y agoLists, extensible protobufs, etc are indeed great in their extensibility. The issue I would see (if I worked at Cloudflare) isn’t to using a list — it’s that the list is the same list that’s used for externally visible HTTP headers.
- hn_throwaway_99 2y agoI don't see that as a problem at all: "Hey any teams out there, before you can add internal headers, you need to register them with some service." Then the list is loaded from static storage at startup time and/or refreshed on some periodic basis.
- Aeolun 2y ago> it also doesn't require any a priori agreement on the structure of a header key - this is probably important when you think about the sheer number of teams at Cloudflare That’s more of an indictment of how little effort is spent aligning things. It’s not that hard to tell every team that any headers have to start with ‘CFint’ and then enforce that.
- ketzo 2y ago> It’s not that hard How many years do you have to spend in software before you stop uttering those four cursed words?
- danbruc 2y agoThis might of course introduce quite a bit of overhead, but the clean solution would arguably be to not mix requests to begin with, the external requests you are processing are the payload of the requests you are sending around internally. This would also allow you to cook up an optimized encapsulation protocol that might be more efficient to process than parsing and modifying HTTP. This admittedly comes from someone with essentially zero knowledge of what exactly Cloudflare is doing to the requests they receive.
- anonymoushn 2y agoX years ago the manager of the cache team raised the lack of data plane and control plane separation as an issue and we came up with various plans to fix things, but I guess nothing has happened since