3 ms·

> Then the paper observes that memory safety is still a significant problem on the server. But instead of pointing out the root cause -- the absence of sandboxi

by jchw 2y ago

> Then the paper observes that memory safety is still a significant problem on the server. But instead of pointing out the root cause -- the absence of sandboxing -- the authors argue that applications should instead be rewritten in go or rust! This is absurd. The kernel already provides strong memory protection guarantees for each process. The kernel also provides hard guarantees for access to devices and the file system. But server software doesn't take advantage of any of these guarantees. When a server process intermixes data of multiple customers and privilege levels then any tiny programming mistake (regardless of memory safety) can result in privilege escalation or catastrophic data leaks. What use is memory safety when your go program returns the wrong user's data because of an off-by-one error? You don't need a root exploit if your process already has "root access" to the database server. Yes, because servers are inherently multi-tenant, you can't inherently avoid the risks. Process isolation can't help you, even if you had the resources to fork off for every single request. If you have a database pool in your process, you can go and access other people's data. There is never going to be a case where having an RCE to a server isn't a serious issue. Also, neither process isolation nor memory safety can guarantee you don't simply return the wrong data for a given customer, so that point really neither here nor there. (I'd also argue that memory safety clearly matters on mobile platforms still anyway. Many of the exploit chains that break kernel protections still rely on exploiting memory bugs in userland first before they can climb their way up. There's also other risks to this. Getting an RCE into someone's Signal process is an extremely dangerous event for the user.)

gizmo 2y ago
Databases also have access controls! Which developers don't use. If you have 10.000 tenants on the same server you can simply have 10.000 database users. And that simple precaution will provide nearly perfect protection against cross-tenant data leaks.
- jchw 2y ago
  The database was merely an example, there are other shared resources that are going to run into the same problem, like caches. Still, it's weird to imply that database users are meant to map to application users without any kind of justification other than "you can do it" but OK, let's assume that we can. Let's consider Postgres, which will have absolutely no problem creating 10,000 users (not sure about 100,000 or 1,000,000, but we can put that aside anyhow.) You basically have two potential options here: - The most logical option is to continue to use database pooling as you currently do, and authenticate as a single user. Then, when handling a user request, you can impersonate a specific database user. Only problem is, if you do this, the protection you get is entirely discretionary: the connection is still absolutely authenticated to a user that can do more, and all you have to do is "reset role" and go on your way. So you can do this, but it doesn't help you with server exploits. - The other way to handle this is by having each request get a separate database connection which is actually authenticated to a specific user. That will work and provide the database level guarantees. However, for obvious reasons, you definitely can't share a global database pool with this approach. That's a problem, because each Postgres connection will cost 5-10 MiB or so. If you had 10,000 active users, you would spend 50-100 GiB on just per-connection resources on your database server box. This solution scales horribly even when the scale isn't that crazy. And this is all assuming you can actually get the guarantees you need just using the database layer. To that I say, good luck. You'll basically need to do most of your logic in the database instead of the application layer and make use of features like row-level security. You can do this, but it's an extremely limiting architecture, not the least of which because databases are hard to scale except vertically. If you run into any scenario where you outgrow a single database cluster, everything here goes out the window. Needless to say, nobody does this, and they're totally right. Having a database assist in things like authorization and visibility is not a terrible idea or anything, but all of this taken together is just not very persuasive. And besides. Postgres itself, along with basically all of the other major databases, are also not necessarily memory-safe. Having external parties have access to your database connection pretty much puts you back at square one for defenses against potential unknown memory safety bugs, making this entire exercise a bit pointless...
  - gizmo 2y ago
    Most server software today intermingles data that should be strictly isolated because it's convenient. I don't buy your arguments about efficiency either, because despite having no data isolation web software is still comically (tragically) slow.
    - jchw 2y ago
      Three things, very briefly: - Scaling is different than efficiency. - Even if you can afford to eat CPU costs, memory limitations are inflexible. - Again, this moves the problem to the database. The database is still written in C, and still deals with multi-tenant workloads in the same process, and you assume your application server might get RCE'd, which means now someone has a connection to your database, already authenticated, and it may very well have memory safety or other bugs. You can't escape this problem by moving it.
      - deleted 2y ago
        [deleted]
graemep 2y ago
> Also, neither process isolation nor memory safety can guarantee you don't simply return the wrong data for a given customer, so that point really neither here nor there. The fact that there can be security issues at the application level is no reason to add memory safety issues to them! You may well have the resources to fork a process per customer, not have a database pool, not cache etc. its a trade-off with resources needed and performance. > Also, neither process isolation nor memory safety can guarantee you don't simply return the wrong data for a given customer, so that point really neither here nor there. You should fix the problems you can, even if there are problems you cannot.
- jchw 2y ago
  > The fact that there can be security issues at the application level is no reason to add memory safety issues to them! > You may well have the resources to fork a process per customer, not have a database pool, not cache etc. its a trade-off with resources needed and performance. This feels like one of those weird things that wind up happening in discussions about a lot of things, including say, renewable energy. People get so hung up about certain details that the discussion winds up going into a direction that really doesn't make sense, and it could only go there incrementally because it would've been very obvious if you followed the entire chain. Please bear with me. Consider the primary problem in the first place: The problem is that we have a bunch of things that are not memory safe, already. Rewriting all of those things is going to take a lot of time and effort. All of these ideas about isolating requests are totally fine, honestly. If you can afford those sorts of trade-offs in your design, then go hog wild. I find many of these architectural decisions to be a bit pointless for reasons I'm happy to dig into in more detail if people really care, but the important thing to understand is that I'm not telling you to not try to isolate requests. I'm just saying that in practice, there are no "legacy" servers that apply this degree of request isolation. They were architected with the assumption that the process would not get RCE'd, and therefore it would require a full rearchitecting to make them safe in the face of that sort of bug. But when we talk about architectures where we avoid connection pooling and enforce security policy entirely in the database layer, that essentially means writing new servers. And if you're writing new servers, why in the world would you go through all of this effort when you don't have to? I'll grant you that the Rust borrow checker has a decently steep learning curve, but I don't expect it to be a serious barrier to experienced C++ programmers if they are really trying to learn and not just looking for reasons to not have to. C++ is not an easy language, either. It would be absolutely insane to perform all of this per-request isolation voodoo and then not start on a clean slate with a memory safe language in the first place. You may as well take both at that point. Fuck it, run each request inside a small KVM context! And in fact, you can do that. Microsoft did that[1], but note that even in their press release for Hyperlight, they are showing it being used in Rust code, because that's the logical order of operations: If you're truly serious about security, and you're building something from scratch, start by preventing vulnerabilities as far left in the process as possible; and you can't go further left than having a programming language that can prevent certain bugs entirely by-design. > You should fix the problems you can, even if there are problems you cannot. That goes both ways. [1]: https://opensource.microsoft.com/blog/2024/11/07/introducing-hyperlight-virtual-machine-based-security-for-functions-at-scale/ https://opensource.microsoft.com/blog/2024/11/07/introducing...

by jchw 2y ago

gizmo 2y ago

Databases also have access controls! Which developers don't use. If you have 10.000 tenants on the same server you can simply have 10.000 database users. And that simple precaution will provide nearly perfect protection against cross-tenant data leaks.

jchw 2y ago
The database was merely an example, there are other shared resources that are going to run into the same problem, like caches. Still, it's weird to imply that database users are meant to map to application users without any kind of justification other than "you can do it" but OK, let's assume that we can. Let's consider Postgres, which will have absolutely no problem creating 10,000 users (not sure about 100,000 or 1,000,000, but we can put that aside anyhow.) You basically have two potential options here: - The most logical option is to continue to use database pooling as you currently do, and authenticate as a single user. Then, when handling a user request, you can impersonate a specific database user. Only problem is, if you do this, the protection you get is entirely discretionary: the connection is still absolutely authenticated to a user that can do more, and all you have to do is "reset role" and go on your way. So you can do this, but it doesn't help you with server exploits. - The other way to handle this is by having each request get a separate database connection which is actually authenticated to a specific user. That will work and provide the database level guarantees. However, for obvious reasons, you definitely can't share a global database pool with this approach. That's a problem, because each Postgres connection will cost 5-10 MiB or so. If you had 10,000 active users, you would spend 50-100 GiB on just per-connection resources on your database server box. This solution scales horribly even when the scale isn't that crazy. And this is all assuming you can actually get the guarantees you need just using the database layer. To that I say, good luck. You'll basically need to do most of your logic in the database instead of the application layer and make use of features like row-level security. You can do this, but it's an extremely limiting architecture, not the least of which because databases are hard to scale except vertically. If you run into any scenario where you outgrow a single database cluster, everything here goes out the window. Needless to say, nobody does this, and they're totally right. Having a database assist in things like authorization and visibility is not a terrible idea or anything, but all of this taken together is just not very persuasive. And besides. Postgres itself, along with basically all of the other major databases, are also not necessarily memory-safe. Having external parties have access to your database connection pretty much puts you back at square one for defenses against potential unknown memory safety bugs, making this entire exercise a bit pointless...
- gizmo 2y ago
  Most server software today intermingles data that should be strictly isolated because it's convenient. I don't buy your arguments about efficiency either, because despite having no data isolation web software is still comically (tragically) slow.
  - jchw 2y ago
    Three things, very briefly: - Scaling is different than efficiency. - Even if you can afford to eat CPU costs, memory limitations are inflexible. - Again, this moves the problem to the database. The database is still written in C, and still deals with multi-tenant workloads in the same process, and you assume your application server might get RCE'd, which means now someone has a connection to your database, already authenticated, and it may very well have memory safety or other bugs. You can't escape this problem by moving it.
    - deleted 2y ago
      [deleted]

graemep 2y ago

> Also, neither process isolation nor memory safety can guarantee you don't simply return the wrong data for a given customer, so that point really neither here nor there. The fact that there can be security issues at the application level is no reason to add memory safety issues to them! You may well have the resources to fork a process per customer, not have a database pool, not cache etc. its a trade-off with resources needed and performance. > Also, neither process isolation nor memory safety can guarantee you don't simply return the wrong data for a given customer, so that point really neither here nor there. You should fix the problems you can, even if there are problems you cannot.

jchw 2y ago
> The fact that there can be security issues at the application level is no reason to add memory safety issues to them! > You may well have the resources to fork a process per customer, not have a database pool, not cache etc. its a trade-off with resources needed and performance. This feels like one of those weird things that wind up happening in discussions about a lot of things, including say, renewable energy. People get so hung up about certain details that the discussion winds up going into a direction that really doesn't make sense, and it could only go there incrementally because it would've been very obvious if you followed the entire chain. Please bear with me. Consider the primary problem in the first place: The problem is that we have a bunch of things that are not memory safe, already. Rewriting all of those things is going to take a lot of time and effort. All of these ideas about isolating requests are totally fine, honestly. If you can afford those sorts of trade-offs in your design, then go hog wild. I find many of these architectural decisions to be a bit pointless for reasons I'm happy to dig into in more detail if people really care, but the important thing to understand is that I'm not telling you to not try to isolate requests. I'm just saying that in practice, there are no "legacy" servers that apply this degree of request isolation. They were architected with the assumption that the process would not get RCE'd, and therefore it would require a full rearchitecting to make them safe in the face of that sort of bug. But when we talk about architectures where we avoid connection pooling and enforce security policy entirely in the database layer, that essentially means writing new servers. And if you're writing new servers, why in the world would you go through all of this effort when you don't have to? I'll grant you that the Rust borrow checker has a decently steep learning curve, but I don't expect it to be a serious barrier to experienced C++ programmers if they are really trying to learn and not just looking for reasons to not have to. C++ is not an easy language, either. It would be absolutely insane to perform all of this per-request isolation voodoo and then not start on a clean slate with a memory safe language in the first place. You may as well take both at that point. Fuck it, run each request inside a small KVM context! And in fact, you can do that. Microsoft did that[1], but note that even in their press release for Hyperlight, they are showing it being used in Rust code, because that's the logical order of operations: If you're truly serious about security, and you're building something from scratch, start by preventing vulnerabilities as far left in the process as possible; and you can't go further left than having a programming language that can prevent certain bugs entirely by-design. > You should fix the problems you can, even if there are problems you cannot. That goes both ways. [1]: https://opensource.microsoft.com/blog/2024/11/07/introducing-hyperlight-virtual-machine-based-security-for-functions-at-scale/ https://opensource.microsoft.com/blog/2024/11/07/introducing...