6 ms·

Orange_slice: Research Kernel and Hypervisor in Rust

126 points by adamnemecek 8y ago

obl 8y ago
Nice. I'm sure you could do a lot of cool things once you have this working well by playing around with the "butterfly effect". For example introduce a single X us jitter in the RTC interrupt and see how long it takes or which part of the machine state diverges from the baseline run.
deleted 8y ago
[deleted]
staticassertion 8y ago
The coolest part of this, to me: > This is going to be developed live? Yup. Check out My Youtube Channel or my Twitter. I announce my streams typically a few hours ahead of time, and schedule the streams on Youtube. Further for streams I think are more impactful, I try to schedule them a few days out. > I'm going to try to do much of the development live, and I'll try to help answer any questions about why certain things are being done. If this project fails, but I teach some people about OS development and get other excited about security research, then it was a success in my eyes. > I have already scheduled a stream for an intro on Wednesday: Intro Video https://github.com/gamozolabs/orange_slice#this-is-going-to-be-developed-live https://github.com/gamozolabs/orange_slice#this-is-going-to-...
- cjbprime 8y ago
  I wonder if they'd be interested in setting up a Patreon, some other regular dev streams I follow have them. This seems like impactful, important work.
  - gamozolabs 8y ago
    I greatly appreciate the interest in support! I don't have a need for funding for projects like these, as I get plenty of enjoyment out of doing them on my own. For now I'd suggest helping the many other projects/blogs/charities out there! If my following grows I'd love to add a Patreon and directly feed any proceeds there back into the community via projects, free trainings, blogs, etc. I'm happy enough with where I am in life, and I'd really like to just give back and teach others. I was fortunate enough to have a mentor at a very young age (and many other great teachers and leaders throughout life), and I recognize the huge advantage it gave me. If I can pass knowledge on to the next generation (and current generations of course), that's the best outcome I can hope for!
  - Nebbers 8y ago
    Would you mind sharing who you watch? I have found a few good ones but feel like there are loads I am missing! Best video I have found to date is https://youtu.be/1rZ-JorHJEY https://youtu.be/1rZ-JorHJEY but there are only 2
    - cjbprime 8y ago
      Harvard cs50 does live infosec CTF: http://www.twitch.tv/cs50tv/v/401612368?sr=a&t=2s http://www.twitch.tv/cs50tv/v/401612368?sr=a&t=2s Lots of game developers everywhere. Someone was building a compiler from scratch in a series and I can't remember the username, will add if I find it.
roca 8y ago
There is one really big technical issue here which I don't see answered on the site: given "it will be multiprocessing from day one", how are they going to handle data races deterministically and efficiently? This is the issue that has made all previous attempts to implement multicore record-and-replay quite inefficient.
- rambojazz 8y ago
  What is multicore record-and-replay?
- sanxiyn 8y ago
  They say 5x. Is 5x unrealistic?
- gamozolabs 8y ago
  My apologies, I wasn't very clear in my readme. It's intended the kernel itself is multiprocessing (support for multiple cores, and multiple VMs). But currently the intent is only to run a single core per VM for now, as getting VT-x or SVM to be deterministic on a single core itself will be hard enough. If we accomplish this, then the next goal would be to maybe look at multiple cores. It is important that I can run multiple single core VMs on a single machine as I plan to use this framework for fuzzing, and retrofitting multiprocessing to a kernel is always a source of many, many bugs. Hope that clears it up a bit!
  - nickpsecurity 8y ago
    I saved a few papers on deterministic multithreading both to knock out concurrency errors and support formal verification. I'm not sure how helpful they are in your use case. I'm just dropping them in case they give you ideas: https://people.csail.mit.edu/mareko/asplos073-olszewski.pdf https://people.csail.mit.edu/mareko/asplos073-olszewski.pdf https://people.cs.umass.edu/~emery/pubs/dthreads-sosp11.pdf https://people.cs.umass.edu/~emery/pubs/dthreads-sosp11.pdf
yazr 8y ago
Why Rust? (In general..) Are C memory-related bugs still such a real problem? IIUC, Rust can prevent these kinds of memory vulnerabilities. But it cant really present the million of other types of vulnerabilities, such as mis-configuration, wrong logic in the code, races, etc Am i understanding this correctly?
- scottlamb 8y ago
  > Are C memory-related bugs still such a real problem? I believe so, yes. https://www.zdnet.com/article/microsoft-70-percent-of-all-security-bugs-are-memory-safety-issues/ https://www.zdnet.com/article/microsoft-70-percent-of-all-se... > IIUC, Rust can prevent these kinds of memory vulnerabilities. Yes: if your Rust code has memory errors, there's a bug in an "unsafe" block (~1% of the code in a project of mine, which I think is typical) or in the compiler. > But it cant really present the million of other types of vulnerabilities, such as mis-configuration, wrong logic in the code, races, etc It can prevent data races. It can't prevent all possible race conditions. https://doc.rust-lang.org/nomicon/races.html https://doc.rust-lang.org/nomicon/races.html
  - oconnor663 8y ago
    > if your Rust code has memory errors, there's a bug in an "unsafe" block Minor clarification around this part. It's possible for a bug in safe code to break an invariant that some correct unsafe code is relying on, if that safe code has private access to something that the rest of the world doesn't. For example, by changing only safe code inside of Vec, you could introduce a bug that set the wrong capacity. Since all the unsafe code in Vec assumes the capacity is correct, that would break everything and cause tons of UB. Vec is sound, though, because the capacity is a private field, and safe code in the caller can't change it. But it does mean that when we're auditing unsafe blocks, we might also need to audit the safe code in the same module, depending on what invariants the unsafe blocks are assuming.
    - glennpratt 8y ago
      I guess that's debatable, your point is that code that should be unsafe might not be, but then that did mean the unsafe code is wrong in a sense. It assumes an invariant that isn't enforced.
peter_d_sherman 8y ago
"The end goal is a deterministic hypervisor, capable of booting Windows and Linux, with less than a 5x performance slowdown to achieve instruction-and-cycle level determinism for cycle counts and interrupt boundaries." You've set some very lofty goals here, that is, you're pushing the bleeding-edge envelope; I hope you succeed wildly, but if you do not, then the next best thing to success is an "Adventurer's Journal". Sort of like you're an explorer, exploring a new land that no one has ever visited. By writing down journal entries of your exploration, you leave both a guide and a map -- for whoever next decides to explore this uncharted territory with your same goals. The history of computing is filled with people who gave us key stepping stones to advance technology in some form or another... If you aren't successful, then becoming one of these people is the next best thing... Anyway, wishing you luck in your adventures, and I hope you succeed wildly!

126 points by adamnemecek 8y ago

obl 8y ago

Nice. I'm sure you could do a lot of cool things once you have this working well by playing around with the "butterfly effect". For example introduce a single X us jitter in the RTC interrupt and see how long it takes or which part of the machine state diverges from the baseline run.

deleted 8y ago

[deleted]

staticassertion 8y ago

The coolest part of this, to me: > This is going to be developed live? Yup. Check out My Youtube Channel or my Twitter. I announce my streams typically a few hours ahead of time, and schedule the streams on Youtube. Further for streams I think are more impactful, I try to schedule them a few days out. > I'm going to try to do much of the development live, and I'll try to help answer any questions about why certain things are being done. If this project fails, but I teach some people about OS development and get other excited about security research, then it was a success in my eyes. > I have already scheduled a stream for an intro on Wednesday: Intro Video https://github.com/gamozolabs/orange_slice#this-is-going-to-be-developed-live https://github.com/gamozolabs/orange_slice#this-is-going-to-...

cjbprime 8y ago
I wonder if they'd be interested in setting up a Patreon, some other regular dev streams I follow have them. This seems like impactful, important work.
- gamozolabs 8y ago
  I greatly appreciate the interest in support! I don't have a need for funding for projects like these, as I get plenty of enjoyment out of doing them on my own. For now I'd suggest helping the many other projects/blogs/charities out there! If my following grows I'd love to add a Patreon and directly feed any proceeds there back into the community via projects, free trainings, blogs, etc. I'm happy enough with where I am in life, and I'd really like to just give back and teach others. I was fortunate enough to have a mentor at a very young age (and many other great teachers and leaders throughout life), and I recognize the huge advantage it gave me. If I can pass knowledge on to the next generation (and current generations of course), that's the best outcome I can hope for!
- Nebbers 8y ago
  Would you mind sharing who you watch? I have found a few good ones but feel like there are loads I am missing! Best video I have found to date is https://youtu.be/1rZ-JorHJEY https://youtu.be/1rZ-JorHJEY but there are only 2
  - cjbprime 8y ago
    Harvard cs50 does live infosec CTF: http://www.twitch.tv/cs50tv/v/401612368?sr=a&t=2s http://www.twitch.tv/cs50tv/v/401612368?sr=a&t=2s Lots of game developers everywhere. Someone was building a compiler from scratch in a series and I can't remember the username, will add if I find it.

roca 8y ago

There is one really big technical issue here which I don't see answered on the site: given "it will be multiprocessing from day one", how are they going to handle data races deterministically and efficiently? This is the issue that has made all previous attempts to implement multicore record-and-replay quite inefficient.

rambojazz 8y ago
What is multicore record-and-replay?
sanxiyn 8y ago
They say 5x. Is 5x unrealistic?
gamozolabs 8y ago
My apologies, I wasn't very clear in my readme. It's intended the kernel itself is multiprocessing (support for multiple cores, and multiple VMs). But currently the intent is only to run a single core per VM for now, as getting VT-x or SVM to be deterministic on a single core itself will be hard enough. If we accomplish this, then the next goal would be to maybe look at multiple cores. It is important that I can run multiple single core VMs on a single machine as I plan to use this framework for fuzzing, and retrofitting multiprocessing to a kernel is always a source of many, many bugs. Hope that clears it up a bit!
- nickpsecurity 8y ago
  I saved a few papers on deterministic multithreading both to knock out concurrency errors and support formal verification. I'm not sure how helpful they are in your use case. I'm just dropping them in case they give you ideas: https://people.csail.mit.edu/mareko/asplos073-olszewski.pdf https://people.csail.mit.edu/mareko/asplos073-olszewski.pdf https://people.cs.umass.edu/~emery/pubs/dthreads-sosp11.pdf https://people.cs.umass.edu/~emery/pubs/dthreads-sosp11.pdf

yazr 8y ago

Why Rust? (In general..) Are C memory-related bugs still such a real problem? IIUC, Rust can prevent these kinds of memory vulnerabilities. But it cant really present the million of other types of vulnerabilities, such as mis-configuration, wrong logic in the code, races, etc Am i understanding this correctly?

scottlamb 8y ago
> Are C memory-related bugs still such a real problem? I believe so, yes. https://www.zdnet.com/article/microsoft-70-percent-of-all-security-bugs-are-memory-safety-issues/ https://www.zdnet.com/article/microsoft-70-percent-of-all-se... > IIUC, Rust can prevent these kinds of memory vulnerabilities. Yes: if your Rust code has memory errors, there's a bug in an "unsafe" block (~1% of the code in a project of mine, which I think is typical) or in the compiler. > But it cant really present the million of other types of vulnerabilities, such as mis-configuration, wrong logic in the code, races, etc It can prevent data races. It can't prevent all possible race conditions. https://doc.rust-lang.org/nomicon/races.html https://doc.rust-lang.org/nomicon/races.html
- oconnor663 8y ago
  > if your Rust code has memory errors, there's a bug in an "unsafe" block Minor clarification around this part. It's possible for a bug in safe code to break an invariant that some correct unsafe code is relying on, if that safe code has private access to something that the rest of the world doesn't. For example, by changing only safe code inside of Vec, you could introduce a bug that set the wrong capacity. Since all the unsafe code in Vec assumes the capacity is correct, that would break everything and cause tons of UB. Vec is sound, though, because the capacity is a private field, and safe code in the caller can't change it. But it does mean that when we're auditing unsafe blocks, we might also need to audit the safe code in the same module, depending on what invariants the unsafe blocks are assuming.
  - glennpratt 8y ago
    I guess that's debatable, your point is that code that should be unsafe might not be, but then that did mean the unsafe code is wrong in a sense. It assumes an invariant that isn't enforced.

peter_d_sherman 8y ago

"The end goal is a deterministic hypervisor, capable of booting Windows and Linux, with less than a 5x performance slowdown to achieve instruction-and-cycle level determinism for cycle counts and interrupt boundaries." You've set some very lofty goals here, that is, you're pushing the bleeding-edge envelope; I hope you succeed wildly, but if you do not, then the next best thing to success is an "Adventurer's Journal". Sort of like you're an explorer, exploring a new land that no one has ever visited. By writing down journal entries of your exploration, you leave both a guide and a map -- for whoever next decides to explore this uncharted territory with your same goals. The history of computing is filled with people who gave us key stepping stones to advance technology in some form or another... If you aren't successful, then becoming one of these people is the next best thing... Anyway, wishing you luck in your adventures, and I hope you succeed wildly!