7 ms·

CMU Computer Systems: Self-Grading Lab Assignments (2018)

206 points by georgecmu 8y ago

proy24 8y ago
15-213 you beauty...
PEJOE 8y ago
Oh malloc lab.
- lasagnaphil 8y ago
  Oh, good times. By the way, CMU's recitation class ppt for this lab is fucking hilarious: http://www.cs.cmu.edu/afs/cs/academic/class/15213-s08/www/recitation/mallocadvice.pdf http://www.cs.cmu.edu/afs/cs/academic/class/15213-s08/www/re...
carlosdp 8y ago
Ah 15-213, these labs were the most well designed assignments I ever had in college.
- deleted 8y ago
  [deleted]
- agiamas 8y ago
  we all remember and respect 15-213 :)
- btucker 8y ago
  13 years on, I continue to reference this class as the gold standard in lab design & execution. It’s the one class I continue to tell stories about to this day.
- ogdrhr 8y ago
  If I recall correctly, I believe the computer architecture course at USC (ducks) also uses these exact CS:APP labs. I loved them, it was fun realizing partway through some of the earlier labs which type of data structure they were having you walk. Decent prep for OS, if you really dig into the malloc lab.
westurner 8y ago
These look fun; in particular the "Attack Lab". Dockerfiles might be helpful and easy to keep updated. Alpine Linux or just busybox are probably sufficient? The instructor set could extend FROM the assignment image and run a few tests with e.g. testinfra (pytest) You can also test code written in C with gtest. I haven't read through all of the materials: are there suggested (automated) fuzzing tools? Does OSS-Fuzz solve? Are there references to CWE and/or the SEI CERT C Coding Standard rules? https://wiki.sei.cmu.edu/confluence/plugins/servlet/mobile?contentId=87152044#content/view/87152044 https://wiki.sei.cmu.edu/confluence/plugins/servlet/mobile?c... "How could we have changed our development process to catch these bugs/vulns before release?" "If we have 100% [...] test coverage, would that mean we've prevented these vulns?" What about 200%?
- yjftsjthsd-h 8y ago
  What on earth would 200% test coverage mean?
  - sk0g 8y ago
    Coverage for all the code that was written, and all the code that has yet to be typed, of course. All thanks to quantum computing :)
    - westurner 8y ago
      ⟨100%| + |100%⟩ = 200%! (Even code with 100% branch coverage may have common weaknesses like those that these (great) labs have students exploit)
Iamhisalt 8y ago
Suggest year added to material: 2015
abhayb 8y ago
Former student and TA here. There is so much great stuff in this class and its assignments! Couple of highlights: - Data lab formally verifies all of the submissions and gives you an exact failing test case if the submission isn't correct. - Malloc lab is run competitively. You don't need to compete to do well, but you do if you want to do great. And the competition drives students to great feats. I know people who rewrote their inner loops in assembly to get better perf. Another built splay trees to get lower fragmentation. They both had comfortable A's in the class at the time. - The automated grading of assignments meant that the class could scale to many hundreds of students while the TAs could dedicate their time to office hours. Turnaround time for exams was sub-one-day. - The instructors are wonderful (and well dressed!) people and they ran the class expertly.
- albntomat0 8y ago
  > Turnaround time for exams was sub-one-day. What's the current status on accurately grading mistakes? From the exams I remember (Fall 2015), the exams autograded perfect answers well, but gave similar marks to nearly correct, but still wrong answers as complete gibberish. I felt that unless I had the answers exactly correct, it was hard to demonstrate my knowledge of the topic.
  - Shoop 8y ago
    Now there is a regrade system that lets you ask for more partial credit if you feel that the autograder was too harsh. That being said, its still relatively easy to get a lot of points off for a small mistake :(. You definitely have to be really careful during 213 exams but thankfully there's plenty of time to redo all the questions 2 or 3 times.
    - josephg 8y ago
      At UNSW we had a similar system. The marking was a test suite, and each passing test would give you a portion of your grade. The system had a couple of simple smoke tests which were ran on submission time (to catch issues that would make the entire test run fail erroneously). As a TA I spent a fair bit of time fixing trivial issues - eg, sometimes a student would get 3/10 on the auto marker with a basically working solution because of a typo or something. I would dock a mark, fix the bug and rerun the tests - leaving the student with 8/10 or something.
akarki15 8y ago
Didn't go to CMU, but I did take cs61 at Harvard extension last year (https://cs61.seas.harvard.edu/site/2018/ https://cs61.seas.harvard.edu/site/2018/). It too had some self grading assignments (time bomb). Thoroughly enjoyed that class.
dooglius 8y ago
I remember when my University copied a couple of these, I decided to go the full mile on the one where you exploit buffer overflow. I got myself a remote connection via execve, and was going to mess around, but soon realized I could also see and edit others' scores which could get me in significant trouble, so I had to make one log-scrubbing attack then stop. I'm surprised they never fixed the hole by containerizing it or something.
- josephg 8y ago
  I ran an in-class test/competition once where students submitted C code to a web server running on my laptop. My laptop compiled and ran their code - then checked it against a test suite and graded it all live. One student asked about security - and we got into a great discussion about permissions and what havoc he could and couldn’t pull off with the restricted user account. I encouraged him to figure out how to write a fork bomb - which he got working after the test. He was nervous about it because he didn’t want to get in trouble. With some reassurances he dove into API documentation and got it working. My computer totally died and needed a reboot. It was a great little teaching opportunity that I’m glad I didn’t pass up
  - techntoke 8y ago
    Ah, how simple it would have been to run in a virtual machine.
    - josephg 8y ago
      How simple it would have been to miss this teaching opportunity! If my laptop were fully protected from that attack, it wouldn't have been as much fun for my student to wreck my machine in front of everyone. And so he might not have done it, and my class would have missed out on a beautiful demonstration and discussion of practical security engineering.
  - paulmd 8y ago
    I forkbombed our server once by accident. I normally code a snippet at a time, see if it works, then go a little bit farther. Unfortunately you don't really want to have a loop with a fork in it with no exit condition. Really failed to think that one through. (in my defense you probably should have ulimits set on a server where students are going to be logging in and working...)
zerr 8y ago
What are the contents of the files behind the wall? Can anyone share those files?
ec109685 8y ago
These would have been fun! For the bomb one, are there protections for taking the binary and running it in an isolated environment?
- MiroF 8y ago
  > For the bomb one, are there protections for taking the binary and running it in an isolated environment? Running it through a debugger is easy enough that this would be sorta unnecessary
- closeparen 8y ago
  You set a breakpoint at the phone-home routine.
  - ollien 8y ago
    My friends and I went the extra mile and overwrote the call to the server with nops.
mips_avatar 8y ago
My professor for my senior operating systems class who was a CMU PhD grad, gave us these assignments (with the permission of CMU). They were some of the most important learning experiences in my undergraduate career.
jibcage 8y ago
Former head TA (Jack) here - 15-213 still remains incredibly foundational to the work I do for my job. Working with the professors of that course (who also wrote the CS:APP textbook, which I still regret never getting signed by them!) was an amazing opportunity and something I hope I never forget. If you want some exercises to help you learn the foundations of computer systems, I honestly cannot think of a better resource than this!
exogeny 8y ago
Another CMU alum here. Just looking at this gave me PTSD. 213, man. It's a killer.
throwaway_7718 8y ago
CS:APP is hands down THE book that every person interested in the low-level stuff in computer systems should own. But heads up, anyone who is considering to buy the book - please get the North American Edition, NOT the global edition. I made the mistake of getting the Global edition, because of its considerably less cost, and because I couldn't afford the North American one - it was only after that I checked out the book site, where the authors mention that the global edition is chock full of errors [0]. I don't blame the authors, nor even the people who were responsible for 'the generation of a different set of practice and homework problems'. I can get printing the book in B&W, reducing paper quality, and publishing as a paperback to cut costs, but it's baffling why the publishers compromise on the actual quality of the content itself. Amazon is full of similar 'PSAs' about not buying the global edition [1]. [0] http://csapp.cs.cmu.edu/3e/errata.html http://csapp.cs.cmu.edu/3e/errata.html [1] https://www.amazon.com/Computer-Systems-Programmers-Perspective-Edition/dp/013409266X https://www.amazon.com/Computer-Systems-Programmers-Perspect...
jamestimmins 8y ago
Is there a way for non-CMU students to take complete these assignments and get scored? I worked through this book a few years back but would love the chance to do actual projects.
- mathieubordere 8y ago
  I worked through the book and did the actual projects, if I remember well, most of the grading logic can be run locally on your machine and is included in the projects.
doktrin 8y ago
Former 15-213 student as well. This was one of the best CS classes I've ever had the pleasure of taking. Also worth noting the plagiarism detector actually did its job, at least in some of the most egregious cases that I heard of.
rogela 8y ago
Took the CSAPP course at Peking University. Easily the most valuable course through 4 years...
voidmain 8y ago
When I took this class many, many moons ago, the close equivalent of the "cache lab" was graded not on the number of cache misses but on the cache hit ratio. When I pointed the problem with this metric out to a TA he blew me off. So I turned in the original code plus a loop that read the same volatile value a few billion times. It took a long time for the cache simulator to grade, but they dutifully gave me a very high grade (something like 200%) on the lab :-)

206 points by georgecmu 8y ago

proy24 8y ago

15-213 you beauty...

PEJOE 8y ago

Oh malloc lab.

lasagnaphil 8y ago
Oh, good times. By the way, CMU's recitation class ppt for this lab is fucking hilarious: http://www.cs.cmu.edu/afs/cs/academic/class/15213-s08/www/recitation/mallocadvice.pdf http://www.cs.cmu.edu/afs/cs/academic/class/15213-s08/www/re...

carlosdp 8y ago

Ah 15-213, these labs were the most well designed assignments I ever had in college.

deleted 8y ago
[deleted]
agiamas 8y ago
we all remember and respect 15-213 :)
btucker 8y ago
13 years on, I continue to reference this class as the gold standard in lab design & execution. It’s the one class I continue to tell stories about to this day.
ogdrhr 8y ago
If I recall correctly, I believe the computer architecture course at USC (ducks) also uses these exact CS:APP labs. I loved them, it was fun realizing partway through some of the earlier labs which type of data structure they were having you walk. Decent prep for OS, if you really dig into the malloc lab.

westurner 8y ago

These look fun; in particular the "Attack Lab". Dockerfiles might be helpful and easy to keep updated. Alpine Linux or just busybox are probably sufficient? The instructor set could extend FROM the assignment image and run a few tests with e.g. testinfra (pytest) You can also test code written in C with gtest. I haven't read through all of the materials: are there suggested (automated) fuzzing tools? Does OSS-Fuzz solve? Are there references to CWE and/or the SEI CERT C Coding Standard rules? https://wiki.sei.cmu.edu/confluence/plugins/servlet/mobile?contentId=87152044#content/view/87152044 https://wiki.sei.cmu.edu/confluence/plugins/servlet/mobile?c... "How could we have changed our development process to catch these bugs/vulns before release?" "If we have 100% [...] test coverage, would that mean we've prevented these vulns?" What about 200%?

yjftsjthsd-h 8y ago
What on earth would 200% test coverage mean?
- sk0g 8y ago
  Coverage for all the code that was written, and all the code that has yet to be typed, of course. All thanks to quantum computing :)
  - westurner 8y ago
    ⟨100%| + |100%⟩ = 200%! (Even code with 100% branch coverage may have common weaknesses like those that these (great) labs have students exploit)

Iamhisalt 8y ago

Suggest year added to material: 2015

abhayb 8y ago

Former student and TA here. There is so much great stuff in this class and its assignments! Couple of highlights: - Data lab formally verifies all of the submissions and gives you an exact failing test case if the submission isn't correct. - Malloc lab is run competitively. You don't need to compete to do well, but you do if you want to do great. And the competition drives students to great feats. I know people who rewrote their inner loops in assembly to get better perf. Another built splay trees to get lower fragmentation. They both had comfortable A's in the class at the time. - The automated grading of assignments meant that the class could scale to many hundreds of students while the TAs could dedicate their time to office hours. Turnaround time for exams was sub-one-day. - The instructors are wonderful (and well dressed!) people and they ran the class expertly.

albntomat0 8y ago
> Turnaround time for exams was sub-one-day. What's the current status on accurately grading mistakes? From the exams I remember (Fall 2015), the exams autograded perfect answers well, but gave similar marks to nearly correct, but still wrong answers as complete gibberish. I felt that unless I had the answers exactly correct, it was hard to demonstrate my knowledge of the topic.
- Shoop 8y ago
  Now there is a regrade system that lets you ask for more partial credit if you feel that the autograder was too harsh. That being said, its still relatively easy to get a lot of points off for a small mistake :(. You definitely have to be really careful during 213 exams but thankfully there's plenty of time to redo all the questions 2 or 3 times.
  - josephg 8y ago
    At UNSW we had a similar system. The marking was a test suite, and each passing test would give you a portion of your grade. The system had a couple of simple smoke tests which were ran on submission time (to catch issues that would make the entire test run fail erroneously). As a TA I spent a fair bit of time fixing trivial issues - eg, sometimes a student would get 3/10 on the auto marker with a basically working solution because of a typo or something. I would dock a mark, fix the bug and rerun the tests - leaving the student with 8/10 or something.

akarki15 8y ago

Didn't go to CMU, but I did take cs61 at Harvard extension last year (https://cs61.seas.harvard.edu/site/2018/ https://cs61.seas.harvard.edu/site/2018/). It too had some self grading assignments (time bomb). Thoroughly enjoyed that class.

dooglius 8y ago

I remember when my University copied a couple of these, I decided to go the full mile on the one where you exploit buffer overflow. I got myself a remote connection via execve, and was going to mess around, but soon realized I could also see and edit others' scores which could get me in significant trouble, so I had to make one log-scrubbing attack then stop. I'm surprised they never fixed the hole by containerizing it or something.

josephg 8y ago
I ran an in-class test/competition once where students submitted C code to a web server running on my laptop. My laptop compiled and ran their code - then checked it against a test suite and graded it all live. One student asked about security - and we got into a great discussion about permissions and what havoc he could and couldn’t pull off with the restricted user account. I encouraged him to figure out how to write a fork bomb - which he got working after the test. He was nervous about it because he didn’t want to get in trouble. With some reassurances he dove into API documentation and got it working. My computer totally died and needed a reboot. It was a great little teaching opportunity that I’m glad I didn’t pass up
- techntoke 8y ago
  Ah, how simple it would have been to run in a virtual machine.
  - josephg 8y ago
    How simple it would have been to miss this teaching opportunity! If my laptop were fully protected from that attack, it wouldn't have been as much fun for my student to wreck my machine in front of everyone. And so he might not have done it, and my class would have missed out on a beautiful demonstration and discussion of practical security engineering.
- paulmd 8y ago
  I forkbombed our server once by accident. I normally code a snippet at a time, see if it works, then go a little bit farther. Unfortunately you don't really want to have a loop with a fork in it with no exit condition. Really failed to think that one through. (in my defense you probably should have ulimits set on a server where students are going to be logging in and working...)

zerr 8y ago

What are the contents of the files behind the wall? Can anyone share those files?

ec109685 8y ago

These would have been fun! For the bomb one, are there protections for taking the binary and running it in an isolated environment?

MiroF 8y ago
> For the bomb one, are there protections for taking the binary and running it in an isolated environment? Running it through a debugger is easy enough that this would be sorta unnecessary
closeparen 8y ago
You set a breakpoint at the phone-home routine.
- ollien 8y ago
  My friends and I went the extra mile and overwrote the call to the server with nops.

mips_avatar 8y ago

My professor for my senior operating systems class who was a CMU PhD grad, gave us these assignments (with the permission of CMU). They were some of the most important learning experiences in my undergraduate career.

jibcage 8y ago

Former head TA (Jack) here - 15-213 still remains incredibly foundational to the work I do for my job. Working with the professors of that course (who also wrote the CS:APP textbook, which I still regret never getting signed by them!) was an amazing opportunity and something I hope I never forget. If you want some exercises to help you learn the foundations of computer systems, I honestly cannot think of a better resource than this!

exogeny 8y ago

Another CMU alum here. Just looking at this gave me PTSD. 213, man. It's a killer.

throwaway_7718 8y ago

CS:APP is hands down THE book that every person interested in the low-level stuff in computer systems should own. But heads up, anyone who is considering to buy the book - please get the North American Edition, NOT the global edition. I made the mistake of getting the Global edition, because of its considerably less cost, and because I couldn't afford the North American one - it was only after that I checked out the book site, where the authors mention that the global edition is chock full of errors [0]. I don't blame the authors, nor even the people who were responsible for 'the generation of a different set of practice and homework problems'. I can get printing the book in B&W, reducing paper quality, and publishing as a paperback to cut costs, but it's baffling why the publishers compromise on the actual quality of the content itself. Amazon is full of similar 'PSAs' about not buying the global edition [1]. [0] http://csapp.cs.cmu.edu/3e/errata.html http://csapp.cs.cmu.edu/3e/errata.html [1] https://www.amazon.com/Computer-Systems-Programmers-Perspective-Edition/dp/013409266X https://www.amazon.com/Computer-Systems-Programmers-Perspect...

jamestimmins 8y ago

Is there a way for non-CMU students to take complete these assignments and get scored? I worked through this book a few years back but would love the chance to do actual projects.

mathieubordere 8y ago
I worked through the book and did the actual projects, if I remember well, most of the grading logic can be run locally on your machine and is included in the projects.

doktrin 8y ago

Former 15-213 student as well. This was one of the best CS classes I've ever had the pleasure of taking. Also worth noting the plagiarism detector actually did its job, at least in some of the most egregious cases that I heard of.

rogela 8y ago

Took the CSAPP course at Peking University. Easily the most valuable course through 4 years...

voidmain 8y ago

When I took this class many, many moons ago, the close equivalent of the "cache lab" was graded not on the number of cache misses but on the cache hit ratio. When I pointed the problem with this metric out to a TA he blew me off. So I turned in the original code plus a loop that read the same volatile value a few billion times. It took a long time for the cache simulator to grade, but they dutifully gave me a very high grade (something like 200%) on the lab :-)