10 ms·

Also: a software rasterizer. Most people refuse to write one because it's so easy not to. Why bother? It will make you a better coder for the rest of your lif

Also: a software rasterizer. Most people refuse to write one because it's so easy not to. Why bother? It will make you a better coder for the rest of your life. Let's make a list of "power projects" like this. A bytecode interpreter, a software rasterizer... What else?

kovrik 10y ago
Your own Lisp-like language (or Scheme implementation). Really recommend it.
- zeveb 10y ago
  Seconded. One of my plethora of hobby projects has been an interpreter or compiler (it changes from month to month) for Common Lisp, for an esoteric platform. It's a huge amount of fun, and I have learnt so much about software from it.
zeta0134 10y ago
I really recommend a raytracer, especially to anyone interested in graphics. It's straightforward, powerful, infinitely expandable with optional features, and opens up a ton of discussion about performance, code complexity, and general organization. Plus it's fun, in an instant gratification kind of way.
- voltagex_ 10y ago
  Every now and then I get interested in demoscene programming. I've never even been able to get a triangle to render on the screen - except with something like XNA. Do you think there's any value in going back to say, DOS based VGA programming? People in #osdev thought I was a bit strange for wanting to write a bootable kernel that only put pixels on the screen, but I really enjoy the idea of starting with plotting pixels, then moving on to more complex effects.
  - chrisseaton 10y ago
    You can write raw pixels to the screen with a HTML canvas and JS - no need to do it with low level code and making your own OS.
    - corysama 10y ago
      I bet you could go from main() to displayBufferOnScreen(unsigned char *buffer) in a couple dozen lines of C using SDL.
      - voltagex_ 10y ago
        This is a good idea, as is the Javascript one above.
    - vram22 10y ago
      Or with different desktop GUI toolkits for different languages: C++ w/Qt, C++ w/ wxWidgets, wxPython, Perl/Tk, Python +Tkinter, Delphi, Lazarus, or many other lang/toolkit combos. Even many BASICs have that ability built-in (from the early days of personal computers).
  - nice_byte 10y ago
    You could skip writing a bootable kernel, and try that with microcontrollers :) It's possible to hook up a tiny lcd screen to an arduino
    - errozero 10y ago
      You can even wire up an arduino into a vga monitor. There are a few vids on youtube showing simple games displayed this way.
  - khedoros 10y ago
    Are you more interested in actually writing to PC VGA registers, or just the idea of writing a pixel-by-pixel renderer? There are bindings for SDL and SFML in a lot of languages, and the web technology option mentioned by the sibling comment. If you like the idea of "DOS-based VGA programming" (taken literally), you could also find a DOS compiler or assembler and run it in DosBox. All of IBM's hardware-level documentation can still be found online: http://www.mcamafia.de/pdf/pdfref.htm http://www.mcamafia.de/pdf/pdfref.htm
- signa11 10y ago
  having written one as a slightly larger python thingy, i can fully attest to that. now that it is kind of done, i want to make it faster :) for example, having a home-grown vector_3d class kind of sucks (performance wise). it might be better to have vector_3d be actually based on, say, numpy.array ? once that is done, and i start seeing some real improvement, it might be possible to go the other route as well i.e. write the hot-spots in c++/c, and interface that with python. or go with lua all the way ? or maybe try a hybrid approach (which would allow you to see how embeddable the language really is) possibilities are endless, and as you so rightly said, gratification is instantaneous :)
  - Marazan 10y ago
    Simply replacing your own vector_3d class with numpy.array() won't actually speed you up that much as the overhead of creating all the tiny 3 element arrays kills you (I think I got only a 2x speed up from going from a pure python vector_3d to numpy arrays). Numpy is optimised for the creation of a small number of big arrays. The massive enormous speed ups come from creating massive arrays and implicitly working on them in parallel. So instead of iterating through every pixel and creating a origin and direction vector for each one you create a Total_number_of_pixels X 3 numpy array and pass that to your ray tracing function. Due to the way numpy broadcasts arrays the amount of rewriting you need to do is incredibly minimal and the speed ups over pure python are astronomical.
    - signa11 10y ago
      > The massive enormous speed ups come from creating massive arrays and implicitly working on them in parallel. So instead of iterating through every pixel and creating a origin and direction vector for each one you create a Total_number_of_pixels X 3 numpy array and pass that to your ray tracing function. Due to the way numpy broadcasts arrays the amount of rewriting you need to do is incredibly minimal and the speed ups over pure python are astronomical. thank you for your insight! this is very useful indeed.
      - Marazan 10y ago
spotman 10y ago
A distributed database. Same category of probably not a great idea to use and claim is better than what's out there ( it won't be ) but boy howdy will it teach you about distributed systems and pitfalls that will absolutely help you when you are working on one at your day job.
voltagex_ 10y ago
I've messed around with a DNS client and DNS server for the same reasons, but I don't think they quite meet the bar for a "power" project.
tptacek 10y ago
Strong agree on emulator and particularly stack vm. Would add: TCP/IP stack.
- nialo 10y ago
  mostly curiosity: why a stack vm in particular? (is it because you then need to write the compiler for it?)
  - tptacek 10y ago
    Because if you're familiar with conventional real register architectures, the stack VM forces you to rethink it, in a much simpler way. It's a useful simplifying abstraction.
    - vram22 10y ago
      There used to be a lot of interest in, and articles about, stack machines (real ones, not VMs) back in the days of mags like BYTE and PC Mag. Good reading.
- vram22 10y ago
  I once corresponded with someone who wrote his own working TCP/IP stack for his own home PC, an IBM PC Jr. See Mike's PCJr page near end of this post: http://jugad2.blogspot.in/2012/09/lissajous-hippo.html http://jugad2.blogspot.in/2012/09/lissajous-hippo.html
  - vram22 10y ago
    Update: Visited his page again. He has now even created a web server for the PCJr and ran his site on it for a while.
keithnz 10y ago
Your own multi tasking OS. Very common to roll your own everything on embedded systems. Especially back in the early 80s/90s
Jasper_ 10y ago
Tracks I've done and suggested to friends and colleagues as learning experiences: * Compression (lossless, lossy, image, audio, texture, video) * Languages (bytecode interpreter, AST interpreter, parser/lexer for a simple language, simple JIT, understanding instruction scheduling) * DSP programming (writing programs for fast, branchless math) * Comfort with binary and binary formats (start with packfiles .zip/.tar, move onto reverse engineering simple formats for e.g. games) * Understanding the difference between RAM and address spaces (e.g. understanding virtual memory, mmap, memory-mapped IO, dynamic linking, the VDSO, page faulting, shared memory) * Device drivers (easier on Linux, understanding userspace/kernel interaction, ioctls, how hardware and registers work, how to read spec sheets and hardware manuals) * Graphics (modern software rasterizer that's not scanline-based, understanding 3D and projective transforms, GPU programming and shaders, basic lighting (reflection and illumination) models, what "GPU memory" is, what scanout is, how full scenes are accumulated all along the stack) I could go heavily into depth for any one of these. Ask me questions if you're interested! They're all fun and I always have more to learn. Also, the longer you get into any given track, the more you realize they all connect in the end.
- kirang1989 10y ago
  This is an awesome list. Thanks!
- Rexxar 10y ago
  Concerning last point, do you know nice example of non-scanline-based 2D renderers ?
  - nadam 10y ago
    A very informative tutorial on high performance edge-function based rasterization: https://fgiesen.wordpress.com/2013/02/17/optimizing-sw-occlusion-culling-index/ https://fgiesen.wordpress.com/2013/02/17/optimizing-sw-occlu... (articles 6, 7, 8, others are not closely related) As far as I know modern hardware has been non-scanline-based for ages btw.
    - 10y ago
skybrian 10y ago
A web server. Use a subset of HTTP 1.0 and make the browser serve pages.
skybrian 10y ago
A simple search engine for a directory of documents. Create the index yourself.
skybrian 10y ago
A chat server. One chat room. Make sure you can handle fast and slow clients, and disconnects.
- fernly 10y ago
  OMG, memories... at once point in the pleistocene era, I was tasked with writing a manual for what was then Informix 4GL[1], and to get comfortable with the language, I spent a week writing a user forum app, with topics and message threads, messages stored as "blobs" in the SQL DB. Tried to get my co-workers in the pubs group to use it. They thought it was cute but all said hahaha no. [1] https://en.wikipedia.org/wiki/IBM_Informix-4GL https://en.wikipedia.org/wiki/IBM_Informix-4GL -- I am flabbergasted to see 4GL is still in use 30 years on.
shermanyo 10y ago
A basic game engine! I was exposed to so many concepts over time, building on a code base I understood from the ground up. It was the first time my code (c++ no less!) felt completely deterministic, that I understood every piece of data down to the byte level, heap/stack/gpu locality at any point of execution, and especially the lifecycle and memory management. If anyone is interested in creating an indie/hobby game engine, I'd recommend the following: - game loop with independent render FPS and a constant simulation tick delta (otherwise it runs non-deterministic due to floating point) - "entities, components, systems" architecture (ECS) - data-driven content (load level and game object data from JSON or similar, and programmatically build your scene) - basic event or messaging system Honestly, I can't think of a field that covers as wide a range of CS topics at the depth I've seen in lower level game development. If you focus on understanding and implementing the popular patterns and algorithms recommended by the indie community, its not the most daunting of projects. There's so much great information, and a few good books that break down and walk through a working implementation you can build on once you understand it from the ground up.
- pooktrain 10y ago
  This seems like lots of fun - mind sharing the books that you alluded to?
  - shermanyo 10y ago
    Sure, the main book I had in mind is "SFML Game Development" https://www.packtpub.com/game-development/sfml-game-development https://www.packtpub.com/game-development/sfml-game-developm... SFML is a C++ library that handles window management, input, sound, etc... For reference, a friend and I went through the book chapter by chapter a couple years ago. He was new to programming, and I'd made a few simple games and read scattered info in the past. The book had a good pace and filled in a lot of the gaps I had. Another great book is "Game Engine Architecture" https://www.crcpress.com/Game-Engine-Architecture-Second-Edition/Gregory/p/book/9781466560017 https://www.crcpress.com/Game-Engine-Architecture-Second-Edi... This book is a lot deeper, but I found it a great reference. I probably spent more time reading this book and going back over my engine code to refactor based on what I learned (rather than use it as a direct example implementation. At that stage, there were a few good articles online that were focused on the ECS pattern. Here's a couple, but try to find something that matches your language, there's plenty of info out there. http://www.dataorienteddesign.com/dodmain/node5.html http://www.dataorienteddesign.com/dodmain/node5.html https://eliasdaler.wordpress.com/2015/08/10/using-lua-and-cpp-in-practice/ https://eliasdaler.wordpress.com/2015/08/10/using-lua-and-cp... https://github.com/junkdog/artemis-odb/wiki/Introduction-to-Entity-Systems https://github.com/junkdog/artemis-odb/wiki/Introduction-to-... Good luck and have fun!
c-smile 10y ago
Hmm, rasterizer was just a starting point for me on the way to understand "how browser works". Having just a rasterizer is like naked VM without bytecode compiler. So I decided to add HTML/CSS engines to it. But if you have HTML/CSS then why not to add scripting to it? So added VM executing bytecodes, GC and compiler producing those bytecodes. Having VM I thought that it would be cool to have built-in persistence in it - you need to persist UI state somehow, right? So it got integrated NoSQL database on board. That's pretty much how http://sciter.com http://sciter.com was born.
- zodiac 10y ago
  I think they meant rasterization as in http://www.scratchapixel.com/lessons/3d-basic-rendering/rasterization-practical-implementation http://www.scratchapixel.com/lessons/3d-basic-rendering/rast... instead of in the context of browsers...it's not a very precise term
richdougherty 10y ago
* A database (transaction, lock managers, buffer/IO management, etc). * An MVC web framework. * A GUI validation library. * A JavaScript UI library. * An iteratee implementation. * An Erlang-style actors library in another language. * Implementing for-yield, async-await, etc on top of delimited continuations. * Interpreters, typecheckers, code generators. * Some of an ECMAScript implementation. * Concurrent data structures: futures/promises, queues, etc. * A simple roguelike game.
- mamcx 10y ago
  I haven't find a good introductory yet complete resource about build a complete database. I'm on the hunt for it (building also a relational language).
  - doozy 10y ago
    While not exactly what you're looking for, The Definitive Guide to SQLite takes you from writing a SELECT statement all the way to an overview of SQLite internals. O'Reilly also published a booklet called Inside SQLite that goes a bit deeper into the subject. I suggest SQLite because the source code is superb (seriously, some of the most readable, most logically organized and best commented C code you'll ever see) and it's a fairly small codebase with a huge community. And then there is Database Systems, The Complete Book where the second half of the book offers in deep coverage of the implementation of database systems.
    - mamcx 10y ago
      This area is one where it look more impenetrable. You find good info about compilers and even build "computers" from scratch but databases is black magic. Specially, because RDBMS look like are at all or nothing. I wonder if is possible to pick "only" the transactional/storage and made by myself the query/optimizer/api on top. I will be happy if is possible to build on top of something like sqlite, unfortunately, I don't know C at all (work with F#,C#, Delphi, Python). In the other hand, know the basic steps could be good enough...
  - 10y ago
bonoboTP 10y ago
* Implementing Internet protocols (TCP, IP, SMTP etc) * Floating point manipulation and numerical computation * Rolling some own crypto (strictly for fun use)
- loup-vaillant 10y ago
  Actually, implementing your own crypto for real is not so crazy. The best algorithms are surprisingly simple, and easy to make side-channel resistant. The only real difficulty I have so far is with modulo arithmetic on big numbers (for elliptic curves and one time authentication). With proper test vectors and some code review, crypto is in fact quite easy. More dangers lie un the use of crypto, especially when the API is flexible or complex. And of course good old exploitable bugs.
  - bonoboTP 10y ago
    No. Never roll your own crypto for production. If you only know one thing about security as a programmer, then it has to be this. But you are right that crypto algorithms aren't specifically hard to implement compared with, say, computer graphics, image processing, gui programming or whatever. But! The big difference is that crypto is attacked by other smart people. The bugs and design flaws in your scientific computing library are not hunted down by intelligent agents to purposefully break it. If people attacked your Paint clone to the same extent they attack computer security programs, then it would become just as hard to write them correctly as it is with crypto. There are so many kinds of ways to get it wrong that beginners don't even know about. It's an "unknown unknowns" situation.
    - loup-vaillant 10y ago
      Dogma, Dogma, Dogma. Have you seen the size of the reference implementations for chacha20, blake2b, or poly1305? Those are tiny, a couple hundred lines for all three! There is very little room for errors to sneak in. Between test vectors, compiler warnings, and Valgrind, we can be pretty sure all bugs have been shaked out. Add in good old code review and the safety of this stuff is pretty much guaranteed. Of course, I would never make the mistake of rolling my own OpenSSL clone (too big, bad API, some bad primitives), or even my own AES implementation (slow or prone to timing attacks). Of course, I don't make the mistake of going blind. I read the literature before making any choice. If I don't know why some construction is considered safe, I don't use it. Due diligence first. And I certainly don't implement the first primitive that my search engine gets its hands on. I go for the most vetted alternatives.
usrusr 10y ago
Reading from a raw filesystem dump was both interesting and rewarding as it works on a problem domain that exists in reality, not just on some simplified playground. If you target something as primitive as FAT, the challenge isn't terribly high.
titzer 10y ago
A virtual memory system for a kernel.
0xcde4c3db 10y ago
A screen-oriented text editor with undo and search/replace. Can you make it handle a 10MB file without "simple" operations having annoying delays? How about 100MB? 1GB? 100GB? With no line breaks in the file?
- mynewtb 10y ago
  Is there any such editor?
  - criddell 10y ago
    I don't think so. Can a text editor do search and replace on a 100 GB file without a delay? You could construct an index, but that's not going to happen instantaneously either.
    - 0xcde4c3db 10y ago
      > Can a text editor do search and replace on a 100 GB file without a delay? I meant that things like simply scrolling through a few pages or inserting/deleting a single character in the middle of the file shouldn't cause noticeable delays. Doing an operation that spans the entire file is different.
- hawski 10y ago
  With both emacs and vim you can certainly edit 500MB file. I concatenated all sources of recent Linux kernel 4.6.3 once. From what I remember it was all .c, .h and .S files. Resulting file has 542MB and 19'726'498 lines. I did it to test the text editor I am working on. Some stats: $ time vim -c 'edit kernel-total' -c q real 0m8.162s user 0m7.687s sys 0m0.398s $ vim kernel-total ^Z $ ps aux | awk "/kernel-total$/ || NR == 1" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND hawski 10467 2.7 17.1 805120 670988 pts/2 T 17:41 0:07 vim kernel-total $ time emacs kernel-total -f kill-emacs real 0m7.155s user 0m6.869s sys 0m0.237s $ emacs kernel-total ^Z $ ps aux | awk "/kernel-total$/ || NR == 1" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND hawski 10825 43.8 14.8 857484 581152 pts/2 T 17:47 0:07 emacs kernel-total With vim editing is quite smooth and with emacs it feels laggy. I only edited a bit, like entering a new line here and there, moving few pages down, going to the end of the file and to the top again. Saving is sluggish.
Johnny_Brahms 10y ago
By far the project that teaches me the most was writing an inliner for a toy language we developed as a part of a course at uni. Most people don't really realise that the more complex an inliner gets, the more similarities it gets to a rabid animal you can just barely restrain on a leash. My prof said he had laughed at my "how hard can it be?"-attitude . I worked my ass off and ended up being asked whether he could use the inliner in a language often mentioned on HN today :) The inliner was superseded by a much better one in 2003 written by a much smarter person than I, though.
Johnny_Brahms 10y ago
By far the project that teaches me the most was writing an inliner for a toy language we developed as a part of a course at uni. Most people don't really realise that the more complex an inliner gets, the more similarities it gets to a rabid animal you can just barely restrain on a leash. My prof said he had laughed at my "how hard can it be?"-attitude . I worked my ass off and ended up being asked whether he could use the inliner in a language often mentioned on HN today :) The inliner was superseded by a much better one in 2003 written by a much smarter person than I, though.
prirun 10y ago
An emulator for a real chip, like a 68000. I wrote one for a Prime minicomputer (Prime died in the early 90's). Telnet to em.prirun.com on port 8001 to try it out! The advantage of emulating a real system/chip is that if you can find old software for it, you avoid the step of having to invent your own programs: real, running programs already exist.

kovrik 10y ago

Your own Lisp-like language (or Scheme implementation). Really recommend it.

zeveb 10y ago
Seconded. One of my plethora of hobby projects has been an interpreter or compiler (it changes from month to month) for Common Lisp, for an esoteric platform. It's a huge amount of fun, and I have learnt so much about software from it.

zeta0134 10y ago

I really recommend a raytracer, especially to anyone interested in graphics. It's straightforward, powerful, infinitely expandable with optional features, and opens up a ton of discussion about performance, code complexity, and general organization. Plus it's fun, in an instant gratification kind of way.

voltagex_ 10y ago
Every now and then I get interested in demoscene programming. I've never even been able to get a triangle to render on the screen - except with something like XNA. Do you think there's any value in going back to say, DOS based VGA programming? People in #osdev thought I was a bit strange for wanting to write a bootable kernel that only put pixels on the screen, but I really enjoy the idea of starting with plotting pixels, then moving on to more complex effects.
- chrisseaton 10y ago
  You can write raw pixels to the screen with a HTML canvas and JS - no need to do it with low level code and making your own OS.
  - corysama 10y ago
    I bet you could go from main() to displayBufferOnScreen(unsigned char *buffer) in a couple dozen lines of C using SDL.
    - voltagex_ 10y ago
      This is a good idea, as is the Javascript one above.
  - vram22 10y ago
    Or with different desktop GUI toolkits for different languages: C++ w/Qt, C++ w/ wxWidgets, wxPython, Perl/Tk, Python +Tkinter, Delphi, Lazarus, or many other lang/toolkit combos. Even many BASICs have that ability built-in (from the early days of personal computers).
- nice_byte 10y ago
  You could skip writing a bootable kernel, and try that with microcontrollers :) It's possible to hook up a tiny lcd screen to an arduino
  - errozero 10y ago
    You can even wire up an arduino into a vga monitor. There are a few vids on youtube showing simple games displayed this way.
- khedoros 10y ago
  Are you more interested in actually writing to PC VGA registers, or just the idea of writing a pixel-by-pixel renderer? There are bindings for SDL and SFML in a lot of languages, and the web technology option mentioned by the sibling comment. If you like the idea of "DOS-based VGA programming" (taken literally), you could also find a DOS compiler or assembler and run it in DosBox. All of IBM's hardware-level documentation can still be found online: http://www.mcamafia.de/pdf/pdfref.htm http://www.mcamafia.de/pdf/pdfref.htm
signa11 10y ago
having written one as a slightly larger python thingy, i can fully attest to that. now that it is kind of done, i want to make it faster :) for example, having a home-grown vector_3d class kind of sucks (performance wise). it might be better to have vector_3d be actually based on, say, numpy.array ? once that is done, and i start seeing some real improvement, it might be possible to go the other route as well i.e. write the hot-spots in c++/c, and interface that with python. or go with lua all the way ? or maybe try a hybrid approach (which would allow you to see how embeddable the language really is) possibilities are endless, and as you so rightly said, gratification is instantaneous :)
- Marazan 10y ago
  Simply replacing your own vector_3d class with numpy.array() won't actually speed you up that much as the overhead of creating all the tiny 3 element arrays kills you (I think I got only a 2x speed up from going from a pure python vector_3d to numpy arrays). Numpy is optimised for the creation of a small number of big arrays. The massive enormous speed ups come from creating massive arrays and implicitly working on them in parallel. So instead of iterating through every pixel and creating a origin and direction vector for each one you create a Total_number_of_pixels X 3 numpy array and pass that to your ray tracing function. Due to the way numpy broadcasts arrays the amount of rewriting you need to do is incredibly minimal and the speed ups over pure python are astronomical.
  - signa11 10y ago
    > The massive enormous speed ups come from creating massive arrays and implicitly working on them in parallel. So instead of iterating through every pixel and creating a origin and direction vector for each one you create a Total_number_of_pixels X 3 numpy array and pass that to your ray tracing function. Due to the way numpy broadcasts arrays the amount of rewriting you need to do is incredibly minimal and the speed ups over pure python are astronomical. thank you for your insight! this is very useful indeed.
    - Marazan 10y ago

spotman 10y ago

A distributed database. Same category of probably not a great idea to use and claim is better than what's out there ( it won't be ) but boy howdy will it teach you about distributed systems and pitfalls that will absolutely help you when you are working on one at your day job.

voltagex_ 10y ago

I've messed around with a DNS client and DNS server for the same reasons, but I don't think they quite meet the bar for a "power" project.

tptacek 10y ago

Strong agree on emulator and particularly stack vm. Would add: TCP/IP stack.

nialo 10y ago
mostly curiosity: why a stack vm in particular? (is it because you then need to write the compiler for it?)
- tptacek 10y ago
  Because if you're familiar with conventional real register architectures, the stack VM forces you to rethink it, in a much simpler way. It's a useful simplifying abstraction.
  - vram22 10y ago
    There used to be a lot of interest in, and articles about, stack machines (real ones, not VMs) back in the days of mags like BYTE and PC Mag. Good reading.
vram22 10y ago
I once corresponded with someone who wrote his own working TCP/IP stack for his own home PC, an IBM PC Jr. See Mike's PCJr page near end of this post: http://jugad2.blogspot.in/2012/09/lissajous-hippo.html http://jugad2.blogspot.in/2012/09/lissajous-hippo.html
- vram22 10y ago
  Update: Visited his page again. He has now even created a web server for the PCJr and ran his site on it for a while.

keithnz 10y ago

Your own multi tasking OS. Very common to roll your own everything on embedded systems. Especially back in the early 80s/90s

Jasper_ 10y ago

Tracks I've done and suggested to friends and colleagues as learning experiences: * Compression (lossless, lossy, image, audio, texture, video) * Languages (bytecode interpreter, AST interpreter, parser/lexer for a simple language, simple JIT, understanding instruction scheduling) * DSP programming (writing programs for fast, branchless math) * Comfort with binary and binary formats (start with packfiles .zip/.tar, move onto reverse engineering simple formats for e.g. games) * Understanding the difference between RAM and address spaces (e.g. understanding virtual memory, mmap, memory-mapped IO, dynamic linking, the VDSO, page faulting, shared memory) * Device drivers (easier on Linux, understanding userspace/kernel interaction, ioctls, how hardware and registers work, how to read spec sheets and hardware manuals) * Graphics (modern software rasterizer that's not scanline-based, understanding 3D and projective transforms, GPU programming and shaders, basic lighting (reflection and illumination) models, what "GPU memory" is, what scanout is, how full scenes are accumulated all along the stack) I could go heavily into depth for any one of these. Ask me questions if you're interested! They're all fun and I always have more to learn. Also, the longer you get into any given track, the more you realize they all connect in the end.

kirang1989 10y ago
This is an awesome list. Thanks!
Rexxar 10y ago
Concerning last point, do you know nice example of non-scanline-based 2D renderers ?
- nadam 10y ago
  A very informative tutorial on high performance edge-function based rasterization: https://fgiesen.wordpress.com/2013/02/17/optimizing-sw-occlusion-culling-index/ https://fgiesen.wordpress.com/2013/02/17/optimizing-sw-occlu... (articles 6, 7, 8, others are not closely related) As far as I know modern hardware has been non-scanline-based for ages btw.
  - 10y ago

skybrian 10y ago

A web server. Use a subset of HTTP 1.0 and make the browser serve pages.

skybrian 10y ago

A simple search engine for a directory of documents. Create the index yourself.

skybrian 10y ago

A chat server. One chat room. Make sure you can handle fast and slow clients, and disconnects.

fernly 10y ago
OMG, memories... at once point in the pleistocene era, I was tasked with writing a manual for what was then Informix 4GL[1], and to get comfortable with the language, I spent a week writing a user forum app, with topics and message threads, messages stored as "blobs" in the SQL DB. Tried to get my co-workers in the pubs group to use it. They thought it was cute but all said hahaha no. [1] https://en.wikipedia.org/wiki/IBM_Informix-4GL https://en.wikipedia.org/wiki/IBM_Informix-4GL -- I am flabbergasted to see 4GL is still in use 30 years on.

shermanyo 10y ago

A basic game engine! I was exposed to so many concepts over time, building on a code base I understood from the ground up. It was the first time my code (c++ no less!) felt completely deterministic, that I understood every piece of data down to the byte level, heap/stack/gpu locality at any point of execution, and especially the lifecycle and memory management. If anyone is interested in creating an indie/hobby game engine, I'd recommend the following: - game loop with independent render FPS and a constant simulation tick delta (otherwise it runs non-deterministic due to floating point) - "entities, components, systems" architecture (ECS) - data-driven content (load level and game object data from JSON or similar, and programmatically build your scene) - basic event or messaging system Honestly, I can't think of a field that covers as wide a range of CS topics at the depth I've seen in lower level game development. If you focus on understanding and implementing the popular patterns and algorithms recommended by the indie community, its not the most daunting of projects. There's so much great information, and a few good books that break down and walk through a working implementation you can build on once you understand it from the ground up.

pooktrain 10y ago
This seems like lots of fun - mind sharing the books that you alluded to?
- shermanyo 10y ago
  Sure, the main book I had in mind is "SFML Game Development" https://www.packtpub.com/game-development/sfml-game-development https://www.packtpub.com/game-development/sfml-game-developm... SFML is a C++ library that handles window management, input, sound, etc... For reference, a friend and I went through the book chapter by chapter a couple years ago. He was new to programming, and I'd made a few simple games and read scattered info in the past. The book had a good pace and filled in a lot of the gaps I had. Another great book is "Game Engine Architecture" https://www.crcpress.com/Game-Engine-Architecture-Second-Edition/Gregory/p/book/9781466560017 https://www.crcpress.com/Game-Engine-Architecture-Second-Edi... This book is a lot deeper, but I found it a great reference. I probably spent more time reading this book and going back over my engine code to refactor based on what I learned (rather than use it as a direct example implementation. At that stage, there were a few good articles online that were focused on the ECS pattern. Here's a couple, but try to find something that matches your language, there's plenty of info out there. http://www.dataorienteddesign.com/dodmain/node5.html http://www.dataorienteddesign.com/dodmain/node5.html https://eliasdaler.wordpress.com/2015/08/10/using-lua-and-cpp-in-practice/ https://eliasdaler.wordpress.com/2015/08/10/using-lua-and-cp... https://github.com/junkdog/artemis-odb/wiki/Introduction-to-Entity-Systems https://github.com/junkdog/artemis-odb/wiki/Introduction-to-... Good luck and have fun!

c-smile 10y ago

Hmm, rasterizer was just a starting point for me on the way to understand "how browser works". Having just a rasterizer is like naked VM without bytecode compiler. So I decided to add HTML/CSS engines to it. But if you have HTML/CSS then why not to add scripting to it? So added VM executing bytecodes, GC and compiler producing those bytecodes. Having VM I thought that it would be cool to have built-in persistence in it - you need to persist UI state somehow, right? So it got integrated NoSQL database on board. That's pretty much how http://sciter.com http://sciter.com was born.

zodiac 10y ago
I think they meant rasterization as in http://www.scratchapixel.com/lessons/3d-basic-rendering/rasterization-practical-implementation http://www.scratchapixel.com/lessons/3d-basic-rendering/rast... instead of in the context of browsers...it's not a very precise term

richdougherty 10y ago

* A database (transaction, lock managers, buffer/IO management, etc). * An MVC web framework. * A GUI validation library. * A JavaScript UI library. * An iteratee implementation. * An Erlang-style actors library in another language. * Implementing for-yield, async-await, etc on top of delimited continuations. * Interpreters, typecheckers, code generators. * Some of an ECMAScript implementation. * Concurrent data structures: futures/promises, queues, etc. * A simple roguelike game.

mamcx 10y ago
I haven't find a good introductory yet complete resource about build a complete database. I'm on the hunt for it (building also a relational language).
- doozy 10y ago
  While not exactly what you're looking for, The Definitive Guide to SQLite takes you from writing a SELECT statement all the way to an overview of SQLite internals. O'Reilly also published a booklet called Inside SQLite that goes a bit deeper into the subject. I suggest SQLite because the source code is superb (seriously, some of the most readable, most logically organized and best commented C code you'll ever see) and it's a fairly small codebase with a huge community. And then there is Database Systems, The Complete Book where the second half of the book offers in deep coverage of the implementation of database systems.
  - mamcx 10y ago
    This area is one where it look more impenetrable. You find good info about compilers and even build "computers" from scratch but databases is black magic. Specially, because RDBMS look like are at all or nothing. I wonder if is possible to pick "only" the transactional/storage and made by myself the query/optimizer/api on top. I will be happy if is possible to build on top of something like sqlite, unfortunately, I don't know C at all (work with F#,C#, Delphi, Python). In the other hand, know the basic steps could be good enough...
- 10y ago

bonoboTP 10y ago

* Implementing Internet protocols (TCP, IP, SMTP etc) * Floating point manipulation and numerical computation * Rolling some own crypto (strictly for fun use)

loup-vaillant 10y ago
Actually, implementing your own crypto for real is not so crazy. The best algorithms are surprisingly simple, and easy to make side-channel resistant. The only real difficulty I have so far is with modulo arithmetic on big numbers (for elliptic curves and one time authentication). With proper test vectors and some code review, crypto is in fact quite easy. More dangers lie un the use of crypto, especially when the API is flexible or complex. And of course good old exploitable bugs.
- bonoboTP 10y ago
  No. Never roll your own crypto for production. If you only know one thing about security as a programmer, then it has to be this. But you are right that crypto algorithms aren't specifically hard to implement compared with, say, computer graphics, image processing, gui programming or whatever. But! The big difference is that crypto is attacked by other smart people. The bugs and design flaws in your scientific computing library are not hunted down by intelligent agents to purposefully break it. If people attacked your Paint clone to the same extent they attack computer security programs, then it would become just as hard to write them correctly as it is with crypto. There are so many kinds of ways to get it wrong that beginners don't even know about. It's an "unknown unknowns" situation.
  - loup-vaillant 10y ago
    Dogma, Dogma, Dogma. Have you seen the size of the reference implementations for chacha20, blake2b, or poly1305? Those are tiny, a couple hundred lines for all three! There is very little room for errors to sneak in. Between test vectors, compiler warnings, and Valgrind, we can be pretty sure all bugs have been shaked out. Add in good old code review and the safety of this stuff is pretty much guaranteed. Of course, I would never make the mistake of rolling my own OpenSSL clone (too big, bad API, some bad primitives), or even my own AES implementation (slow or prone to timing attacks). Of course, I don't make the mistake of going blind. I read the literature before making any choice. If I don't know why some construction is considered safe, I don't use it. Due diligence first. And I certainly don't implement the first primitive that my search engine gets its hands on. I go for the most vetted alternatives.

usrusr 10y ago

Reading from a raw filesystem dump was both interesting and rewarding as it works on a problem domain that exists in reality, not just on some simplified playground. If you target something as primitive as FAT, the challenge isn't terribly high.

titzer 10y ago

A virtual memory system for a kernel.

0xcde4c3db 10y ago

A screen-oriented text editor with undo and search/replace. Can you make it handle a 10MB file without "simple" operations having annoying delays? How about 100MB? 1GB? 100GB? With no line breaks in the file?

mynewtb 10y ago
Is there any such editor?
- criddell 10y ago
  I don't think so. Can a text editor do search and replace on a 100 GB file without a delay? You could construct an index, but that's not going to happen instantaneously either.
  - 0xcde4c3db 10y ago
    > Can a text editor do search and replace on a 100 GB file without a delay? I meant that things like simply scrolling through a few pages or inserting/deleting a single character in the middle of the file shouldn't cause noticeable delays. Doing an operation that spans the entire file is different.
hawski 10y ago
With both emacs and vim you can certainly edit 500MB file. I concatenated all sources of recent Linux kernel 4.6.3 once. From what I remember it was all .c, .h and .S files. Resulting file has 542MB and 19'726'498 lines. I did it to test the text editor I am working on. Some stats: $ time vim -c 'edit kernel-total' -c q real 0m8.162s user 0m7.687s sys 0m0.398s $ vim kernel-total ^Z $ ps aux | awk "/kernel-total$/ || NR == 1" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND hawski 10467 2.7 17.1 805120 670988 pts/2 T 17:41 0:07 vim kernel-total $ time emacs kernel-total -f kill-emacs real 0m7.155s user 0m6.869s sys 0m0.237s $ emacs kernel-total ^Z $ ps aux | awk "/kernel-total$/ || NR == 1" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND hawski 10825 43.8 14.8 857484 581152 pts/2 T 17:47 0:07 emacs kernel-total With vim editing is quite smooth and with emacs it feels laggy. I only edited a bit, like entering a new line here and there, moving few pages down, going to the end of the file and to the top again. Saving is sluggish.

Johnny_Brahms 10y ago

By far the project that teaches me the most was writing an inliner for a toy language we developed as a part of a course at uni. Most people don't really realise that the more complex an inliner gets, the more similarities it gets to a rabid animal you can just barely restrain on a leash. My prof said he had laughed at my "how hard can it be?"-attitude . I worked my ass off and ended up being asked whether he could use the inliner in a language often mentioned on HN today :) The inliner was superseded by a much better one in 2003 written by a much smarter person than I, though.

Johnny_Brahms 10y ago

prirun 10y ago

An emulator for a real chip, like a 68000. I wrote one for a Prime minicomputer (Prime died in the early 90's). Telnet to em.prirun.com on port 8001 to try it out! The advantage of emulating a real system/chip is that if you can find old software for it, you avoid the step of having to invent your own programs: real, running programs already exist.