> I'm still looking for ways to optimize things Re-implement your crawler on a FPGA. Just kidding :) It's great that you got what you were expecting, but as P

> I'm still looking for ways to optimize things Re-implement your crawler on a FPGA. Just kidding :) It's great that you got what you were expecting, but as Paul said, you have to profile before you optimize (otherwise you will "optimize" useless stuff, and not only waste your time but also likely do counter-optimizations). Anyway, your post was very interesting. Because a lot of people assume they have to use layers on top of the OS, while modern Unix systems have good file systems and memory managers (maybe have a look at DragonFly BSD; they are going into an interesting direction).