4 ms·
Yea, fetching one URL at a time was a major issue, but things like the executable size in memory was also a problem. Remember, this is an old machine. Mysql t
by scumola 18y ago
Yea, fetching one URL at a time was a major issue, but things like the executable size in memory was also a problem. Remember, this is an old machine. Mysql takes 400M for its index for my data, my other processes run on the same machine also and all take memory. I initially added swap for things, but that's clearly not the right direction.
Also, I'm 40 years old. When I went to college, C programming was the norm, so it wasn't that difficult for me and didn't take long to implement and seemed like a great place to start to optimize things. It's fast, saturates my home Comcast pipe (they might cap my bandwidth, which I'm now worried about) and takes about 200M of memory with 30k URLs loaded into the queue. I suppose that I could reduce the memory requirements of the app some more by keeping the URLs in a BerkDB or something, but I think that 200m is acceptable for my needs now. For me, it's all about getting the most out of my available resources without throwing more money at it.
Re: gnu utils and 60's and 70's comments ... Yea, I admit that my knowledge of the gnu utils started in 1990 and I know that unix has been around since the 60's, so my initial post stating that the utils have been around since the 60's and that CPU speed and memory sizes were larger than they actually were back then were incorrect. My bad. ;)
- jshen 18y agoeven if you don't have funding how hard is it to get some old desktops that can have a couple of gigs of memory?
- fizx 18y agoYour urls don't need to be in BDB. Just write them to a text file and stream from that. If you want to check for duplicates, just find a simple bloom filter library. There, now you're under a megabyte. :)
- scumola 18y agodbserver swf # ps auxw | grep crawl root 6984 3.0 0.5 215832 5168 pts/5 Sl+ 12:59 0:00 ./crawl I'm using a linked-list in C to store all of the queued-up URLs so all of the threads can mark them as being worked on or whatever. Also, the app and curl lib takes a few megs too. I'm sure that the heap and multi-threadding libs are responsible for some of the resident memory usage. :) the on-disk binary is only 12k if that make a difference. ;) The struct: typedef struct my_url { char * url; int retcode; double old_size; double new_size; status status; struct my_url * next; char * filename; int flashurl_id; int depth; } my_url; I also like keeping them in memory as opposed to a BDB because of access speed - 25+ threads don't have to wait when it's all in memory, but 25+ threads hitting a BDB would be much nastier on the OS.
- barryfandango 18y agoFunny how these 2.0 young bucks consider a C-rewrite extreme, when for many coders is is the lingua franca, the bread and butter. I got started on high-level languages but I've made an effort to get familiar with C because it's small, beautiful, and fast as all hell. Fast is powerful.
- gnaritas 18y agoSpeed comes in two forms, execution time and development and maintenance time, C is only good for one of these. Python or Ruby would be better for the other.
- jawngee 18y agoWhy do you think C is such a maintenance issue?
- jshen 18y agoI like it's well accepted that garbage collection is a huge boon for productivity and stability
- jawngee 18y agoC has a variety of garbage collectors, here's one: http://www.hpl.hp.com/personal/Hans_Boehm/gc/ http://www.hpl.hp.com/personal/Hans_Boehm/gc/ I wonder if it's really more of an intimidation factor than anything (and I'm not saying this to be a jerk, I am truly curious why people have the perception that C is such a nightmare to maintain).
- jshen 18y agoAre you suggesting that you can be as productive in C as I can be in ruby? If not then you answered your own question. Second, are those garbage collectors widely used in C code and if not why?
- 18y ago
- spc476 18y agoI'm curious as to what you are storing besides the URL. Given the two figures, (200M RAM, 30k URLs) I calculate that you are using 7k per URL, which seems a bit excessive to me. Or does the 200M include other data besides the URLs?
- paul 18y agoYes, that's the right question to be asking. There's no reason why anything should be taking 7k per url.
- jhancock 18y agofully agree. From one 40 year old hacker to another, C is a very meaningful and useful language. It is not a "premature optimization" for someone who really gets unix/linux. I have built http://shellshadow.com http://shellshadow.com with C and have been very happy with the decision to get "closer to the metal" from the beginning. My server processes are lean and mean. They don't crash for inexplicable reasons. When I have a bug, there are not many layers of technology to slug through.
- kingkongrevenge 18y agoA server written with boost's asio in C++ will be just as lean and mean, but a fraction of the code.