8 ms·

Command line tools for web developers

322 points by coderholic 15y ago

gluecode 15y ago
siege is a very useful tool for benchmarking. I use it a lot.
- ez77 15y ago
  How come sending concurrent requests, as explained in the article, doesn't flag you as a DoS offender?
  - mnutt 15y ago
    I was surprised that he ran siege against www.google.co.uk without some kind of "don't do this" disclaimer to new users. Running it against other people's websites is pretty poor form.
    - sebmck 15y ago
      I'm pretty sure Google can handle it.
    - rorrr 15y ago
      Google has rate limiting protection from a single IP. If you make too many requests, they send you to a captcha page.
  - idonthack 15y ago
    If you run it against a public site, that may be what happens. However it's intended for stressing your own sites in a test environment, in which case you have full control over the config. Setting up your test environment to block your stress tester as a DoS offender would just be silly.
  - rorrr 15y ago
    If it brings down the site, you might get into trouble. It's totally consider DoS.
- dstein 15y ago
  Does siege support benchmarking gzipped HTTP responses? This is something ab doesn't seem to do and it's starting to annoy me.
flexterra 15y ago
screen is another tool I use a lot, specially when connecting to remote servers over ssh.
- trb 15y ago
  Screen is a bit like a window manager for consoles. I use it to start multiple (software) servers without daemon mode. I can then switch between their outputs with screen. Screen also detaches the console from your ssh session. In my case this means that the servers keep running if I loose my ssh connection to the (hardware) server. It's a very handy tool and definitely belongs in the articles list of tools.
  - yuvadam 15y ago
    screen -d -m processName is practically the easiest way to run and daemonize a process.
    - icebraining 15y ago
      nohup processName
      - iam 15y ago
        But then the output goes into a black hole after you close your shell
        eis 15y ago
        You can always redirect stdout/stderr to a file, e.g. nohup someapp > someapp.log
        LiveTheDream 15y ago
        The output goes into nohup.out automatically.
- reinhardt 15y ago
DevX101 15y ago
Are there any graphical interfaces for load testing, in lieu of siege?
- sebmck 15y ago
  Even if there is, a gui never compare to the power and control you get when you use a cli.
- pbogdan 15y ago
  You might want to have a look at JMeter (http://jakarta.apache.org/jmeter/ http://jakarta.apache.org/jmeter/).
hasenj 15y ago
`curl ifconfig.me` is pretty awesome But I didn't find anything else to be interesting.
- ams6110 15y ago
  ngrep was news to me. I think I've heard of it once before, recently, but didn't really realize what it was. Having not tried it, it sounds like a really nice option when something like Wireshark or Ethereal would be overkill or just too much effort to bother with.
cultureulterior 15y ago
http://freshmeat.net/projects/apachetop/ http://freshmeat.net/projects/apachetop/ I find it amazingly useful.
Pewpewarrows 15y ago
I prefer curl'ing icanhazip.com, and I make it a quick alias in my .bashrc: alias myip="curl icanhazip.com" Apachebench (ab) is a decent alternative to siege, and knowing how to use tcpdump and netcat comes in handy for debugging. Other than that, my new favorite command line tool over the past several months has to be vagrant, which lets you script and streamline VM creation and builds from the command line. If I need to completely reproduce my production environment on a test box, it's my utility of choice.
- RyanKearney 15y ago
  That only returned my IPv6 address so I can't get my IPv4 address with that.
  - madars 15y ago
    Try ipv4.icanhazip.com and ipv6.icanhazip.com .
- LiveTheDream 15y ago
  I use http://whatismyip.org/ http://whatismyip.org/ frequently.
- mwill 15y ago
  Theres also http://ifconfig.me/ http://ifconfig.me/ which acts similarly. The page has a list of curl commands you can issue and what your output will be.
tecoholic 15y ago
Coderholic.com is Timing out, Anyone else having issues?
- sarp 15y ago
  Database Error
- brs 15y ago
  Perhaps everyone is trying out siege.
- LiveTheDream 15y ago
  Works for me, but here's a cache just in case: http://liveweb.archive.org/http://www.coderholic.com/invaluable-command-line-tools-for-web-developers/ http://liveweb.archive.org/http://www.coderholic.com/invalua...
perlgeek 15y ago
curl -I and wget -S are particularly helpful when debugging redirects. Sometimes I migrate URL schemes, and set up permanent redirects in my .htaccess files. Testing them in a browser is a real pain, because browsers cache the redirect (which is the point of having a permanent redirect), so even if you change the .htaccess, you still get the old response. And pressing the refresh button is no help, because that reloads the destination page, not the source of the redirect. That's when a non-caching command line client saves your day.
vld 15y ago
also useful: wget post-file.it --post-file /path/to/file
dexen 15y ago
lftp is a practical SFTP, FTPS and FTP transfer program, including automatic upload/download resumption and synchronization (mirror) mode. Good for both interactive use and scripting.
- nikcub 15y ago
  lftp can also continue downloads and run multiple concurrent download threads, acting as a download accelerator. for eg. lftp :~> pget -c -n 10 http://warez.com/large_file.mp4 will start 10 concurrent download processes (n) and continue (c). lftp is full of hidden gems.
- LiveTheDream 15y ago
  Synchronization (mirror) mode is absolutely invaluable when you only have access to a machine via FTP.
NathanKP 15y ago
For me the most invaluable command line tool that I have ever used is S3 Tools: http://s3tools.org/s3tools http://s3tools.org/s3tools It provides a great suite of tools for interacting with S3, and is best used on an EC2 instance you are connected to via SSH. It is also ridiculously fast, much faster than trying to interact with S3 from a local FTP browser, or even from Amazon's own S3 dashboard. For example on my computer using Amazon's own web facing dashboard it will take about 30-45 minutes to make 14000 files on S3 public, versus via the command line tool downloaded and running on one of my EC2 instances it can make those files public within minutes. I assume this is because it is local network traffic. Anyway, if you are ever in a bind and need to move a bunch of files to S3, I highly recommend S3 Tools. It has saved me many times. Along those lines wget is the most powerful command line tool I've ever used, with so much capability. It is simply incredible when combined with S3 tools, allowing you to easily grab gigabytes of images off of a personal server or staging location and upload them to S3 very quickly. And if you need to do more than move files around you can manage even more aspects of AWS including EC2 instances from the command line using this powerful command line tool: http://www.timkay.com/aws/ http://www.timkay.com/aws/ Its great for writing automated bash scripts to manage your instances via cron jobs.
- epi0Bauqu 15y ago
  My main complaint with s3cmd is you can't easily throttle it, and on a busy machine its fast speeds can cause issues.
  - NathanKP 15y ago
    Good point. I guess in the past I have spun up a micro instance to run it, so I didn't have to run my S3 operations using one of my main production servers, and hence I didn't run into any problems with its unthrottled speed.
  - delinka 15y ago
    Perhaps lowering its priority would help. nice s3cmd ...
  - spiffytech 15y ago
sfoguy 15y ago
iMacros firefox addon for web automation via command line (when curl is not enough): http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Support http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Sup...
ryduh 15y ago
Note: ngrep can also be installed using brew on OS X.
- LiveTheDream 15y ago
  Also available via MacPorts.
genieyclo 15y ago
My most used cli tool outside of the default nuts and bolts is dtrx, the best and easiest file extractor for *nix. No more fiddling with flags or looking up, handles issues with putting lots of things in different directories or the wrong permissions on files. It has saved me a ton of time over the years. http://brettcsmith.org/2007/dtrx/ http://brettcsmith.org/2007/dtrx/
- divtxt 15y ago
  Wow! Just wow! Bookmarked! A mere upvote is not enough for this! Thank you!!! Questions: - Is there a site pointing out amazing command line tools like this? (their existence as opposed to usage examples like at http://www.commandlinefu.com/ http://www.commandlinefu.com/) - Is there a site listing the OS X equivalents for Linux command line tools? (e.g. what's the OS X equivalent for dos2unix / flip?) (edit: merge 2 comments into 1)
  - genieyclo 15y ago
    Yep, check out http://onethingwell.org http://onethingwell.org which is exactly that. Software (incl cli) that do one thing well, as according to the UNIX philosophy. It's nicely tagged as well, so you can click the 'osx' tag and get tools just for OS X and so forth. I actually found dtrx on hacker news though a few years ago, but it's also on onethingwell I believe.
    - BasDirks 15y ago
      Amazing, cheers.
  - deleted 15y ago
    [deleted]
- ludwigvan 15y ago
  Actually, for extracting tar files, using xf is enough. tar xf archive.tar.gz tar xf archive.tar.bz2 Still, this will be useful for other formats, I think. Thanks.
marcusramberg 15y ago
This was worth it for me for the reminder about siege alone, as 'ab' seems to have some serious issues in Lion.
bretthopper 15y ago
Ever wanted to know the progress of a long running operation? Copying a file, importing a mysql db, etc. Pipe Viewer (pv) is what you need: http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml. It lets you monitor the progress of any piped command and gives you time elapsed, speed, time left, and a progress bar (wget style). Copy a file: pv sourcefile > destfile Import mysql db: pv dump.sql| mysql -uroot -p db More tricks: http://blog.urfix.com/9-tricks-pv-pipe-viewer/ http://blog.urfix.com/9-tricks-pv-pipe-viewer/ edit: To install on OS X just do brew install pv
- brendano 15y ago
  Pipe Viewer dramatically increased my productivity for large scale data processing. In particular, it lets you quickly know whether something will take 5 minutes, or 2 hours, so you can plan accordingly. It's painful watching people try to do this without pv.
nobuff 15y ago
Oh-my-zsh with git and git flow plugins work well so far. https://github.com/robbyrussell/oh-my-zsh https://github.com/robbyrussell/oh-my-zsh
HerraBRE 15y ago
It's been mentioned on Hacker News a few times before, but my project PageKite (and showoff.io and localtunnel) is designed to help out with quick web demos and collaboration. Getting started is now two lines: curl http://pagekite.net/pk/pagekite-0.4.py >pagekite.py python pagekite.py 80 SOMENAME.pagekite.me ... answer a few questions and whatever is running on port 80 will be visible as https://SOMENAME.pagekite.me/ https://SOMENAME.pagekite.me/ within moments, almost no matter what kind of network connection you have. :-) There are also .deb and .rpm packages available for heavier users.

322 points by coderholic 15y ago

gluecode 15y ago

siege is a very useful tool for benchmarking. I use it a lot.

ez77 15y ago
How come sending concurrent requests, as explained in the article, doesn't flag you as a DoS offender?
- mnutt 15y ago
  I was surprised that he ran siege against www.google.co.uk without some kind of "don't do this" disclaimer to new users. Running it against other people's websites is pretty poor form.
  - sebmck 15y ago
    I'm pretty sure Google can handle it.
  - rorrr 15y ago
    Google has rate limiting protection from a single IP. If you make too many requests, they send you to a captcha page.
- idonthack 15y ago
  If you run it against a public site, that may be what happens. However it's intended for stressing your own sites in a test environment, in which case you have full control over the config. Setting up your test environment to block your stress tester as a DoS offender would just be silly.
- rorrr 15y ago
  If it brings down the site, you might get into trouble. It's totally consider DoS.
dstein 15y ago
Does siege support benchmarking gzipped HTTP responses? This is something ab doesn't seem to do and it's starting to annoy me.

flexterra 15y ago

screen is another tool I use a lot, specially when connecting to remote servers over ssh.

trb 15y ago
Screen is a bit like a window manager for consoles. I use it to start multiple (software) servers without daemon mode. I can then switch between their outputs with screen. Screen also detaches the console from your ssh session. In my case this means that the servers keep running if I loose my ssh connection to the (hardware) server. It's a very handy tool and definitely belongs in the articles list of tools.
- yuvadam 15y ago
  screen -d -m processName is practically the easiest way to run and daemonize a process.
  - icebraining 15y ago
    nohup processName
    - iam 15y ago
      But then the output goes into a black hole after you close your shell
      - eis 15y ago
        You can always redirect stdout/stderr to a file, e.g. nohup someapp > someapp.log
      - LiveTheDream 15y ago
        The output goes into nohup.out automatically.
reinhardt 15y ago

DevX101 15y ago

Are there any graphical interfaces for load testing, in lieu of siege?

sebmck 15y ago
Even if there is, a gui never compare to the power and control you get when you use a cli.
pbogdan 15y ago
You might want to have a look at JMeter (http://jakarta.apache.org/jmeter/ http://jakarta.apache.org/jmeter/).

hasenj 15y ago

`curl ifconfig.me` is pretty awesome But I didn't find anything else to be interesting.

ams6110 15y ago
ngrep was news to me. I think I've heard of it once before, recently, but didn't really realize what it was. Having not tried it, it sounds like a really nice option when something like Wireshark or Ethereal would be overkill or just too much effort to bother with.

cultureulterior 15y ago

http://freshmeat.net/projects/apachetop/ http://freshmeat.net/projects/apachetop/ I find it amazingly useful.

Pewpewarrows 15y ago

I prefer curl'ing icanhazip.com, and I make it a quick alias in my .bashrc: alias myip="curl icanhazip.com" Apachebench (ab) is a decent alternative to siege, and knowing how to use tcpdump and netcat comes in handy for debugging. Other than that, my new favorite command line tool over the past several months has to be vagrant, which lets you script and streamline VM creation and builds from the command line. If I need to completely reproduce my production environment on a test box, it's my utility of choice.

RyanKearney 15y ago
That only returned my IPv6 address so I can't get my IPv4 address with that.
- madars 15y ago
  Try ipv4.icanhazip.com and ipv6.icanhazip.com .
LiveTheDream 15y ago
I use http://whatismyip.org/ http://whatismyip.org/ frequently.
mwill 15y ago
Theres also http://ifconfig.me/ http://ifconfig.me/ which acts similarly. The page has a list of curl commands you can issue and what your output will be.

tecoholic 15y ago

Coderholic.com is Timing out, Anyone else having issues?

sarp 15y ago
Database Error
brs 15y ago
Perhaps everyone is trying out siege.
LiveTheDream 15y ago
Works for me, but here's a cache just in case: http://liveweb.archive.org/http://www.coderholic.com/invaluable-command-line-tools-for-web-developers/ http://liveweb.archive.org/http://www.coderholic.com/invalua...

perlgeek 15y ago

curl -I and wget -S are particularly helpful when debugging redirects. Sometimes I migrate URL schemes, and set up permanent redirects in my .htaccess files. Testing them in a browser is a real pain, because browsers cache the redirect (which is the point of having a permanent redirect), so even if you change the .htaccess, you still get the old response. And pressing the refresh button is no help, because that reloads the destination page, not the source of the redirect. That's when a non-caching command line client saves your day.

vld 15y ago

also useful: wget post-file.it --post-file /path/to/file

dexen 15y ago

lftp is a practical SFTP, FTPS and FTP transfer program, including automatic upload/download resumption and synchronization (mirror) mode. Good for both interactive use and scripting.

nikcub 15y ago
lftp can also continue downloads and run multiple concurrent download threads, acting as a download accelerator. for eg. lftp :~> pget -c -n 10 http://warez.com/large_file.mp4 will start 10 concurrent download processes (n) and continue (c). lftp is full of hidden gems.
LiveTheDream 15y ago
Synchronization (mirror) mode is absolutely invaluable when you only have access to a machine via FTP.

NathanKP 15y ago

For me the most invaluable command line tool that I have ever used is S3 Tools: http://s3tools.org/s3tools http://s3tools.org/s3tools It provides a great suite of tools for interacting with S3, and is best used on an EC2 instance you are connected to via SSH. It is also ridiculously fast, much faster than trying to interact with S3 from a local FTP browser, or even from Amazon's own S3 dashboard. For example on my computer using Amazon's own web facing dashboard it will take about 30-45 minutes to make 14000 files on S3 public, versus via the command line tool downloaded and running on one of my EC2 instances it can make those files public within minutes. I assume this is because it is local network traffic. Anyway, if you are ever in a bind and need to move a bunch of files to S3, I highly recommend S3 Tools. It has saved me many times. Along those lines wget is the most powerful command line tool I've ever used, with so much capability. It is simply incredible when combined with S3 tools, allowing you to easily grab gigabytes of images off of a personal server or staging location and upload them to S3 very quickly. And if you need to do more than move files around you can manage even more aspects of AWS including EC2 instances from the command line using this powerful command line tool: http://www.timkay.com/aws/ http://www.timkay.com/aws/ Its great for writing automated bash scripts to manage your instances via cron jobs.

epi0Bauqu 15y ago
My main complaint with s3cmd is you can't easily throttle it, and on a busy machine its fast speeds can cause issues.
- NathanKP 15y ago
  Good point. I guess in the past I have spun up a micro instance to run it, so I didn't have to run my S3 operations using one of my main production servers, and hence I didn't run into any problems with its unthrottled speed.
- delinka 15y ago
  Perhaps lowering its priority would help. nice s3cmd ...
- spiffytech 15y ago

sfoguy 15y ago

iMacros firefox addon for web automation via command line (when curl is not enough): http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Support http://wiki.imacros.net/iMacros_for_Firefox#Command_Line_Sup...

ryduh 15y ago

Note: ngrep can also be installed using brew on OS X.

LiveTheDream 15y ago
Also available via MacPorts.

genieyclo 15y ago

My most used cli tool outside of the default nuts and bolts is dtrx, the best and easiest file extractor for *nix. No more fiddling with flags or looking up, handles issues with putting lots of things in different directories or the wrong permissions on files. It has saved me a ton of time over the years. http://brettcsmith.org/2007/dtrx/ http://brettcsmith.org/2007/dtrx/

divtxt 15y ago
Wow! Just wow! Bookmarked! A mere upvote is not enough for this! Thank you!!! Questions: - Is there a site pointing out amazing command line tools like this? (their existence as opposed to usage examples like at http://www.commandlinefu.com/ http://www.commandlinefu.com/) - Is there a site listing the OS X equivalents for Linux command line tools? (e.g. what's the OS X equivalent for dos2unix / flip?) (edit: merge 2 comments into 1)
- genieyclo 15y ago
  Yep, check out http://onethingwell.org http://onethingwell.org which is exactly that. Software (incl cli) that do one thing well, as according to the UNIX philosophy. It's nicely tagged as well, so you can click the 'osx' tag and get tools just for OS X and so forth. I actually found dtrx on hacker news though a few years ago, but it's also on onethingwell I believe.
  - BasDirks 15y ago
    Amazing, cheers.
- deleted 15y ago
  [deleted]
ludwigvan 15y ago
Actually, for extracting tar files, using xf is enough. tar xf archive.tar.gz tar xf archive.tar.bz2 Still, this will be useful for other formats, I think. Thanks.

marcusramberg 15y ago

This was worth it for me for the reminder about siege alone, as 'ab' seems to have some serious issues in Lion.

bretthopper 15y ago

Ever wanted to know the progress of a long running operation? Copying a file, importing a mysql db, etc. Pipe Viewer (pv) is what you need: http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml. It lets you monitor the progress of any piped command and gives you time elapsed, speed, time left, and a progress bar (wget style). Copy a file: pv sourcefile > destfile Import mysql db: pv dump.sql| mysql -uroot -p db More tricks: http://blog.urfix.com/9-tricks-pv-pipe-viewer/ http://blog.urfix.com/9-tricks-pv-pipe-viewer/ edit: To install on OS X just do brew install pv

brendano 15y ago
Pipe Viewer dramatically increased my productivity for large scale data processing. In particular, it lets you quickly know whether something will take 5 minutes, or 2 hours, so you can plan accordingly. It's painful watching people try to do this without pv.

nobuff 15y ago

Oh-my-zsh with git and git flow plugins work well so far. https://github.com/robbyrussell/oh-my-zsh https://github.com/robbyrussell/oh-my-zsh

HerraBRE 15y ago

It's been mentioned on Hacker News a few times before, but my project PageKite (and showoff.io and localtunnel) is designed to help out with quick web demos and collaboration. Getting started is now two lines: curl http://pagekite.net/pk/pagekite-0.4.py >pagekite.py python pagekite.py 80 SOMENAME.pagekite.me ... answer a few questions and whatever is running on port 80 will be visible as https://SOMENAME.pagekite.me/ https://SOMENAME.pagekite.me/ within moments, almost no matter what kind of network connection you have. :-) There are also .deb and .rpm packages available for heavier users.