Using wget
Posted on March 13th, 2006 by Erin
I use wget from time to time (via cygwin when I am in the office) but the most I have used on its parameter is the -c option. This post is a reminder of the options and tips that I could use and have used for the wget utility. Why not just consult the manpage? It is because it is easier to put it here for further reference. There are a thousand wget tutorial sites out there, so why waste space here? The answer to that is because I can. 
Usage: wget [options] URL
- Instead of specifying the proxy using the environment variable http_proxy, consider putting it in /etc/wgetrc instead. Not advisable if you dont have a consistent single source of connection (e.g. prepaid cards,etc)
- -c – continue the file being downloaded. nifty if you are downloading a big file although it requires the download site to support resuming.
- –user-agent=”UA_STRING” – based on a tip from the PLUG list. Spoofs the user agent to look like the download request is from a browser. This is good for sites that disallow dowloads from download managers. Example would be
-user-agent="Mozilla/4.0 (compatible; MSIE 5.0b1; Windows NT 5.1*) " - –limit-rate=[number] – limits the download rate so wget does not hog all the bandwidth. Appending a k at the end of the number signifies kilobytes per second.
More updates as I use more options. 
[...] This week I encountered an error when writing the wget post since saving or publishing it causes a page not found (or 404 in techie parlance) error. I found this thread in the wordpress support forum and tested the “limit” scenarios which did not fully apply in my case. Posting a short test message works but somehow even one paragraph from my wget post causes the 404 message. [...]