want to mirror a website and ignore robots.txt:

wget -mkEpnp -e robots=off http://www.site.com