Crawling content by wget

2018-01-28 22:43:47

Here we will find how to use wget to scrape a target site. The process creates a mirror of the content on the local disk. you can use the tree utility to show the directory structure.

Crawling content by wget

wget -r -m -nv example.com

Show the directory structure


Useful grep search patterns

grep -r -i '<script'
grep -r -i '<script type="text/javascript" src="'
grep -r -i 'type=hidden'
grep -r '<!--'