[ILUG] wget receipe

[Thomas Pedoussaut]

Justin MacCarthy wrote:
> Hi
> 
> I want to use wget to give me a list of all URIs referenced on my domain 
> (all only my domain), so I can remove old files. I'm sure there is way 
> to do this with wget, but I just can't see it, (really bad flu today) I 
> don't want to download anything just list the documents so I can clean 
> up a local copy

I didn't wanted to go to this quick and dirty hack, but because nobody 
came with better in 24 hours...

The idea came from a section of wget man page.

wget -r -nd --reject gif,jpg,png --delete-after http://whatever.com/

That will request all the linked documents on you site, but won't store 
the retrieved data.
Then just after that, retrieve the logfile of your webserver, grep the 
IP you connected from, cut the 7th field (cut -d' ' -f7) and you'll get 
your full list.

You may also ask wget no to retriev files that are obviously "leaves" of 
you web tree like images.

-- 
Thomas