[Webdev] Robots

[Anton McKee]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Thanks for all the responses. Yeah I think some playing around with
the dB and then Mod_Rewrite will probably do want I want. Then a large
custom robots.txt with user_agent rules. The biggest problem is
working out the URL mappings but it should not be too difficult.

Thanks
Aj

- -----Original Message-----
From: John McCormac [mailto:jmcc at hackwatch.com]
Sent: Thursday, July 05, 2001 3:49 PM
To: 'webdev at linux.ie'
Subject: Re: [Webdev] Robots


Anton McKee wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi Guys,
> 
> Hope everyone is well. We should have a f2f someday ya know.
> 
> Anyhow I was wondering, is there a way of telling visiting spiders
etc
> to ignore content in a page like a text menu etc.

The only way that I can think of is by using frames. The subframe
would
have the META tags to NOFOLLOW,NOINDEX or FOLLOW,NOINDEX. The
robots.txt
I think only applies to directories and agents. Alternatively what you
could do is to feed the search engines a fodder page based on agent
detection.

Regards...jmcc
- -- 
********************************************
John McCormac            * Hack Watch News
jmcc at hackwatch.com       * 22 Viewmount, 
Voice:   +353-51-873640  * Waterford,
BBS&Fax: +353-51-850143  * Ireland
http://www.hackwatch.com/~kooltek
********************************************

- -----BEGIN PGP PUBLIC KEY BLOCK-----
Version: 2.6

mQCNAzAYPNsAAAEEAPGTHaNyitUTNAwF8BU6mF5PcbLQXdeuHf3xT6UOL+/Od+z+
ZOCAx8Ka9LJBjuQYw8hlqvTV5kceLlrP2HPqmk7YPOw1fQWlpTJof+ZMCxEVd1Qz
TRet2vS/kiRQRYvKOaxoJhqIzUr1g3ovBnIdpKeo4KKULz9XKuxCgZsuLKkVAAUX
tCJKb2huIE1jQ29ybWFjIDxqbWNjQGhhY2t3YXRjaC5jb20+tBJqbWNjQGhhY2t3
YXRjaC5jb20=
=sTfy
- -----END PGP PUBLIC KEY BLOCK-----

_______________________________________________
Webdev mailing list
Webdev at linux.ie
http://www.linux.ie/mailman/listinfo/webdev

-----BEGIN PGP SIGNATURE-----
Version: PGP 6.0.2
Comment: http://www.broadcom.ie/keys/Anton-McKee.asc

iQA/AwUBO0V2K3bVSPTJOVkJEQIYiACfRnTXnLiTPjpJkelwCkKrg6QMfgMAoIou
vplv/Jwv80mAeza84uzOiPFy
=uMNy
-----END PGP SIGNATURE-----