[ILUG] Personal web indexer

[Burke, Gary : GIG IS&T]

]> ... and history database too, to pick up URLs, I think the hist db is
]> longer-lived...
]
]Possibly true, but then it would have to go and download all those
]pages that have history entries but no cache entries.  Not nice over
]a dialup.

Agreed.  Unless if can do it in the background when it knows the connection
isn't busy (if indeed it can know!).  So while you are reading the big long
Slashdot article, or whatever, this is continuing to catalog and index your
history.


]> Sounds cool. But have you worked out if it's possible to get netscape to
]> dump the URLs/content of pages it's looking at?  Or do you mean use this
]> prog as a proxy? That'd slow down browsing a little :(
]
]A proxy is the best design, because it is then browser-independent
]and could run on a different machine.  It wouldn't slow things
]down very much, unless you've got a seriously fat pipe to the net.

I wonder if it would be possible to do this using a few threads.  One would
get the info from the site and immediately pass it back to the browser, but
at the same time send a copy to another thread which would do the processing
of the page, or store it for later processing, or whatever.

Main problem I can see is - what happens when you click 'stop' on your
browser?  I don't know much about proxys - does the browser send a stop
command which will tell the proxy to stop download that page, or will the
proxy continue to download everything?

Another possible idea with a proxy is that it can capture addresses for you,
and intrepret them.  So you can send it http://proxy_command/whatever, and
the proxy will not search on the internet for it, but return a page
responding to that command, eg. http://proxy/search, http://proxy/flush,
http://proxy/list, etc.  Stopping Netscape from trying to auto-make it into
a www.proxy.com might be difficult, if indeed it is necessary.



Gary
**********************************************************************
This e-mail and any attachment contains information which is private
and confidential and is intended for the addressee only.
If you are not an addressee, you are not authorised to read, copy or 
use the e-mail or any attachment.  
If you have received this e-mail in error, please notify the sender 
by return e-mail and then destroy it.
**********************************************************************