I'm encountering a problem getting the spider to work. Creating a robots.txt file did not seem to help.
Help: Choosing good keywords in Google Adwords::
Link Spider http://www.linkspider.co.uk/map/investing/ This one is very interesting as Does this help? Please do not hesitate to contact for a further
http://answers.google.com/answers/threadview/id/107346.htmlHOME |
Here is my robots.txt:
User-agent: *
Disallow: /include/
When I tell it to spider localhost or the hostname of the local server (depth=5) always results in this:
Spidering in progress...
--------------------------------------------------------------------------------
Search engines penalize dynamic pages?::
So the page should look 'static' to the search engine spider. Therefore I do not think that the quoted statement above applies to the sites I create.
http://answers.google.com/answers/threadview/id/252653.htmlHOME |
SITE : http://localhost/
Exclude paths :
- include/
There are no CPU cycles being used and the page is fully loaded. What else can I check? How will I know that the spider process is actually doing something?
Google Florida Update - Dropped Position::
Google has steadfastly refused to make any change that does not offer a benefit to In an ideal situation, a search engine spider might detect duplicate
http://answers.google.com/answers/threadview?id=283860HOME |
Thanks
acidelic
In config.php set up define('USE_IS_EXECUTABLE_COMMAND','0');
(It is the best search script!)
That did the trick!!! The spider page didn't load, but if I refresh the admin page I can see the counts going up. And I can actual search for things on the site. Thanks for your help Siava!
I took a look at the article and I don't think that is the problem. I have a single web server with one NIC. I'm trying to spider the local web server, but I never see any results. I don't see any progress on the page (which also says it is "Done") and it will sit there forever doing nothing. I tried spidering some other sites but nothing happens with those either.
What else is there that I can check?
Hi. Try this (http://www.phpdig.net/showthread.php?threadid=514) thread.
 Nortel Unveils Vision, Strategy for Israeli High-Performance Net
 Busy Friday Leads to Strong Close for Net Stocks |
HOME