hi ,
as all the web-masters i submitted my sitemap to google ,
and yahoo before 3 months maybe , i didnt face any problem
with "google" , but i do face a lot of problems with "yahoo spiders"
everyday , i mean the forum now got 4 members online , and 370 yahoo spider,
and the spiders are showing the index of the forum , nothing else
How do I know my facial skin type?::
Please note that the majority of skin care regimen sites on the web are .. " When combined with a good diet, the right dietary supplements can help keep
http://answers.google.com/answers/threadview?id=440352HOME |
and before couple of days there was about 500 yahoo spider in my forum
not doing anything , just bothering me i guess .
is there anything i can do to reduce "yahoo" spiders in my forum?
I only allow Google to index my forum. Here is what my robots.txt looks like.
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
I only see on the average 15 or 20 yahoo slurp bots at one time. Maybe 2 Google bots and the same for MSN in a 24 hour period.
Maybe it's your subject matter. Care to toss up a link? Wholesale distributors or manufacturers of 1/8 ::
I also used an incorrect e-mail address for the exact same question a little earlier vcedwards@yahoo.com (PLEASE DO NOT RESPOND TO THAT EMAIL ADDRESS).
http://answers.google.com/answers/threadview?id=250845HOME |
Help me find this movie's title!::
You guys have all got me thinking, and I stumbled upon a review of "The Omega Man" with Please read carefully the Google Answers Terms of Service.
http://answers.google.com/answers/threadview/id/391219.htmlHOME |
I only see on the average 15 or 20 yahoo slurp bots at one time. Maybe 2 Google bots and the same for MSN in a 24 hour period.
Maybe it's your subject matter. Care to toss up a link?
i see 511 yahoo bots at one time , i dont know why ,
and i see 3-4 google bots at one time , and same for msn
and googlebots and msnbots are normal , but 511 yahoo bot at one time
is not normal.
You can find a list to stop spiders from going to certain areas.
http://www.vbulletin.com/forum/showthread.php?t=76662
Googlebot crawls (http://www.google.com/support/webmasters/bin/answer.py?answer=48620)
Yahoobot crawls (http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html)
Msnbot crawls (http://search.msn.com.my/docs/siteowner.aspx?t=SEARCH_WEBMASTER_FAQ_MSNBotIndexi ng.htm&FORM=WFDD#D)
I would just complain to yahoo, as they are just trying to index the heck of every thing right now (Microsoft Takeover)
thanks ,
but this code or whatever :
User-Agent: *
Disallow: /
disallows all the spiders from indexing my site i guess , including google right?
and i found "set crawl rate " on google , but i didnt find it on yahoo .
any idea where is it?
User-Agent: *
Disallow: /
You can find a list to stop spiders from going to certain areas.
http://www.vbulletin.com/forum/showthread.php?t=76662
Googlebot crawls (http://www.google.com/support/webmasters/bin/answer.py?answer=48620)
Yahoobot crawls (http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html)
Msnbot crawls (http://search.msn.com.my/docs/siteowner.aspx?t=SEARCH_WEBMASTER_FAQ_MSNBotIndexi ng.htm&FORM=WFDD#D)
I would just complain to yahoo, as they are just trying to index the heck of every thing right now (Microsoft Takeover)
 Nortel Unveils Vision, Strategy for Israeli High-Performance Net
 Busy Friday Leads to Strong Close for Net Stocks
|
