Skip to main content

Yahoo's Crawler Not Listening To Robots.txt Directive?


A WebmasterWorld thread reports that Yahoo may not be fully listening to the robots.txt directive to block their spider, Yahoo Slurp.
The thing is, Yahoo spider isn't all that active these days - because Bing is now powering much of Yahoo and thus BingBot is most active.
The webmaster said:
Depending on the Host and UA, the official Yahoo! Slurp apparently does whatever it wants to. Note the subtle differences in the subdomains and UAs...
This morning, the only Host to read/heed robots.txt was:
b3091154.crawl.yahoo.net [67.195.112.189] Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
These retrieved graphics by the pageful, over 60 total:
b5101137.yst.yahoo.net [98.137.72.218] b5101139.yst.yahoo.net [98.137.72.228] Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
I am not sure if this is a widespread issue or something that is just a smaller bug.
The main question is, should you care of Yahoo is crawling your site when Bing is? That discussion is also taking place in the forum thread. The answer is, it depends.
Forum discussion at WebmasterWorld.
Labels:

Comments

Post a Comment

Popular posts from this blog

List of Real Estate Forums

Hello SEO, find below list of real estate forums which will help you generate valuable links for your seo promotion. http://mypropertyreviews.com http://www.indiahousing.com/india-real-estate/forums.html http://www.realestateforum.com/forum.php http://www.realestatetalks.com/ http://www.realestatewebmasters.com/forum/ http://www.agentsonline.net/forums/ubbthreads.php http://www.topix.com/forum/business/ http://www.reiclub.com/forums/index.php http://www.indianrealestateforum.com/
5 comments
Read more

PPC Training Course sector 22 Noida

Pay Per Click or simple PPC is method of paid advertisement on search engine and their partner website. Need of ppc increasing every day because of frequent changes in google updated which punish low quality websites. PPC (Pay Per Click) is method for instant leads generation, you can start getting leads within 2 hours and it is highly measurable with the use of ppc management tools like google adwords and bing ads center. It is most cheapest method of marketing. We are starting PPC training course at sector 22 noida with the aim to provide quality ppc training to our students which will leads good job opportunity in reputed companies. Best part is that you doesn’t need any type of technical education to learn ppc. Anyone who is 12th pass, graduate, under graduate, post graduate can join this course and change his fortune. To Join: Call at: +91-9555758092    Visit: www.pavasha.in
23 comments
Read more

Keyword Rich Domain Names Next To Be Hit By Google

It should come to no surprise of readers of this site that Google and Matt Cutts next target will be domain names ranking too well on the merit of the keywords being in the domain name. We covered several times already on how well keyword match domain names rank in the Google search results. We also documented that Google's Matt Cutts said he   would look at keyword match domains   while giving a presentation at   PubCon. In any event, new news. Google's Matt Cutts posted   a video   recently saying they are going to look at this stuff, again. Here is the video:
5 comments
Read more