Author Topic: Importance of robot.txt  (Read 632 times)

Offline simonjhons16

  • linkGroup
  • *
  • Posts: 17
  • Gender: Female
    • View Profile
    • Dynaweb
Re: Importance of robot.txt
« Reply #5 on: February 04, 2012, 10:01:25 AM »
Hello Friends..........

Using a robots.txt file I think is really up to the individual site. I use it to disallow images and certain files as well as the ia_archiver.

Thanks

Offline simonjhons16

  • linkGroup
  • *
  • Posts: 17
  • Gender: Female
    • View Profile
    • Dynaweb
Re: Importance of robot.txt
« Reply #4 on: January 27, 2012, 02:10:37 PM »
Hello Friends..........

It is a simple text file that is created using any text editors. This file provided essential information to the search engine .This is mainly used for Search Engines to Navigate and Index a website. Best Practice robots.txt file improve search engine rankings for the site. So this is one of the important file to have nice ranking on Search engines and for SEO.

Thanks

Offline carolinehill

  • linkGroup
  • *
  • Posts: 36
  • Gender: Female
    • View Profile
Re: Importance of robot.txt
« Reply #3 on: January 03, 2012, 01:21:59 AM »
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.yourwebsite.com/robots.txt

___________________________________________________
Mobile App Development | iPhone App Development | Web Designers

Offline jaman017

  • linkGroup
  • *
  • Posts: 34
    • View Profile
    • Email
Re: Importance of robot.txt
« Reply #2 on: September 16, 2011, 10:18:28 PM »
hello
Robots.txt file is a very important file if you want to have a good ranking on search engines, many websites don't offer this file. A Robots.txt file is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc. It defines which paths are off limits for spiders to visit. This is useful if you want to hide some personal information or some secret files.

Webmaster Questions

Offline kimanderson

  • Newbie
  • *
  • Posts: 1
    • View Profile
    • Email
Importance of robot.txt
« Reply #1 on: September 13, 2011, 04:22:09 PM »
Hi Friends

Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.
« Last Edit: September 13, 2011, 04:24:55 PM by kimanderson »

 

Entrecard



Our Sponsers

BLOGS 4 SEO
visit tour trekking tourism in nepal

Add to Google Reader or Homepage

Subscribe in a reader

   
anything