Topic: Importance of robot.txt

[simonjhons16]

Hello Friends..........

Using a robots.txt file I think is really up to the individual site. I use it to disallow images and certain files as well as the ia_archiver.

Thanks

[simonjhons16]

Hello Friends..........

It is a simple text file that is created using any text editors. This file provided essential information to the search engine .This is mainly used for Search Engines to Navigate and Index a website. Best Practice robots.txt file improve search engine rankings for the site. So this is one of the important file to have nice ranking on Search engines and for SEO.

Thanks

[carolinehill]

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.yourwebsite.com/robots.txt

___________________________________________________
Mobile App Development | iPhone App Development | Web Designers

[jaman017]

hello
Robots.txt file is a very important file if you want to have a good ranking on search engines, many websites don't offer this file. A Robots.txt file is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc. It defines which paths are off limits for spiders to visit. This is useful if you want to hide some personal information or some secret files.

Webmaster Questions

[kimanderson]

Hi Friends

Robots.txt file contains some kind of text(not HTML)which you put on your site to tell to search engine which page you would like them to visit and which page you wouldn't like them to visit.