| Advertising Networks / Pay Per Click / SEO / Traffic Discuss AdSense and other advertising networks. Search Engines Optimization Problems & Solutions. How to monetize your websites. Pay Per Click Discussions. Best ways to attract / generate traffic. Best ways to make money with AdSense and other networks. Where and how to get backlinks. |
 |
| | LinkBack | Thread Tools | Display Modes |
10-15-2011, 09:20 PM
| #1 (permalink) |
| Status: Member Join Date: Oct 2011 Posts: 43
 |  robots.txt file is important for all websites ? I read robots.txt is used to allow , or disallow crawler to particular page....so i dont add robot.txt file to my site . robots.txt file is important for all websites ? thanks in advance
__________________ Mua ban nha dat |
|         |
|
04-09-2012, 02:56 AM
| #2 (permalink) |
| Status: Member Join Date: Oct 2011 Posts: 49
| yes robots.txt file is important but not mandatory for websites. it prevents your site from bots. Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from XnYnZ.com |
|
| |
|
| Thread Tools | |
| Display Modes | |
| |
Linear Mode
