Preventing Google and Other Search Engines From Indexing Your Website Using Meta Tags
August 16th, 2009
2 comments
Not everything you post on the internet needs to be crawled, indexed, and cached by Google or any other search engines. For that very reason a robots.txt file was created, but sometimes we don’t need to get to granular in the robots.txt file, or we may not have access to edit it. For that very reason we can specific information to our META tags in our web pages. The robots field is a comma separated list, if you do not set it, it will automatically default to ALL, this means that the page can be indexed and that all links on the page can be crawled. Here is the api for the robots field… Read more…