Webmaster Papers








Effectively Using Robots Meta Tags


The "robots" meta tag, when used properly, will tell the search engine spiders whether or not to index and follow a particular page. For the purposes of this article, we will be using the "( )" symbols to represent the "< >" in html coding.

Some examples of robot usage are as follows:

(meta name="robots" content="index,follow")

(meta name="robots" content="noindex,follow")

(meta name="robots" content="index,nofollow")

(meta name="robots" content="noindex,nofollow")

Let us first examine what these terms mean before we explain the usage for each one:

"index"- This directive tells the search engine robots (or spiders) that it is okay to index the page. Another words, you are allowing the search engine to include your page within their search directory.

"noindex"- Using this tag, you are informing the robots that this page should not be indexed. Simply put, this page will not appear in their search directory.

"follow"- When you use this tag, you are telling the search engines that you want their robot to follow any links that are found on that page.

"nofollow"- The opposite of the above definition, this directive will tell the robots not to follow any links on your page.

Putting It All Together:

With the robots tags explained, let's examine the usage for each one.

1. (meta name="robots" content="index,follow")

This tag will be used when you want the search engine spiders to index the page and follow the links to other pages. Most search engines use this setting as a "default" setting. It is possible that you may not even need to use this tag if you want the search engines to follow and index the page. However, an article at Search Engine World (searchengineworld.com/metatag/robots.htm) suggests that Inktomi does not use this as their default setting. Instead, they use the "index, nofollow" tag.

Better safe than sorry!

There has been much debate over whether or not it is necessary to use this tag. If there is even a slight possibility that some search engines do not use this as the default setting, then it would only make sense to include this tag if you want your page included in their search directory AND your links to be followed. Do the research and decide for yourself.

2. (meta name="robots" content="noindex,follow")

This tag can be used to tell the search engines that you do not want the page included in their directory, but you DO want them to follow the links that lead to other pages. A good example of its usage would be your disclaimer or privacy policy pages. You may not want these pages to show up in the search engines if they are only important to your actual visitors. However, if the links on these pages point to other pages that you want the search engines to find, then you would still want the spiders to "follow" those links.

3. (meta name="robots" content="index,nofollow")

This tag will allow your page to be indexed in the search engines, but any links on that page will not be followed.

4. (meta name="robots" content="noindex,nofollow")

When using this tag, the search engine spiders will not include this page in their directory and will not follow any links on the page either.

Where does the "robots" tag belong?

The "robots" meta tag should be used within the (head) and (/head) tags of your page. These tags are located at the top of the html coding. It will look something like this:

(html)
(head)
(title)Title of your page goes here(/title)
(meta name="keywords" content="word1,word2,word3,word4")
(meta name="description" content="A brief description of the content of this page.")
(meta name="robots" content="index,follow")
(/head)
(body)
Your webpage information here.
(/body)
(/html)

More Robots Tags

Google automatically archives a page as it crawls it. This is called a "cached" version of the page. Visitors can retrieve the archived version of the page by clicking on the "cached" link within Google's search results. If you do not want your content to be archived, you can use the following tag:

(meta name="robots" content="noarchive")

*This will only prevent your page from being "cached". If you do not want your page to be indexed at all, you will still need to include the "noindex" tag.

Another alternative to the above tag is the tag that specifically addresses Google only. If you want other search engine robots to archive your site, but you would like to prevent Google from doing so, then you can use the following tag:

(meta name="googlebot" content="noarchive")

The Misuse of Robots Tags

Something that has been popping up on websites everywhere is the Google indexing tag. This is a silly little tag that is not necessary. Some people think this tag helps Google to spider your site, but this simply isn't true. The tag looks like this: (meta name="googlebot" content="index,follow"). Some website owners believe that by specifying "googlebot" that their site has the advantage of being spidered faster and listed by Google. According to Google's web crawler information, you may use the noindex, nofollow, or noarchive tags when you DO NOT want Google to cache, index, or follow that page. Google's default setting is to index and follow the links on the page, so this "so called" googlebot index/follow tag that some site owner made up one day is completely unnecessary.

Another silly little tag--- The "Revisit-After" Tag

(meta name="revisit-after" content="90 days")
(meta name="revisit-after" content="15 days")

I'm not sure where this myth was started. Today, you will find this tag all over the Internet. Webmasters have even promoted it, claiming that it actually works. Are we so naive to believe the search engine spiders need to know when to come back? I have never used this tag, and my site has no problem with being crawled on a regular basis. Even some SEO (search engine optimization) sites are claiming its value. This comes back to the importance of researching the topic thoroughly first. My research came from some interesting information on WebmasterWorld.

It is important to examine the correct usage of the "robots" tag before applying it to your website. Incorrect usage of tags could result in spidering errors that cause robots to completely ignore the page all together. If you are interested in learning more about web robots, this great little site will provide you with the information you need to use them effectively: www.robotstxt.org/wc/robots.html

Stephani Richardson is a work at home mother of 4 who has been actively involved with affiliate marketing and home business opportunities since December 1999. She owns and operates several business related websites including 1 Work At Home Dot Com.

RELATED ARTICLES


HTML Editors 101 - Smaller Is Better
SiteSpinner
Optimal Website Design
Optimal website design is the art of logical navigation.
Server Stats ? Analyzing Traffic To Your Site
Analyzing traffic to a site is a key factor in tweaking search engine optimization and marketing campaigns. Determining how people get to your site is critical. Surprisingly, few sites take advantage of the data available on their servers.
How to Get a Better Web Site ROI
Better Web Site ROI
Getting One-way Inbound Links: the 5 Major Strategies
With search engines putting a damper on direct reciprocal links, the hunt for the elusive one-way inbound link is on.
Art, Artists, and the Web:Part: 2--First Steps in Building an Artist?s Website
What you should do and know if you are an artist and you have decided to have a website.
Ten Major Tips to Develop a Multilingual Web Site to Work
If you are living in a country that its native language is something rather than English language, then you may like to develop your website to offer content in the language of your own country.
Improve Conversion Rates ? Load Times
Your site is getting traffic, but conversion rates are horrible. Everything seems to be working, so what's wrong? Your site may simply be to slow.
Running An Effective Website
Running an effective website is a continuous process. I have compiled five (5) ways of doing it on a regular basis.
9 Keys to an Effective Website
Many small companies choose not to invest in a professionally designed website, although a well-designed Web site may raise the company's level of professionalism and exposure ? AND drive business and sales!
How To Make Your Website More Successful? (Part I)
Building a website and getting it online is easy. Driving visitors to it is the more difficult part. Most people are not patient enough when it comes to build up traffic. They expect thousands of visitors a week after they go live with their website. But that is not how it works. We share some secrets of how to make your website more successful.
Why Do I Need A Web Site?
Even though the Internet has been around for a long time and many people are "educated" about the Internet, most have little knowledge about what a web site is and what can it do for their business.
High Quality Website at Low Cost
Although designing a great web site is an important part of developing a successful online business, it is often becomes a problem for newcomers to the internet business.
A Web Site for Your Small Business: How to Set Up a Website
Your web site will be your window to the world. Even if you are not selling any product online still you need your customers to access you 24 hours a day 365 days in a year and you can pass important information to them by hosting frequently asked pages, product descriptions, product improvements etc. Your web site will be your office or interface to the world even when you are away in weekends, sleeping at nights or in an important assignment for your next business expansion. Then how to go about it? We will discuss some basic points here.
Whats Your Business Online Attitude?
This article was prompted by some customer support work I was involved in for a private members site. One of the paid subscription members wanted someone to look over his website and give an opinion of what we thought of its design.
You Don?t Have to be Amazon.com to Achieve 12% Conversion Rates!
That's right. According to a recent study by Nielsen/NetRatings, Amazon.com converts 12.8% of its visitors into sales.
How to Listen on the Web
When building a relationship, listening is more important than talking.
The Top Ten Benefits of Having a Web Site
Do you need a web site? Are you considering getting one but are unsure? Here we take a look at the main reasons why a web site could be beneficial to your organisation.
Entangled in the World Wide Web
I am just a learner who likes to delve in websites as a hobby. Hobbies are what make life so enjoyable for me. I have tried many other things - watercolor and pastel painting, organic gardening, kayak making, dinghy sailing, kite making, origami, stock photography, cartooning, carpentry, rock collecting, snake rearing and many other unusual pursuits. Somehow, as time goes on, and for some particular reason, they gradually become just bits of historical and passing interests for me. Amidst these passing interests, the Internet comes along.
Website For The Company Store
If your company own one or more stores, you might be wondering if it would be a good idea to have a website. You might also be wondering if a website for the company store would be profitable for your business.