Webmaster Papers








The Role of the Robots.txt File to Improve Site Ranking!


Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located.

Here is examples of the use of the robots.txt file:

To block out a directory in a robots.txt file, such as a subdirectory for your online store called /store/ you would do the following: Disallow: /store/

Another example to block out your stats directory: Disallow: /stats/

You may also want to disallow individual files that you do not want searched by the search engines. For example you dont want search.php to be parsed by the Search Engines. To do this you type in the following on its own line:

Disallow: /search.php

Following the rules outlined and creating the robots.txt file, you will keep search engine spiders out of unwanted files and directories, and letting them go through the important files to see what your web site is all about!

Michael Kralj is owner of Emenki Web Solutions and Domains at Retail. Emenki Web Solutions are web site designers and programmers based in Hamilton, Ontario, providing businesses with an informative and strategic approach to establishing an online presence on the web.

Please visit Emenki Web Solutions & Web Design in Hamilton, Ontario on the web http://www.emenki.com

Please visit Domains at Retail - Cheap Domain Name Registration on the web: http://www.domainsatretail.com

RELATED ARTICLES


Analyzing Googles Backlinks Is Close To Worthless

Determining the Value of Your SEO Service
Every once in a while--and probably more often than we should--we find ourselves reviewing our SEO pricing models. Pricing SEO has always been a real sticking point for me because there is no one-size-fits-all pricing metric. As I began our most recent review of our pricing something really starting to become quite obvious; search engine optimization is requiring more and more research and analysis than it ever did before.
7 Essential SEO techniques
1) Title Tag ? When we're talking about SEO Technique, the Title tag is one of the best and most powerful tags that you can use. Every page should have it's own title tag, each tag should include the keyword that you are targeting, along with sub-keywords associated with the main keyword that you seek to drive traffic from.
Link Popularity Pitfalls
As we all know Google uses their PageRank technology to measure link popularity by counting the number of inbound links to your web pages, and it is one of the many factors influencing your ranking. Most website owners do not utilize properly what PageRank they already have. Their linking campaigns could be in vain if they let a large number of outbound links drain their existing PageRank. If their site were a bucket it would be full of holes, to illustrate this point, if you had a large number of outbound links, which are not reciprocated it would reduce your overall PageRank. Be diligent in identifying sites that are no longer linking back to you anymore. Also keep in mind putting more than 50 outbound links on any page is not advisable. Also be wary of sites that link to you from pages with more than 50 outbound links. Google's PagerRank is based on incoming links, but not only on the number of them. Instead PageRank is also based on the PageRank of the page on which your link is placed. For example a link to your site could be more valuable from a PR4 page with no other links than a link from a page with PR5 and 60 other links.
Finding Profitable Keyword Phrases
So you have a site concept developed that you are ready to start writing great content for.
How to Avoid Being Dropped by the Search Engines
For websites, one of the most important things in their existence is their ranking with the search engines. The reason why this is so important is because when websites are ranked high by the search engines, they get flooded with free, targeted web traffic from visitors who are looking for information or products. Obviously, any kind of activity that might hurt a websites chance of getting a premium listing should be avoided at all costs. The following tips are things websites should completely avoid if they want to optimize their results with the search engines.
How to Avoid the Google Duplicate Content Filter?
More and more webmasters are building websites with publicly available content (data feeds, news feeds, articles). This results in websites with duplicate content on the Internet. In cases of websites build on news feeds or data feeds you can even find websites that match each other 100% (except for the design). Several copies of the same content in a search engine does not really do any good and so Google apparently decided to weed out some of this duplicate content to be able to deliver cleaner and better search results.
Spamglish; A Search Engine Comedy With A Language All Its Own
When the movie Spanglish hit the screens in 2004, it was dubbed "A comedy with a language all it's own." I don't think the producers even knew they were slipping a lesson for website owners who want better search engine listings into the movie.
HTML Title Tags Dictate Your Rankings
There are a lot of things in Search Engine Optimization when done to your site that can increase you rankings. But one aspect that always comes up and that always affects your rankings in a huge way is altering the title tag.
Reality Check: A Straightforward Guide to Keywords and Search Engine Optimization (SEO)
Web site owners need to have realistic expectations when it comes to the world of keyword selection.
Ten Steps To A Well Optimized Website - Step 4: Content Optimization
Welcome to part four in this search engine positioning series. Last week we discussed the importance of the structure of your website and the best practices for creating an easily spidered and easily read site. In part four we will discuss content optimization.
The Search Engine Of The Future: Mobility Has A Pricetag
We all know the ease of using a search engine to garner information from around the world in microseconds. We've gotten accustomed to accessing these for the price of our hookup to the internet, be it dial-up, cable or wireless. But as more of our world connects via wireless media, what is included in our 'free look' may be changing.
Non-Reciprocal Link Building For Higher Search Engine Positioning
Non-Reciprocal Link Building For Higher Search Engine Positioning By Dave Davies, Beanstalk Search Engine Positioning, Inc.
Search Engine Optimization Strategies To Drive More Targeted Traffic To Your Website
Want to drive more targeted traffic to your web site?
Banned from Google and Wondering Why?
There are those that get on the computer one night and find that all of their Web pages have disappeared from Google. While, others are still in the search engine index, but don't rank high for nothing, not even for their Web site's name. It's a Web site owners worst nightmare, getting kicked out of the search engines.
Google is Quickly Changing...
Google is quickly changing...
How to Create Killer Keywords to Get FREE Traffic to Your Website
When exploring good keyword choices there are several steps one must consider. The most important step is putting your self in the shoes of the type of customer you want to target. Just make a list of the keywords or phrases you think that customer would enter into a search engine or directory when searching for your company or site. A good tool is WordTracker.com which will help you brainstorm different key phrases. Ask friends and associates which words they would use to search for your site on the internet, if they were looking for a site similar to yours.
9 Steps to Getting Better Search Engine Rankings
You finally have a website and you are ready to sit back and let the visitors start rolling in. How does that saying go? "If you build it they will come." This may be true in Hollywood but certainly not true in the case of your website. You need to constantly work at improving your site to ensure that your website achieves a decent search engine ranking. This is a process that doesn't happen over night but with some concentrated effort you can get better search engine rankings. Here are some tips on getting to the top of the search engines.
Search Engines Are Going to Love You for This
What's the most potent free traffic source on the web?
Search Engines: Different Types, Different Strategies
There are four basic types of Search Engines: