Webmaster Papers








SEO Expert Guide - Search Engines Explained (part 1/10)


Before we explore the world of search engine optimization, it is vital that you know a little about how search engines work and their relative market shares. It will help you to prioritize your activities later!

(a) What are Search Engines and who powers them?

There are essentially four different parts to a typical large search engine; the crawler, the directory, sponsored results and the search engine itself.

Crawlers (e.g. Google) automatically visit web pages to compile their listings, making use of a so-called robot or spider (eg. Googlebot), which follows links from one website to another, ultimately compiling an index of all the pages and sites on the internet. These crawlers provide an index, which can then searched by the search engine. You may find that several or all of the pages on your site are indexed in thisway. Some search engines have their own crawler and others buy-in crawler results from others.

Human-powered directories, such as the Open Directory, rely on submissions from the public, which are reviewed by editors for inlusion in the directory. If you get included in a directory, generally only one page from your site (usually your home - or index - page) will be listed.

Crawled results are combined with sponsored results, supplied by pay-per-click (PPC) advertisers, and the results from human-maintained directories to complete the search engine index. Check out the Search Engine Reationship Chart at Bruce Clay inc. for the latest picture on who powers whom. You will note a couple of things right away. Firstly, the dominance of the Google and Yahoo! crawlers and secondly the importance of DMOZ directory results as a back-door for many search engines.

(b) How do Search Engines find and rank sites?

Search engines do not really search the web directly, but rather an index database of the full text of web pages, which itself is drawn from the billions of web pages on the internet's servers. Search engine databases are selected and built by computer robot programs called spiders.

If a web page is never linked to by any other page, spiders cannot find it, unless the (usually new) site is submitted manually by a human at the search engine's "add URL" page. All search engine companies offer ways to do this.

After spiders find pages, they pass them on to another computer program for "indexing." This program uses an "algorithm" to assess the text, links, and other content in the page for "key words" that might be searched on at the engine. This allows the search engine to order results served by their "relevancy" to the search terms used. As each search engine has a different algorithm, it will index sites in a different way and thus serve up different relevant results.

Some types of pages and links are excluded from most search engines by policy. Others are excluded because search engine spiders cannot access them. Generally, the use of frames, flash graphics and dynamic URLs all get in the way of effective spidering and should thus be avoided.

In addition to indexing pages, most algorithms seek to establish the "authority" of a site. A site which is linked to by many other sites (using keyword-rich anchor text) is assumed to be of greater merit than one with no links at all. This activity is called "ranking" and helps search engines to sort otherwise similar results into ever-more relevant and authoratative results.

(c) Which Search Engines are the most popular?

Based on US analysis in January 2005, the top search engines (by share of total searches at home and work) are as follows:

Google Search - 47%

Yahoo! Search - 21%

MSN Search - 13%

All Others - 19%

These shocking figures do not convey the true dominance of the top players, as you have seen from the interdependence of search provision in section (a) above. You could be searching at AOL (part of the "other" 19%) and viewing Google results, for example.

There is also strong anecdotal evidence that Yahoo! and MSN tend to send more searchers through to their sponsored (or paid) results than do Google (due to the prominence of these results on their results pages). As such, for a typical small webmaster who does not use pay-per-click (PPC) advertising, they might get up to 80% of all their traffic from Google's various sites across the world.

Now you understand the market a little better, you will perhaps understand the obsession many webmasters have with Google! A top-10 position at Google for your key search terms can make your online business fly. If you drop out of that top-10, your business can literally collapse overnight!

Don't forget these key stats as you embark on your optimization journey...

Navigate the guide

Previous : SEO Expert Guide - Index of Contents

Next: SEO Expert Guide - Proposition Development (part 2/10)

About the author:

David Viney (david@viney.com) is the author of the Intranet Portal Guide; 31 pages of advice, tools and downloads covering the period before, during and after an Intranet Portal implementation.

Read the guide at http://www.viney.com/DFV/intranet_portal_guide or the Intranet Watch Blog at http://www.viney.com/intranet_watch.

RELATED ARTICLES


Can Invisible Text in CSSs Slip Under Search Engine Radar?
I'm literally inundated with questions on the subject of invisible text & hosting so in I thought I'd debunk some myths and give you the facts straight up.
The Ultimate Free Google Ranking Tool
The first months my website was online, I was constantly checking the search engines to see if my site was listed under the keywords that I was targeting. And always with the same negative results.
Yahoo!/Overture Site Match: A License To Steal
Unless you've been living in a cave somewhere, I'm sure you've heard by now, Overture now offers the Yahoo! Search Inclusion under its own branded name--Site Match.
Site Maps: Let Search Engines Find Your Pages
With 40 million websites in existence, and more than 3 billion web pages indexed by Google at the time of this writing (July 2003), it's no wonder that more and more people are relying on search engines to find their way through the unruly world that the web has become.
Going To Market: Keywords and Backlinks, Part 1
As I waded into the 'Make Money Online' waters on the Internet, I felt I needed to do some research on what keywords might help me generate traffic for my websites. I am very, very new to this whole Internet marketing arena but at the simplest level it seems that the keys to generating income are:
Do the Robot!
Everyone should realize that the search engines (sponsored ads aside)are not tools for advertisement, they are meant to be tools for everday web users. Users who search the web are looking for information, thats it. They may want information on how to buy something, or they may simply want to what the weather will be later that day, but the fact is they want information, and search engines are out there to help them find the information!
How to Boost Your Traffic and Profits with Content!
Are you aware of how vitally important and valuable CONTENT is to your online business? In fact, content can do more to build your business and profits than just about any other resource or service available.
3 Principles Of Google
When online "Use it. Use it. Use it."
Optimizing Your Website
Search Engine Optimization, optimizing your website for it to be visible in the search results of a search engine's query or in a search result of a directory.
How to Avoid the Google Duplicate Content Filter?
More and more webmasters are building websites with publicly available content (data feeds, news feeds, articles). This results in websites with duplicate content on the Internet. In cases of websites build on news feeds or data feeds you can even find websites that match each other 100% (except for the design). Several copies of the same content in a search engine does not really do any good and so Google apparently decided to weed out some of this duplicate content to be able to deliver cleaner and better search results.
Tread Towards A Successful ?Internet Research?
Internet is a terrific resource containing billions of web pages dedicated to thousands of topics. Since the amount of information available on the Internet is so vast and mind baffling you may feel lost.
Link Popularity Pitfalls
As we all know Google uses their PageRank technology to measure link popularity by counting the number of inbound links to your web pages, and it is one of the many factors influencing your ranking. Most website owners do not utilize properly what PageRank they already have. Their linking campaigns could be in vain if they let a large number of outbound links drain their existing PageRank. If their site were a bucket it would be full of holes, to illustrate this point, if you had a large number of outbound links, which are not reciprocated it would reduce your overall PageRank. Be diligent in identifying sites that are no longer linking back to you anymore. Also keep in mind putting more than 50 outbound links on any page is not advisable. Also be wary of sites that link to you from pages with more than 50 outbound links. Google's PagerRank is based on incoming links, but not only on the number of them. Instead PageRank is also based on the PageRank of the page on which your link is placed. For example a link to your site could be more valuable from a PR4 page with no other links than a link from a page with PR5 and 60 other links.
Improve Search Engine Rankings - The Real Deal!
Ok, here's the deal, follow these steps and shoot me if your rankings doesn't improve. I know that there's been so many articles on how to improve your search engine rankings but most of them are either incomplete or untrue. So I've put up a list of what works best to improve your rankings and I'm telling you now this works but it's no walk in the park.
Got Spiders?
Many internet marketers blow mountains of start-up cash on their websites just trying to break into search engine rankings. I was one of these internet marketers.
Analyzing Googles Backlinks Is Close To Worthless

SEO and the Outsourcing of Inbound Link Building
Search Engine Optimization nowadays has a lot to do with building inbound links to your website. Building inbound links is a cumbersome tasks and webmasters have always been looking for shortcuts to do this. Webmasters buy links (as advertising as an example) or contact other webmasters to exchange links with them. The need for inbound links has created a new business opportunity in the search engine optimization industry. The outsourcing of link building emerged from the fact that many inbound links mean a high search engine ranking and/or a high Google PageRank.
Click Click Boom: a Linking Strategy that will Blow Away Your Competition
Web marketers, do you hear what I hear?
Ten Steps To A Well Optimized Website - Step 1: Keyword Selection
This is part one of ten in this search engine positioning series. In part one we will outline how to choose the keyword phrases most likely to produce a high ROI for your search engine positioning efforts. Over this ten part series we will go through ten essential elements and steps to optimizing a site. Some steps take a few hours, some may take months depending on the competition, but in the end and if done correctly you will have a well optimized site that will place well and hold it's positioning.
A Classified Way To Drive Business To Your Web Site
There are more than 105 million of them in the United States. Worldwide, there could be at least 250 million of them. Them, according to statistics from the Nielsen/Net Ratings service, is the number of active Web surfers. 250 million in the whole world? The figure is more than the populations of Canada, Australia, Great Britain, and a few non-English speaking countries combined. That's a lot of them!
Top 5 Search Engine Optimization Mistakes
There are a lot of ways to promote your website and, unfortunately, a lot of these methods are mistakes. Here is a list of some of the more common mistakes (often referred to as Black Hat SEO) that you should steer well clear of.