Introduction to search engines

This page is part of Ron Mertens' web page guides.

The internet today is dominated be search engines, and in particular Google (although they only have around 50% of the search market. Others are Yahoo and MSN and many more small engines). It's utterly important that you understand how the search engines work.

History

In the beginning (something like 1994 or so) there was nothing. You had two options of finding a web page - using the domain name (ah, the good old days of 'football.com') or asking Yogev, our real world web site directory friend.

Then the internet started to spawn directory sites. Yahoo is one of them - they just collected a lot of links and put them in their page.

After the directories, came the search engines. These sites "crawled" the web, and collected all the web sites automatically (using the links to hop from site to site, just like humans). You could them search for keywords and find relevant sites. The search worked by giving a score for each site, based on the site attributes - keywords in the text, domain name, meta information in the page, etc. I think the most known web search engine at the time was AltaVista. It was really cool, wasn't it?

Google

And then came Google. Google made a revolution because it was really quick, it had collected a lot of web pages and most importantly, the results were really accurate. So how did they do it?

The idea was that ranking sites according to the content inside them is simply not good enough. Google wanted to have some way of defining whether a site was "popular" or not. They used links to do it. If my site has 5 links to it, it must be more popular than your site that has only 1. The idea came from academic research, where an article is measured by the amount of times it is mentioned in other article. Pretty cool.

PageRank

Google works this way. It gives a pagerank to every page. This is a number. Then let's say you have PR = 100. If you have 2 links coming out of your site, each of them "contribute" 50 to the site they are pointing to. Google crawls the web, and computes the page rank in a recursive way, all the time. The search results are hugely effected by this. Note - the above explanation is hugely simplifying things, but it's good enough I believe. Google publishes a "simplified" page rank information, on a logarithmic scale (0..10). You can download the Google toolbar, and enable site information, and you'll see the page rank of every page you visit. My highest PageRank site is MRAM-Info which has PR=6.

Obviously if a high page rank is linking to you, that's great for your Google search position. Of course Google uses a really complicated algorithm to compute the search results - using Page rank, keywords, site analysis, and what not.

SERP

You'll probably stumble across the term "SERP". It means - "Search Engine Results Page". People will say "I've gone down in the SERPS for my keywords" which means that if you search for their keyword, their position in the result list has gone down.

Your position in the SERP of a search engine is called your "Organic Listing". It means that people will find your site when they search, and you don't have to pay anything. Obviously the number one site gets most of the searches, and that's the holy grail of your site promotion - getting to be number 1, especially in Google. There's nothing like free publicity!

SEO

Getting to be on the top SERP is not so easy, especially on competitive keywords. So how do you do it?

This "art" is called SEO - Search Engine Optimization. The idea is that you tweak your site form, content and other stuff, in order to get the search engines to realize you are the best site for a specific set of keywords.

In the old "AltaVista" days, people were repeating their keywords a lot of times, and they were trying tricks such as writing a keyword is very small text (So the search engine will see it, but the users won't) on with a "transparent text" (white text on white background for example).

The SEO is always a cat-and-mouse game. Someone thinks about something clever, and then the search engines notices that, and add some algorithm to eliminate it. For example, after Google came, people found out that Links are the most important thing. The obvious thing is to do a link exchange - I link to you, and you link to me, and everyone gets better SERP at Google. But after a short while, Google discounted links that were coming both ways.

There are companies that specialize in SEO, and there is a lot of discussion and action in this rather big market.
I'm not an expert SEO, but I do know that the best SEO tactic is to write a web page that is good. With good content, and good design. Make people want to visit your page. Then you'd be able to get good, real links from pages, and your SERP will rise.
Content is king!

How to get listed?

The first question people ask me sometimes is how to get listed in Google or other search engines?
Once you get links to your site, you will be crawled by the search engines sooner or later. You can also submit manually to most pages (Google has a web-site submit page) if you do not yet have any links to your site.

Google 'do not like' new pages. Usually it takes a lot of time to get listed for good keywords. They 'punish' new sites, and put them in a 'sandbox' for some time. It can be a while - sometimes even a year or more before your site is even listed in the good keywords.

What to do? Start early! don't delay, and just start your web site, even if the content or design is not 100% ready yet.

What's next?

One of the way to get visitors is to pay for them. And there's nothing like Google's AdWords system...