SEO, Lies and Video Tape (part 3)

In the first first post of this series, I pointed out that SEO companies sell features that can be done easily by most people, thus avoiding cost. As a reminder, those 4 simple, easy-to-accomplish techniques are:

• Having a descriptive domain name
• Creating and submitting a sitemap
• Descriptive titles and meaningful content
• Getting other websites to link to you

Last time I discussed domain names. In this post I want to explore the next item:

Creating and submitting a sitemap

A sitemap is what the name implies - a map of your site. The point is that the automated indexing routines ("crawlers") from Google, Yahoo, Bing and others can work faster if they have a list of web pages to scan. And that's what a sitemap is.

NOT having a sitemap doesn't mean your site won't get indexed. But it does mean that pages might be overlooked or that updates to your site won't show up on search queries as quickly - all of which translate to lost visitors.

Submitting your sitemap isn't necessary - all the search engines will find it eventually unless you've named it something completely weird and/or stuck it into a stupid directory name like "golf scores". That having been said if you are being a diligent web designer, you can push the issue so that you remove all doubt.

In most cases, you will need an account with each of the search engines in order to submit your sitemap. That also shouldn't be an issue for you since, as a web designer, you ought to have those accounts anyway.

Also, let's be honest: Google is THE game in town. So it behooves you to get a Google Webmaster account (as well as a Google Analytics account). Neither cost you anything. I'm not going to take time here to go over all the bells and whistles of these tools, but you can get the ball rolling by going to http://www.google.com/webmasters/

To add your sitemap to Google:

1.     Sign in to your Google Webmaster account.
2.     From the dashboard, click the "Add A Site" button
3.     Go through the steps to verify the site
4.     Click on the site to bring up it's specific stats
5.     Click "Site Configuration" from the sidebar to expand the list
6.     Click "Sitemaps"
7.     Click the "Submit a Sitemap" button and follow the prompts

To add your sitemap to Yahoo!:

1.     From the Search Engines page, copy the link to your Sitemap file.
2.     Sign in to your Yahoo! account.
3.     Enter the URL for your site in the Submit Site feed field (e.g., http://www.yourdomain.com)
4.     Click Submit Feed.

Bing?

Creating a sitemap is very simple. The instructions below really depend on whether your site is a "regular" static site made up of a bunch of pages, or if it's more like a blog.

Regular Sitemaps

To create a sitemap for your regular site you have to generate it or write it by hand. If you immediately thought "oh let's do it by hand, that sounds exciting" then I'm done speaking to you. Please leave my website. I'll wait.

OK, now that the mouth-breathing village idiots have left the building, we can move on.

The easiest way to do this as a one-time-shot is to use one of the (many) online sitemap generators. For the sake of example, I'm using http://www.xml-sitemaps.com/ . But feel free to use any one you want.

1. Go to http://www.xml-sitemaps.com/
2. Enter the webiste URL
3. Enter your change frequency (that tells the web crawlers how often to come back and recrawl the site.
4. Fill in any other options based on the Sitemap generator you are using and click Go/Start/Run/Whatever
5. Once the process is complete, you be presented with downloadable versions of your sitemap in at least a couple of formats (xml, txt, html). Go ahead and pull them down to your local computer
6. Now upload them to the root folder on your website.
7. Finally, using the steps I outlined earlier, submit your sitemap to Google, Yahoo, and wherever else your fancy deems important.

Blog Sitemaps

Setting up a sitemap for a blog is even easier than for a regular site, because most blog software have add-ons or plugins to do the job for you.

For example, in Wordpress I recommend adding the pluging "Google XML Sitemaps". From there the options are very straightforward and it even submits the sitemap to Google for you.

TRICK: Skipping webpages and folders with robots.txt

While it might sound counter-intuitive at first blush, every site has folders and even web pages that you DON'T want to have show up on search results. Things like the "images" folder where you put all your webpage graphic elements, or the webpage "testme.html" which you use to test out new stuff before adding it to the live pages, or the "documentation" folder where you store all the design information about the website

(What's that, you don't HAVE documentation on your website? Here's some advice: Don't say that out loud to your customer.)

To get Google, Yahoo, Bing, etc to overlook pages, you use a robots.txt file. This file - also found at the root of your website - tells web crawlers which pages to look at and which to skip. In it's simplest form, it looks like this:

User-agent: *
Disallow:

This tells the web crawler that this file applies to ALL search agents, and there are NO pages disallowed. Using my example above, let's say you wanted to tell google NOT to index /images, /documentation and testme.html. Your robots.txt file would look like this:

User-agent: *
Disallow: /images/
Disallow: /documentation/
Disallow: /testme.html

While there is a lot more the robots.txt file can do for you, I want to leave you with one reminder: Robots.txt is a well-known filename that anyone can pull up on your website. So don't use it to try to hide things from visitors because robots.txt  is basically a big fat finger pointing to those directories saying "look here for good stuff". 

Make sure you check back here (or better yet, use "sign up" options in the sidebar to add this to your RSS feed or receive email notifications) for the next installment where we rise up out of the weeds of step by step instructions to talk about descriptive titles and meaningful content.