Isn’t amazing to think that despite the millions of different websites and billions of different web pages that somehow, someway the search engines still manage to find your site? I personally find this to be rather mind-boggling. In fact, it kinda makes my brain hurt just thinking about it.

Searchbot 101

Search engines have programs called searchbots that send out “spiders” to crawl the Internet and visit these millions of different websites. Once a spider finds your site, it reads every page and then follows every link on your page and reads those pages, as well. A spider then will revisit your site on a regular basis looking for new content and changes to existing content. After it visits your site, your content is then indexed on a search engine. Only after your content has been indexed, can it actually be found on search results of Google or Yahoo or MSN. When a user types in a search term, the search engine itself analyzes these millions of pages that have been indexed to match results of the search term and determine how relevant your site is to that search term. This is how your site will or will not appear in the search results of any given query.

Search Me, Baby!

A robots.txt file is used to tell the searchbots what file and directories on your site can and cannot be searched and indexed. You may have directories, like an admin folder, that you don’t want search engines crawling. It’s in this file that you make that clear to the searchbot.

There’s really not much to the robots.txt file. It’s a simple text file and usually only has a few lines in it, unless your site is unusually big. A typical file looks like this:

User-agent: *
Disallow: /wp-admin
Disallow: /images

User-agent:* means that following instructions applies to all robots. Then you list, specifically, each directory or file that robots are not allowed to index using the Disallow keyword. Once you’re done with that, save the file and make sure to name it ‘robots.txt’ and then upload it to the root of your site. You can use a tool like Google’s Webmaster Tools to validate it.

Click here for more detailed information.

1. Facebook
   I’m not a huge Facebook person, but if you’re somebody who needs to check on everyone’s status all the time, then this is a good one to have. It’s the most fully-featured Facebook app available, from what I know. Uploading photos to your Facebook account really simple. It has message notifications. It will interact with your phone’s calendar and contacts. Very solid app.
2. Ubertwitter
   This is, by far, my favorite Twitter app for the blackberry and it has been since they first released it. With every new release, they seem to be really addressing the existing problems while still adding lots of new features. As twitter apps go, this one is as solid and fully-featured as it gets.
3. Pandora
   I can’t say enough about this app. I have discovered so much new music from listening to Pandora. Even before it really became a mainstream product it was a very useful and enjoyable app. But then when it became a mobile app, it changed everything. I love, love, love the fact that I can listen to Pandora in my car. I probably listen to Pandora more than I listen to my personal library. On a side note, however, I would like to try Grooveshark, so if any of you have downloaded that app, please post your opinions and tell us what you think and how it compares to Pandora.
4. Poynt
   Poynt is a wonderful little app to have on your phone. They keep adding new features to it. Most recently they added gas prices and weather. What I like about it the most, is that they really seem to have thought of everything the user would want to do after they’ve found whatever it was they were looking for. For example, after finding a restaurant, it gives you every option you want like saving it to your address book or calling directly from the app or mapping a route to the listing or looking at the restaurant details. If it’s available, they’ll have a link to OpenTable so you can make a reservation. The list goes on an on. It’s a really nice app and very well developed.

When you’re first starting a blog in a particular niche, you may initially want to look at other the bloggers, in your niche, as competitors and why wouldn’t you? Doesn’t competition basically drive our country’s entire economy and aren’t we competing for our readers attention?

Where that is true, with blogging, it’s a little different. Mainly, because your “product” is not exactly the same as your competitor’s product. Sure maybe you’re both blogging about the Mexican Walking Fish, but your content is not going to be exactly the same as your competitor and readers know that. In fact, most readers are going to follow as many different blogs on Mexican Walking Fish as they possibly can. And wouldn’t they? In most cases, it’s free and products like Google Reader make it incredibly easy to follow multiple blogs.

So…how do I gain the lion’s share of readers?

You do this in two ways. The first one you should be doing already and that’s focusing on writing interesting, value-rich content. You should be focusing on crafting a writing style that makes users want to come back again and again because they just enjoy reading your blog.

The second is by creating a network with your fellow bloggers. Simply knowing that the average reader is going to follow multiple blogs, is exactly why you should help promote the other blogs in your niche instead of fighting them. Work together to give your readers as much interesting information about Mexican Walking Fish as you can. Building a community of readers will only benefit you in the end with reader loyalty and ultimately revenue.

Plus, all those links back to your site will help you move up in the search engines because backlinks are very important in search engine rankings.

Where there are a lot of really great things designers have come up with to help improve the look and feel of websites there are just as many things that need to go away and never come back. Here are five things that could gracefully disappear into the web abyss and I wouldn’t be sad about it at all.

1. Uninvited Music
I’m fine with music on websites, but not when it starts up immediately and without warning. It does nothing for user experience except start it off on a bad note. (No pun intended!)
2. Clip Art
Doesn’t the name say it all. There’s nothing worse and unappealing than the use of clip art. If you can’t design artwork yourself or have someone do it for you then just do without.
3. Frames
The use of frames is bad enough as is, but when it’s not seamless and there are scrollbars in the middle of the page, that is completely unacceptable. There are very few instances where frames are acceptable.
4. Hit Counters
You don’t see hit counters very often anymore and there’s good reason for that. They’re pointless and superficial. Focus more on providing quality content to your readers.
5. Ugly Backgrounds
This goes right along with clipart. Backgrounds should not take the focus off the actual content. When I keep looking at the background instead of the site itself that’s not a good thing.

Keep these five things off your site and you’ll be instantly ahead of the curve. For a good laugh, here’s a site that pretty much covers all five of these items. Enjoy!

You have to see this!