Isn’t amazing to think that despite the millions of different websites and billions of different web pages that somehow, someway the search engines still manage to find your site? I personally find this to be rather mind-boggling. In fact, it kinda makes my brain hurt just thinking about it.
Searchbot 101
Search engines have programs called searchbots that send out “spiders” to crawl the Internet and visit these millions of different websites. Once a spider finds your site, it reads every page and then follows every link on your page and reads those pages, as well. A spider then will revisit your site on a regular basis looking for new content and changes to existing content. After it visits your site, your content is then indexed on a search engine. Only after your content has been indexed, can it actually be found on search results of Google or Yahoo or MSN. When a user types in a search term, the search engine itself analyzes these millions of pages that have been indexed to match results of the search term and determine how relevant your site is to that search term. This is how your site will or will not appear in the search results of any given query.Search Me, Baby!
A robots.txt file is used to tell the searchbots what file and directories on your site can and cannot be searched and indexed. You may have directories, like an admin folder, that you don’t want search engines crawling. It’s in this file that you make that clear to the searchbot.
There’s really not much to the robots.txt file. It’s a simple text file and usually only has a few lines in it, unless your site is unusually big. A typical file looks like this:
User-agent: *
Disallow: /wp-admin
Disallow: /images
User-agent:* means that following instructions applies to all robots. Then you list, specifically, each directory or file that robots are not allowed to index using the Disallow keyword. Once you’re done with that, save the file and make sure to name it ‘robots.txt’ and then upload it to the root of your site. You can use a tool like Google’s Webmaster Tools to validate it.
Click here for more detailed information.
You Should Also Check Out This Post:
- My favorite social media apps for the Blackberry
- Don't Fight 'Em, Join 'Em
- 5 Design Objects That Annoy Me
My name is Matthew Bauer and I've been working in the web industry for over 10 years. Professionally, I work full-time as a web developer in Cincinnati, Ohio. Home to Skyline Chili, Nick Lachey and Jerry Springer. For fun, I write this blog. I really enjoy the topics of blogging, social media and design. Where I don't consider myself a designer, it has always fascinated me. I'm not a very serious person. Mostly everything I write is lighthearted in nature.....