Why Can't I Get All My Web Pages Indexed By The Search Engines?

Search engine optimization is about paying attention to the basics. From making sure that you vary your anchor text in your inbound links to ensuring that your keyword density is between 2 and 5 percent, it is all about getting the details right. The only other thing required than that is time.

As part of search engine optimization, there is one factor that is often either missed, or not done properly. I’m talking about maintaining a good site layout. There are several very real, money-valuable benefits to having a good site layout.

Search Engine Indexing

Proper layout techniques, such as having a site map and executing a proper, planned linking strategy throughout your website will not only get your pages indexed easily (but not quicker), but in some cases proper linking will squeeze out every last sliver of ‘votes’ towards your important pages. I’ll talk more about site maps and linking strategies later on.

Conversion Rate

A good site layout is all about converting your visitors into customers. By making an easy to- use, uncluttered and user-centric layout, you are increasing the chances of leading your customer to into making the ‘critical move’, whether it is signing up to your newsletter, filling a survey or buying your product.

User Satisfaction

User satisfaction should be central aim when designing a site layout. Put yourself in the shoes of your visitors, and decide what you want from your website. It is a subtle shift in perception, but it will help you decide whether you really need all those extra menu options on the left or if the design could be simplified by placing those extra links at the bottom of the page; out of immediate view, thus reducing clutter and confusion but within reach if the user needs extra information.

A good site layout will improve the image of your website. Don’t just think about search engine rankings – keep your users as your first priority and ensure that your visitors do not go away without being impressed by the clarity and simplicity of your design. Word-of-mouth marketing (either through natural linking or plain blog and forum activity) is a power marketing tool that is largely based on how user-friendly and helpful your website actually is.

Site Map

A site map has been widely proclaimed as the basic linking tool for site-wide search engine optimization, and with good reason. It presents your website’s content – the linking structure of your website – on one single page for search engines and users alike.

Accessibility

Your site map is something like a table of contents for your website. While not the first resort for users looking for information on websites, today’s increasingly-aware user audience will more and more turn to a site map if they cannot find something on your website through the traditional menu structure, or if they need to get somewhere really quickly.

Search Engines

A site map, properly mapped and linked from your home page, is the search engine’s guide to the depth and breadth of your website. When a search engine spider finally decides that your site in interesting (read important) enough to be indexed further, it will start by exploring links on your home page. Through the site map, it gains immediate one-link access to your complete website, and this greatly speeds up the indexing of all your pages.

Even when the spider does not engage in a deep crawl, a two-level initial crawl is not uncommon, and that will invariably give the spider the opportunity to see all the pages. For examples of good site maps, check out the following site maps:

· http://www.google.com/sitemap.html – Google Site Map

· http://pages.ebay.com/sitemap.html – eBay Site Map

· http://www.apple.com/find/sitemap.html – Apple Site Map

Optimal Indexing

To ensure that your website is optimally indexed, there are some specific linking strategies that you need to follow. It is NOT as difficult as you might expect. At the very basic level, there are two things you must take care of.

Template-based web design

Design templates before you start designing your website. Using templates to add new pages to your website will not only bring in consistency, but also allow you to standardize the optimal pattern of in-site linking. This might sound terribly complicated unless you base all pages on a template. With a well-designed template the process is simplified to just updating the placeholder hyperlinks. Then create sub-templates for categories of pages (main category pages, subcategory pages, etc.) to further ease your burden of reconfiguring each page manually.

Site Structure

A template-based design should, apart from speeding up the design process, focus on optimizing your sitewide linking. This will not only with indexing, but also help in increasing SERPS placement due to extra inbound links for your important pages. Base your site structure on solid, site-wide linking strategies like these:

· Each page should link back to the home page.

· Each page should further link back to its main category page.

· Each category page should provide clear links to any sub categories.

· If possible, each page should have the main menu structure – so as to give maximum link exposure to the most important pages of your website.

· Each page should further link to those important pages on your website that do not have any clear category (privacy policy, Help section, user guide, search page, members section, etc.).

If your template is properly designed and as mentioned earlier you specialize your template into subtemplates, your site structure will become more defined and manageable, and your linking strategy will help in both improving the search engine indexing and increasing your rankings.

Reality Check

Don’t spend more time than necessary on site structure a