August 9, 2013: Search engine spiders – the Google Bot being among the most important because of Google’s enormous reach — are small programs sent out by search engines to probe, record, and report back information about your site. These friendly beasts are programmed to address basic questions about your site, such as What it about? Is it alive or dead? In a good or bad neighborhood? Who are its friends?
Here are 5 top-level tips for making this peculiar beast your best friend.
1. Don’t Block the Bot
If you’re using a standard platform for creating web content (e.g. WordPress, Blogger, or other popular CMS), you don’t have to worry too much about blocking search spiders from recording the correct signals from your site, because these platforms were designed with spiders in mind. E-commerce site proprietors and webmasters with dynamic content have had a more difficult job because URLs generated can sometimes block the bot with a URL with special characters. Non-HTML documents (such as PDFs) used to pose a problem for spiders but they’ve grown a lot more tolerant in the past several years. Unless you’re deliberately trying to thwart the spider from seeing what’s on your site (e.g. using the No Robots tag to tell it to go away), you should be OK.
2. Don’t Adulterate The Meal
SEO forums are full of messages about how large and rich the meals you prepare for the spider should be. Will a 300 word article smothered in choice keywords be satisfying enough or too rich? Will a bland 10,000 word long form article cause the spider to gag? Don’t obsess too much over this issue, because the spider isn’t going to be able to digest all of your articles equally well. Write naturally, write articles that are long enough to make your point, and avoid putting too much “keyword spice” into your content meals, because the spider might conclude – rightly or wrongly, that you’re “adulterating” what you serve with artificial ingredients.
3. Make It Fresh
Search spiders love fresh content, and that’s so many big brands are behaving more like newspapers than merchants these days. Not every business can afford to serve the spider a fresh new meal each day. But the advantage of posting content on a regular basis (say once or twice a week) is that you can train search spiders to spider your site more frequently. Being able to post content to your site regularly also gives you a powerful advantage on social networks which prize newness, novelty, and fresh content.
4. Avoid Junk Food
Search spiders aren’t intelligent enough to distinguish between low-rent, robotically-generated content and high-quality, original prose. All they do is record and travel back to Search Engine HQ, where subsequent analysis will determine whether you’ve fed the spider a good or bad meal. At this point, Google’s higher-level beasts, Penguin and Panda – which compare the information gathered by the spiders with information gathered from other spiders and human editors — will decide your site’s fate. Feeding the spider nothing at all is better than feeding it junk food. What’s junk food? Stuff that other robots spit out at the so-called “Content Farms” loathed by Penguin/Panda.
5. Nudge The Spider
If you’re publishing content regularly, the spider will show up eventually (although if your site is new it may take awhile for this first visit to occur). But you can “nudge” the spider into visiting, both by communicating this fact directly to Google (via site map upload), and by integrating your publishing system with Google Plus. Many people put down Google Plus because fewer folks socialize there than on Facebook, Twitter, and LinkedIn. But notice what happens when you hook up the Google Plus Plug-In – whenever you publish a new article on it, the spider comes by – often within a few minutes. That spider is reporting back to Google’s Index that you’ve created new content. For this reason alone, you should connect your publishing system to Plus.
Your search spider can be a faithful pal. Feed it well and feed it often.
- 10 Mistakes to Avoid When Using QR Codes for Marketing - September 20, 2023
- Kevin Lee on How AI Changes the SEO Landscape - August 31, 2023
- The Power of Compound Marketing: Kevin Lee Presents @ 1MediaWorld 2023 Global Conference - March 7, 2023