A crawler is a program of search engines which visit on the web pages automatically to read, store and index them; whatever on the web pages. Its another name are web spider, web robot, ants, bots, worms etc. but here I am talking about the name of major search engines spider. Below is the list -
----------------------------------------
Search Engine Spider Simulator
A lot of Content and Links displayed on a webpage may not actually be visible to the Search Engines, eg. Flash based content, content generated through javascript, content displayed as images etc.
This tool Simulates a Search Engine by displaying the contents of a webpage exactly how a Search Engine would see it.
It also displays the hyperlinks that will be followed (crawled) by a Search Engine when it visits the particular webpage.
-----------------------------------------------------------------
Googlebot, MSNbot, and Yahoo Slurp can be welcomed guests. These three search engine bots gather (harvest) information about your page for their respective search engine. Seeing these spiders more often is also desirable because this means that you are being indexed more often and more likely to show up quickly in the SERPs (search engine results page).
A spider is nothing more than a computer program that follows certain links on the web and gathers information as it goes. For example, Googlebot will follow HREF or SRC tags to find pages and images that are associated with any given site. Because these crawlers are merely computer programs, they aren't always the smartest of creatures and may get caught in endless loops built by dynamically created webpages.
Robots.txt
While having Googlebot index your site more quickly is almost always a good thing, there are times when you don't want certain pages or images indexed. Most "reputable" spiders will obey a directive given by the robots.txt file. This file is document that tells spiders what they may and may not index. You can also explicitly instruct a robot not to follow any of the links on a page by the following meta tag:META NAME="Googlebot" CONTENT="nofollow".
Because of how these bots work and the importance they place on text links, many people have begun placing keyword filled text links to their website in their signatures on blogs and other comment sections. To reduce the impact that these have, you can instruct spiders not to follow one specific link by placing the following in the anchor tag:rel="nofollow". This will reduce the outgoing number of links and help you to maintain your pagerank.
Bad SPAM bots
Now just as in life, not all bots are good. There are "bad" bots that don't care about your robots.txt and are only out there to harvest your email address. To fight these "bad" SPAM bots, some people use javascript to "hide" their email addresses. However, anything that can be written to avoid a bad bot can be broken by an even worse bot. One company is fighting bots by giving them just what they want, email addresses, and lots of them. However, they are all email addresses of known SPAMers. I found the sight to be quite clever.
Hopefully this will clear up some confusion as to what a bot, crawler, spider is and how they go about collecting information. If you have any questions, post them below and we will try to answer as quickly as possible. If you need help with SEO (search engine optimization), we would love to help show you ways to increase the frequency and number of times Googlebot, Yahoo Slurp, and MSNbot index your site.
------------------------------------------------------------------
Search Engines -- Spiders or Crawler Name
Google -- Googlebot
Yahoo -- Yahoo Slurp
MSN -- MSNbot
Ask -- Teoma
Excite -- Architext spider
Inktomi.com -- Slurp
Alexa -- ia_archiver
Altavista -- Scooter
Lycus -- Lycos_Spider_(T-Rex)
Cuil -- Twiceler (New Search Engine)
Looksmart -- MantraAgent
source: various source
Monday, September 29, 2008
What is name of major search engines spider crawler
Posted by
Abhay Mathur
at
5:12 AM
1 comments
Labels: difference in robot crawler spider, What is name of major search engines spider crawler
Friday, September 26, 2008
These 10 SEO Tactics Bring Me Over 2000 Visitors Daily
No matter how hard some people try to mystify SEO, it is not as complicated as many would lead you to believe. Despite all the techno jargon that many in the field will throw at you: SERPs, SEM, PageRank, Keyword Density, Vertical Search, Algorithms... SEO is really simple to do if you understand some basic concepts and follow some easy steps.
Search Engine Optimization is getting your content listed in all the top positions in Google, Yahoo and MSN for your targeted keywords. When someone does a keyword search in a search engine for your particular subject or niche - you want your site or content to be at the top of the líst.
Here are 10 SEO tactics that have worked and are working for me at this moment in time.
1. Quality Content is and always will be your number one factor for getting high rankings and keeping them. You must understand search engines are simply businesses that supply a product like any other company. That product is information. They must provide quality results to anyone using their service to solve a problem, answer a question or to buy a product. The more relevant, the more targeted the search solution they return, the higher the overall quality of their product and the more popular their search engine will become. Providing quality content is vital for SEO success.
2. Keywords are your number one tools for achieving high rankings. You must understand keywords and how they work on the web. You must know how many searches are made each day for your chosen keywords. Sites like Wordtracker and Seobook will give you a rudimentary number of searches. Design your pages around your targeted keywords and don't forget to do some deep-linking to these pages on your site. Find and build backlinks to these interior keyword pages and not just to your home page or domain URL. Picking keywords with medium to low competition has worked out well for me. So too has using the more targeted and higher converting "long-tail" keywords been very beneficial for me.
3. Onpage Factors and site design will play a major role in the spidering and indexing of your site/content. Make sure all your pages are SEO friendly, make sure all your pages can be reached from your homepage and no pages should be more than three levels away from it - keeping a sitemap listing of all your major pages makes the search engines happy. Make sure you have all your meta tags such as title, description, keywords... optimized (Title = around 65 characters, Description = around 160 characters).
Remember, your title and description should be keyword targeted and are the first contact/impression anyone will have of your site - make sure you use them to draw and entice interested visitors to your site and content. Also make sure your title and URL are keyword matched for maximum effect. Having your major keyword in your Domain Name also helps, using a pike | to separate different elements of your title has helped my rankings, so too does having your keyword in the first and last 25 words on your pages.
4. Google will send you the most qualified traffic so concentrate the majority of your SEO efforts on Google. Don't ignore Yahoo! or MSN but Google is king of search so give it the respect it deserves. With its new browser, Google's influence will only grow stronger so you must optimize your pages for Google. Use Google's Webmaster Tools and Google Analytics to fine-tune your pages/content. I also use Google Alerts to keep up on my niche keywords and for comment link-building on the newly created pages Google is indexing.
5. Link Building is still the most effective way to boost your search rankings. Make sure you get backlinks from relevant sites related to your niche market and make sure the 'anchor text' is related to your keywords but don't ignore the text and overall quality of the content linking to you. The anchor text is the underlined/clickable portion of a link. Don't forget linking is a two-way street, make sure you link out to high quality, high ranked relevant sites in your niche.
6. Article Marketing is a well established method of getting quality backlinks and it still works. Writing short 500 - 700 word informative helpful articles with your backlinks in the resource box is still very effective for getting targeted traffíc and backlinks. Longer articles have also worked for me and I use an extensive network of distribution, including SubmitYourArticle, Isnare, Thephantomwriters... plus other major online sites. Don't forget the whole element of blogging and RSS feeds in your article distribution. And always remember you're also using these articles to pre-sell your content or products. Don't forget to leverage sites like Squidoo, Hubpages... to improve your rankings and traffic.
7. Onsite Traffic Hubs have worked extremely well for me. These traffic hubs are whole sections of your site devoted to one sub-division of your major theme. For example, if you have a site on Gifts, then wedding gifts could be a separate section. This would be fully fleshed out with extensive pages covering everything dealing with wedding gifts - a self-contained keyword rich portion of your site on wedding gifts. Works similar as a sub-domain, but I prefer using a directory to divide it up, such as yourdomain/wedding_gifts. (Most experts suggest always using a hyphen in your urls, but underscores have worked fine for me.) Search engines love these keyword/content rich hubs but keep in mind you're creating content to first satisfy your visitors.
8. WordPress blog software is extremely effective for SEO purposes. WordPress software is easy to install on your site even if you haven't any knowledge of installing server-side scripts. Besides, search engines love these highly SEO friendly blogs with their well structured content and keyword tagging. I have at least one of these on all my sites to draw in the search engines and get my content indexed and ranked. I also use Blogger (owned by Google), Bloglines and other free blogs to help distribute my content.
9. Social Bookmark/Media Sites are becoming very important on the web. These include a whole range of social sites like MySpace, FaceBook, Twitter... media news sites like Digg, SlashDot, Technorati... you must get your content into this whole mix if you want to take full SEO advantage of Web 2.0 sites. You should be joining these sites and using them. It's time consuming, but it will keep you in the swing of things. One simple thing you must do is to put social bookmark buttons on all your pages so that your visitors can easily bookmark your content for you. You can use a WordPress plug-in. I like using the simple free site/service from Addthis.com which gives me a simple button to put on all my content.
10. Masterplan! Many webmasters and site owners forget to develop or have an overall masterplan/strategy when it comes to SEO. You must have an understanding of what SEO is and what it can do for you and your site. More importantly, you just don't want SEO - you want effective SEO. In order to achieve effective SEO you must have three things: Relevance, Authority and Conversions.
First, your content/site must be relevant to the topic or niche area you're pursuing - your content must fit in and be related to all the other sites in your niche. That's why closely themed sites do so well in the search engines. They give only relevant content to what's been searched for or discussed.
Second, your content/site must be perceived as an authority site on your subject or niche. Establish this authority position and the search engines will love you and your content. One way to develop this authority, besides offering superior content, is to form links/partnerships with other perceived authority sites in your field. Always strive to make your site an authority site - tops in your niche - the one site everyone has to check before drawing or forming a conclusion.
Third, conversions should be your main goal of any SEO efforts because you want to convert your targeted traffíc into site members, subscribers, buyers or just repeat visitors. If you're into online marketíng, conversions will be the most important element of the whole SEO process because you want buyers, not just visitors coming to your site.
Most of all, you must convince yourself Search Engine Optimization is not difficult, nor is it the equivalent of the online bogeyman as many would like you to believe. Used effectively, SEO can give you the targeted traffíc you're seeking. Just follow some of the outlined steps/tactics listed above and you will have SEO working for you and your site in no time at all.
About The Author
The author is a full-time online marketer who has numerous websites. For the latest web marketing tools try: http://www.bizwaremagic.com .
Posted by
Abhay Mathur
at
2:49 PM
0
comments
Labels: get thousands 1000 visitors by these seo steps, get tons of visitors using these easy steps
Your Website from the Ground Up in 10 Steps
So, you have finally decided to build yourself a website, but really have no idea where to start. The following 10 steps will give you some insight into what you need to address when creating a search friendly website from the ground up.
While this article is not an exhaustive líst of everything you need to know, it does touch base on many of the important aspects of creating a new website.
Step 1 - Keyword Research
Even before you choose your domain name, you should put a little time into some keyword research. Research all the possible keywords that will fit your industry and the website you plan on building. Having a clear idea of what your end targets are will make the rest of your job much easier and help things to just fall into place. Take a look at Keyword Research for SEO, written earlier this summer, for more help on this.
Step 2 - Domain Selection
If you already have an established brick and mortar business and the website will be an extension of that business, using your company name as the domain name is in most cases the best idea. If your company name is either irrelevant, or simply unavailable, you may want to consider a domain that has your target phrase listed as part of the domain.
A great example of this is if your site is focused geographically. Using the location as part of the domain when possible will give you a little extra juice with the search engines and help draw people to your site as they instantly can see the relevance in the domain.
Let's say that you are building a website focused on your home town, "Somewhere USA". A domain you may consider could be "Somewhere.com"; however, this would probably be already taken. Other options such as "SomewhereInfo.com" or "SomewhereGuide.com" may be good alternatives. The same goes for retail stores. "SomewhereCameras".com or "SomewhereBakery".com would also be good choices.
Avoid excessive use of hyphens; sometimes it is appropriate to use one, but if you can help it, avoid more than that as it can appear messy and even spammy in some cases.
Step 3 - System Back End
If you plan on having a large scale website that will grow and change constantly then you may want to consider a content management system (CMS) such as Joomla. If you decide to go this route, you want to ensure that whichever CMS you choose is search engine friendly and offers items such as unique title tags, custom URL's, and full control over content, heading tags, image alt tags, etc.
Starting a website using a non-friendly CMS is like buying a car without an engine. Sure it may look great, but it won't get you anywhere.
Step 4 - Site Structure & Navigation
This is really one of the most fundamental aspects of your site creation. If the structure of your site does not work well, then your site may be doomed from the very beginning.
Take a look back at your keyword research and brainstorm all the areas of your site that you may want to develop content for. In some cases you may find valuable keywords that would fit perfectly into a few pages of content for your site. If the phrase and the content would be a good match for the theme of your site, go ahead and note them as pages to create. Get a líst, or flowchart, of all the content you plan on adding and sort those pages into relevant categories.
Be sure your site files are saved in a way that makes sense - this includes both the file name, and the complete path to the file. Save files, including similar content in a relevant subdirectory, with simple file names representing each. Let's take an example of an informational site dedicated to a specific geographic location. If you have a series of pages dedicated to recreation, you may save them as:
/recreation/parks.html
/recreation/trails.html
/recreation/beaches.html
Keeping your URL structure clean and tidy can not only help with search engine rankings, but it will give a good visual impression to the site visitor as well. Often, using each of these categories as main points for your primary site navigation may make the most sense.
Also be sure to keep your site relatively flat, with as few layers as possible. Don't make the search engines follow a dozen links to get to the deepest levels of your site. Unless the site is literally tens of thousands of pages, there is no need to click more than 2 or 3 links to get to any deep content. The shorter the path to an internal page, the more credít by the search engines.
Step 5 - Navigation
When developing the end site, you also want to make sure that your site navigation is search engine friendly - this is critical if you ever want free organic listings.
If possible, use a text based form of navigation. You can use CSS to style the text links to fit into your graphical design. Text links are the best method, but image based navigation and even some forms of drop down menus are also search engine friendly.
If you choose to use image based navigation be sure to include image alt text relevant to the link to give something for Google to associate with the linked page. If you absolutely must use Flash, or any form of navigation not friendly for search engine spiders, be sure to supplement this with text based links on another location of the page.
Step 6 - Analytics
It isn't ever too early to start thinking about your stats. Before your site goes live you must have some form of accurate analytics in place so you can measure your site's traffic and progress.
There is an endless supply of analytics options out there to choose from. You can simply use the stats software that comes free with your web hosting, however, more often than not, they tend to be very basic with no flexibility. Advanced choices such as ClickTracks can give you rather in-depth statistics, but for a very small mom and pop operation it may be too expensive. Google Analytics, is a free option that can give you most, if not all the data you will need and does not require access to your raw log files.
If you do opt for an option such as ClickTracks, check with your host to ensure that you will have the raw log files you require. StepForth can also help you with your statistical analysis and offers a number of inexpensive
Step 7 - Content Creation
Now is the time to get that new, fresh content posted to your site with the SEO in mind. Make sure that the content you write reflects your industry and target keywords. You don't need to flood the content with your target keywords, but make sure they are in there a few times. When it makes sense to do so, also include some acronyms to help Google establish the overall relevance.
If you perform a search in Google for "~keyword", any words that Google bolds in the search results will be acronyms that Google deems relevant. Include some of these where possible.
Step 8 - Basic SEO
Ensure that ALL pages of your site include unique