Why Using Robots.txt for Chinese Search Engine Baidu?

Submitting your site to Baidu through the website submission form notifies Baidu search engine to crawl and index your website. To exclude specific content (or web pages) from being crawled by Baiduspider, Baidu’s search engine robot / spider / user agent, use Robots.txt.

• Using Robots.txt is optional.
• Include a Robots.txt file only if your site has content that you do not want Baidu to index.
• If you want Baidu to access your entire website’s content, do not include a Robots.txt file.
• Place Robots.txt file under your website’s root directory. Before crawling your website’s pages, Baiduspider first checks the root directory of your site’s domain if a plain text file called “robots.txt” can be found.
• Robots.txt can improve your site’s Baidu SEO traffic and ranking when it is done right.
• Robots.txt blocks your web pages’ content from being crawled or indexed by Baidu, but Baiduspider may still index the URLs if they can be found on other web pages on the web. So the “blocked” web pages’ URLs may appear in Baidu’s organic search results pages.

Using Robots.txt for Baiduspider

Baiduspider follows two basic rules in Robots.txt files:

• User-agent: the robot the following rule applies to
• Disallow: the URL you want to block

To block your entire site from Baidu:

User-agent: Baiduspider
Disallow: /

To block your entire site from all search engine spiders, but Baiduspider:

User-agent: Baiduspider
Disallow:

User-agent: *
Disallow: /

To block a directory of your site and all the files in it, from Baiduspider:

User-agent: Baiduspider
Disallow: /cgi-bin/

To block a directory of your site, but some of the URLs in it, from Baidu:

User-agent: Baiduspider
Allow: /cgi-bin/tmp-1
Allow: /cgi-bin/tmp-2
Disallow: /cgi-bin/

To block a web page from Baidu:

User-agent: Baiduspider
Disallow: /my-page.html

Baiduspider supports using wildcard symbols including “*” and “$” to match URLs:

• “*” matches zero or more arbitrary characters.
• “$” matches the line terminating character(s).

To block access to all dynamic URLs (i.e. all URLs that contain “?”) by Baiduspider:

User-agent: Baiduspider
Disallow: /*?*

To block access to certain file types, but to allow other types of files, by Baiduspider:

User-agent: Baiduspider
Allow: .gif$
Allow: .jpg$
Disallow: .jpeg$
Disallow: .png$
Disallow: .bmp$

Other Baiduspider Bots

While Baiduspider is responsible for crawling web search content, Baidu uses different search engine spiders/bots to crawl different types of content:

• Baiduspider-image crawls images
• Baiduspider-mobile crawls mobile search content
• Baiduspider-video crawls videos
• Baiduspider-news crawls news content
• Baiduspider-favo crawls bookmarks
• Baiduspider-sfkr crawls Baidu PPC/ads
• Baiduspider-cpro crawls Baidu’s contextual advertising network

Robots.txt Examples on Large Chinese Websites

• Baidu.com is blocking Baiduspider to access some of the site’s directories: http://www.baidu.com/robots.txt
• Taobao.com at the time is blocking Baiduspider through Robots.txt at the root directory: http://www.taobao.com/robots.txt
• Alibaba China at the time is blocking the entire site from certain bots/spiders: http://china.alibaba.com/robots.txt

Guidelines to Robots.txt

• Baidu offers guidelines in Chinese on how you should use Robots.txt to block Baiduspider.
• Robotstxt.org and Searchtools.com both provide comprehensive guidelines to create Robots.txt.

Besides your website’s URL, you will have to enter a randomly generated captcha passcode when submitting through Baidu’s URL submission page. However, Baidu’s URL submission page is simple and it won’t require you to understand any Chinese characters.

Go to Baidu.com and check your website with the “Site:” command after about one month:

site:your-site.com

To speed up your site’s inclusion in Baidu’s index, ensure your site’s content in written in Chinese (Simplified). Alternatively, get a link from a Chinese (Simplified) web site that is already indexed in Baidu to point to your new Chinese website.

Unlike Google, Yahoo or Bing, Baidu currently offers no official search engine webmaster tools and no URL submission through sitemap in XML format.

Use Baidu’s Keyword Research Tool in Baidu PPC Account

This Baidu keyword research tool suggests new keywords to you in two ways:

• Generate keywords from a “core keyword” that you enter
• Generate keywords from a URL that you enter

Run Baidu’s Keyword Research Tool

The keyword research tool returns a list of related keywords based on your “core keywords”. You can download the new SEO keywords as a spreadsheet.

Understand the Correlation Among Baidu SEO, Baidu PPC and Baidu Search Open Platform

To maximize your Baidu SEO effort, you will have to understand how the two Baidu PPC Auction Systems work. Also for some search queries, universal search results (or one-box results) may show up. Baidu has made the submission through Baidu Search Open Platform available.

If you are running Baidu PPC alongside Baidu SEO, Baidu’s keyword research tool allows you to add the new keywords directly into one of your Baidu PPC ad groups and/or campaigns.

Get Localized Keyword Data

If you run a web site that provides local business for certain regions in China, Baidu keyword suggestion tool can narrow down by regions and/or cities and show you the “geo-targeted” keyword data.

Google’s Keyword Tools as Alternative

Google provides several great SEM/SEO keyword research tools in which you can discover more new Chinese keywords.

Baidu Search’s open platform allows both webmasters and Baidu together to build universal search results. Instead of Baidu’s search algorithm trying to figure out what should be displayed as the universal search results, webmasters collectively are to submit data as “suggestions” and some of the submitted data is believed to become Baidu’s universal search results.

To use the open platform, sign up and verify the ownership of your website. You will have an option to either upload a file to your website’s server, or insert a verification code into your header META tag. You will also be required to submit a personal identity document.

• Decide the keywords that you want to be included within Baidu’s universal search results
• Choose a template
• Create and submit the XML files

The default templates that Baidu offers, includes the one-line template:

The festival template:

And the score chart template:

If none of the templates are suitable for your keywords, you can even ask Baidu to create customized templates. Tencent’s QQ.com has the customized service telephone number template below:

Once approved, your universal search results will be displayed above Baidu’s organic search results.

Baidu does have guidelines to determine the “relevance” to the universal search results, so not all submissions will show up among Baidu’s organic search results. Baidu especially requires the data being submitted to be:

• Comprehensive
• Accurate
• Unique
• Up-to-date

Another tip before submission is that you should always consider search engine user experience in general.