Google Panda Algorithm to Affect Content Farm Sites

When the Panda Algorithm lowers a website’s ranking in Google’s search engine results pages (SERP), the website’s visits and page views from Google SEO will drop as a direct result. Sites / companies that suffered hard because of Google’s Panda Algorithm include:

• About.com
• Some of Yahoo’s content pages
• Demand Media

People working on the following roles within the Internet industry should pay attention to Google’s Panda Algorithm updates:

• Professionals whose role is to manage search engine optimization (SEO) for websites that have large quantity of text content.
• Webmasters who own and manage multiple websites with multilingual content.

Google Panda Algorithm Updates

• Panda 1.0 – The first version of Google’s Panda Algorithm was rolled out in February 2011 that hit content farms which caused irrelevant organic search results to show up in Google and affected 12% of searches in the US.
• Panda 2.0 – Released in April 2011 to all English language search queries worldwide including: British English (google.co.uk), Australian English (google.com.au) and English language search results (e.g. google.fr, google.com.hk).
• Panda 2.1, 2.2, 2.3 – Several minor algorithm updates in May, June and July 2011.
• Panda 2.4 – Released in August 2011 for English language search queries globally and non-English language search queries except for Chinese, Japanese, and Korean, and affected 6-9% of search queries.
• Panda 2.5 and Panda Flux – Released from September to November 2011 and affected 2% of search queries.
• Panda 3.1 – After Panda 2.5, it went straight to Panda 3.1 (released in November 2011) without an official Panda 3.0 algorithm update. Since Panda 3.1, algorithm updates have entered a period of Panda Flux where updates started to happen more frequently and were relatively minor.
• Panda 3.2 – Released in January 2012 and this Panda data update was confirmed by Google.
• Panda 3.3 – Released in February 2012 as a Panda Flux.
• Panda 3.4 – Released in March 2012 and affected 1.6% of search queries.

SEOmoz provides a list of all Google Algorithm changes that includes the incremental Panda Algorithm updates.

SEO Experts’ Suggestions for Google Panda Algorithm

Suggestions from Danny Sullivan on Google’s Panda Algorithm updates:

• Google does make small algorithm updates from time to time to lower websites’ rankings in Google’s SERP, but may be unrelated to the Panda algorithm.
• Google may update a site’s PageRank from time to time that may have negative effect to a site’s ranking in Google’s SERP, but may be unrelated to the Panda algorithm.
• Google confirmed it may be possible for some websites to be negatively affected by Panda updates to improve their SERP rankings, if the sites have made significant improvements to the quality of content.
• Monitor your site’s Google organic traffic fluctuation, instead of focusing only on your web pages’ SERP ranking.

Six Revisions’ suggestions on creating websites that are optimized for Google’s Panda Algorithm:

• Design for Engagement and User Experience
• Spelling and Grammar are Important
• Focus on Content Quality
• Avoid Too Many Ads
• Avoid Duplicate Content
• Less is More
• Ensure High-Quality Code

• Web Analytics Principles and Scope
• Google Analytics Basic Setup
• Web Analytics Reports & KPI Metrics for Web Analysts focusing on SEO and SEM
• Google Analytics Tips
• Google Analytics Advanced Setup

Web Analytics Principles and Scope

• What is and why Web Analytics?
• Tracking methods: Web log files vs JavaScript tags
• Benefits to JavaScript-based tags: URL and cookie-based, no data storage issue, easy to install
• Disadvantages to JavaScript-based tags: URL structure planning, No search engine spider, no page error data, JavaScript codes slow down web page load data
• Web Analytics: Google Analytics, Yahoo Web Analytics, Baidu Analytics, 99Click, Site Catalyst, Piwik

Google Analytics Basic Setup

• How does Google Analytics work?
• Google Analytics account and profile setup
• Google Analytics Tracking Code (GATC) setup: Standard, multiple sub-domains, multiple domains, Chinese search engines
• Google Analytics profile filter setup: Lowercase URLs, full path URLs

Web Analytics Reports & KPI Metrics for Web Analysts focusing on SEO and SEM

• What is a web analyst?
• Basic reports: Site Traffic (Visit) Trends, Direct vs. Search Engines vs. Referring Sites, New Visitors vs. Returning Visitors, Visitor Frequency (counts of visits) and Visitor Recency (days since last visit), Geography, Browsers, Languages, Mobile Devices, Top Landing Pages vs. Top Pages, SEO/SEM Visit Trends, Top SEO/SEM Keywords, Site Search, In-page Analytics
• Metrics: Visits, Visitors, Page Views, Unique Page Views, New Visitors, Returning Visitors, Bounce Rate, Exit Rate, Time on Site, Time on Page, Clicks
• Goals: URL destination, visit duration, page / visit, event
• Goal funnels
• Segmentation: Advanced segments
• Direct Traffic
• SEM Tracking
• SEO reports: Traffic trends, spider behavior (pages indexed), page errors, keyword ranking
• Google Webmaster Tools

Google Analytics Tips

• New Google Analytics: Dashboard, Real-time, Visitors Flow, Multi-channel Funnels, Custom reports
• Google Analytics Cookies: __utma, __utmb, __utmz
• Google Analytics Sampled Data

Google Analytics Advanced Setup

• Social media tracking: Facebook Like/Share buttons, Twitter Tweet button, Google +1 button
• URL Tracking

The social media tracking tools:

• Google Analytics
• URL Shorteners: Bit.ly, Hootsuite, Google URL Shortener
• Social Influence Measurement tool: Klout

Social influence metrics:

• True reach
• Amplification
• Network impact

Why Adwords Conversion Rates Can Be Larger Than 100%?

It is possible to have Adwords conversion rate that is larger than 100% and Google official explains how it can happen:

• A user clicked one of your Adwords ads, arrived at your site, completed a purchase on your site and reached the conversion confirmation page (or thank you page) of your site (the web page where you have Adwords conversion tracking code inserted). The user left your site and then came back directly to your site without clicking any of your Adwords ad again. This time, the user completed a second purchase and reached the conversion confirmation page again. The result is 2 Adwords conversions are counted, with 1 ad click.
• A user clicked one of your Adwords ads, arrived at your site, completed a purchase on your site and reached the conversion confirmation page (or thank you page) of your site (the web page where you have Adwords conversion tracking code on). The user clicked away from the conversion confirmation page but then clicked on the browser’s back button to return to the conversion confirmation page (where Adwords conversion tracking code is placed). The result is 2 Adwords conversions are counted, with 1 ad click.

Conversions 1-per-click and Conversions Many-per-click

Conversions 1-per-click and conversions many-per-click are two of the metrics in Google Adwords conversions.

• Conversions 1-per-click: A conversion is counted for each Adwords ad click leading to a conversion within 30 days of the first ad click. Subsequent conversions that are the result of this first ad click will not be counted as conversion 1-per-click, as conversions 1-per-click will count at most one conversion per ad click. Conversions 1-per-click is most useful when measuring conversions approximating the number of unique customer acquisitions.
• Conversions Many-per-click: A conversion is counted for each Adwords ad click leading to a conversion within 30 days of the first ad click, and conversions many-per-click will count multiple conversions per click. Conversions Many-per-click is most useful when measuring conversions that happen every time which is actually B2C website transactions.

Metrics related to conversion 1-per-click:

• Conversion Rate 1-per-click = Conversions 1-per-click / clicks
• Cost-per-conversion 1-per-click = Cost / conversions 1-per-click

Metrics related to conversion many-per-click:

• Conversion Rate Many-per-click = Conversions Many-per-click / clicks
• Cost-per-conversion Many-per-click = Cost / conversions Many-per-click

Conversion Rate Many-per-click may be larger than 100% as you may receive more than one conversion per click.

PPC Metrics

PPC metrics explains conversion, conversion rate and other metrics including Google Adwords Quality Score, impression share and clickthrough rate.

• Search Engine Marketing, Inc.
• Don’t Make Me Think
• Rocket Surgery Made Easy
• Advanced Web Metrics with Google Analytics
• Web Analytics – An hour a day
• Web Analytics 2.0
• Landing Page Optimization
• Always Be Testing
• Freakonomics
• Super Freakonomics
• Small is the New Big
• Meatball Sundae
• Linchpin
• Outliers
• Blink
• The Tipping Point
• The World is Flat
• The Google Story
• Online Marketing Heroes
• Click
• Code Complete
• Rapid Development
• The Data Warehouse Lifecycle Toolkit

How many books can you recognize?

While books provide solid background and/or knowledge of a field (i.e. Web Analytics, SEO, landing page optimization, etc) the list of SEM blogs offer similar topics that are updated more often.

google / cpc

However, Google Adwords clicks can show up in the Google Analytics keyword report as “(not set)” entry, because Google Analytics couldn’t associate a keyword or any search campaign information with the visits (i.e. Google Adwords clicks).

Link Adwords to Google Analytics

Make sure your Adwords account is always linked to your Google Analytics account. If you run search campaigns on multiple Adwords accounts for your website, then link all multiple Adwords accounts to the Google Analytics account.

Auto-tagging Google Adwords Destination URLs

Turn on auto-tagging and test if auto-tagging works on your site (in case some of the web pages on your website have URL redirection enabled):

• Append ?gclid=test or &gclid=test parameter to the end of one of your Adwords destination URLs.
• Put the URL with the appended ?gclid=test or &gclid=test parameter into a web browser address bar.
• If the web page shows up correctly and the appended ?gclid=test or &gclid=test parameter are still in the web browser address bar, then auto-tagging works and keywords will properly show up in Google Analytics keyword report (within being attributed to the “(not set)” entry).

Manually Tag Google Adwords Destination URLs

In some cases when you have auto-tagging enabled, the amount of “(not set)” entry can become significantly large in the Google Analytics keyword report. You should:

• Disable auto-tagging in your Google Adwords account.
• Tag all the destination URLs manually or through Google’s URL Builder, with source (e.g. google), medium (e.g. cpc), campaign, keyword/term and ad/content.

Google Adwords Cost Data in Google Analytics Campaigns Report

For Adwords cost data to show up in Google Analytics campaigns report, when linking Adwords account(s) to Google Analytics account, you should select to apply Adwords cost data.

• If Adwords cost data is applied and auto-tagging is enabled, then both Adwords cost data and visits data will appear in Google Analytics campaigns report.
• If Adwords cost data is applied and manual-tagging is used, then both Adwords cost data and visits data will appear in Google Analytics campaigns report.
• If Adwords cost data is not applied and auto-tagging is enabled, then Adwords cost data will not appear and Adwords dimensions (i.e. campaign name, ad group, keyword) will appear as “(not set)” in Google Analytics campaigns report.
• If Adwords cost data is not applied and manual-tagging is used, then Adwords visits data without cost data will appear in Google Analytics campaigns report.

http://www.google.cn/landing/shuixia/

Start Google Underwater Search

A search for “Google Analytics” followed by “Barack Obama” and “China” on Underwater Search.

Google’s Underwater Web Search doesn’t actually return any real search results, but it is rather a joke for April’s Fool Day.

• Serving web page content faster to your site’s visitors and improving conversion rates
• Improving Google SEO – Refer to Optimize website page load time

Google has even made site load speed data available through Google Webmaster Tools and Google Analytics.

Yahoo Developer Network has compiled this document on web performance optimization – 35 best practices for speeding up load time of your website and/or web pages.

• Minimize HTTP Requests
• Use a Content Delivery Network
• Add an Expires or a Cache-Control Header
• Gzip Components
• Put Stylesheets at the Top
• Put Scripts at the Bottom
• Avoid CSS Expressions
• Make JavaScript and CSS External
• Reduce DNS Lookups
• Minify JavaScript and CSS
• Avoid Redirects
• Remove Duplicate Scripts
• Configure ETags
• Make Ajax Cacheable
• Flush the Buffer Early
• Use GET for AJAX Requests
• Post-load Components
• Preload Components
• Reduce the Number of DOM Elements
• Split Components Across Domains
• Minimize the Number of iframes
• No 404s
• Reduce Cookie Size
• Use Cookie-free Domains for Components
• Minimize DOM Access
• Develop Smart Event Handlers
• Choose <link> over @import
• Avoid Filters
• Optimize Images
• Optimize CSS Sprites
• Don’t Scale Images in HTML
• Make favicon.ico Small and Cacheable
• Keep Components under 25K
• Pack Components into a Multipart Document
• Avoid Empty Image src

Minimize HTTP Requests

80% of the end-user response time is spent on the front-end. Most of this time is tied up in downloading all the components in the page: images, stylesheets, scripts, Flash, etc. Reducing the number of components in turn reduces the number of HTTP requests required to render the page. This is the key to faster pages.

One way to reduce the number of components in the page is to simplify the page’s design. But is there a way to build pages with richer content while also achieving fast response times? Here are some techniques for reducing the number of HTTP requests, while still supporting rich page designs.

Combined files are a way to reduce the number of HTTP requests by combining all scripts into a single script, and similarly combining all CSS into a single stylesheet. Combining files is more challenging when the scripts and stylesheets vary from page to page, but making this part of your release process improves response times.

CSS Sprites are the preferred method for reducing the number of image requests. Combine your background images into a single image and use the CSS background-image and background-position properties to display the desired image segment.

Image maps combine multiple images into a single image. The overall size is about the same, but reducing the number of HTTP requests speeds up the page. Image maps only work if the images are contiguous in the page, such as a navigation bar. Defining the coordinates of image maps can be tedious and error prone. Using image maps for navigation is not accessible too, so it’s not recommended.

Inline images use the data: URL scheme to embed the image data in the actual page. This can increase the size of your HTML document. Combining inline images into your (cached) stylesheets is a way to reduce HTTP requests and avoid increasing the size of your pages. Inline images are not yet supported across all major browsers.

Reducing the number of HTTP requests in your page is the place to start. This is the most important guideline for improving performance for first time visitors. As described in Tenni Theurer’s blog post Browser Cache Usage – Exposed!, 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these first time visitors is key to a better user experience.

Use a Content Delivery Network

The user’s proximity to your web server has an impact on response times. Deploying your content across multiple, geographically dispersed servers will make your pages load faster from the user’s perspective. But where should you start?

As a first step to implementing geographically dispersed content, don’t attempt to redesign your web application to work in a distributed architecture. Depending on the application, changing the architecture could include daunting tasks such as synchronizing session state and replicating database transactions across server locations. Attempts to reduce the distance between users and your content could be delayed by, or never pass, this application architecture step.

Remember that 80-90% of the end-user response time is spent downloading all the components in the page: images, stylesheets, scripts, Flash, etc. This is the Performance Golden Rule. Rather than starting with the difficult task of redesigning your application architecture, it’s better to first disperse your static content. This not only achieves a bigger reduction in response times, but it’s easier thanks to content delivery networks.

A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content more efficiently to users. The server selected for delivering content to a specific user is typically based on a measure of network proximity. For example, the server with the fewest network hops or the server with the quickest response time is chosen.

Some large Internet companies own their own CDN, but it’s cost-effective to use a CDN service provider, such as Akamai Technologies, EdgeCast, or level3. For start-up companies and private web sites, the cost of a CDN service can be prohibitive, but as your target audience grows larger and becomes more global, a CDN is necessary to achieve fast response times. At Yahoo!, properties that moved static content off their application web servers to a CDN (both 3rd party as mentioned above as well as Yahoo’s own CDN) improved end-user response times by 20% or more. Switching to a CDN is a relatively easy code change that will dramatically improve the speed of your web site.

Add an Expires or a Cache-Control Header

There are two aspects to this rule:

• For static components: implement “Never expire” policy by setting far future Expires header
• For dynamic components: use an appropriate Cache-Control header to help the browser with conditional requests

Web page designs are getting richer and richer, which means more scripts, stylesheets, images, and Flash in the page. A first-time visitor to your page may have to make several HTTP requests, but by using the Expires header you make those components cacheable. This avoids unnecessary HTTP requests on subsequent page views. Expires headers are most often used with images, but they should be used on all components including scripts, stylesheets, and Flash components.

Browsers (and proxies) use a cache to reduce the number and size of HTTP requests, making web pages load faster. A web server uses the Expires header in the HTTP response to tell the client how long a component can be cached. This is a far future Expires header, telling the browser that this response won’t be stale until April 15, 2010.

Expires: Thu, 15 Apr 2010 20:00:00 GMT

If your server is Apache, use the ExpiresDefault directive to set an expiration date relative to the current date. This example of the ExpiresDefault directive sets the Expires date 10 years out from the time of the request.

ExpiresDefault "access plus 10 years"

Keep in mind, if you use a far future Expires header you have to change the component’s filename whenever the component changes. At Yahoo! we often make this step part of the build process: a version number is embedded in the component’s filename, for example, yahoo_2.0.6.js.

Using a far future Expires header affects page views only after a user has already visited your site. It has no effect on the number of HTTP requests when a user visits your site for the first time and the browser’s cache is empty. Therefore the impact of this performance improvement depends on how often users hit your pages with a primed cache. (A “primed cache” already contains all of the components in the page.) We measured this at Yahoo! and found the number of page views with a primed cache is 75-85%. By using a far future Expires header, you increase the number of components that are cached by the browser and re-used on subsequent page views without sending a single byte over the user’s Internet connection.

Gzip Components

The time it takes to transfer an HTTP request and response across the network can be significantly reduced by decisions made by front-end engineers. It’s true that the end-user’s bandwidth speed, Internet service provider, proximity to peering exchange points, etc. are beyond the control of the development team. But there are other variables that affect response times. Compression reduces response times by reducing the size of the HTTP response.

Starting with HTTP/1.1, web clients indicate support for compression with the Accept-Encoding header in the HTTP request.

Accept-Encoding: gzip, deflate

If the web server sees this header in the request, it may compress the response using one of the methods listed by the client. The web server notifies the web client of this via the Content-Encoding header in the response.

Content-Encoding: gzip

Gzip is the most popular and effective compression method at this time. It was developed by the GNU project and standardized by RFC 1952. The only other compression format you’re likely to see is deflate, but it’s less effective and less popular.

Gzipping generally reduces the response size by about 70%. Approximately 90% of today’s Internet traffic travels through browsers that claim to support gzip. If you use Apache, the module configuring gzip depends on your version: Apache 1.3 uses mod_gzip while Apache 2.x uses mod_deflate.

There are known issues with browsers and proxies that may cause a mismatch in what the browser expects and what it receives with regard to compressed content. Fortunately, these edge cases are dwindling as the use of older browsers drops off. The Apache modules help out by adding appropriate Vary response headers automatically.

Servers choose what to gzip based on file type, but are typically too limited in what they decide to compress. Most web sites gzip their HTML documents. It’s also worthwhile to gzip your scripts and stylesheets, but many web sites miss this opportunity. In fact, it’s worthwhile to compress any text response including XML and JSON. Image and PDF files should not be gzipped because they are already compressed. Trying to gzip them not only wastes CPU but can potentially increase file sizes.

Gzipping as many file types as possible is an easy way to reduce page weight and accelerate the user experience.

Put Stylesheets at the Top

While researching performance at Yahoo!, we discovered that moving stylesheets to the document HEAD makes pages appear to be loading faster. This is because putting stylesheets in the HEAD allows the page to render progressively.

Front-end engineers that care about performance want a page to load progressively; that is, we want the browser to display whatever content it has as soon as possible. This is especially important for pages with a lot of content and for users on slower Internet connections. The importance of giving users visual feedback, such as progress indicators, has been well researched and documented. In our case the HTML page is the progress indicator! When the browser loads the page progressively the header, the navigation bar, the logo at the top, etc. all serve as visual feedback for the user who is waiting for the page. This improves the overall user experience.

The problem with putting stylesheets near the bottom of the document is that it prohibits progressive rendering in many browsers, including Internet Explorer. These browsers block rendering to avoid having to redraw elements of the page if their styles change. The user is stuck viewing a blank white page.

The HTML specification clearly states that stylesheets are to be included in the HEAD of the page: “Unlike A, [LINK] may only appear in the HEAD section of a document, although it may appear any number of times.” Neither of the alternatives, the blank white screen or flash of unstyled content, are worth the risk. The optimal solution is to follow the HTML specification and load your stylesheets in the document HEAD.

Put Scripts at the Bottom

The problem caused by scripts is that they block parallel downloads. The HTTP/1.1 specification suggests that browsers download no more than two components in parallel per hostname. If you serve your images from multiple hostnames, you can get more than two downloads to occur in parallel. While a script is downloading, however, the browser won’t start any other downloads, even on different hostnames.

In some situations it’s not easy to move scripts to the bottom. If, for example, the script uses document.write to insert part of the page’s content, it can’t be moved lower in the page. There might also be scoping issues. In many cases, there are ways to workaround these situations.

An alternative suggestion that often comes up is to use deferred scripts. The DEFER attribute indicates that the script does not contain document.write, and is a clue to browsers that they can continue rendering. Unfortunately, Firefox doesn’t support the DEFER attribute. In Internet Explorer, the script may be deferred, but not as much as desired. If a script can be deferred, it can also be moved to the bottom of the page. That will make your web pages load faster.

Avoid CSS Expressions

CSS expressions are a powerful (and dangerous) way to set CSS properties dynamically. They were supported in Internet Explorer starting with version 5, but were deprecated starting with IE8. As an example, the background color could be set to alternate every hour using CSS expressions:

background-color: expression( (new Date()).getHours()%2 ? "#B8D4FF" : "#F08A00" );

As shown here, the expression method accepts a JavaScript expression. The CSS property is set to the result of evaluating the JavaScript expression. The expression method is ignored by other browsers, so it is useful for setting properties in Internet Explorer needed to create a consistent experience across browsers.

The problem with expressions is that they are evaluated more frequently than most people expect. Not only are they evaluated when the page is rendered and resized, but also when the page is scrolled and even when the user moves the mouse over the page. Adding a counter to the CSS expression allows us to keep track of when and how often a CSS expression is evaluated. Moving the mouse around the page can easily generate more than 10,000 evaluations.

One way to reduce the number of times your CSS expression is evaluated is to use one-time expressions, where the first time the expression is evaluated it sets the style property to an explicit value, which replaces the CSS expression. If the style property must be set dynamically throughout the life of the page, using event handlers instead of CSS expressions is an alternative approach. If you must use CSS expressions, remember that they may be evaluated thousands of times and could affect the performance of your page.

Make JavaScript and CSS External

Many of these performance rules deal with how external components are managed. However, before these considerations arise you should ask a more basic question: Should JavaScript and CSS be contained in external files, or inlined in the page itself?

Using external files in the real world generally produces faster pages because the JavaScript and CSS files are cached by the browser. JavaScript and CSS that are inlined in HTML documents get downloaded every time the HTML document is requested. This reduces the number of HTTP requests that are needed, but increases the size of the HTML document. On the other hand, if the JavaScript and CSS are in external files cached by the browser, the size of the HTML document is reduced without increasing the number of HTTP requests.

The key factor, then, is the frequency with which external JavaScript and CSS components are cached relative to the number of HTML documents requested. This factor, although difficult to quantify, can be gauged using various metrics. If users on your site have multiple page views per session and many of your pages re-use the same scripts and stylesheets, there is a greater potential benefit from cached external files.

Many web sites fall in the middle of these metrics. For these sites, the best solution generally is to deploy the JavaScript and CSS as external files. The only exception where inlining is preferable is with home pages, such as Yahoo!’s front page and My Yahoo!. Home pages that have few (perhaps only one) page view per session may find that inlining JavaScript and CSS results in faster end-user response times.

For front pages that are typically the first of many page views, there are techniques that leverage the reduction of HTTP requests that inlining provides, as well as the caching benefits achieved through using external files. One such technique is to inline JavaScript and CSS in the front page, but dynamically download the external files after the page has finished loading. Subsequent pages would reference the external files that should already be in the browser’s cache.

Reduce DNS Lookups

The Domain Name System (DNS) maps hostnames to IP addresses, just as phonebooks map people’s names to their phone numbers. When you type www.yahoo.com into your browser, a DNS resolver contacted by the browser returns that server’s IP address. DNS has a cost. It typically takes 20-120 milliseconds for DNS to lookup the IP address for a given hostname. The browser can’t download anything from this hostname until the DNS lookup is completed.

DNS lookups are cached for better performance. This caching can occur on a special caching server, maintained by the user’s ISP or local area network, but there is also caching that occurs on the individual user’s computer. The DNS information remains in the operating system’s DNS cache (the “DNS Client service” on Microsoft Windows). Most browsers have their own caches, separate from the operating system’s cache. As long as the browser keeps a DNS record in its own cache, it doesn’t bother the operating system with a request for the record.

Internet Explorer caches DNS lookups for 30 minutes by default, as specified by the DnsCacheTimeout registry setting. Firefox caches DNS lookups for 1 minute, controlled by the network.dnsCacheExpiration configuration setting. (Fasterfox changes this to 1 hour.)

When the client’s DNS cache is empty (for both the browser and the operating system), the number of DNS lookups is equal to the number of unique hostnames in the web page. This includes the hostnames used in the page’s URL, images, script files, stylesheets, Flash objects, etc. Reducing the number of unique hostnames reduces the number of DNS lookups.

Reducing the number of unique hostnames has the potential to reduce the amount of parallel downloading that takes place in the page. Avoiding DNS lookups cuts response times, but reducing parallel downloads may increase response times. My guideline is to split these components across at least two but no more than four hostnames. This results in a good compromise between reducing DNS lookups and allowing a high degree of parallel downloads.

Minify JavaScript and CSS

Minification is the practice of removing unnecessary characters from code to reduce its size thereby improving load times. When code is minified all comments are removed, as well as unneeded white space characters (space, newline, and tab). In the case of JavaScript, this improves response time performance because the size of the downloaded file is reduced. Two popular tools for minifying JavaScript code are JSMin and YUI Compressor. The YUI compressor can also minify CSS.

Obfuscation is an alternative optimization that can be applied to source code. It’s more complex than minification and thus more likely to generate bugs as a result of the obfuscation step itself. In a survey of ten top U.S. web sites, minification achieved a 21% size reduction versus 25% for obfuscation. Although obfuscation has a higher size reduction, minifying JavaScript is less risky.

In addition to minifying external scripts and styles, inlined <script> and <style> blocks can and should also be minified. Even if you gzip your scripts and styles, minifying them will still reduce the size by 5% or more. As the use and size of JavaScript and CSS increases, so will the savings gained by minifying your code.

Avoid Redirects

Redirects are accomplished using the 301 and 302 status codes. Here’s an example of the HTTP headers in a 301 response:

HTTP/1.1 301 Moved Permanently
Location: http://example.com/newuri
Content-Type: text/html

The browser automatically takes the user to the URL specified in the Location field. All the information necessary for a redirect is in the headers. The body of the response is typically empty. Despite their names, neither a 301 nor a 302 response is cached in practice unless additional headers, such as Expires or Cache-Control, indicate it should be. The meta refresh tag and JavaScript are other ways to direct users to a different URL, but if you must do a redirect, the preferred technique is to use the standard 3xx HTTP status codes, primarily to ensure the back button works correctly.

The main thing to remember is that redirects slow down the user experience. Inserting a redirect between the user and the HTML document delays everything in the page since nothing in the page can be rendered and no components can start being downloaded until the HTML document has arrived.

One of the most wasteful redirects happens frequently and web developers are generally not aware of it. It occurs when a trailing slash (/) is missing from a URL that should otherwise have one. For example, going to http://astrology.yahoo.com/astrology results in a 301 response containing a redirect to http://astrology.yahoo.com/astrology/ (notice the added trailing slash). This is fixed in Apache by using Alias or mod_rewrite, or the DirectorySlash directive if you’re using Apache handlers.

Connecting an old web site to a new one is another common use for redirects. Others include connecting different parts of a website and directing the user based on certain conditions (type of browser, type of user account, etc.). Using a redirect to connect two web sites is simple and requires little additional coding. Although using redirects in these situations reduces the complexity for developers, it degrades the user experience. Alternatives for this use of redirects include using Alias and mod_rewrite if the two code paths are hosted on the same server. If a domain name change is the cause of using redirects, an alternative is to create a CNAME (a DNS record that creates an alias pointing from one domain name to another) in combination with Alias or mod_rewrite.

Remove Duplicate Scripts

It hurts performance to include the same JavaScript file twice in one page. This isn’t as unusual as you might think. A review of the ten top U.S. web sites shows that two of them contain a duplicated script. Two main factors increase the odds of a script being duplicated in a single web page: team size and number of scripts. When it does happen, duplicate scripts hurt performance by creating unnecessary HTTP requests and wasted JavaScript execution.

Unnecessary HTTP requests happen in Internet Explorer, but not in Firefox. In Internet Explorer, if an external script is included twice and is not cacheable, it generates two HTTP requests during page loading. Even if the script is cacheable, extra HTTP requests occur when the user reloads the page.

In addition to generating wasteful HTTP requests, time is wasted evaluating the script multiple times. This redundant JavaScript execution happens in both Firefox and Internet Explorer, regardless of whether the script is cacheable.

One way to avoid accidentally including the same script twice is to implement a script management module in your templating system. The typical way to include a script is to use the SCRIPT tag in your HTML page.

<script type="text/javascript" src="menu_1.0.17.js"></script>

An alternative in PHP would be to create a function called insertScript.

<?php insertScript("menu.js") ?>

In addition to preventing the same script from being inserted multiple times, this function could handle other issues with scripts, such as dependency checking and adding version numbers to script filenames to support far future Expires headers.

Configure ETags

Entity tags (ETags) are a mechanism that web servers and browsers use to determine whether the component in the browser’s cache matches the one on the origin server. (An “entity” is another word a “component”: images, scripts, stylesheets, etc.) ETags were added to provide a mechanism for validating entities that is more flexible than the last-modified date. An ETag is a string that uniquely identifies a specific version of a component. The only format constraints are that the string be quoted. The origin server specifies the component’s ETag using the ETag response header.

HTTP/1.1 200 OK
Last-Modified: Tue, 12 Dec 2006 03:03:59 GMT
ETag: "10c24bc-4ab-457e1c1f"
Content-Length: 12195

Later, if the browser has to validate a component, it uses the If-None-Match header to pass the ETag back to the origin server. If the ETags match, a 304 status code is returned reducing the response by 12195 bytes for this example.

GET /i/yahoo.gif HTTP/1.1
Host: us.yimg.com
If-Modified-Since: Tue, 12 Dec 2006 03:03:59 GMT
If-None-Match: "10c24bc-4ab-457e1c1f"
HTTP/1.1 304 Not Modified

The problem with ETags is that they typically are constructed using attributes that make them unique to a specific server hosting a site. ETags won’t match when a browser gets the original component from one server and later tries to validate that component on a different server, a situation that is all too common on Web sites that use a cluster of servers to handle requests. By default, both Apache and IIS embed data in the ETag that dramatically reduces the odds of the validity test succeeding on web sites with multiple servers.

The ETag format for Apache 1.3 and 2.x is inode-size-timestamp. Although a given file may reside in the same directory across multiple servers, and have the same file size, permissions, timestamp, etc., its inode is different from one server to the next.

IIS 5.0 and 6.0 have a similar issue with ETags. The format for ETags on IIS is Filetimestamp:ChangeNumber. A ChangeNumber is a counter used to track configuration changes to IIS. It’s unlikely that the ChangeNumber is the same across all IIS servers behind a web site.

The end result is ETags generated by Apache and IIS for the exact same component won’t match from one server to another. If the ETags don’t match, the user doesn’t receive the small, fast 304 response that ETags were designed for; instead, they’ll get a normal 200 response along with all the data for the component. If you host your web site on just one server, this isn’t a problem. But if you have multiple servers hosting your web site, and you’re using Apache or IIS with the default ETag configuration, your users are getting slower pages, your servers have a higher load, you’re consuming greater bandwidth, and proxies aren’t caching your content efficiently. Even if your components have a far future Expires header, a conditional GET request is still made whenever the user hits Reload or Refresh.

If you’re not taking advantage of the flexible validation model that ETags provide, it’s better to just remove the ETag altogether. The Last-Modified header validates based on the component’s timestamp. And removing the ETag reduces the size of the HTTP headers in both the response and subsequent requests. This Microsoft Support article describes how to remove ETags. In Apache, this is done by simply adding the following line to your Apache configuration file:

FileETag none

Make Ajax Cacheable

One of the cited benefits of Ajax is that it provides instantaneous feedback to the user because it requests information asynchronously from the backend web server. However, using Ajax is no guarantee that the user won’t be twiddling his thumbs waiting for those asynchronous JavaScript and XML responses to return. In many applications, whether or not the user is kept waiting depends on how Ajax is used. For example, in a web-based email client the user will be kept waiting for the results of an Ajax request to find all the email messages that match their search criteria. It’s important to remember that “asynchronous” does not imply “instantaneous”.

To improve performance, it’s important to optimize these Ajax responses. The most important way to improve the performance of Ajax is to make the responses cacheable, as discussed in Add an Expires or a Cache-Control Header. Some of the other rules also apply to Ajax:

• Gzip Components
• Reduce DNS Lookups
• Minify JavaScript
• Avoid Redirects
• Configure ETags

Let’s look at an example. A Web 2.0 email client might use Ajax to download the user’s address book for autocompletion. If the user hasn’t modified her address book since the last time she used the email web app, the previous address book response could be read from cache if that Ajax response was made cacheable with a future Expires or Cache-Control header. The browser must be informed when to use a previously cached address book response versus requesting a new one. This could be done by adding a timestamp to the address book Ajax URL indicating the last time the user modified her address book, for example, &t=1190241612. If the address book hasn’t been modified since the last download, the timestamp will be the same and the address book will be read from the browser’s cache eliminating an extra HTTP roundtrip. If the user has modified her address book, the timestamp ensures the new URL doesn’t match the cached response, and the browser will request the updated address book entries.

Even though your Ajax responses are created dynamically, and might only be applicable to a single user, they can still be cached. Doing so will make your Web 2.0 apps faster.

Flush the Buffer Early

When users request a page, it can take anywhere from 200 to 500ms for the backend server to stitch together the HTML page. During this time, the browser is idle as it waits for the data to arrive. In PHP you have the function flush(). It allows you to send your partially ready HTML response to the browser so that the browser can start fetching components while your backend is busy with the rest of the HTML page. The benefit is mainly seen on busy backends or light frontends.

A good place to consider flushing is right after the HEAD because the HTML for the head is usually easier to produce and it allows you to include any CSS and JavaScript files for the browser to start fetching in parallel while the backend is still processing. Example:

... <!-- css, js -->
</head>
<?php flush(); ?>
<body>
... <!-- content -->

Yahoo! search pioneered research and real user testing to prove the benefits of using this technique.

Use GET for AJAX Requests

The Yahoo! Mail team found that when using XMLHttpRequest, POST is implemented in the browsers as a two-step process: sending the headers first, then sending data. So it’s best to use GET, which only takes one TCP packet to send (unless you have a lot of cookies). The maximum URL length in IE is 2K, so if you send more than 2K data you might not be able to use GET.

An interesting side affect is that POST without actually posting any data behaves like GET. Based on the HTTP specs, GET is meant for retrieving information, so it makes sense (semantically) to use GET when you’re only requesting data, as opposed to sending data to be stored server-side.

Post-load Components

You can take a closer look at your page and ask yourself: “What’s absolutely required in order to render the page initially?”. The rest of the content and components can wait.

JavaScript is an ideal candidate for splitting before and after the onload event. For example if you have JavaScript code and libraries that do drag and drop and animations, those can wait, because dragging elements on the page comes after the initial rendering. Other places to look for candidates for post-loading include hidden content (content that appears after a user action) and images below the fold.

Tools to help you out in your effort: YUI Image Loader allows you to delay images below the fold and the YUI Get utility is an easy way to include JS and CSS on the fly. For an example in the wild take a look at Yahoo! Home Page with Firebug’s Net Panel turned on.

It’s good when the performance goals are inline with other web development best practices. In this case, the idea of progressive enhancement tells us that JavaScript, when supported, can improve the user experience but you have to make sure the page works even without JavaScript. So after you’ve made sure the page works fine, you can enhance it with some post-loaded scripts that give you more bells and whistles such as drag and drop and animations.

Preload Components

Preload may look like the opposite of post-load, but it actually has a different goal. By preloading components you can take advantage of the time the browser is idle and request components (like images, styles and scripts) you’ll need in the future. This way when the user visits the next page, you could have most of the components already in the cache and your page will load much faster for the user.

There are actually several types of preloading:

• Unconditional preload – as soon as onload fires, you go ahead and fetch some extra components. Check google.com for an example of how a sprite image is requested onload. This sprite image is not needed on the google.com homepage, but it is needed on the consecutive search result page.
• Conditional preload – based on a user action you make an educated guess where the user is headed next and preload accordingly. On search.yahoo.com you can see how some extra components are requested after you start typing in the input box.
• Anticipated preload – preload in advance before launching a redesign. It often happens after a redesign that you hear: “The new site is cool, but it’s slower than before”. Part of the problem could be that the users were visiting your old site with a full cache, but the new one is always an empty cache experience. You can mitigate this side effect by preloading some components before you even launched the redesign. Your old site can use the time the browser is idle and request images and scripts that will be used by the new site.

Reduce the Number of DOM Elements

A complex page means more bytes to download and it also means slower DOM access in JavaScript. It makes a difference if you loop through 500 or 5000 DOM elements on the page when you want to add an event handler for example.

A high number of DOM elements can be a symptom that there’s something that should be improved with the markup of the page without necessarily removing content. Are you using nested tables for layout purposes? Are you throwing in more <div>s only to fix layout issues? Maybe there’s a better and more semantically correct way to do your markup.

A great help with layouts are the YUI CSS utilities: grids.css can help you with the overall layout, fonts.css and reset.css can help you strip away the browser’s defaults formatting. This is a chance to start fresh and think about your markup, for example use <div>s only when it makes sense semantically, and not because it renders a new line.

The number of DOM elements is easy to test, just type in Firebug’s console:

document.getElementsByTagName('*').length

And how many DOM elements are too many? Check other similar pages that have good markup. For example the Yahoo! Home Page is a pretty busy page and still under 700 elements (HTML tags).

Split Components Across Domains

Splitting components allows you to maximize parallel downloads. Make sure you’re using not more than 2-4 domains because of the DNS lookup penalty. For example, you can host your HTML and dynamic content on www.example.org and split static components between static1.example.org and static2.example.org

For more information check “Maximizing Parallel Downloads in the Carpool Lane” by Tenni Theurer and Patty Chi.

Minimize the Number of iframes

Iframes allow an HTML document to be inserted in the parent document. It’s important to understand how iframes work so they can be used effectively.

iframe pros:

• Helps with slow third-party content like badges and ads
• Security sandbox
• Download scripts in parallel

iframe cons:

• Costly even if blank
• Blocks page onload
• Non-semantic

No 404s

HTTP requests are expensive so making an HTTP request and getting a useless response (i.e. 404 Not Found) is totally unnecessary and will slow down the user experience without any benefit.

Some sites have helpful 404s “Did you mean X?”, which is great for the user experience but also wastes server resources (like database, etc). Particularly bad is when the link to an external JavaScript is wrong and the result is a 404. First, this download will block parallel downloads. Next the browser may try to parse the 404 response body as if it were JavaScript code, trying to find something usable in it.

Reduce Cookie Size

HTTP cookies are used for a variety of reasons such as authentication and personalization. Information about cookies is exchanged in the HTTP headers between web servers and browsers. It’s important to keep the size of cookies as low as possible to minimize the impact on the user’s response time.

For more information check “When the Cookie Crumbles” by Tenni Theurer and Patty Chi. The take-home of this research:

• Eliminate unnecessary cookies
• Keep cookie sizes as low as possible to minimize the impact on the user response time
• Be mindful of setting cookies at the appropriate domain level so other sub-domains are not affected
• Set an Expires date appropriately. An earlier Expires date or none removes the cookie sooner, improving the user response time

Use Cookie-free Domains for Components

When the browser makes a request for a static image and sends cookies together with the request, the server doesn’t have any use for those cookies. So they only create network traffic for no good reason. You should make sure static components are requested with cookie-free requests. Create a subdomain and host all your static components there.

If your domain is www.example.org, you can host your static components on static.example.org. However, if you’ve already set cookies on the top-level domain example.org as opposed to www.example.org, then all the requests to static.example.org will include those cookies. In this case, you can buy a whole new domain, host your static components there, and keep this domain cookie-free. Yahoo! uses yimg.com, YouTube uses ytimg.com, Amazon uses images-amazon.com and so on.

Another benefit of hosting static components on a cookie-free domain is that some proxies might refuse to cache the components that are requested with cookies. On a related note, if you wonder if you should use example.org or www.example.org for your home page, consider the cookie impact. Omitting www leaves you no choice but to write cookies to *.example.org, so for performance reasons it’s best to use the www subdomain and write the cookies to that subdomain.

Minimize DOM Access

Accessing DOM elements with JavaScript is slow so in order to have a more responsive page, you should:
Cache references to accessed elements

• Update nodes “offline” and then add them to the tree
• Avoid fixing layout with JavaScript
• For more information check the YUI theatre’s “High Performance Ajax Applications” by Julien Lecomte.

Develop Smart Event Handlers

Sometimes pages feel less responsive because of too many event handlers attached to different elements of the DOM tree which are then executed too often. That’s why using event delegation is a good approach. If you have 10 buttons inside a div, attach only one event handler to the div wrapper, instead of one handler for each button. Events bubble up so you’ll be able to catch the event and figure out which button it originated from.

You also don’t need to wait for the onload event in order to start doing something with the DOM tree. Often all you need is the element you want to access to be available in the tree. You don’t have to wait for all images to be downloaded. DOMContentLoaded is the event you might consider using instead of onload, but until it’s available in all browsers, you can use the YUI Event utility, which has an onAvailable method.

For more information check the YUI theatre’s “High Performance Ajax Applications” by Julien Lecomte.

Choose <link> over @import

One of the previous best practices states that CSS should be at the top in order to allow for progressive rendering.

In IE @import behaves the same as using <link> at the bottom of the page, so it’s best not to use it.

Avoid Filters

The IE-proprietary AlphaImageLoader filter aims to fix a problem with semi-transparent true color PNGs in IE versions below 7. The problem with this filter is that it blocks rendering and freezes the browser while the image is being downloaded. It also increases memory consumption and is applied per element, not per image, so the problem is multiplied.

The best approach is to avoid AlphaImageLoader completely and use gracefully degrading PNG8 instead, which are fine in IE. If you absolutely need AlphaImageLoader, use the underscore hack _filter as to not penalize your IE7+ users.

Optimize Images

After a designer is done with creating the images for your web page, there are still some things you can try before you FTP those images to your web server.

You can check the GIFs and see if they are using a palette size corresponding to the number of colors in the image. Using imagemagick it’s easy to check using

identify -verbose image.gif

When you see an image useing 4 colors and a 256 color “slots” in the palette, there is room for improvement.

Try converting GIFs to PNGs and see if there is a saving. More often than not, there is. Developers often hesitate to use PNGs due to the limited support in browsers, but this is now a thing of the past. The only real problem is alpha-transparency in true color PNGs, but then again, GIFs are not true color and don’t support variable transparency either. So anything a GIF can do, a palette PNG (PNG8) can do too (except for animations). This simple imagemagick command results in totally safe-to-use PNGs:

convert image.gif image.png
"All we are saying is: Give PiNG a Chance!"

Run pngcrush (or any other PNG optimizer tool) on all your PNGs. Example:

pngcrush image.png -rem alla -reduce -brute result.png

Run jpegtran on all your JPEGs. This tool does lossless JPEG operations such as rotation and can also be used to optimize and remove comments and other useless information (such as EXIF information) from your images.

jpegtran -copy none -optimize -perfect src.jpg dest.jpg

Optimize CSS Sprites

Arranging the images in the sprite horizontally as opposed to vertically usually results in a smaller file size.

Combining similar colors in a sprite helps you keep the color count low, ideally under 256 colors so to fit in a PNG8.

“Be mobile-friendly” and don’t leave big gaps between the images in a sprite. This doesn’t affect the file size as much but requires less memory for the user agent to decompress the image into a pixel map. 100×100 image is 10 thousand pixels, where 1000×1000 is 1 million pixels

Don’t Scale Images in HTML

Don’t use a bigger image than you need just because you can set the width and height in HTML. If you need

<img width="100" height="100" src="mycat.jpg" alt="My Cat" />

then your image (mycat.jpg) should be 100x100px rather than a scaled down 500x500px image.

Make favicon.ico Small and Cacheable

The favicon.ico is an image that stays in the root of your server. It’s a necessary evil because even if you don’t care about it the browser will still request it, so it’s better not to respond with a 404 Not Found. Also since it’s on the same server, cookies are sent every time it’s requested. This image also interferes with the download sequence, for example in IE when you request extra components in the onload, the favicon will be downloaded before these extra components.

So to mitigate the drawbacks of having a favicon.ico make sure:

• It’s small, preferably under 1K.
• Set Expires header with what you feel comfortable (since you cannot rename it if you decide to change it). You can probably safely set the Expires header a few months in the future. You can check the last modified date of your current favicon.ico to make an informed decision.

Imagemagick can help you create small favicons.

Keep Components under 25K

This restriction is related to the fact that iPhone won’t cache components bigger than 25K. Note that this is the uncompressed size. This is where minification is important because gzip alone may not be sufficient.

For more information check “Performance Research, Part 5: iPhone Cacheability – Making it Stick” by Wayne Shea and Tenni Theurer.

Pack Components into a Multipart Document

Packing components into a multipart document is like an email with attachments, it helps you fetch several components with one HTTP request (remember: HTTP requests are expensive). When you use this technique, first check if the user agent supports it (iPhone does not).

Avoid Empty Image src

Image with empty string src attribute occurs more than one will expect. It appears in two form:

1. straight HTML
<img src="">

2. JavaScript
var img = new Image();
img.src = "";

Both forms cause the same effect: browser makes another request to your server.

• Internet Explorer makes a request to the directory in which the page is located.
• Safari and Chrome make a request to the actual page itself.
• Firefox 3 and earlier versions behave the same as Safari and Chrome, but version 3.5 addressed this issue[bug 444931] and no longer sends a request.
• Opera does not do anything when an empty image src is encountered.

Why is this behavior bad?

• Cripple your servers by sending a large amount of unexpected traffic, especially for pages that get millions of page views per day.
• Waste server computing cycles generating a page that will never be viewed.
• Possibly corrupt user data. If you are tracking state in the request, either by cookies or in another way, you have the possibility of destroying data. Even though the image request does not return an image, all of the headers are read and accepted by the browser, including all cookies. While the rest of the response is thrown away, the damage may already be done.

The root cause of this behavior is the way that URI resolution is performed in browsers. This behavior is defined in RFC 3986 – Uniform Resource Identifiers. When an empty string is encountered as a URI, it is considered a relative URI and is resolved according to the algorithm defined in section 5.2. This specific example, an empty string, is listed in section 5.4. Firefox, Safari, and Chrome are all resolving an empty string correctly per the specification, while Internet Explorer is resolving it incorrectly, apparently in line with an earlier version of the specification, RFC 2396 – Uniform Resource Identifiers (this was obsoleted by RFC 3986). So technically, the browsers are doing what they are supposed to do to resolve relative URIs. The problem is that in this context, the empty string is clearly unintentional.

HTML5 adds to the description of the tag’s src attribute to instruct browsers not to make an additional request in section 4.8.2:

• The src attribute must be present, and must contain a valid URL referencing a non-interactive, optionally animated, image resource that is neither paged nor scripted. If the base URI of the element is the same as the document’s address, then the src attribute’s value must not be the empty string.

Hopefully, browsers will not have this problem in the future. Unfortunately, there is no such clause for <script src=”"> and <link href=”">. Maybe there is still time to make that adjustment to ensure browsers don’t accidentally implement this behavior.

This rule was inspired by Yahoo!’s JavaScript guru Nicolas C. Zakas. For more information check out his article “Empty image src can destroy your site“.

SEO Glossary has a short description of log file:

A Log file (or web server log) records all information about a website’s incoming and outgoing traffic, including search engine spider activities, that can be analyzed by webmasters to improve a site’s SEO.

What is IIS?

Wikipedia defines “Internet Information Services”:

Internet Information Services (IIS) – formerly called Internet Information Server – is a web server application and set of feature extension modules created by Microsoft for use with Microsoft Windows. It is the most used web server after Apache HTTP Server.

If your website is hosted on a Microsoft Internet Information Services (IIS), you should:

• Log (record) all the requests to your site
• Extract the information from the log files
• Review and analyze the log files on a regular basis

IIS Log Format, IIS Log Field Definitions

An IIS log file can have 22 fields. Microsoft’s official IIS log site provides definitions for all the IIS log fields:

==============

EXAMPLE

FIELD – [Field Name]

Definition.

==============

Date – [date]

Date when the request occurs.

Time – [time]

Time (Coordinated Universal Time, UTC) when the request occurs.

Service Name and Instance Number – [s-sitename]

Internet service name and instance number that is running on the client.

Server Name – [s-computername]

Name of the server where the log file entry is generated.

Server IP Address – [s-ip]

Server’s IP.

Method – [cs-method]

The method used when the web page is requested, e.g. Get method or Post method.

URI Stem – [cs-uri-stem]

A target (e.g. a file) on your site is requested, e.g. If the home page of your site is requested: index.asp.

URI Query – [cs-uri-query]

A Universal Resource Identifier (URI) query is needs only when the client requests dynamic web pages. The URL Query (cs-uri-query) contains the the portion after the “?” symbol of the URL.

Server Port – [s-port]

Server port number configured for the service.

User Name – [cs-username]

Name of the authenticated user who accesses the server. Anonymous users are indicated by a hyphen (-).

Client IP Address – [c-ip]

Client’s IP.

Protocol Version – [cs-version]

Protocol version used when the request is made by the client. Protocol can either be HTTP or FTP, e.g. HTTP/1.1

User Agent – [cs(User-Agent)]

User agent is the browser type that the client used.

• When a human user requests a specific web page (URL) of your site through a browser, the name and version of the browser are recorded.
• When a search engine spider (e.g. Googlebot) requests a specific web page (URL) of your site, the search engine spider information/name is recorded.

Cookie – [cs(Cookie)]

Content of the cookie sent or received.

Referrer – [cs(Referer)]

Referrer is the website that referred the visit to your site.

• The IIS log captures the referrer as a referral URL.
• Web analytics tools (e.g. Google Analytics) also provide referral site / referral URL information.
• If a user is referred through a search engine’s organic results (e.g. Google’s organic SERP), then the user’s search query will be included in the referrer. Google Webmaster Tools does provide Google users’ search query report.

Host – [cs-host]

The host’s header name, e.g. www.gordonchoi.com

HTTP Status – [sc-status]

HTTP status code.

• Microsoft provides this official list of HTTP Status Codes on 1xx, 2xx, 3xx, 4xx, and 5xx.

Protocol Substatus – [sc-substatus]

Substatus error code.

Win32 Status – [sc-win32-status]

Windows status code.

Bytes Sent – [sc-bytes]

Number of bytes the server sends to the client during the request, in bytes.

Bytes Received – [cs-bytes]

Number of bytes the server receives from the client during the request, in bytes.

Time Taken – [time-taken]

Length of time the request takes, in milliseconds (ms).

• Google has confirmed website’s page load time is a factor in Google’s ranking algorithm.

Log File Analysis

Through the IIS log files:

• Webmasters can review site and/or web page errors through errors that show up in HTTP Status Codes (sc-status). Actions should be taken depending on the HTTP status code errors (e.g. 302, 404, 500, 503).
• Webmasters / SEO professionals can review different search engine spiders’ behavior to the site through the cs(User-Agent) field. After analyzing the search engine spider information, actions may be taken to optimize how Googlebot or other spiders access/crawl your site.

Tools to analyze IIS log files:

• Log Parser – You will need to apply some specific SQL queries when Log Parser.
• Most of the programming languages (e.g. C, PHP, Python) allow you to apply customized queries to IIS logs, but coding skills/experience is required to the programming languages you choose to use.
• Nihuo Weblog Analyzer – Allows you to generate a wide range of website statistics from your IIS log file and present in over 80 different reports, without SQL query writing skills or programming language experience.

Search Engine Spiders in IIS

The behavior of search engine spiders to your Internet Information Services (IIS) hosted website can be found in your site’s IIS log files. In the IIS logs, the cs(User-Agent) field records the search engine spiders.

• Google’s spider: Googlebot
• Bing’s spider: msnbot or Bingbot
• Yahoo’s spider: Slurp
• Yandex’s spider: YandexBot
• Baidu’s spider: Baiduspider
• Sogou’s spider: Sogou+web+spider or New-Sogou-Spider
• Soso’s spider: Sosospider
• Youdao’s spider: YoudaoBot
• Naver’s spider: Yeti
• Daum’s spider: Daumoa

Search engine spiders of each of the search engines look like the below in IIS log’s cs(User-Agent) field.

Google’s Search Engine Spiders

Google Web Search:

• Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)

Google Image:

• Googlebot-Image/1.0

Google Feed:

• Feedfetcher-Google;+(+http://www.google.com/feedfetcher.html;+feed-id=xxxxx)

Google Ads:

• AdsBot-Google+(+http://www.google.com/adsbot.html)
• Mediapartners-Google

Google Ads for Mobile:

• AdsBot-Google-Mobile+(+http://www.google.com/mobile/adsbot.html)+Mozilla+(iPhone;+U;+CPU+iPhone+OS+3+0+like+Mac+OS+X)+AppleWebKit+(KHTML,+like+Gecko)+Mobile+Safari

Bing Search Engine Spiders

Bing Web Search:

• Mozilla/5.0+(compatible;+bingbot/2.0;++http://www.bing.com/bingbot.htm)
• msnbot/2.0b+(+http://search.msn.com/msnbot.htm)

Bing Ads:

• msnbot-media/1.1+(+http://search.msn.com/msnbot.htm)

Yahoo Search Engine Spider

Yahoo Slurp Web Search:

• Mozilla/5.0+(compatible;+Yahoo!+Slurp/3.0;+http://help.yahoo.com/help/us/ysearch/slurp)
• Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)

Baidu Search Engine Spiders

Baidu Web Search:

• Baiduspider+(+http://www.baidu.com/search/spider.htm)
• Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)

Baidu Video Search:

• Baiduspider-video+(+http://www.baidu.com/search/spider.htm)

You can block Baiduspider’s access to your web pages by writing instructions in Baidu robots.txt.

Sogou’s Search Engine Spider

Sogou Web Search:

• New-Sogou-Spider/1.0+(compatible;+MSIE+5.5;+Windows+98)
• Sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

Soso’s Search Engine Spider

Soso Web Search:

• Sosospider+(+http://help.soso.com/webspider.htm)

Yahoo China’s Search Engine Spider

Yahoo Slurp China:

• Yahoo!+Slurp+China

Youdao Search Engine Spider

Youdao Web Search:

• Mozilla/5.0+(compatible;+YoudaoBot/1.0;+http://www.youdao.com/help/webmaster/spider/;+)

Yandex’s Search Engine Spider

Yandex Web Search:

• Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)

Naver’s Search Engine Spider

Naver Web Search:

• Yeti/1.0+(NHN+Corp.;+http://help.naver.com/robots/)

Daum’s Search Engine Spider

Daum Web Search:

• Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;++http://ws.daum.net/aboutWebSearch.html)+Daumoa/2.0

• Google+ posts
• Google Search Plus Your World

Google Search Plus Your World

Google Search Plus Your World was launched with official explanation from Google. Three new features are to show up on Google’s search results pages:

• Personal Results – Which enable you to find information just for you, such as Google+ photos and posts—both your own and those shared specifically with you, that only you will be able to see on your results page.
• Profiles in Search – Both in autocomplete and results, which enable you to immediately find people you’re close to or might be interested in following.
• People and Pages – Which help you find people profiles and Google+ pages related to a specific topic or area of interest, and enable you to follow them with just a few clicks. Because behind most every query is a community.

Google Search Plus Box Ranking Factors

Conversation Marketing explains the ranking factors for your Google Plus Profiles to show up on Google Search Plus Your World’s “Plus Boxes”. The ranking factors from the most important to least important:

• Reach
• Circle membership – Circle influential people with big circle membership.
• Freshness
• Post frequency – At least a few times a day. Remember, these posts can be re-shares or links to other interesting stuff you found on the web. It doesn’t have to all be original. Try to gain relevance for the non-commodity term that fits your industry. “Design”, not “Designer”. “Fashion”, not “Jeans”.
• Average plus ones per post
• Re-shares and replies (comments) per post – Comment on and re-share their stuffs that will get your followers to respond to them, and get you noticed. It’ll also help your own reach, as your re-shares are more likely to get passed around.

Factors that have little or no effect to Google+ SEO:

• Keyword relevance of your ‘about’ tab: Four profiles get consistent plus box placement for ‘seo’. Only two of them shows ‘SEO’ as the #1 topic in their about tab. Every profile has SEO as a topic, according to the content analysis API. But in two cases, it’s the third-ranking topic or lower.
• Topical relevance of your ‘about’ tab: That said, all plus box profiles for every test query had about tabs relevant to the overall ‘space’. All ranking ‘seo’ profiles, for example, cluster around search, search marketing, the industry and related topics. All ‘movies’ profiles cluster around show business. Meh. That could mean anything. But at least a bike shorts manufacturer won’t start showing up in SEO rankings (we hope).
• Keyword relevance of your posts: Totally meaningless. Write whatever you want. Seriously. Just make it good.

Google Plus SEO Best Practices

Blind Five Year Old has listed the 9 items of Google+ SEO Best Practices:

• Optimize your Google+ Profile – Fill out your Introduction, Employment, Education and Places lived understanding that the text in these sections is used for Google+ search. If you want to be returned for the term ‘SEO’ you need to have that term in one of these fields. Placing the term in two or more fields seems to deliver some added benefit. Do it if it makes sense but don’t go overboard.
• Get Verified – A verified name with that small checkmark icon seems to have some influence on Google+ search. When I figure out exactly how to make this happen I’ll let you know.
• Confirm Authorship – Google+ is an identity platform that allows you to claim Authorship of your content. Doing so delivers a type of rich snippet in search results that will increase the visibility and click-through rate on your content today and may put you in the catbird’s seat when Google implements AuthorRank.
• Use the +1 Button – Put the +1 button on your site prominently so users can deliver your content to Google+. Not only will this result in traffic to your site, it allows Google to understand how people are engaging with your content.
• Optimize Social Snippets – It’s not enough to just create content and have the +1 button on your site. Optimize your social snippet to ensure you’re getting the most engagement out of your content.
• Create Great Content – Put this in the ‘no duh’ category, but really, create great content. It’s not enough on its own but everything gets easier if you do this.
• Share Great Content – Great content takes time so you won’t always be sharing your content. Instead, share the great content of others. Make your Google+ feed valuable and interesting and you’ll be rewarded.
• Format Google+ Posts – Just putting a URL as your Google+ post won’t cut it. Create a bold title for each of your (longer) posts. Not only is this easier to read it’s what Google will use as the browser Title on Search+. You should also start to use hashtags, sparingly, to ensure your posts are aggregated appropriately.
• Use Google+ – Really use Google+. Using it for the express purpose of SEO won’t be successful. Do or do not. There is no try.

Search Engine Land has summarized Blind Five Year Old’s investigation on the possible ranking considerations when searching inside Google+ itself – Factors that influence Google’s search results when you type “SEO” into the Google+ search box and look for relevant other Google Plus users:

• The search term must appear in one of these sections of your profile: Introduction, Employment, Education or Places. For example, Danny Sullivan didn’t show up in Google+ searches for “SEO” until he added that keyword into the Introduction section of his Google+ profile.
• Using the keyword in more than one of those fields helps.
• The “Occupation” field isn’t used.
• There’s already a fair amount of spamming of these profile fields happening.

Google Plus SEO Example

Smart Insights demonstrates a Google Plus example on H&M in which the brand has managed to hijack a large portion of the search engine results page for ‘David Beckham’. If you count up the individual items, there are more than ten individual links through to H&M’s Google+ content when logged in, zero when not logged in.

H&M has achieved 2 things at the most basic level:

• Grown a large Google+ following – Big enough that they affect more than 500,000 peoples’ search results when they are logged into G+.
• Shared a lot of content related to David Beckham on Google+.

And when H&M’s followers reshare their Google+ content, and you’re following their sharers, the effect spreads much further.

Google Plus Growth Data (from Compete.com)

Compete.com reported on Google Plus data:

• Unique visits in December 2011: Google Plus (20 million) vs Twitter (40.4 million)
• Google+’s December unique visits have increased 4 times from July’s 5 million
• The plusone.google.com sub-domain had 60 million unique visits in December 2011

Google Plus for Social Media Marketing

For Google+ social media marketers to get the most out of Google Plus:

• Add Google +1 Buttons to Websites – Add +1 buttons to your site
• Google Plus Profile URL Shorterner – Make your Google+ Profile easy to remember
• Google Analytics Social Reports for Google +1 – Review the performance of the +1 social engagement, social actions and social pages in Google Analytics
• Google Plus Keyboard Shortcuts – Learn the keyboard shortcuts
• Google Plus Invitations – Invite more people to join Google+
• Google Plus Experts on Google Plus Social Network – Follow the Google Plus Experts

Tnooz

Tnooz – Global provider of news, analysis, commentary, data and business services to the travel, tourism and hospitality industry, focusing on travel technology, web strategy, social media and marketing.

Hotel Marketing

Hotel Marketing – Curated online travel and online marketing news for hotel marketers and hoteliers since 1997.

Travel Weekly

Travel Weekly – Travel trade weekly publication for the travel industry, covering topics in hotels, aviation, cruise, ground transportation, travel technology, online travel, and corporate travel.

Gadling

Gadling – Interesting, relevant travel stories, from travel tips to travel technology, and from budget to adventure travel.

Travel Rants

Travel Rants – Deals with travel problems and holiday complaints.

Space Travel

Space Travel – Space tourism destination website with resource for space adventurers seeking a personal perspective in space travel, allowing travelers to share space experience and trip reviews.

Airline Trends

Airline Trends – Commercial innovations and developments in the global aviation industry for marketers, product developers, managers, researchers, and suppliers.

eHotelier

eHotelier – Hospitality industry news for hoteliers, executives, travel companies, research firms, marketing groups, technology/service providers, publishers and hotel schools/universities, since 1999.

Rezgo T4 Blog

Rezgo T4 Blog – T4 stands for travel and tourism technology trends, created by Stephen Joyce. T4 is a blog sponsored by Rezgo, a next generation web 2.0 tour and activity booking engine for SME travel suppliers and tour operators.

Lonely Planet Blog

Lonely Planet Blog – Travel advice, information and inspiration from Lonely Planet’s online community.

United States Travel Association

US Travel Answer Sheet (PDF) – Facts about a leading American industry that’s more than just fun, created by US Travel Association.

Definitions of Adobe’s Flash and HTML5 on Wikipedia:

Adobe Flash (formerly Macromedia Flash) is a multimedia platform used to add animation, video, and interactivity to web pages. Flash is frequently used for advertisements, games and flash animations for broadcast. More recently, it has been positioned as a tool for “Rich Internet Applications” (“RIAs”).

HTML5 is a language for structuring and presenting content for the World Wide Web, and is a core technology of the Internet originally proposed by Opera Software. It is the fifth revision of the HTML standard (created in 1990 and standardized as HTML4 as of 1997) and as of December 2011 is still under development. Its core aims have been to improve the language with support for the latest multimedia while keeping it easily readable by humans and consistently understood by computers and devices (web browsers, parsers, etc.).

Adobe’s Chief Engineer Mike Chamber Clarifies Future of Flash

Adobe’s Chief Engineer Mike Chamber says:

• We are focusing work around the Flash Platform on mobile applications created with Adobe AIR and expressive content (particularly games and video) in the browser on the desktop via the Flash Player.
• We are further increasing the amount of resources (both money and engineer) toward HTML5 tools, solutions and browsers.
• We are no longer going to be actively developing the Flash Player for mobile browsers.

Why did Adobe decide to no longer develop the Flash Player for mobile browsers?

The Flash Player was not going to achieve the same ubiquity on mobile as it has on the desktop

This one should be pretty apparent, but given the fragmentation of the mobile market, and the fact that one of the leading mobile platforms (Apple’s iOS) was not going to allow the Flash Player in the browser, the Flash Player was not on track to reach anywhere near the ubiquity of the Flash Player on desktops.

This effectively meant that if you wanted to use Flash to deliver a rich web experience in the browser on mobile devices you would have to provide both a Flash based, as well as HTML5 based solution. Given the strong support for HTML5 across modern mobile devices, it simply made more sense to create an HTML5 based solution. Now, there are some exceptions to this, especially around advanced video content, but it is very clear that HTML5 is the solution to turn to if you want to provide a richer browser based experience that works across browsers on mobile devices.

Just to be very clear on this. No matter what we did, the Flash Player was not going to be available on Apple’s iOS anytime in the foreseeable future.

Ubiquity of HTML5 on mobile browsers

Related to the point above, HTML5 has very strong support on modern mobile devices and tablets. Indeed, on mobile devices, it has a level of ubiquity similar to what the Flash Player has on the desktop. While performance and implementations haven’t always been great or consistent across devices, they have continued to improve at a pretty dramatic rate (just look at the insane Canvas performance increases between iOS 4 and 5).

This new generation of smart phones and tablets (ushered in by the original Apple iPhone) are only a couple of years old. Because of this, the rendering engines deployed on these devices (most WebKit based) were all also relatively new and modern. The end result is that when developing for mobile devices and tablets today, you don’t have to deal with legacy browsers as you do on the desktop.

On mobile devices HTML5 provides a similar level of ubiquity that the Flash Player provides on the desktop. It is the best technology for creating and deploying rich content to the browser across mobile platforms.

Our goal has always been to obtain the same level of ubiquity for the Flash Player on mobile browsers, but, at the end of the day, it is something that did not, and was not going to happen.

Differences in how users consume rich content on mobile devices compared to the desktop

On the desktop, users are used to consuming rich content (such as games and applications) via both the browser and native applications. However, on mobile devices users are much more likely to look exclusively toward applications for consuming rich content. The mobile platforms make it very easy to discover new content and applications by providing tight integration between the app stores (Apple App Store, Android Marketplace, etc..) and the mobile operating system. In general, users do not look to the web on mobile devices for finding and consuming rich content (such as games and applications).

There are a number of reasons for this, including:

• Differences in screen sizes, resolution and interaction models between mobile devices and desktop PCs.
• Generally slower, and higher latency network connections (which is often metered) on mobile devices, which makes it cumbersome, sometimes expensive, and sometimes impossible to repeatedly load rich content from the web on demand.
• The tight integration with the underlying operating systems that native applications provide.
• The tight integration between mobile app stores and the mobile operating systems, which removes most of the friction for discovering new content.

When a user wants to play a game on a mobile device they turn to the app store for their platform. This makes it very easy for them to discover and install new content. This content can then be quickly accessed regardless of their network connectivity.

Essentially, users’ preferences to consume rich content on mobile devices via applications means that there is not as much need or demand for the Flash Player on mobile devices as there is on the desktop.

Scalability of developing plugins for mobile browsers

Developing the Flash Player for mobile browsers has proven to require much more resources than we anticipated. When building the player for desktop browsers, we can target well defined plugin APIs provided by the browsers. While we do have close relationships will all of the browser vendors (including Google, Apple, Firefox, Microsoft), as a general rule we can do most of our development using the existing APIs.

However, in the mobile ecosystem, we have to work very closely with other companies engineers on a number of levels:

• Mobile Operating System Vendors (such as Google and RIM)
• Hardware Device Manufacturers (such as Motorola and Samsung)
• Component Manufacturers (such as NVIDIA)

While we have good relationships on all levels of this ecosystem, having to do specific work for different combinations of OS, Hardware and event components has taken a significant amount of resources. For each new device, browser and operating system released, the resources required to develop, test and maintain the Flash Player also increases. This is something that we realized is simply not scalable or sustainable.

I have seen a couple of questions asking how Adobe AIR is different. There are a couple of differences which make AIR development significantly less resource intensive, including a more well defined API that we can target, as well as not needing to worry about differences in browsers or new browser versions. Ultimately though, developers are building successful applications with Adobe AIR, and thus it makes sense for us to continue to invest in it.

Shifting some Resources from Flash to HTML5

Finally, given the growth of HTML5 on both mobile and desktop browsers, we decided to more evenly balance our resources dedicate between Flash and HTML5.

Halting development on the Flash Player for mobile devices frees up resources for HTML5 development (tooling, frameworks, browsers).

I understand that not everyone may not agree with all of the conclusions drawn above. However, given these points, along with the increasing complexity and costs of developing the Flash Player for mobile browsers, we decided that further development was not the best use of our engineering resources.

Steve Jobs’ Thoughts on Flash

Apple’s Steve Jobs has expressed his thoughts on flash in May 2010:

I wanted to jot down some of our thoughts on Adobe’s Flash products so that customers and critics may better understand why we do not allow Flash on iPhones, iPods and iPads. Adobe has characterized our decision as being primarily business driven – they say we want to protect our App Store – but in reality it is based on technology issues. Adobe claims that we are a closed system, and that Flash is open, but in fact the opposite is true. Let me explain.

Open:

Adobe’s Flash products are 100% proprietary. They are only available from Adobe, and Adobe has sole authority as to their future enhancement, pricing, etc. While Adobe’s Flash products are widely available, this does not mean they are open, since they are controlled entirely by Adobe and available only from Adobe. By almost any definition, Flash is a closed system.

Apple has many proprietary products too. Though the operating system for the iPhone, iPod and iPad is proprietary, we strongly believe that all standards pertaining to the web should be open. Rather than use Flash, Apple has adopted HTML5, CSS and JavaScript – all open standards. Apple’s mobile devices all ship with high performance, low power implementations of these open standards. HTML5, the new web standard that has been adopted by Apple, Google and many others, lets web developers create advanced graphics, typography, animations and transitions without relying on third party browser plug-ins (like Flash). HTML5 is completely open and controlled by a standards committee, of which Apple is a member.

Apple even creates open standards for the web. For example, Apple began with a small open source project and created WebKit, a complete open-source HTML5 rendering engine that is the heart of the Safari web browser used in all our products. WebKit has been widely adopted. Google uses it for Android’s browser, Palm uses it, Nokia uses it, and RIM (Blackberry) has announced they will use it too. Almost every smartphone web browser other than Microsoft’s uses WebKit. By making its WebKit technology open, Apple has set the standard for mobile web browsers.

Full Web:

Adobe has repeatedly said that Apple mobile devices cannot access “the full web” because 75% of video on the web is in Flash. What they don’t say is that almost all this video is also available in a more modern format, H.264, and viewable on iPhones, iPods and iPads. YouTube, with an estimated 40% of the web’s video, shines in an app bundled on all Apple mobile devices, with the iPad offering perhaps the best YouTube discovery and viewing experience ever. Add to this video from Vimeo, Netflix, Facebook, ABC, CBS, CNN, MSNBC, Fox News, ESPN, NPR, Time, The New York Times, The Wall Street Journal, Sports Illustrated, People, National Geographic, and many, many others. iPhone, iPod and iPad users aren’t missing much video.

Another Adobe claim is that Apple devices cannot play Flash games. This is true. Fortunately, there are over 50,000 games and entertainment titles on the App Store, and many of them are free. There are more games and entertainment titles available for iPhone, iPod and iPad than for any other platform in the world.

Reliability, Security and Performance:

Symantec recently highlighted Flash for having one of the worst security records in 2009. We also know first hand that Flash is the number one reason Macs crash. We have been working with Adobe to fix these problems, but they have persisted for several years now. We don’t want to reduce the reliability and security of our iPhones, iPods and iPads by adding Flash.

In addition, Flash has not performed well on mobile devices. We have routinely asked Adobe to show us Flash performing well on a mobile device, any mobile device, for a few years now. We have never seen it. Adobe publicly said that Flash would ship on a smartphone in early 2009, then the second half of 2009, then the first half of 2010, and now they say the second half of 2010. We think it will eventually ship, but we’re glad we didn’t hold our breath. Who knows how it will perform?

Battery’s Life:

To achieve long battery life when playing video, mobile devices must decode the video in hardware; decoding it in software uses too much power. Many of the chips used in modern mobile devices contain a decoder called H.264 – an industry standard that is used in every Blu-ray DVD player and has been adopted by Apple, Google (YouTube), Vimeo, Netflix and many other companies.

Although Flash has recently added support for H.264, the video on almost all Flash websites currently requires an older generation decoder that is not implemented in mobile chips and must be run in software. The difference is striking: on an iPhone, for example, H.264 videos play for up to 10 hours, while videos decoded in software play for less than 5 hours before the battery is fully drained.

When websites re-encode their videos using H.264, they can offer them without using Flash at all. They play perfectly in browsers like Apple’s Safari and Google’s Chrome without any plugins whatsoever, and look great on iPhones, iPods and iPads.

Touch:

Flash was designed for PCs using mice, not for touch screens using fingers. For example, many Flash websites rely on “rollovers”, which pop up menus or other elements when the mouse arrow hovers over a specific spot. Apple’s revolutionary multi-touch interface doesn’t use a mouse, and there is no concept of a rollover. Most Flash websites will need to be rewritten to support touch-based devices. If developers need to rewrite their Flash websites, why not use modern technologies like HTML5, CSS and JavaScript?

Even if iPhones, iPods and iPads ran Flash, it would not solve the problem that most Flash websites need to be rewritten to support touch-based devices.

Most Importantly:

Besides the fact that Flash is closed and proprietary, has major technical drawbacks, and doesn’t support touch based devices, there is an even more important reason we do not allow Flash on iPhones, iPods and iPads. We have discussed the downsides of using Flash to play video and interactive content from websites, but Adobe also wants developers to adopt Flash to create apps that run on our mobile devices.

We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps and hinders the enhancement and progress of the platform. If developers grow dependent on third party development libraries and tools, they can only take advantage of platform enhancements if and when the third party chooses to adopt the new features. We cannot be at the mercy of a third party deciding if and when they will make our enhancements available to our developers.

This becomes even worse if the third party is supplying a cross platform development tool. The third party may not adopt enhancements from one platform unless they are available on all of their supported platforms. Hence developers only have access to the lowest common denominator set of features. Again, we cannot accept an outcome where developers are blocked from using our innovations and enhancements because they are not available on our competitor’s platforms.

Flash is a cross platform development tool. It is not Adobe’s goal to help developers write the best iPhone, iPod and iPad apps. It is their goal to help developers write cross platform apps. And Adobe has been painfully slow to adopt enhancements to Apple’s platforms. For example, although Mac OS X has been shipping for almost 10 years now, Adobe just adopted it fully (Cocoa) two weeks ago when they shipped CS5. Adobe was the last major third party developer to fully adopt Mac OS X.

Our motivation is simple – we want to provide the most advanced and innovative platform to our developers, and we want them to stand directly on the shoulders of this platform and create the best apps the world has ever seen. We want to continually enhance the platform so developers can create even more amazing, powerful, fun and useful applications. Everyone wins – we sell more devices because we have the best apps, developers reach a wider and wider audience and customer base, and users are continually delighted by the best and broadest selection of apps on any platform.

Conclusion:

Flash was created during the PC era – for PCs and mice. Flash is a successful business for Adobe, and we can understand why they want to push it beyond PCs. But the mobile era is about low power devices, touch interfaces and open web standards – all areas where Flash falls short.

The avalanche of media outlets offering their content for Apple’s mobile devices demonstrates that Flash is no longer necessary to watch video or consume any kind of web content. And the 250,000 apps on Apple’s App Store proves that Flash isn’t necessary for tens of thousands of developers to create graphically rich applications, including games.

New open standards created in the mobile era, such as HTML5, will win on mobile devices (and PCs too). Perhaps Adobe should focus more on creating great HTML5 tools for the future, and less on criticizing Apple for leaving the past behind.

Periscopic – Flash vs HTML5 vs Apple iOS

Periscopic has this comparison among Flash, HTML5 and iOS.

Flash:

Flash is an established and unified platform that reaches 99% of desktop and laptop browsers with a broad and consistent feature set matched with solid performance. Applications can be developed for Flash with fewer technical restrictions and greater speed than is currently possible in HTML5. Currently, 85% of the most-visited web sites use Flash, and 75% of web video is published for Flash Player.

Perhaps the most significant drawback to developing with Flash is that some mobile browsers, specifically iOS Safari, lack support for the technology. However, while the mobile iOS market is growing, it currently only accounts for approximately 1% of the total browser market.

HTML5:

HTML5 is a synthesis of several new and existing technologies (primarily JavaScript, CSS3, Canvas, and SVG), and as such, has a broad potential developer base. It allows for the deployment of an immersive application in modern browsers that either do not, or cannot, have the Flash Plugin installed – most notably Safari for the iPad which has a screen size large enough to interact with an immersive application.

The HTML5 development landscape is still somewhat lawless, and there are known inconsistencies between browser implementations of core features. Being an aggregate of technologies, it lacks an established and unified methodology, which can increase the development needed for cross-browser compatibility.

Apple iOS:

Among mobile devices and tablets, Apple’s iPod Touch, iPhone, and iPad account for a large and growing market. The default web browser on these devices, iOS Safari, accounts for about 20% of the total mobile browser market. To date, Apple has sold over 100 million iPhones and nearly 20 million iPads.

Immersive content intended to target these devices must be developed in HTML5, or as a downloadable app. Unfortunately, the support of HTML5 on iOS Safari is incomplete, specifically in the realm of communication and animation protocols.

• Giving credits to the original writers/bloggers by linking to the web pages.
• Including part of their writings in this blog post as snippets.

Best Software Engineering Books

Noop.nl has this best software engineering books ever (the top 10):

• 1. Code Complete: A Practical Handbook of Software Construction – Steve McConnell
• 2. Head First Design Patterns – Elisabeth Freeman
• 3. Rapid Development – Steve McConnell
• 4. Design Patterns: Elements of Reusable Object-Oriented Software – Erich Gamma
• 5. Applied Cryptography: Protocols, Algorithms, and Source Code (2nd Edition) – Bruce Schneier
• 6. Agile Software Development: Principles, Patterns and Practices – Robert C. Martin
• 7. Joel on Software – Joel Spolsky
• 8. Peopleware: Productive Projects and Teams (2nd Edition) – Tom DeMarco and Timothy Lister
• 9. The Mythical Man-Month, Anniversary Edition (2nd Edition) – Frederick P. Brooks
• 10. Refactoring: Improving the Design of Existing Code – Martin Fowler

Popular Programming Languages for Software Engineers

Tiobe Index reveals/updates a list of the most popular programming languages for programmers / software engineers as well as this answer to the question “What programming languages should a modern day programmer have in his/her arsenal?” by Joshua Levy on Quora.

• C for general familiarity with systems programming, and understanding how compilers, memory management, and the stack work.
• C++ because any programmer who works with C should also, as a practical matter, know C++, and the values and drawbacks of its features relative to C. Performance-sensitive or large-scale data processing (such as code at Google, Facebook, or most other serious tech companies) is still often best written in (a subset of) C++. It also teaches some useful facts about messy topics like code generation (templates) and multiple inheritance.
• Python or Ruby for familiarity with mainstream, modern scripting languages. Dynamic typing, metaprogramming, rapid prototyping, and Web-oriented development are all easily learned with either of these languages. (Avoid debates on which is better. They are both good to know, and more similar than they are different.)
• JavaScript because you can’t be a well-rounded developer without knowing basic Web development. It’s an interesting language in itself, albeit with numerous flaws, and not all that different from Python or Ruby. It can also be used server side.
• Java or C# for mainstream, modern, imperative, garbage-collected languages. Development in a higher-level, more modern language like these is not just popular, but stylistically different enough from C++ that you probably shouldn’t avoid learning it. Java especially can introduce you to heavyweight, enterprise-style development (for better and for worse). They acquaint you with extensive sets of libraries that are good to know about, so you don’t waste time building things that already exist.
• Bash, at least at a basic level. This includes use of common Unix/Linux command-line tools like ls, grep, and sort. This is needed to be proficient in general problem-solving, debugging, and quick data processing on any Linux system. Without this, you may find yourself taking hours or more on something a hacker familiar with Bash could do in minutes.
• At least one or two other languages of a different family. It doesn’t matter if they are popular or esoteric, but you should have familiarity with functional languages (Lisp, Scheme, Clojure, etc.) and with type inference (ML, OCaml, Haskell, Scala), and ideally with more specialized languages (Prolog, Erlang, XSLT, etc.). Mathematical and statistical tools like Mathematica, Matlab, R, or Maple are also good. Some might say this isn’t important since industry doesn’t use these languages broadly, but it improves your understanding of programming enough that it can make a big difference in the long run.

You should also know one or two of the languages above at a deeper level. This would mean you have read and written a large amount of code in that language, know its subtleties, and have worked with or read the code of people who have expert-level proficiency in it. This also means knowing something of how the language internals work — at least look at code for the language’s compilers or interpreters, for its built-in libraries, and (if applicable) its native bytecode or assembly output.

Also, athough these are not usually called programming languages, a good programmer should also know database/query languages, formatting languages, and data formats. Minimally: SQL, HTML, CSS, XML, XPath, and regular expressions.

A couple other languages are important but less essential:

• PHP is the most popular server-side Web development language. However, it’s not in the essential list above, simply because it’s not a good first language due to its poor design and shortcomings, and because once you know Python or Ruby, it is not too difficult to pick up.
• Perl has a venerable history and in some cases is still the quickest tool for a job. It’s also fast. However, it is a poor choice in most situations due to its age and poor readability and maintainability.

Solid experience in all of this takes a while, but it’s one of those situations where being ahead keeps you ahead. Once you have breadth and depth, you’ll be stronger and more flexible than most professional programmers, which will help you find better and more challenging programming jobs, which will allow you to learn from the work and from others and keep improving your skills quickly.

Python vs Ruby

What are the advantages of Python over Ruby? Robert Cezar Matei’s answer on Quora:

• The two are more similar than they are different, in everything from design to disadvantages to common uses – you can’t really go wrong either way, and shouldn’t base your decision on syntactical differences.
• As a Rubyist, Python’s main advantage has nothing to do with the languages’ features. It’s more subjective: it seems to me that Python has more momentum amongst serious computer scientists. It’s increasingly popular in academic and scientific applications, and a lot of the technologists I respect the most seem to prefer it. By comparison, the Ruby community feels more designer-y and relatively more novice.
• What this means is that while the Ruby world has very slick out-of-the-box product solutions, the Python world seems to produce more exceptionally well-written components like Tornado (web framework). Combined with it being used at Google and the potential for stuff like LiveNode to be released as open-source, I’d cast my lot with Python if I were starting today.

What are the advantages of Ruby over Python? Jonathan Ragan-Kelley’s answer on Quora:

• Broadly, Ruby and Python implement very similar models, and have similarly powerful libraries and communities. The biggest distinction is in syntax.
• Ruby includes several syntactic features which make dynamic extension of and higher-order interaction with external (library) code more straightforward.
• Blocks allow arbitrary, multi-line closures to be cleanly passed in-line nearly anywhere. (Closures are anonymous functions which are lexically scoped inside the point at which they are defined, so they can access enclosing stack variables.) Python allows two syntaxes for in-line closure declaration: lambdas and nested def statements. Lambdas are equivalent to Ruby blocks, in that they are expressions which evaluate to a closure (and so can be used in-line), but syntactically they can only easy express a single expression, not multiple nested statements. Nested definitions have the full syntax of any other function definitions, and so easily support multiple statements and control flow within themselves, but they are statements not expressions, so they cannot be used in-line.
• Mix-ins allow imported classes and modules to be extended and overridden by user code after it is imported. Python’s runtime model is sufficiently dynamic to inject new or altered methods and fields into an external class or object, but its syntax is procedural (assigning values to fields of an object), rather than declarative and identical to the standard class definition syntax.

Python vs PHP

Why is Python better than PHP? Owen Yamauchi’s answer on Quora considers both technical reasons and cultural reasons.

Technical reasons:

• Python has a real module system.
• Python distinguishes indexed arrays from associative arrays.
• Python is stricter about dumb error conditions; e.g. undefined variables.
• Python’s philosophy is more general-purpose than PHP’s originally was and thus it does not have weird artifacts like magic global variables $_GET and $_POST.
• Python has functional-like features that are distinctive among mostly-imperative scripting languages, like list comprehensions. PHP has no such things.
• Python has fewer easily abusable ways to write shitty code like extract().
• Python does not exhibit as many head-scratchingly strange behavioral oddities as PHP, for example the fact that in PHP, you cannot use the array-index operator on a function call expression.
• Python generally has a nicer standard library.
• Python ships with a REPL, which includes an interactive documentation viewer. Facebook had to write their own for PHP.
• Python has passable Unicode support. PHP (5.2 anyway) is no better than C in this regard.

Cultural reasons:

• Python was not developed by a man who has publicly stated that he does not enjoy programming, and who clearly does not understand how to properly design a programming language.
• Python does not suffer from the “hobbyist” problem to the same extent that PHP does. This is the problem wherein someone with no real training in programming works through a PHP tutorial on the Internet (many of which were written by hobbyists who do not really understand programming) and then thinks they understand programming.
• On the flip side of this, Python is generally more popular in the open-source community, so there are more cool open-source Python projects than PHP.

Why would someone choose PHP over Python? Allen Cheung’s answer on Quora:

• More familiarity/comfort with C/C++ syntax
• Rapid prototyping, knowing the code probably won’t survive for long
• Writing against an API that has a mature PHP API and an immature Python one (e.g. the Facebook developer API in its early days)
• Choice of multiple mature web frameworks

Java, Shortcomings

Why do some people hate Java? Sergei Turin’s answer on Quora:

• Sometimes, the elegant implementation is just a function. Not a method. Not a class. Not a framework. Just a function. — John Carmack
• Back when I was starting out in computer science I thought by today we’d be writing a few lines of code to accomplish much. Instead, we write hundreds of thousands of lines of code to accomplish little. — Lispian
• Such is modern computing: everything simple is made too complicated because it’s easy to fiddle with; everything complicated stays complicated because it’s hard to fix. — Rob Pike
• Java is like a variant of the game of Tetris in which none of the pieces can fill gaps created by the other pieces, so all you can do is pile them up endlessly. — Steve Yegge
• Languages that try to disallow idiocy become themselves idiotic. — Rob Pike
• The problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle. — Joe Armstrong
• So much complexity in software comes from trying to make one thing do two things. — Ryan Singer
• The purpose of software engineering is to control complexity, not to create it. — Dr. Pamela Zave
• “Design patterns” are concepts used by people who can’t learn by any method except memorization, so in place of actual programming ability, they memorize “patterns” and throw each one in sequence at a problem until it works. — Jason Garrett-Glaser (aka Dark_Shikari)
• Under the doubtful disguise of the holy “code reuse” an insane amount of gratuitous complexity is added to our environment, which makes necessary industrial quantities of syntactical sugar to make the ensuing mess minimally manageable. — Uriel
• It is not that uncommon for the cost of an abstraction to outweigh the benefit it delivers. — John Carmack
• The beauty of small and simple code is that you can bend or break the rules as long it stays small and simple. Rules allow people to write code without thinking. [And when] you dont think [...] you get bloated code that just concatenates stupid patterns. People stop thinking and questioning [and] then its just worshipping some rules without any pruporse. — Cinap Lenrek
• The object-oriented model makes it easy to build up programs by accretion. What this often means, in practice, is that it provides a structured way to write spaghetti code. — Paul Graham
• We ignore the real problems and instead invent problems for which we know how to solve–and create a spaghetti tangled mess instead, thinking that our ‘invented’ problem is more interesting. We then even go on to congratulate ourselves for solving the “fake” problem, leaving the real problem alone. — William Woody
• The notion of OOP was that you could leverage other people’s code and get those advantages. But that’s not the case. Instead you find code that “almost” does what you want, and then tweak and adjust it. And most of the time, it’d have been faster to simply rewrite it — and the code base would have been smaller. — Lispian

C Programming Language – Building Blocks

What programming language has the maximum credibility today? Kevin Ernest Long’s answer on Quora:

• It’s still C. Assuming you mean the credibility of the language and not the credibility its use might confer upon the user.
• If I told you I was going to write the most efficient web server ever or a faster 3d shader algorithm and I was going to write in the “X” programming language. The most credible programming language would be C. This of course will vary based on your goals and constraints.
• My answer assumes that you are looking to solve hard problems rather than common problems.
• C is the most general purpose of all computer languages and is commonly used to create the operating systems and core system libraries that are the building blocks that more specialized languages, niche languages and DSLs (Domain Specific Languages) rely upon.
• If the task is unknown and the constraints unknown then C has the most credibility when it comes to the odds of success.
• Even more often than other languages; programming in C is a lot like creating your own mini-language to solve the problem at hand. I’ve programmed in over thirty languages and dialects most of which are derived from C in some way.
• I mostly program in languages and against libraries that are more specific to my problem domain (these days Python and JavaScript) as these are a more efficient way to solve the common problems most of us face.

Google’s C++ Crawler

Why did Google move from Python to C++ for use in its crawler? Benjy Weinberger and Eisar Lipkovitz both have answers on Quora:

Software Engineer at Foursquare, formerly at Twitter and Google, Benjy Weinberger:

• Google has very robust, state-of-the art C++ libraries that handle a great many of the concerns of large-scale distributed systems: networking, monitoring, logging and so on.
• The best-maintained versions of the client libraries for the internal systems the crawler interacts with are in C++.
• Even small performance gains add up to a lot of money when you multiply them by many thousands of machines.
• Google does not optimize for speed of development. Code is written once, read many times and run many, many, many times. The time it takes to write code is rarely the limiting factor.

Engineering VP at Google, Eisar Lipkovitz:

• Crawling is relatively small part of the “building an index” process.
• Python was “abandoned” from the core search stack around 2000.
• Crawling main bottleneck is concurrency. We crawl MANY pages per second and each one of them might take minutes to successfully crawl [they are server bound]. It’s entirely possible modern Python allows for massive parallelism per machine/process, but certainly wasn’t the case a decade ago.
• HTTP and other related protocols didn’t change much in over a decade so the lower level crawling code doesn’t benefit from the supposed faster development cycle of Python.

No Better Programming Language to Replace C++

Why hasn’t a fast, practical, and reasonable language supplanted C++? Joshua Levy’s answer on Quora:

• Massive investment is required. Bringing a new programming language into widespread use requires a large amount of effort and time. Usually about a decade, typically with the support of either a large company or community. This must include the design and development of the language, the development of (or integration with) standard libraries, the building of a solid compiler/interpreter, and — last but not least — the community development needed to popularize the language. Look at the languages in widespread use now — C, C++, Java, Python, Ruby, PHP, C#, JavaScript, Perl — and this is quite clear.
• Language superiority rarely drives adoption. Again, look historically. Lisp had garbage collection and the dynamic capabilities now appreciated in Ruby or Python, with better performance, yet failed to maintain traction. Hardly anyone thinks PHP is a well designed language, yet it’s still going strong (with the support of giants like Facebook), even when similar alternatives like Python or Ruby are by most assessments preferable. In fact, when it comes to adoption, practically every big factor is more important than the fundamental design of a language, including backing by businesses or communities (Sun, Microsoft, the Python community), tools (Eclipse, Visual Studio), library support (C++, PHP, Java, etc.), killer apps (Rails), platform (Mac/iPhone, .NET), and historic niche expertise (physicists with Fortran).
• Lack of expertise. Surprisingly, considering the number of people making their living programming, broad knowledge of programming languages is remarkably rare. Most developers know at most two or three of them well, and often they are from the same family (C++ and Java, for example). Partly this is due to the large investment needed to learn languages, and partly it is cultural insularism. (As an illustration, see how many Java programmers you can find who use REPL-style exploratory development, such as with BeanShell, Rhino, or whatever.) It’s harder to promote the importance of first-class functions, type inference, closures, metaprogramming, continuations, or around methods, for example, when they are unfamiliar concepts. Even worse, the designers of successful programming languages themselves often seem not to have seriously surveyed other languages for ideas. (There are of course exceptions here.) The consequence is, features that seem “reasonable” in retrospect were ignored at the time they could have been added.
• Difficulty of design. Even if you have brilliant and informed designers with plenty of support, it’s still hard to make good choices when designing a language, because it takes so long to learn your mistakes, both of omission and commission. Fundamentally, language design is a case where tiny details compound to have giant effects. It’s hard to predict what a half million-line codebase in a new language will look like, and what the problems will be. And by the time you know, it’s usually too late to change anything substantial.
• Difficulty of implementation. Writing a good compiler or interpreter and platform is hard. And without a full-featured and solid platform, the language will go nowhere. It’s easy to start, and run a few toy programs, but the expertise required to build, say, an optimizing compiler that competes with gcc on optimization, or a virtual machine that competes with Sun’s JVM, is immense. And it needs to be built solid enough, or at least have enough support, that people will trust it enough to use it.
• Rapid software business cycle. In the software industry, perhaps because it is so volatile, businesses often avoid long-term investments of this sort. Although there are almost certainly massive productivity gains to be had by improving the state of programming languages, the investment is on such a long scale relative to other factors (such as evolving use of the Web, or mobile technologies, or consumer products) that such investments usually seem expensive and risky.

It’s all too easy to forget just how large these barriers are when excited by a new programming language (or frustrated with the obvious shortcomings of an old one). Really, unlike many situations in software engineering or programming, where quick hacks can have big payoff, getting widespread adoption of a new langauge may be more akin to building a large company, with the same challenges of having a good solution, a strong organizational leader, an angle for breaking into the market and then maintaining and growing user base, and so on.

.Net Programmers

Why we don’t hire .NET programmers from Expensify Blog explains why .Net programming don’t fit with tech start-ups:

• If you ever want to work in a startup, avoid .NET. It does you no favors. If you are a start-up looking to hire really excellent people, take notice of .NET on a resume, and ask why it’s there.
• Yes, .NET developers are great at what they do. It’s the most modern platform for application development on the planet. Microsoft has always produced the best tools for building internal business applications, and .NET is their masterpiece. There’s a reason why they own that space; they earned it. That space employs millions of people, and those people are unquestionably the masters at what they do.
• But what they do is very, very rarely start-ups. I’m sorry if that sounds offensive, but that’s a simple fact. You can measure this yourself: go to any list of startups, maybe look at YCombinator’s recent graduate class, or Scoble’s list of picks for 2010, or Sequoia’s list of seed companies – or go find a list you like better. Do a curl (or your .NET equivalent) on each domain, and see how many are running a Windows server: I think you’ll find the fraction very small. Don’t get me wrong: there are a lot of people who aren’t interested in working at a startup, for a lot of very valid reasons. But if you are the odd person who is interested, it’s worth asking: why do so few use .NET?
• Because .NET is designed to extend, not disrupt. The same could largely be said about J2EE, but in general anybody who “grows up” with these self-contained, highly-automated platforms can’t help but view computers in a different way than those who start from a lower, less automated, more difficult level. As one of the coherent commentators says below, Joel Spolsky himself laments schools teaching Java with the same basic reasoning of my article above, albeit more diplomatically stated. .NET is designed to tightly integrate with and seamlessly extend the Microsoft stack in extremely powerful but ultimately incremental ways. Again, there’s nothing wrong about that if that’s what you want to do. But if that’s not what you want to do, then .NET probably isn’t the right choice, as evidenced by how few people in the startup world choose it.

Getting into Programming

Software engineers / programmers all have stories on how they got into programming in the first place.

Jean Hsu’s “It’s not too late to learn how to code”:

• It is difficult! Things that take awhile at first will come naturally to you later on. Of course some people are more naturally disposed toward the type of logical thinking that programming demands, but I believe that in the majority of cases, people assume they aren’t cut out for it before giving it a fair shot. Though of course, having people believe that programming is “too difficult” and that the average person is “not smart enough” strokes our egos and makes us feel like we’re part of some super-elite hyper-intelligent group.
• Mentally prepare yourself for roadblocks. I used to think I was cursed, because every time I tried to setup something new (new development environment, tool, library, etc), something would always go wrong. It took many years for me to realize that with all the different combinations of user operating systems, software versions, etc, the documentation was often outdated or not comprehensive.
• Do you like it? Forget about if you think it’s too hard; do you think it’s fun? When you struggle for hours debugging something, and finally, it runs as expected, do you feel a rush of excitement? Do you look forward to having a few hours to figure something out? Those are the things that attracted to me to software–I just didn’t worry too much about if I could make it or not.

Matt DeBoard’s “How I became a programmer”:

• How did you learn Django in 11 weeks? I want to make it clear that I didn’t set out to learn Django per se. Django is just a very nice toolkit of abstractions that makes creating web applications easy using Python. As far as I’m concerned learning Django was incidental to learning to program. I did not – and still don’t – want to be considered a “Django developer.” I’m not even sure I want to refer to myself as “a Python programmer.”
• In other words, I do not feel that I would be as modestly competent as I am today if I had spent an inordinate time becoming an expert at the abstraction layer of Django, instead of learning the concepts that make Django work.
• How did you show the company your skills? Did you show them the projects you’ve made? Github, Github, Github. I can’t emphasize it enough. Make stuff, put it on github, show people you’re passionate and smart and curious.
• If I had to summarize the big overview of how I did what I did, I’d say: Ask questions, be curious, be passionate.
  Learn a language, not a web framework for god’s sake. Work hard. Network, attend meetups, tweet, blog, be social and show people you’d be fun to work with, and a credit to team. (Optional) Put yourself in a position of desperation, so there is no choice but to succeed.

Older Software Engineers/Developers

Lessons of Failure’s Five Pervasive Myths About Older Software Developers shows why older software engineers are invaluable for a tech company:

• Yes, older software developers have higher salaries than younger ones. But what exactly are you paying for here? With an experienced software developer, you’re paying for all the experience that comes with past project successes and failures. Those are expensive lessons if you want to pay for them directly during your tenure as a manager. But if you buy into an experienced worker, that’s like getting insurance against some of those classic mistakes in project management and software development that you don’t have to repeat. Meaning you look better on your annual review because you hired smart people that know how to get the job done.
• It’s actually because of their past experience, that more experienced software developers can migrate to new technologies, frameworks, and systems more quickly and in greater depth. For example, if you learn a GUI framework in C/C++, you have a mental model about message passing, event handling, MVC patterns used to design the system and separate presentation from back-end logic. The very first time you learn a GUI framework, in addition to the syntax, the examples, and the quirks of the library, you also need to learn all the conceptual stuff. And after two, three or more GUI frameworks, you notice that they all have very deep similarities outside of the syntax. You might even notice newer frameworks have overcome substantial limitations that you used to have to work around with complicated hacks. Those insights are lost on someone new to the framework. And those insights can really boost productivity in ways you’re not able to directly measure.
• I think it would be fair to state that experienced software developers are less willing to work those long, painful hours because they’ve learned the hard way that there are productive limits to pushing yourself 80 hours a week for months on end. It’s called burnout, and I’m willing to bet that anyone who has already experienced it in the past simply isn’t as eager to go there again. But with that said, the supposed reason of “family commitments” is bogus. High-quality, experienced software engineers are ruthless time managers, and those with families are even more motivated to get things done in allotted times. They may have dance recitals and soccer games to attend to, but they also make up that time in off hours and highly focused work during the 40 hours they’re given in a week. Good software engineers with families must become highly efficient with personal time management or they quickly get buried with the deluge of work coming their way.
• Lost mental agility is a poor excuse to not hire an older software engineer in light of the fact they’ve seen, done, and lived many more successes and failures than a younger developer. Experienced developers have tons of past projects to draw from and assist in avoiding bad decisions today. Younger developers have new ideas which are important, but often untested and unproven. Having both perspectives on your team is of great value.
• Anyone who believes this is probably someone who doesn’t like their ideas criticized by those who’ve been around long enough to see really stupid decisions put into practice again and again. Experienced software developers smell crap a mile away. They don’t buy your stories about how the product isn’t well received in the marketplace because they’ve been interacting with the customers for years and know you’re trying to cover up a future layoff in the company. They won’t put up with managers asking them to work 80 hours a week because the customer wants the software next month and they already told you it will take 3 more months to complete with the features agreed upon.

Non-Technical Founders for Tech Start-ups

YCombinator.com’s advice for founders of tech start-ups who initially had no technical / coding experience at all:

• If you have a startup idea and you don’t know how to code, learning how to code enough to implement the idea all on your own is probably not cost-effective from a time standpoint and you’re more likely to end up with a mess of non-functioning code than anything else.
• That being said, at least to me, knowing even a layman’s amount about programming will gain major points with the potential technical co-founders you do meet.
• I get approached by friends/acquaintances on a weekly basis with some startup idea. And I mostly think, “So basically you’re suggesting I spend 10 hours a week of my free time for six months to build Facebook For Cats, while you make some half-assed attempt to do some marketing or whatever, and if there are any technical issues I can’t even discuss them with you because it’ll go over your head, and if there are any tedious technical issues you can’t even help with those.” It’s pretty much a non-starter right there.
• But knowing even a little bit about programming could go a long way. It means your idea is probably a little bit better than “Facebook For Cats,” because maybe in programming you learned about some company’s API and how to leverage it. It means I can split up work and give you some of the easier programming tasks and feel like we’re putting in close to the same effort. And it means I can tell you things like, “the service doesn’t always return well-formed XML so we should find a validator and then run it on the response before we insert it into the database,” and you’ll know what I’m talking about.
• It all starts with the product, and the product usually means code, and having one and a half heads coding will usually be more productive than just one. Learn enough to be that half a head and you’ll go far.

The real-time data is available in 4 major reports:

• Overview – Includes active visitors on site, pageviews per minute and per second, top referrals, top active pages, top keywords, and top locations
• Locations – Includes active visitors on site by geographical location (e.g. country)
• Traffic Sources – Includes active visitors on site by traffic source (e.g. Organic – Google)
• Content – Includes active visitors on site by active page

Google provides official introduction and interpretation to Google Analytics Real-time Reports.

Google Analytics Real-time Reports in Screenshots

Google Analytics Real-time Overview Report:

Google Analytics Real-time Overview Report (Top Locations Map):

Google Analytics Real-time Locations Report:

Google Analytics Real-time Locations Report (Google Earth):

Google Analytics Real-time Traffic Sources Report:

Google Analytics Real-time Content Report:

Bing has been offering SEO data to webmasters through Bing Webmaster Tools long before Yahoo Site Explorer got shut down. With Bing’s SEO data, webmasters can understand more about:

• Bing’s search algorithm for ranking sites (and/or web pages) – Compare Google and Bing search algorithms
• Bing’s organic search query – One alternative in keyword research is through Bing’s keyword tool

Bing’s Webmaster Center Blog has listed 2011′s updates of Bing Webmaster Tools regarding:

• Traffic Reports Depth
• HTML5
• Crawler Control
• Index Tracker
• User Permissions
• Sitemaps
• User Experience and Communications
• Block URLs
• Improved Speed
• Know More, Faster
• Expanded Crawl Details
• URL Normalization
• Webmaster Account Verification
• Adcenter CPC Integration

Traffic Reports Depth

Early in the year saw us adding more information in the Traffic reports. This includes the new detailed traffic stats feature. We are showing much more information to the webmasters including:

• Average impression and click positions for their top 100 queries.
• The specific pages showing up for a particular query, including their impressions, clicks, CTR, average impression and click positions, and position details.
• Their top 100 pages including their impressions, clicks, CTR, and average impression and click positions.
• The specific queries showing up for a particular page, including their impressions, clicks, CTR, average impression and click positions, and position details.
• Trending over time for their top 100 queries.

These additions made it easier for Webmasters to see which phrases were driving traffic, and to which pages on their websites that traffic was going.

HTML5

Around this same time, maybe a little earlier, we addressed the need to install Silverlight to view the reports by moving the tools to HTML5. With this move, we made it easier for users to interact with our reports and saw report load times slightly improve as well. This move meant our reports were now viewable in all modern browsers and on smart phones as well.

Crawler Control

Users asked loud and clear for ways to control the crawler, so in addition to the usual avenue open via your robots.txt, we built a tool that allows you to easily drag and drop a graph to set the crawl rate specific to your website. It’s as simple as creating the crawl patter you want. This means it’s easy to tailor our crawling efforts to times when your bandwidth is least affected. Telling us to stay away during peak business hours, and visit when everyone is asleep is as easy as clicking your mouse. We added a checkbox for you to let us know if we’d encounter AJAX URLs along the way, and there is an option to simply let us figure it all out for you.

The times shown were all GMT, so we added a layer to display your server time relative to GMT, hopefully making it easier to understand when your peak business hours hit, and allowing you to build a control graph around those hours.

Index Tracker

Our Index Tracker tool received a thorough backend rewrite, improving coverage, performance, depth and latency. When these improvements released, even brand new websites saw all of their data as the Bing index held it. No longer having to wait 48+ hours meant Webmasters could understand what Bing was actually indexing and how it was performing much faster.

User Permissions

We introduced user controls and permissions. Site owners could now grant admin, read / write, or read-only access to other users for their site using their existing verification code. This made it much easier for businesses to manage their accounts, share access and control who could turn the knobs and pull the levers. This layer of control allowed businesses to manage account ownership and access at a new level, ensuring, as employees moved around, access and control of these tools remained in hand.

Sitemaps

We enhanced the Sitemap UX by showing not only user submitted sitemaps, but all sitemaps we know of. We also expanded the types of feeds we accept enabling folks to submit Atom, RSS and xml.

User Experience and Communications

Working to make it easier to navigate inside the tools, we added a second layer to the tabbed navigation, which you see when logged in.

In an effort to improve communications, we enabled the ability to tell us where to email you and what to email you. If there was an alert appearing inside your account for malware, as an example, you can now opt to receive an email alert to the address of your choice. This made it much easier to know when something was happening.

Block URLs

Our “Block URLs” feature got a backend overhaul to include more built-in safeguards, helping to protect webmasters from accidently blocking large numbers of URLs. While the control is still in your hands, there are added layers of checks on our end that prompt confirmations before actions take place.

Improved Speed

We did some work on how we cache all of your website data, as well, resulting in a dramatic drop in latency from, in some cases 5 seconds for a report to load, to 400 milliseconds for the same report now. That dramatically improved the performance of reporting.

Know More, Faster

Later in the year, we expanded user email communication preferences and controls. It’s easy to stay up to date on the latest announcements and alerts applied to your account. You can set frequency preferences, select options on which alerts you want notifications for and set a dedicated email and contact number.

Expanded Crawl Details

The expansion of Crawl Details meant you would now see all URLs attributed to header codes and notations made. This was a great step forward to understanding which URLs Bing was having issues accessing, and makes it much easier to understand if any high value URLs are being seen by us in ways you’d like to change.

URL Normalization

To help you manage parameters in a more in depth manner, we expanded the number of parameters you could assign us to manage from 25 items to 50. This made it easier for larger and older sites to manage legacy URLs.

Webmaster Account Verification

This was one folks were asking for all year, and we managed to get it in with one of our Fall releases. Being able to verify a website via a DNS change. Past choices included adding a snippet to your <head> code or uploading a dedicated XML file to the root of your site, and both options remain as valid ways to verify your domains. This third option will allowed you to place a discrete CNAME record to your DNS to validate a domain as well.

Adcenter CPC Integration

In an effort to help businesses understand how they can easily expand their inbound traffic, we began showing CPC data from Adcenter inside your Webmaster accounts. When looking at Traffic data, to the left of the keywords you now see CPC amounts associated with those keywords in Adcenter. This is an excellent way to understand the practical cost of driving more traffic through paid search, and can help in targeting keywords you perform well for in organic search, thereby increasing your business’s footprint on the search results page.

When building a website, focus and review the items below that will improve the quality of your site to visitors and SEO for Bing.

Crawlability

If a crawler can’t access your content, the content won’t be indexed by search engines, nor will it be ranked. Enable and use XML sitemaps with a low error rate to build trust with search engines. Make sure your website navigation is clean and strive for a simple, search-friendly URL structure. This means the URL is keyword rich and avoids session variables or docIDs. We suggest you use robots.txt files to instruct the crawlers on how to interact with your webpage and more easily find your content.

Site Structure

Provide a site structure with strong functionality, which helps encourage link building. Linking to both trusted outside sources and internal content shows a search engine you care about users getting the best data around their query. In addition, HTML sitemaps ensure a good user experience and help search engines discover all your pages and content.

Content Hierarchy

When you plan your website’s content hierarchy, take care in aligning your content with what searchers are looking for and their intent during their journey on your URL. Basic keyword research can also help you understand how searchers are interacting with search engines, and can help craft your content strategy. Also avoid placing links and content inside rich media applications, such as Flash and Silverlight, which make it almost impossible for crawlers to find and read the content. Just gotta have your rich media? Not to worry. Make sure a solid downlevel experience exists and we’ll still find your content.

RSS Feeds

Get them up and running and keep them clean. By following your feeds, it’s easier for the engine to get your latest content. This means we see it faster, so indexing, ranking and showing in the SERPs can happen faster. Want to really impress Bing? Get into your Bing Webmaster account and insert your RSS feed URL into the sitemap submission flow.

Rel=canonical

Being able to tell the engine which version of your URL you’d like to have attributed as the original is pretty useful. This handy command can help you build value on the version of the URL that matters most to you, and help combine value attributed to many version of the URL into one location, helping boost the rank of that one, original version of the URL.

The tough part here is usually getting the code installed on each page, and of course, each instance needs to point to one selected URL for this to work. We’d rather you didn’t use the rel=canonical to cover issues where your CMS needs work. If the CMS is causing instances of duplicate URLs to occur, you should fix the problem. We see increasing usage of the rel=canonical across huge numbers of pages on large websites. While we don’t really like this, either, we can work with it, as we understand the need to balance the workload and the returns.

The bottom line, though, remains that you need to be able to manage the rel=canonical, and if you don’t have control over when its deployed, which URLs they point to and when it is used, you need to work it out.

Robots.txt

Seems like a no-brainer, this one, but so many websites remain without a robots.txt file. In some cases it’s a purely missed opportunity, or the site owner is unaware of what a robots.txt file is. In other cases, though, it’s the inability to place a file on the root of your domain.

Regardless of the blocker, the robots.txt file is one of the most important files you can place on a web-server. Given it is the location search crawlers reference to understand how to interact with the website, it’s a pretty powerful document. If you do not have access to place your robots.txt in the correct location, you need to understand why this control is lacking. Then solve the problem.

Sitemap.xml

This is another file missing from a huge number of websites today. Another important file the search crawlers look for. One that is referenced inside the robots.txt mentioned above, and one which can help get more of your pages into the search index. Overall, it’s almost as important as the robots.txt file, and if you cannot place these files in a location the crawlers can find, you need to fix this issue.

This file typically lives on the root of the domain, but for larger websites, where multiple files may exist to capture all of the URLs present, maybe only a sitemap directory file is on the root. Whichever your case, it’s important he crawlers can find it, and if you cannot access the root on your server to place files there, it’s a missed opportunity.

Rich Snippets

Marking up your content has been around for a few years now, and with the launch earlier this year of www.schema.org, the major engines have made a clear statement there is value in marking up your content. By embedding these elements in your page code, we can extract information more accurately and use that information to provide increasingly richer search results. You are credited as the source for the data used. This is important work for sites seeking to differentiate themselves from the pack as we move into 2012.

Websites need to balance the future value versus the workload to implement, and still need to keep in mind that implementing these elements won’t immediately increase rankings. This works helps us better understand relevancy.

What’s important here is planning for the work and ensuring you have buy in across your organization.

Title, Description, Alt Tags, etc.

Managing your title tags, meta descriptions, alt tags, etc. is still important. All these basic, on page/technical SEO factors add up to help us understand what your content is relevant for. The bottom line with these items is you need to be able to manage them. If you cannot change these elements on a per-page basis, you lack needed control. We only mention three here, but you can think of all of them.

That meta description you don’t care to alert and let appear across all you pages? While writing unique ones for each page won’t suddenly vault your pages to number 1 in the rankings, it can make the difference between a searcher clicking on your result in the search results or not. Think of the meta description as the “call to action” then, when read by a searcher, tells them why they should click your result. Better to have your words appearing in our search results than random text we take from the page because your meta description was low quality.

We use the meta description as an example, but the same level of thought should be applied to all of your on page optimization efforts.

Content

This might seem pretty obvious, but with so many website still aggregating content or using article services to build out pages, its worth mentioning. We talked about how to build good content a little while back and the value of “article-site content“, but we still see websites trying to get ahead in the rankings by basing their websites on a thin content model. Such sites are often very polished looking, and while may provide value in aggregating a lot of items in one location, they still aren’t adding anything new and unique to the conversation. A website designed primarily to hold affiliate links that get the user off the website quickly and into a shopping cart on another website is an example of a thin content website, though not the only example. Affiliate links on a website can be completely useful, but when the content on the site is duplicating that from the original website, there’s simply no reason why that thin content site should outrank the real deal.

Control as applied to content means you can influence the creation of quality content on the website. If the website is not producing unique, quality content, it won’t last long in the search results.

Verification Access

This is pretty straight forward. You need to be able to place the verification code in place to use webmaster tools. This could be in the form of embedding a tag in the <head> code of your webpage, or by a notation added in the DNS for the website. No matter the option used, you need to have access to make this happen, and if you do not have that access, you lack the control needed to then be able to access some of the richest data about your website online – our webmaster tools data.

UX Improvements

If you don’t have a website your visitors love, you’re missing an opportunity. Get cracking on a user experience review and see where you’re bleeding users. By staying tuned in to what users like and dislike about your website, you can make the myriad small changes needed to field a UX-winning site. And if you keep visitors happy, they share you more often with friends, netting you more links. Visitors are also more likely to come back to you again in the future if they like the site and find it easy to get what they’re after.

It might seem like a small thing, to focus on the user experience, but that UX directly influences the happiness of visitors and the engines can see that in how they reach to your site when they see it ranked in search results. If you don’t have input on UX improvements, you need to push. This is an important aspect of optimizing any website.

Social Sharing Integration

This almost goes without saying, but we still see so many websites not involved socially with their visitors. Social isn’t going away folks, and while it does take work, skipping social integration is a missed opportunity. People share what they like with friends. If you have social sharing icons embedded in your pages, you are much more likely to get shared by visitors. At the very least, you need to cover this angle. Get the buttons embedded into your pages so your visitors can share you content with their friends and followers.

One of the major SEO issues to any large websites is duplicate content/URLs. Bing Webmaster Central Blog reveals how to handle duplicate content URLs and how canonical URLs, 301 permanent redirects and 302 temporary redirects work:

301 redirect

The 301 redirect defines a redirect which tells the search engine the content has moved permanently to a new location. This is preferred as it clearly states the intent to move and instructs the engine to transfer any value the URL has accrued from the old URL to the new location. It’s important to know that the 301 redirect does not pass all of the value from an old URL to a new one. The new URL does need to build its own level of trust with the engines, which is why we won’t simply transfer full value to new URLs.

302 redirect

The 302 redirect defines a redirect which states the content has moved temporarily and will return to its original URL shortly. This redirect will still move people to your content, but the engines are essentially being told to hold their assigned values on the original URLs and wait for the content to return to the original URL. This is not what you want. Be careful when requesting redirects be implemented, and clearly define that you need to have a 301 in place. Otherwise, you can lose value from your original URLs, leaving your new ones to struggle on their own.

rel=canonical

It’s important to understand that a rel=canonical is not a true redirect. There has been a lot written that it’s “basically like a 301 redirect”, which can be misleading. The purpose of the rel=canonical is to help the engines understand when an individual URL is essentially a duplicate of another. The rel=canonical element does suggest to the engine that any value assigned to the duplicate URL be assigned to the original URL, though. This is similar to how the 301 functions, which is the origin of the over-simplification noted above.

The biggest difference between the two is that while a 301 redirect physically moves a visitor to the new URL, the rel=canonical does not physically move anyone anywhere.

Something else you need to keep in mind when using the rel=canonical is that it was never intended to appear across large numbers of pages. We’re already seeing a lot of implementations where the command is being used incorrectly. To be clear, using the rel=canonical doesn’t really hurt you. But, it doesn’t help us trust the signal when you use it incorrectly across thousands of pages, yet correctly across a few others on your website.

A lot of websites have rel=canonicals in place as placeholders within their page code. Its best to leave them blank rather than point them at themselves. Pointing a rel=canonical at the page it is installed in essentially tells us “this page is a copy of itself. Please pass any value from itself to itself.” No need for that.

We do understand that doing work at scale requires some compromises, as it’s not easy to implement anything on a large site page by page. In such cases, leave the rel=canonical blank until needed.

Review the definitions of 301/302 redirects in HTTP status codes 2xx, 3xx, 4xx, 5xx.

Other Search Engines’ Webmaster Tools

Besides Bing, other search engines also offer webmaster tools to webmasters:

• Google, World’s largest search engine (with multiple languages): Google Webmaster Tools
• Yandex, Russian’s number one search engine: Yandex Webmaster Tool (English)
• Baidu, Chinese’s leading search engine: Baidu Webmaster Tools