Has IE8 run its course as a default test browser?

WebPagetest enthusiasts – a group that includes 100% of Strangeloop employees - have grown accustomed to using the popular site to get a reliable measurement of how fast pages load across different browsers. Pat Meenan and his team have done a great job of maintaining relevance in the face of constantly shifting browser trends.

Recently, however, some here at Strangeloop began wondering why WebPagetest still uses Internet Explorer 8 as its default browser, when both Chrome and Firefox exceed it in popularity.

So we started digging…

Why is this a relevant question?

While modern browsers increasingly embrace common standards, they’re far from equal. As web pages become increasingly complex, front-end optimization (FEO) techniques that can make pages faster in some browsers can slow them down, or even break them, in others.

The problem with conducting all your tests on a single browser - such as IE8 - is you might just be getting a pinhole view of your site’s performance. This problem worsens if the majority of your traffic uses Chrome or Firefox, as it renders the findings largely irrelevant. 

How many people use WebPagetest’s default settings?

To get an idea of just how many people rely on default test settings, here’s a quick breakdown of roughly 24,000 public tests on WebPagetest from July 6th.

As this graph shows, most users are indeed using the default settings:

Global browser usage

Compare these findings with the global browser breakdown for July 1-5, courtesy of Statcounter:

Comparing WebPagetest and global browser stats

A quick side-by-side comparison of these two data sets shows some striking discrepancies: 

• Just 13% of global internet users use IE8, whereas roughly 65% of WebPagetests are done on IE8.
• Chrome 20 holds the largest global usage share at 23%. IE9 is second at 17%, and Firefox 13 is third at just under 15%.
• While WebPagetests are split almost equally between IE9 and IE7, Statcounter indicates IE9 to be vastly more popular.
• Further to the preceding point, the number of Safari iPad users is almost double the number of IE7 users, at 2.52% and 1.38% respectively.

Here’s a graphical illustration of the differences:

Based solely on this data, an ideal set of testing options for WebPagetest might look something like this:

• Add the capability to test on Chrome 20, and make this the default browser.

• Add Firefox 13, Safari 5.1, and Safari iPad.

• Implement the ever-popular Browser Tax on those who test on Firefox 6.

Or would we?

These global usage numbers do not reflect the real-world data seen here daily at Strangeloop, so we gathered traffic data for the same time period from a handful of Strangeloop customers - approximately 350,000 unique visits in total - and broke down the data by browser version:

 

We then compared these numbers to data gathered by Akamai IO between July 1-5:

While Akamai’s results aren’t identical to ours, they’re quite similar, further validating our belief that the sample data generated represents a typical user segment in the ecommerce market.

WebPagetest versus Strangeloop browser stats

We then overlaid Strangeloop’s browser usage data on WebPagetest’s data:

 

The data shows strong correlation on IE8 usage, however, there’s a significant gap when it comes to IE9. Most of the browsers we believe to be the most widely used - Chrome 20, Firefox 13, and Safari — are not yet on WebPagetest’s map.

Conclusions 

1. IE8 remains relevant as a default test browser, but IE9 is the future.

2. WebPagetest could generate more accurate results if they offered different defaults based on country and/or user/application type.

The bottom line

Browsers matter, and before running tests, it's wise to examine your analytics to to get a better picture of who your users are.

Questions about browsers and web performance?

Talk to a Strangeloop Performance Expert.