Case Studies from the Mobile Frontier: The Relationship Between Faster Mobile Sites and Business KPIs
In this presentation at the 2011 Velocity Berlin conference, Strangeloop president Joshua Bixby discusses research findings on the relationship between faster mobile sites and business metrics such as revenue, page views, and customer abandonment. This data was gathered over many months of the beta of our Mobile Site Optimizer.
Steve Souders: I like fast. Okay so that was good morning we’re running a little late but that’s okay. Anyone wish we would cut the content back to speed it up. Alright then we would go ahead with the content that we have. So our next speaker this could be very good Joshua Bixby from Strangeloop Networks. They have, I don’t know if you…he has got a great blog, I don’t know if you follow his blog and the work that Strangeloop is doing but the thing I really like the work that they are doing and the thing that they do really well is talking about the business effects of performance and so for me this year has been all about mobile, that has been my focus. So I am very excited to have Joshua here he is here he is going to talk about Business KPIs and mobile. Please let me welcome Joshua Bixby from Strangeloop. [Sound of applauding]
Joshua Bixby: Thanks Steve. Hmm…it is a pleasure to be here. Berlin holds a special place for me. My grandmother is here, was born here and raised here so it’s wonderful to come realize why she is so crazy and wonderful and artistic and everything else. It is a pleasure to be here. Hmm…it’s nice to see so many familiar faces I have 56 odd slides and only half an hour. Many of you, who I do know, know that I am the president and cofounder of the Strangeloop; we are in the business of automating performance making it faster. What some of you don’t know is that my previous career was in development economics, working in the microfinance industry. Microfinance for those who don’t know is giving out small loans often to women often in rural areas and I was stationed in West Africa doing that work and I had 80,000 women that we are trying to turn into entrepreneur as is an entrepreneur myself now. I realized that is a pretty tricky thing to do. One of the things that I didn’t listen to too much of what my professors told me but one of the things I did listen to was the fact that in development we make the same mistakes over and over again. We go to countries, we try to teach them things that work in our country, and often the teachers are people who have never done it themselves. So I was in the World Bank teaching women to be entrepreneurs and I had never entrepreneured no one in the World Bank office had ever entrepreneured and I decided in the 1990s that I was going to come home and learn to entrepreneur myself and I happen to find myself in the middle of a revolution. That revolution was the revolution of the internet. It was an incredibly exciting time a lot of money was made and lost and people came in said you are so you know, subsequently people come to me and say you are so lucky to have been in the internet. You are in you know a shift change and how the world looked it at technology and here you will see there have been a few. Now we are a lucky generation we get to see two of them. We’re right in the middle of one right now and when young entrepreneurs come to me I say we are right in the middle of a revolution and an evolution of a huge change and that’s that we are in the middle of mobile. Now mobile isn’t good enough. I was talking to one of my friends in the VC Community last week and he said you know, it is funny because the greatest minds of our generation are trying to figure out how to get people to click on Banner Ads and the reality is if you have a dumb phone if you got an old phone you don’t have a browser and you can’t click on ads and you can’t make the smartest people of our generation money. So what we really need is, we need a mobile revolution but we also need Smartphones and so what I want to do is and when I introduce this type of graph, we will take a look at this in a few occasions I want to look at Smartphone adoption versus dumb phones. Over the last four years and I will take an example of Western Europe and Asia. I want you to pay attention to two things; one is when we hit the tipping point when more than 50% of phones sold in Western Europe and use that as a proxy for any developing and developed country became Smartphone, Blackberries, iPhone, Androids and how that relates to other parts of the world like Asia where that penetration has not been as significant. Let’s take a look. This is the baseline this is where 2008 starts…now few things to notice here. One is we hit the tipping point in the Western World where most of us obviously everyone in this room I’m assuming, but most of the world actually owns a phone and a phone that can browse and that’s really important as I start talking about the KPIs that affect people’s business, we need phones that can browse, we need to be to click on things, and we need to be able to buy, but this has not affected the whole world. I was very surprised when I saw this graph and took a look at Asia and said well that’s surprising to me. I spent a lot of time in Asia, I spent a lot of time with people with mobile phones I don’t…and Smartphone. So if we have this revolution and we need Smartphones there is also been a dramatic change in who owns that phone the actual underlying operating system. So if you take a look at this you’ll see a huge change over the last six years. What you are going to see is the sudden and dramatic rise of iOS and Android you’re going to see crushing defeat to Symbian and Linux based operating system again this is the evolution of the Smartphones taking over from the dumb phones. You’ll see Blackberry growing not sure how this is going to look in four years for them and you’ll see windows with a very similar percentage of market. So as we are caring through here and we are going to, and when we talk about case studies it is important to understand who owns that underline platform, who runs these devices, and how this market works?
Now the next thing I want to talk about is the question that always comes up if we start talking about okay I get it people own these phones do they go to sites and this is where this data today the state of data starts getting really weak. So let’s take a look at three brands. Three brands that I applaud for actually publishing these numbers lets take a look at the percentage of the mobile traffic that they get over the last four years.
Now I spend a lot of my time trying to sell the mobile is important and I use stats like this and this is where all the buts come in. But it’s an App. But it’s a social network. But it’s Twitter. Pandora for those who don’t know is a movie sharing site. So these stats start getting us insight that some people are using their phones and are they buying well they are certainly looking at sites. Let’s take a look at the question of whether they are buying. I still can’t convince my buyers that mobile is important enough in their business with stats like this. Let’s start looking to the question of whether people spend money? Again let’s take a look at four brands and their revolution over the last three years. How much money they are making?
So nice growth, nice doubling, year in year out, but ultimately I get the same buts. But it’s Amazon, but it’s eBay, that’s not me, and that’s why we need to look at real work case studies. And so what we are going to do is we are going to take a look at two in depth case studies and take a look at are people buying and most importantly what is the effect of performance on the key business metrics for that business. So first a bit about data, so one of the things that we are in the stage of big data and I am very excited about. We are able to correlate between what users do, we are able to look at how you know, their performance metrics and for people this morning talking about real end user metrics obviously that’s important and we’re capturing those that data in all of the statistics that we are looking at, we are also capturing what somebody’s latency and bandwidth is or correlate that is all the business metrics. So in order to get this information we need to combine a whole bunch of what today’s siloed but hopefully tomorrow won’t be which is real end-user monitoring plus the business metrics and who buys how much they buy along with things like latency and bandwidth. So if you assume we have all of that together let me introduce you our first customer. Now one of the things about the introduction of customers is that when I start sharing intimate details about retailers they don’t like me mentioning their name. Now I’m going to try to give you as much of a picture of this retailers as I can, but I can’t tell you who they are. This is a retailer that does over three billion dollars a year in revenue. They have 30,000 employees. They have got stores; bricks and mortor stores, as well in the US, as well as online presence that serves both the US and Canada and Western Europe. Let me give you a feel for this customer in terms of who that shopper is. This site has 22,000 products on it. The average shopper spending $100 and they are predominantly a woman, in their 40s, who makes a lot of money. This is a very good niche market if you are in the marketing world you would say this is fantastic. These are people I want to target.
They have three and one of the reasons I picked them for case studies is they have three avenues which mobile browsers can interact with, they have got a mobile site, they have got a full site and they have an App and that’s what made this case particularly interesting to me, I get a lot of questions about the interactions between these three platforms. Lets take a look at how many page views come through each of these. There are few things to note. They launched a mobile site in January and the App in May. These are just mobile visitors. You can see that the full site takes the brunt of the mobile load. Although they have an MDOT site, which captures some traffic we will see the lot of that traffic gets funneled off to the full site and because a lot of their traffic comes from search particularly in mobile devices those unique products are just often go to the full sites as well and I’m definitely seeing a trend in our industry moving towards full site functionality that might degrade for the device but not individual siloed sites with different code bases. Now let’s take a look at the important question. Should this customer care for every $100 spent how much is spent on one of these devices because ultimately that’s what drives innovation and investment. So in 2010, in September 2010, 50 cents were spent on a mobile device. This was not getting a lot of attention and I was on with the media yesterday and I was getting some questions about mobile is this the year, is this the year, I get that question all the time amd the reality is we are seeing dramatic growth. One year later this company is making $7 out of every $100 through a mobile device. That doesn’t talk about who many people are searching in store for a product or price comparing at a competitor and choosing not to buy doesn’t give us any of those statistics. These are just the raw numbers that drive this business and what so interesting is to look where those come from. About 50 cents comes from the App, about a dollar comes from mobile site and all of the rest comes from the purchases on the full site. Again this evolution that we're seeing mobile is important it is growing 14 times for this customer over the last year and the vast majority of that growth is coming on the full site trends that I’m seeing across many customers. Now let’s return to this question of the mobile site and this is a trend that I’m also seeing. For every 100 visitors who come just to the mobile site I see that 35 of them – I'm sort of delayed on the clicker here. Hmm…35 of them go to the full site because there is a view full site link. So people who are coming specifically get redirected to the mobile site a lot of them want to go see the full site and we see a whole bunch of people that bounce on both the first view as well as the second view and the number of people that buy; one person that buys for every 100 people that come and buys on the mobile site. So I want to take a look and lets go back a slide here. I want to take a look at this market this graph that we saw in the context of the customer. So if this is what the market looks like what does this customer’s page views look like across these different browsers? So what I want to do is want to take all these operating systems and I can’t actually look at all of them. So I’m going to take out other and I’m going to take out Linux. I’m going also unfortunately take out windows because it just isn’t enough traffic. I am going to divide iOS into two categories because I want to talk about the iPad and the iPhone separately and I want to look at this page views by mobile browser over the last two and a half years, two years in essence. So up top on the right you’re going to see the total page views to the entire site that get through that comes from mobile device and then we are going to see the page growth in page views across all of these browsers. In your mind think about the seminal moments over the last year and a half or two years in the evolution of the mobile so we are sort of start at a point where Jobs and the Google Boys haven’t got along and Androids coming and I mean it is a 3GS environment for the iPhone. The iPad really isn’t out. Think about those seminal moments and watch how this customer site interacts with those seminal moments. These are page views for mobile on the specific devices. Let’s take a look….hopefully.
This is an iPhone world to start with. Here comes Android and the first version of the iPad comes out. I’m now almost doubled my page views I’m almost six months into it. Big marketing campaign, the iPhone and the Android are together the mother that we are talking about just got an iPad for Christmas and has just learned to use it oh my goodness. It is absolutely remarkable what happens to the iPad’s evolution here now they are getting training from their teenage son or daughter and they’re really using the iPad and it absolutely starts to dominate and four times about three times higher than I originally was in terms of page views and I end this not coincidentally with 8½ percent of this customers mobile traffic coming over mobile devices or traffic coming over mobile devices this is starting to be substantial and most importantly it is really interesting to see that the tight raise between the iPhone and the Android I was very happy to see this because I hate pulling one out of the other if I had to talk about Android versus iPhone I get yelled at continuously as does anyone that blog about that and the absolute dormancy iPad. So I have started to try to set the stage now lets look at some of the key business metrics that affect his customer. Before that I want to talk about one thing which is when I looked at this date and let me just go back I see Blackberry and Symbian I shake my head and think there is something wrong here. How can Blackberry and Symbian be such a large part of the market share and not representative at all in this study and so what we did is we have seen other studies Cotendo did one where they looked at this Java script based analysis and the fact that some older Symbian browsers and many Blackberries done support Java script and I asked myself the question are we dramatically under reporting the quantity of traffic from these browsers. So we looked at one analysis, we looked at the Java script information, we looked at another analysis where we did a log analysis of this site for specific month and you can see we are dramatically under reporting Symbian and Blackberry users in that analysis. Now to be fair the fact that combined their 3% I’m pretty comfortable with the general conclusions here, but one of the things to notice we saw is that there is a discrepancy something to keep you know, to pay some attention to it. So let’s talk about HTML delay experiments. This is the greatest you know, when Google and Bing did this I was very, very excited it is a way to really look at the direct impact of delay and so what we did is we convinced this customer to do this and people asked me how we did it because this customer is very interested in figuring out the value of time for their business. If they have more time they can give a richer experience. So what we do is we delay the HTML we are going to delay by 200 milliseconds, 500 milliseconds and a 1000 milliseconds on a certain percentage of users and we are very lucky to have I’m sure many of you who listen to these talks in Google and Facebook oh we have this experimentation platform and your are jealous as I’m but we finally got this ability for customers like this whether they can take small percentage of their traffic delay them speed them up and figure out what the impact is and the impact is dramatic. Let’s take a look at, you know, the statistically significant impact from these three delays over this 12 week experiment.
Again this is on mobile devices. So when I delay a few things to note about this and I had still do double takes when I look at these findings. The strong negative impacts across almost every number here and also the really interesting linearly this customer was blown away that they are going to lose 3½ percentage of their conversions if their site is delayed by one second on a mobile device. This was a very impact full stat for them and it is helping to change their business and how they view mobile. Obviously I encourage everyone to these kinds of experiments, but sometimes it’s hard to convince your boss to do it. Every single metric was impacted. Now I wasn’t satisfied with this because I know that when I go to sites on a mobile device occasionally they will take 15 seconds and sometimes they'll take two and no customer is going to let me slow their site down by 13 seconds to see really what happens. So we are going to get to how I can try to sort of proxy that. Let us take a look before that statistically like bounce rate. So here I am looking at the percentage change and something like bounce rate over the two main delays that we have looked at the 500 milliseconds and the one second this is the change. So my baseline would be zero and you can see as marketing does a better or worse job the bounce rates going up to paying on the clients that they are targeting and coming to the site but you can see strong linearity here. Strongly linear the trend maintains. That 1000 second is worse than 500 milliseconds and they are going up and down so this tells me that at week one I had a 10% change in bounce rated at one second delay and then in week four I had a two second difference in bounce rates. You can see these going up and down. Again strong linearity and strong negative effect. Now we wanted to look beyond just the effect of delay in the immediate timeframe. So you know, the question was more like it this like a band aid being pulled off or is this like me telling my wife I don’t want another kid. One has a fact short term and one creates years of pain. It really depends and so we wanted to really figure out which was this. So we looked at the stat that was really important to this customer, which was what is the percentage chance of the user coming back and we looked that during the experiment period and after the experiment and what we saw was there that there was a long term effect. If I slowed you down you had a smaller chance of coming back and if you don’t come back you don’t buy. So we saw not only in the experiment period but even beyond that strong long term effect from delay. Now as I said I wasn’t particularly happy about the fact that no one lets me slow down their site by more than a second so and I say we the team at Strangeloop decided to think of a few innovative ways to try to proxy. I can’t do a/b tests here but can I do something and this is going to take a second to graph. What I am going to do here is I am going to take a look at the iPad, the Android phones and the iPhone and I am going to look at bounce rate and performance. But instead of looking at time which we've looked at in all of the other graphs I am going to look at network quality. So what I am going to do is I am going to take a vast data set and I am going to divide it into cohorts based on your network quality. What I know, in this case the first cohorts that we are going to look at is people coming in that measure 250 kilobytes per second. Now this is a range and sort of 200 to 300 kilobytes per second so a slow connection and obviously there is an inherent latency window that will move as well. What you are going to see is you are going to see as the speed goes up as I go from a really crampy modem to a really fast connection. What happens to bounce rate and the average performance across those groups? This is a way to try to try to simulate trend lines across a larger gap of change in terms of performance. So I am going to press playing and what you notice is these dots are going to start moving down to the left because bounce rates can come down as performance goes up because performance impacts the key metrics. Let’s take a look.
As many of you know I am really in to this kind of analysis where I take different metrics it is really interesting to see these lines add most of the trend lines here to make them a little easier to see. So I have at any point in time I have got a bounce rate for that particular user group over a network quality and I also have performance and it is really interesting to see the slope of these lines. This tells me that the performance matters across the entire spectrum and it also tells me that for example the iPad users up here are of much less patience at high speeds, at low speeds. When things are really slow iPad users go away much faster than Android and iPhone users. But you can se as the time gets faster iPad user tends to stay and the bounce rates dramatically go lower. So this is an interesting evolution and an interesting way to take a look at stats to really show to this customer hey it is not jut about 500 millisecond increments or 1000 or one second increments when I look across the entire scope of performance I can see bounce rates going from 24% all the way down to for example 5% as network quality gets better. And this is a really and you can look at many metrics this way and I really like looking at network quality give me a broader picture. I want to move on to the second company and this is one that is not in e-commerce Company. This is an enterprise more like an enterprise application. Here is an organization that is quite large they are a task based organization. You go from step one to step two to step three to step four to step five. Harder obviously to measure the KPIs. This is a voluntary application. So you could do this task by calling the sales rep or a customer service rep. You could also do that task in the traditional from and faxing and to this business when you call a rep it's expensive when you fax it is expensive the internet is by far the least expensive way for this customer to do their business. So they have launched a site in order to accomplish that and this is really excited. This is a very large organization sort of fortune 500 and this was one specific app that targets Europe, United States, and in Asia and it was launched when we started the study. So it is pretty cool to be there early. So the first thing I want to show you is their page views and their page views are obviously in week one we can see you know, it is interesting to see word of mouth. Somebody said the iPad worked on this app and then they start telling their friends and very, very quickly the iPad absolutely dominates the transactions here. So you can see that this is the type of sites that you know, phones work but it is probably a little difficult to use but certainly the iPad is really, really works, really, really well. And we asked ourselves what kind of experiment can we take, can we do on a site that is task based and has step one and then step two and then step three and step four and it reminds me of a conversation that I had with Steve. I was playing hooky one day and I was on the phone with Steve and I was in the line up for Alcatraz. You know, if anyone has been to Alcatraz it’s a prison in San Francisco and I was talking to Steve and we were talking about this idea of what happens when you slow down a page in a flow. You know, is it about the first page is it about how fast or slow the fastest page is what it is that really determines the key metrics for a business. So I took that inspiration and we decided to do a analysis of a flow and this is pretty cool. So what you are going to see on the left is the representative use or a cohort of users and I am going to walk you thorough the five step process. And I am going to start with a base line where every page is fast. Every page is sub five seconds. So we take a vast amount of data we find all of the users where every page is sub five seconds and we plot them here in terms of that groups. So let’s take a look these users are going to flow across this diagram they are going to turn red when they bounced and they have just given up on the task. And they have chose to either not to do it or they chose to do it through the telephone or fax really expensive ways for this company get their job done and at the end we will see how many make it through. Let’s take a look. Four of these cohorts bounced on the first step. So I wind up with five of this representative user base that finishes the task successfully. One out of every three which this organization would like to get better but this is a good start that saved them they can quantify who much money they can save them and save productivity time for their other employees this had a business value for this customer and the same way the conversion had same value for the E-commerce customer we talked about. So now let’s look at what happens when I slow down step one. So this is about slowing down one of the steps in the process. Instead of a four second page let us add two seconds of latency. Again before the HTML is shipped we received a request we stopped, pause and then we ship it out. Let’s take a look at what happens at the end of the day when I slow down for the first step for this organization. Again by two seconds. In theory I will start….here we go. My bounce rate almost double I loose as many as twice users on that first page.
So this is how an organization can quantify performance. We are not you know, it is not very exciting and sexy to talk about conversion and bounce rate metrics like this. But this gives us a real idea for what happens what is two seconds of latency due or you know, added time due to mobile devices on a task base application and this is dramatic. I've lost more than 50% of the users who would finish the task. This is a huge loss of an organization. So I didn’t end this study there because I actually was interested in you know, the conversation that Steve reminded me, I don’t want to slow down the first page we should all know at this point that slow performance for the landing page makes a huge difference. What happens when I slow down a page in the middle of the flow? So I am going to slow down step three now by two seconds. Let’s take a look at the outcome for this company. I again almost double the bounce rate on that step. And so it is bad to slow down you bounce rate, your first page but it is equally bad to have a slow page in the middle of your flow although it is not as detrimental obviously you can see from the results it really is terrible for the organization. And imagine that each of the users was worth $200 to this organization they can easily quantify the business pain of performance. I want to end with a few observations. As I said I have seen a dramatic change to people wanting full site functionality and I have seen apps being very specific only for certain groups where you know, you can see from the results and again I recommend everyone shared their own results if they are able that speed matters, mobile matters it is not the be all and end all we are not seeing 50% of the site traffic on the customers we have looked at come over mobile but the growth is phenomenal and I think when I come back here in a year we will see a big, big dramatic difference. I want to end up with one thing I want people to measure their mobile performance. And I want you to measure it because if you don’t monitor it so if you all came to this point if you don’t monitor you don’t measure it you don’t change it what we have seen in our world is when we measure things we see change. And I don’t see enough organizations measuring their mobile performance. It has been a pleasure. Thank you very much.