Theo Schlossnagle (OmniTI)

Podcast Transcription

Joshua Bixby: Hello. It’s Joshua again. President of Strangeloop Networks.  Welcome back to the next installment in the webperformancetoday podcast series. When I was at Velocity in London in October, I had a chance to sit down with Cliff Crocker.  Cliff, who is a friend, has a really neat perspective on performance because he has worked on both sides of the wall.  He has spent time as a consultant and worked at Keynote, then crossed the wall and started to work at Walmart where he was the senior engineering manager, which is where I met him selling front-end optimization into Walmart.  We became friends. We were both driving towards the same goal, which was trying to show Walmart that performance mattered. It was friendship that struck up there and has continued through his new role where he is VP Product at SOASTA, which is a cool company and I’m gonna spend a bit of time talking on the podcast about them in one of the podcast coming up. I got a chance to talk to Cliff about his experience in all three companies, the emergence of big data, among other things. I hope you enjoy. Here at Velocity Europe with my friend, Cliff Crocker, formerly of Walmart fame, currently of more famous SOASTA fame. Hey, buddy, how are you doing?
Cliff Crocker: Doing well, doing well. How are you?
Joshua Bixby: I’m great. Thanks for joining me. Tell me about Velocity Europe. How is it going for you?
Cliff Crocker: It’s my first time to Europe period and to come here for a web performance conference I think just really makes it kind of the best trip that I’ve been on for a while. It is going great.  I think that Velocity Europe seems to be raising the bar from what I have seen in terms of the talks and the presentation and slides that came out of Berlin last year and then the quality of the talks and the people here speaking today, just, it’s amazing. It’s going really well.
 Joshua Bixby: Nice. See, they’re loving us in the background too. We’re actually, like, right in the middle here, you can hear the applause from the crowd.
Cliff Crocker: Yeah.
Joshua Bixby: That wasn’t for us, but…
Cliff Crocker: I think it was.
Joshua Bixby: You think we can take credit for that? People are filing past us as we’re chatting here. We’re sort of tucked into corner. I want to rewind to Keynote days.
 Cliff Crocker: Yeah, absolutely. 
Joshua Bixby: How long were you in Keynote? 
Cliff Crocker: I was at Keynote for six years, I believe.
Joshua Bixby: And you started as?
Cliff Crocker: I started as actually doing load testing, as a load-testing consultant. I was coming in to do web performance consulting, but interestingly enough that team wasn’t growing and wasn’t big enough at the time, although a really good friend of mine there, Ned Rushlow [Phonetic] [0:02:22] was doing a great job, kind of, early days doing evangelism, so I jumped onto the load testing team and worked there and did load testing for several e-commerce sites for a few years then switching over and switching into web performance consulting.
Joshua Bixby: So, tell me about the early days of web performance consulting. This would have been, what, 2008?
Cliff Crocker: Yeah, I guess it would’ve been 2008-2007 potentially.
Joshua Bixby: How was the world different back then than it is now?
Cliff Crocker: Well, it is interesting. I think, this industry moved so fast; however, I think, honestly, we still have so many of the same things that we’re still talking about today that we were talking back then. So, it was interesting. I think people kind of were not as attuned to the fact that performance had an impact on the bottom line, so it was a bigger challenge back then than it is today, but I still find myself having the same conversations that I had, you know, six year ago.
Joshua Bixby: So, fundamentally, the similar dialogue?
Cliff Crocker: Absolutely.
Joshua Bixby: Hey, speed’s important, why, maybe you didn’t have the artillery before to tell people, now there seems to be a lot of artillery out there, but…
Cliff Crocker: Yeah, and I think that the performance of browsers back then made it a lot easier to be a consultant, because there were so many best practices that you could take into account and things you could do and now that the browsers are getting better and faster, there is obviously still a lot of optimizations that hold true, but at the time it was a lot easier to kind of go through your checklist prior to or right around the YSlow days and things as they were coming out and right around, you know, the book, when the book was published.
Joshua Bixby: The book. I like that. Steve would like that. What were your lessons from those web consulting days, like, what do you take into SOASTA now that you guys are doing, you know, also in the performance business, what from that time, when that new kid comes into SOASTA and works in your division and you’re saying, listen man, over a beer, let me tell you about the good old days, there were some key lessons I learned, what would you share, grandfather like?
Cliff Crocker: I think the biggest thing, because interestingly, and I will rewind for one second, before SOASTA, after leaving Keynote, I went to Walmart, so really I was able to, and I knew this was part of my journey and part of my career is that I wanted to hop on to the other side of the fence and kind of, you know, see what it was like and what I have been telling people all these years, what were the real challenges, why was it so hard to, you know, combine the JavaScript or, you know, do any number of optimizations that should be simple and easy to do, so I wanted to get a taste for why that was hard and I think what I learnt at the enterprise level and specially when we’re dealing with a site that is so large and sort of forced to moving so slow, there is a lot of things you learn about patience and there is a lot of things you learn about choosing your battles and it is the same old truce, so now that I’m a vendor again and we’re back on that other side, I think that, you know, you can’t make the assumption that you know exactly what’s going on, on the other side of the fence and just because something is easy to do in practice doesn’t mean that it is easy to do in process.
Joshua Bixby: Yeah, and it doesn’t mean they’re idiots.
Cliff Crocker: Yes, exactly, right cos they’re not, I mean, they’re really smart guys, I mean…
Joshua Bixby: So, you evolved out of keynote. How did you get the Walmart gig?
Cliff Crocker: Actually it as a guy that I had been doing consulting with on the Walmart side that finally said, man, why don’t you just come over, just come over here and do this for us, so a lot of that was because they wanted to focus a lot of load testing as well and sort of build up a center of excellence, but, Subir, my boss at the time, Subir Sengupta, brought me in and was just great and kind of gave me a bunch leash and just let me run and build a team and really work on sort of brining performance in as a culture more than anything at Walmart.
Joshua Bixby: And you guys did some amazing stuff. You were there for two years, three years?
Cliff Crocker: Yep, two years.
Joshua Bixby: I mean, you guys, as far as I can tell were one of the leading, if not the leading organization looking at this stuff.
Cliff Crocker: Well, thanks. I think definitely we tried to raise the level of awareness and we tried to, again, change the culture and I think that we got some sponsorship, executive sponsorship really, interestingly enough, not in the engineering side, but more on the business side that said, you know what, this is important, you guys run with it. It’s okay if you go and talk about it. It’s okay if you go and talk about the fact that Walmart isn’t the fastest site on the Internet and the things that we’re trying to do to fix that, so I think it was about community, it was about awareness. A big huge thing for me was actually the web perf meetups that are happening and specifically the San Francisco web perf meetup that Aaron Kulick founded, so I got to meet Aaron very early on when I was actually looking for performance engineers and then started really getting tied into that community that’s where I met Buddy who you’ll talk about from LogNormal, you know, the company that SOASTA has just acquired, as well as Philip Tellis and then, you know, Aaron Kulick who is, you know, a very dear friend and still fighting the fight with Walmart.
Joshua Bixby: Brilliant, brilliant man.
Cliff Crocker: Yes, he is. Yes, he is.
Joshua Bixby: Where did vendors go wrong at Walmart, I mean, as you cross the bridge and were on the Walmart side, you were attacked by vendors, of course, you were a target. What did they do wrong? What were the flaws that you saw continuously from enterprise sales guys trying to sell to Walmart?
Cliff Crocker: You know, I guess, there was a lot assumption, there was a lot of fear tactics that were tried, right? I mean, the vendors coming in and saying, you know, are you just gonna let your site crash or you just gonna this, the other guys are, you know, are better than you, they’re faster than you. I think that that there was just not as much empathy and there wasn’t as much trying to really understand the positioning and understand that, you know, Walmart is a large organization, they’re not gonna move extremely fast on the cell side, but that shouldn’t discourage them from actually coming in and trying to work with them and I think patience was probably the biggest thing.
Joshua Bixby: Just didn’t last…
Cliff Crocker: It’s a long sales cycle and I hate it on this side, I hated it on the Keynote side, but I understand where it comes from.
Joshua Bixby: I know what it is like to sell into Walmart. You know that.
Cliff Crocker: Absolutely. Absolutely, you do. You do. But, you know, you guys kept smiling and there definitely was a lot of patience.
Joshua Bixby: I think it’s the Canadian side of us.
Cliff Crocker: I think it is.
Joshua Bixby: Because there is a side of that. Coming back to Walmart side cos I am always fascinated by the business model that Walmart has, forget the technical side, which is using some of this data, working with vendors very closely to optimize the shopping experience and the price for the shopping experience, I mean, everything that I hear about how they treat vendors, whether you like it or not is they’re close relationships, there’s strong direction, was part of the culture being brought over on the technical side? I mean, was part of the idea that we figured out data for warehousing and shipping to stores and we should figure out how data can be used on the website, was there any crossover there, was there anything you pulled from that culture?
Cliff Crocker: I would say, well, we definitely pulled a lot of things from the vendor management perspective and some of the people there actually really care about on the vendor management side today made it very clear that, you know, our success is depending on the shoulders of all these vendors that we work with, so that culture was definitely there. However, I think that some of the separation between Bud and Bill and, you know, e-commerce side at Walmart and the website itself, there was a pretty big disconnect.  It wasn’t until the formation of Walmart labs that I started to see some that come on and actually teams that were dedicated to data and dedicated to big data, so, that’s where we gotta start playing with people and playing with the cool toys and, you know, the very large city hoop clusters and obviously we got Boomerang up and running at Walmart, started collecting all this rich data and now that is actually driving a lot of what’s going on on that side. It is just about better understanding the customer, being closer to the customer, who is this customer that is shopping at a Walmart versus an Asda in the UK versus, you know, Sam’s Club and that type of a thing, so and then the mobile store gets extremely interesting and, you know, you saw, I don’t know if you saw Dion and Ben talking this morning, but they’re always great to watch and great to talk to.
Joshua Bixby: They are. They have a great interplay between the two of them, like, it’s a good, sort of, on stage gig they have.
Cliff Crocker: We were chatting last night in the lobby and they were finishing each other’s sentences every other word. Obviously those guys have been doing some cool stuff for years, but the things they’re doing and the opportunities they have there in mobile, if you think about the number of people that are in a Walmart store or in some type of a property all across the US at any time with a smart phone in their pocket, you start to connect the dots and see what a huge opportunity that is and innovation that goes on there between those gentlemen as well as the great team that they have there, the mobile team at Walmart labs, you know, it’s fun stuff. It’s pretty cool.
Joshua Bixby: I don’t know if you read that article in New York Times about Target and how they use data…
Cliff Crocker: Yeah, the whole thing about the dad finding out that his daughter is pregnant, yeah.
Joshua Bixby: How did that type of thing resonate in how you guys were using data and thinking about data, you know, and I ask this also from a perspective of SOASTA, I mean, there are things we can find out about people, what they’re looking at, how much they’re spending, when you think of that whole challenge of privacy around this, how do you think about that?
Cliff Crocker: I think, you know, there is a creepiness factor, right? There is always a creepiness factor and a big brother factor thinking about, oh my gosh, you know, just me as a consumer thinking about I don’t know that I want everyone knowing all this stuff about me and, you know, predicting, you know, what I’m doing. I just joined a CrossFit last week and I need to, you know, be buying these products or, you know, longer socks or whatever it might be, but to be honest I think that it’s actually, it’s not an intention of trying to, you know, invade privacy or, you know, predict, you know, all types of things to really maximize revenue, it’s more of a competitive advantage and something that, you know, all companies are having to do if they want to serve the customer better, right? So, I love it. I think that it’s great. I think big data is amazing. I think that what we’re doing at SOASTA or beginning to kind of embark on with Buddy and Philip from LogNormal and me coming in and having experience from Walmart and really delivering that product line, I think it’s not always gonna be about that performance, it’s gonna really be about human behavior and how can we sort of predict what that user is going to do next or what their behavior might be on a site, so we can think more about what should we actually be testing, on which we actually should be spending our time, what should we be optimizing and, you know, what kind of things and behaviors are driving people away. Buddy introduced something yesterday that he has talked about before called the LD50. The lethal dose basically where you guys have the performance poverty line at Strangeloop, out stick may be the LD50 where you’re looking at what point the user starts bouncing or exiting a site when performance gets to that level and sort of understanding that in a broad spectrum, but in a multifaceted way where you’re looking at multiple dimensions, whether it is browser, device type, geography, whatever it might be, it’s just really about creating better quality and also providing more input into that whole development life cycle where functional requirements come in from the business and marketing and all the way through to where we’re supporting production.
Joshua Bixby: Yeah. I saw an interesting startup today on the New York Times, New York based.  Its goal is to allow people to sell their information, so, you know, I could track all of my browsing history and then sell it back to either a conglomerate of vendors or vendors who might want to buy that information.
Cliff Crocker: Wow.
Joshua Bixby: Yeah. I don’t know where it’s gonna go, but I thought, you know, this whole world of privacy and tracking and what we do and how we interact with it, how we time it, it’s interesting and I thought that, you know, it definitely caught my interest. I was like wow, that’s interesting.
Cliff Crocker:  Yeah. Crowdsourcing is very powerful and I think what we wanna do is, well, start to feed all that data back into the industry whether it’s in a way that’s free, that we can all kind of consume it and understand how different industries perform, different verticals perform so we can kind of provide more contextual intelligence and get more contextual intelligence to our own data. It’s extremely powerful and really why I’m doing it cos the questions always change and more things you can do with the data.
Joshua Bixby: So, you have this great gig at Walmart, you’re collecting information, you’re analyzing it, you’re in big data heaven and SOASTA obviously had something pretty attractive. What attracted you, I mean, I don’t think SOASTA is a household name for most people yet, so I guess, give us, what was attractive about it and why, like, you’re a high-value asset, why did you go there?
Cliff Crocker: You know, I think that I had given myself a timeline and said, hey, I wanna do this for a while, but I don’t wanna stay here, I wanna move, I wanna actually, you know, do something that’s gonna have more impact at a global level as opposed to use within one organization, so it was parting on great terms first of all with Walmart, but really the reason that I thought SOASTA was so attractive was back to your point about vendors and partnerships. Before I even get into the technology and the things they’re good at, they were a company with a lot of integrity, you know, when I was dealing with a sales rep it wasn’t dealing with a sales rep, it was dealing with an account manager and a guy that I could call and talk to and, you know…
Joshua Bixby: Who knew something about the product.
Cliff Crocker: Exactly, yeah.
Joshua Bixby: Not just what the discounts were this quarter.
Cliff Crocker: Exactly. Exactly. You know, not how much he was gonna let me beat him up on price or whatever, it was more about getting the work done and just the innovative people there and how much the people really loved working there and really loved the direction of the company. But all that aside and that’s all nice and warm and fuzzy, these are guys are really smart, these guys have done this, you know, the two founders, Ken Gardner, who I report to, the executive chairman, and Tom Lounibos, have been doing this for a while and have had several other successful startups and the thing that Ken, really attracting about Ken and the reason that we work well together was really his history with real-time analytics and data and that was something that they started doing years ago and had perfected with one of their previous companies and we started putting the dots together and realizing, hey, we can start to deal with this data in a real-time way and also do it with some real-time visualizations and the visualization part of it kind of had me sold in terms of, you know, playing with the data, making it beautiful, breaking the mold, you know, not just developing yet another monitoring product, but doing something what could be different and really kind of change the way people look at real user measurement.
Joshua Bixby: And then you guys obviously bought a real user measurement company two days, yesterday. Tell me about that.
Cliff Crocker: Yeah. I might’ve had a little something to do with that. So, Buddy was actually the guy who I brought on to help me to get up and running with Boomerang at Walmart and I, again, met him through the web perf meetup through Aaron’s meetup there and I always loved working with Buddy, but he wasn’t a guy that I was ever gonna be able to hire and he and Philip started this great thing with LogNormal. I loved the way that they were actually, the approach that they were taking with the data, very statistical and analytical approach to measuring data and a sensible way of doing it, so, you know, rather than kind of strike out and, you know, try and find and build a team of people that could adapt and learn about performance and really sort of build that up again, which will take some time and is hard to do, I thought what a great idea to actually go out and talk to these guys and see if your interests are in line enough for them to actually come join us and do this together.  So, it was a process of several months. Taking about partnership, looking, you know, do we do this ourselves, do we, you know, how do we actually win in this space and it just ended up making sense and I, you know, couldn’t be happier about the decision. I’m very excited to work with those guys, very humbled and, you know, when I sit down and talk with Philip or Buddy about their ideas and their thoughts, it’s just exciting and it’s fun again, so.
Joshua Bixby: And, I mean, I’ve met Buddy a number of occasions and I echo your sentiments, smart dude, we’re actually gonna have him on the podcast here, so, people will get to hear for themselves. Tell me about where measurements goes, I mean, as you said, and I’ve always thought this, the market hasn’t changed that much over the last 10 years, I mean, the browsers have changed, but not much else has. We have more simulation of real end users, we’re getting closer to, you know, people are understanding that you have to use a real browser and it’s important to have a location that mimics a real end user and a bandwidth that mirrors a real end user, what’s changing?
Cliff Crocker: Well, I think what’s changing is the fact that simulations, you know, while they have served us well and they have done a good job to this point, there are absolutely no replacement for human behavior and I think that’s really what’s driving this, is that instead of a world where we base everything on synthetic and live and die by our Gomez Keynote web page test numbers, I’d rather live in a world that we’re actually basing it on what we’re measuring from the end user. So, I think leading with that as opposed to leading with synthetic is really what’s changing and how it is going to be different and also, you know, setting some new metrics or coming up with some new metrics that just make more sense.  I think what we’re big on is the fact that, you know, whether you’re at Walmart or, you know, Target or whatever sites you might be as our test go, what’s important to that company is their number and what their goal is and really looking at their own data instead of trying to base their, you know, their studies on something that Google did or Amazon did or Shopzilla did or Walmart did, when I take that same model and really see, you know, what makes sense for our customers and our users, where is the LD50 or performance poverty line for our businesses as opposed, you know, the rest of the industry.
Joshua Bixby: And do you think it’s varied? I mean, do you think Walmart is different than Target?
Cliff Crocker: Well, I think that everyone and every company that I’ve ever talked to or consulted with thinks they’re different and I think that’s…
Joshua Bixby: Are they?
Cliff Crocker: I think that there are different types of customers. I think that, you know, whether you’re…if you’re selling something at Walmart that someone could easily pop over to Amazon and buy because the page is slower and it’s not loading, it’s different than if you’re a specialty store and selling something that someone absolutely has to have from that store, like a cheese shop or something, right? So, I think that they are different. I think that what’s not different is really the fact that we do see some level of drop off, we do see some level of engagement where people are just getting more and more and more impatient or more expecting, you know, to be delighted as Philip would put it, you know, by providers like, you know, Walmart and everyone else, so I think that expectations are growing at the same rate, but I think that behaviors might vary enough and not even just between industries, but, you know, one of the study that we did that you guys have posted on your blog as well and talked about was basically where performance changes is depending on the type of product you’re looking at, that patience or that tolerance for someone who is buying something like an iPad is going to be, you know, much greater than someone who is buying PowerBait or, you know, something else off the site, right?
Joshua Bixby:  Yeah and that’s very interesting, almost within segments of the product line one has different tolerance, right?
Cliff Crocker:  Exactly.
Joshua Bixby:  I know that for myself.  I’m buying a car right now and I’ll spend a good 20 minutes on a page, you know, and I’ll wait for it because I want to see what capabilities and…
Cliff Crocker: A very Flash-heavy page.
Joshua Bixby: Yeah, I mean, I don’t like it, but I definitely wait because, you know, it’s a big purchase and I’m gonna take some time, so I get that, I mean, as somebody who preaches that every second counts, I must admit, in my own behavior sometimes I definitely will spend more time on one thing than I will on the other. That makes a lot of sense to me.
Cliff Crocker: And I think, you know, back to sort of your original point, that it hasn’t really changed that much.  I think that it has and it hasn’t, but also this whole mobile thing that, you know, isn’t just a fad anymore, has really changed the game as well because it has made it harder again for developers to get give a fast user experience that people are expecting, you know, same speeds over carrier lines that they are over, you know, DSL or whatever.
Joshua Bixby: Yeah. No, I know, that’s something I’ve been spending my time thinking about the presenting on, which is how do cell phones work and why are they slow and, you know, that’s definitely a real area of interest.
Cliff Crocker: Right.
Joshua Bixby: What have you learnt at Velocity that’s new, net new, like, you’re gonna take back to the shop and say, holy smokes! Everyone needs to download this slide, anything?
Cliff Crocker: Well, actually, I, embarrassingly, wasn’t able to attend your talk.
Joshua Bixby:  So, there you go.  That was like six of the slides I was thinking, that was like a soft ball for you.  Other than the genius that I presented, any other one or two slides that stick out?
Cliff Crocker: And it was only because Steve organized the track in a way that LogNormal presenting at the same time as you were, otherwise I would’ve been there.
Joshua Bixby: That’s true. That’s true. You’re one of the only ones that has an excuse, although someone else had to do an intro so I figured they had an excuse too, I can’t remember who it was.
Cliff Crocker: I think aside from that, which I’m sure to be inspired by and motivated by…
Joshua Bixby: I love it. This is good. Keep going. I can handle this forever.
Cliff Crocker:  I think, actually, one session I just came out of with Pat Meenan who I’m a huge fan of, he’s my hero.  He was doing his whole presentation that had nothing to do with Webpagetest, but had everything to do with single point of failure and SPOF-O-Matic and I think that’s been introduced, you know, for a while and then it sort of died off and now it’s getting really hot again at Velocity in US and here and something that I can talk to customers about from a performance perspective, in terms of beingready for holiday or something.  It is not just about load testing, but look for these things and so I think SPOF-O-Matic is great.  I’m excited about that.
Joshua Bixby: That’s cool. I was beta testing that early on and loved it, like, I was so excited using it, so I’m a big fan of that stuff too.
Cliff Crocker: Yeah. Absolutely. So, if I had to say there’s obviously something I learn with every talk even if it’s someone who has recycled their slides 12 times and I’m not gonna call out any names or anything, but I think that every single one of those talks…
Joshua Bixby:  We all recycle a bit, but there should be some, you know, especially a conference of this magnitude, you should put some original thought into I think.
Cliff Crocker: Yeah. I was actually, you know, sort of silently referring to Steve’s slides that aren’t so great, but… 
Joshua Bixby: Of all the guys, he is kind of known for the same shtick.
Cliff Crocker:  Absolutely.
Joshua Bixby: It’s like when you go McDonalds, you expect a burger to be the same every time, you know.
Cliff Crocker: But you get something new every single time. I take something new from it every time I hear it or I hear something in a different way or I think of a new question to ask and what I love of Velocity is the fact that it’s the hallway track…
Joshua Bixby: Yeah, we can hear it in the background here, you can hear the hallway track, we’re in the hallway track.
Cliff Crocker:  Absolutely. It’s the networking, it’s the talking, it’s the ideas and the community and the companies that get started and then companies that have successful exits and all circle around Velocity that I think is just amazing. So, it’s an amazing community. It’s been amazing to me, I mean, I have a lot to be thankful for this whole movement or whatever you wanna call it because it certainly has created endless opportunities for me.
Joshua Bixby: No, it’s a good tribe. Cliff, thank you.
Cliff Crocker: Yeah.
Joshua Bixby: Take care. I want you to get out, I’m gonna stop this so you can get out.  I see some clouds over there…
Cliff Crocker: That’s right.
Joshua Bixby:  …kind of in the background.
Cliff Crocker: That’s right, yeah.
Joshua Bixby: So, get out there and enjoy London.
Cliff Crocker: And what was the name of the place again, one more time?
Joshua Bixby: Borough Market.
Cliff Crocker: Borough Market, got it, got it. 
Joshua Bixby: Right near London Bridge. 
Cliff Crocker: Thanks.
Joshua Bixby: That’s my current…I don’t know when it’s open till, but it’s definitely a lunch place.  It’s fantastic.
Cliff Crocker: Okay. Excellent. Thanks for having me.
Joshua Bixby: Okay. Take care. Well, that was great. Thanks for listening and thanks again to Cliff for taking the time out of what was a very, very busy schedule to chat at Velocity. All the links and slides that Cliff and I spoke about are available on the blog,  If you have a suggestion for a future podcast, topic or guest, please drop me a line at  Any and all suggestions are welcome.  I’ve had some crazy ones and I’m trying to book some of those crazy chats and I look forward to more.  Have a wonderful day.  Thanks for listening.

Joshua Bixby:  Hello everyone it's Joshua again, President of Strangeloop Networks and blogger at  Welcome to another edition of the Web Performance Today podcast series.  I had a great opportunity this week to sit down with Theo Schlossnagle, CEO of OmniTI.  Theo is a geek among geeks, you probably have a tab in one of your browsers right now open to an application he has helped scale, an infrastructure, he has helped build or a distributed system that he has spoken to or architected himself.  He has been doing long, list longer than most people since 1997 probably longer than you and we talked about what he actually does in the day, his tech predictions for the New Year as well as a really interesting beer project where he – a big data beer project that he has been involved in, that’s very personal, I hope you enjoy. 

Joshua Bixby: Hello everyone.  Welcome to another installment of the Web Performance Today podcast.  I am here joined by Theo on the phone, Theo Schlossnagle.  Theo please pronounce your last name for me?

Theo Schlossnagle:  I think you nailed it, Theo Schlossnagle.

Joshua Bixby:  Theo is CEO of OmniTI, many of you will know him from his incredible presentations at Velocity, but probably don’t know a lot about what he does every day and frankly neither do I.  I have spent time with Theo in Beijing, what I guess Berlin, London, Munich during the Velocity Conferences but I haven’t got a chance to really spend much time talking about OmniTI and performance.  So Theo give us the, give us the digest version, tell us about what you do every day?

Theo Schlossnagle:  What I do every day.  I have the challenges of a CEO in a company that’s growing, I do all sorts of undesirable things like look at contracts and talk to landlords and things like that.  On the technical side, I think that I split the gamut in a pretty interesting way.  Most of it is related to performance and scalability of large scale web apps.  Everything from infrastructure design and automation up through kernel level stuff, so I do a lot of work in the ilumos kernel for our datacenters and then up the stack, you know, through databases, through web servers all the way in to JavaScript to browsers, so I kind of float around from task to task with my eye specifically focused on performance and scalability.

Joshua Bixby:  And the customers who you know – if I was a customer how would I know if I was well targeted to come to you guys, what would I look like, am I a Twitter, Facebook, am I you know more like an enterprise, is there a sweet spot for you guys?

 Theo Schlossnagle:  I think that we have three types of customers and they kind of classify outside of market verticals.  We have customers that have large scale applications today that you know they are lumbering along and they are wondering if they have done everything right the technology is five years old or ten years old, they want to know if – basically they need an assessment of their architecture.  A lot of times that happens for acquisition purposes as well.  Then we have the other type of group that are looking to build a very large scale web app; they have a prototype they've tested it with a hundred or thousand users and they would like to you know take that to the moon, so they are looking at picking on 10 million or 50 million customers and they’re very concerned about their ability to address that demand as it increases the thundering herd of the internet is a scary thing.

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  And then the last group of people who have an application that has been hit with a thundering herd and been trampled to death and they come to us with a thousand pieces and say please put this together again.  But in the Enterprise versus Dotcom sort of world, we have customers that that range that you know that that cover that pretty much the whole set.  We service about 15% of top Alexa 100 and so we deal with very, very large websites.  We also deal with a lot of enterprise accounts that most people are on, you know, there is a lot that you would find rather boring.

 Joshua Bixby:  I’m sure.  So part of sort of establishing that streak grid because I don’t think people know a lot about you guys is to dig into what are you seeing out there I mean you guys have been doing this since what ‘97, has the world changed, I mean as you see these presentations at Velocity or people talking about performance are the same basic principles at play or has there been or will there be some dynamic market shift in how these things are architected, built you know related to performances.  Is it still the same thing as it was 10 years ago?

 Theo Schlossnagle:  I think from the scalable systems engineering side, we are largely seeing the same thing.  The question to scale out or scale up is still there; people feel like they are a little bit more confident in their answers but I think they are no more correct than they were then.  They are being lured into the scale out always mentality just because software allows them to do that now when scaling things out makes things more difficult to bug, distributed systems to bugging is you know a nightmare always.

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  So if you could scale vertically you choose to but that’s the same you know in 1997 or in 2000 when things started to get pretty serious, you always wanted to scale up until you couldn't and then you scaled out and you know that’s pretty straightforward.  I think the radical difference is that back in the, you know, late 1990’s there was so much less complexity being sent down to the world’s largest distributed computer which is, all of your consumers browsers.  So the world of JavaScript running on the client side today is it, you know, its night and day compared to what it was and so that’s radically different.

 Joshua Bixby:  Is that a trend I mean, you have probably seen some of the post I just did recently, did one talking about the growth in not only the size of pages but JavaScript specifically and the complexity of that.  Is, does that hit you know a point of diminishing rates of growth.  Do we, you know, what – is that just going to keep going up, getting bigger?

 Theo Schlossnagle:  I think that we are still in that uncomfortable space where we do not have comprehensive development environments, best practices, IE’s and things like that for developing very, very rich applications on the client side.  People are still kind of evolving their applications.  They are building largely HTML applications, HTML5 leaving the JavaScript out of it and then adding JavaScript for a flare and then adding it for features and then adding it for complex features and then driving business lodging into it and it's kind of a snowball effect and then you are left with a half a meg of JavaScript on the client side where there aren’t really the same the quality control tools that we have on the backend side.  So if you are building a database and see for example or writing an operating system kernel, the landscape of software engineering practice is there, it's so much more mature than the landscape of software engineering when it comes to shipping JavaScript to do that with the client.

 Joshua Bixby:  One of the things that’s always intrigued me because you work on, you know, the development and operational side, you seem to have, you know, seem to run the gamut but you spend a lot of time talking about the operational side.  So for the people in the operational side of the business, what do they do when they are handed that app that has the half a meg of JavaScript.  I mean are you helping people find solutions to that problem sort of assuming okay it's a black box and this is what it comes with or are you finding that you are constantly in discussion with developers?

Theo Schlossnagle:  Oh constantly in discussion with developers.  I think that days operations and development and engineering being separate are, those days are over. I think they were numbered sometime ago, I think the numbers have run out.  It is very difficult to compete today without shipping rather complex JavaScript down the pipe that stuff cannot really be, like any software it cannot be magically optimized.  The algorithms and data structures that youre chosing because the code is definitely complicated enough to necessitate algorithms and data structures on the JavaScript side that that is not right for you know magical optimization.  We have gotten as far as we will go with things like the VA chip.  I mean they will steadily improve, but there is not double digits percentage there left I don’t think. 

 So really it's about good software strategies strategies and you know don’t repeat things, basic software engineering premise, how do you measure that because the one thing that developers don’t really have it's a perspective of how their code is running universally right.  They can test it on their browser, they can test it on five or six different browsers but there is difference between their software and what we used to write 10 years ago is that that JavaScript is in the client because it needs to run there and it's not just because we don’t want to run it on the server side.  It's that there is an active program running on the client side to receive sometimes arbitrary data and we are sending them very complex streams of data that were you know assimilating into something useful and you have a million or ten million customers that are all getting different data streams, it's a very I mean the richest, you know the wealth of all of the performance analysis of this software is actually in the wild, which you want is empirical data about how your performances are, is operating, how your java script code is operating, how it's functioning, what it's errors are, all of the performance profiling needs to come from the real world.

 Joshua Bixby:  And that really changes the game for sure?

 Theo Schlossnagle:  Yeah it absolutely changes, I mean so we, we – most of the clients we work with have some sort of bespoke implementation of that. 

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  Right there is some sort of telemetry being collected, not just like how long that they are taking down with the JavaScript.  But there are you know meters and metrics being counted inside the JavaScript code and that stuff being passed back up to the server to the infrastructure being analyzed by developers and when you look at that analysis of data that looks like what ops have been doing for the past fifteen years or twenty five or thirty years on other equipment and we are looking at network switches and CPU’s and disk IO and things like that.  It's what database engineers have been looking at when it comes to you know, database performances you know like four hundred graphs for any descent Oracle DBA.

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  That is immature on the JavaScript side and it's evolving rapidly and what we have noticed is that like all of the tools are very, very similar and it's actually valuable to have all of that data just, not just you know the systems and network information but also the database information, the distribution database infrastructure information and the JavaScript performance profiling stuff, all of that really needs to be in one tool because they are all tightly interrelated.

 Joshua Bixby:  So I had a really interesting discussion with a gentleman who remained unnamed out of Microsoft really senior architect who basically said ops guys are used to dealing with tons of data and devs aren't and what we are seeing is a culture where devs are being exposed to big data which has always existed on the ops side.  Do you think there is some truth to that?

 Theo Schlossnagle:  I think it's a really interesting - actually I had a discussion like that today.  I think that that’s true but that sword has two edges because developers hadn’t been indoctrinated with crappy tools, their eyes are not, they are not winking in pain.  So you look at network operations, you look at systems operations they are used to collecting telemetry up with every machine they have, you know they sample everything every you know one second or ten seconds or sixty seconds and they have got graphs coming up their ears and you give that to a developer and they would go wow that’s really useful.  They don’t stop there, they ask the next exact, you know that there is the next question which is well I don’t have enough data, actually I need all of this extra data, I need this deeper data, I need this to be more dimensional.

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  And then you know the OP guys are like well that’s how the way these tools work, which is sort of interesting because then the software engineers go and they build the tools, right cause that’s what they do.  So what we have been doing for a while now with one of our companies Sekonic is trying to take those more evolved telemetry analysis tools, the ones that engineers have designed to answer their questions and push them back into the ops group to make the decision making capabilities faster and more rich.

 Joshua Bixby:  If only the world could see that there is all this you know incredible value and collaboration, I mean I keep coming into organizations and admittedly more enterprise type organizations where it feels like there is just a complete separation of the worlds; there is the velocity world which is big data iterate, fails quickly, fails cheaply that whole lean startup mentality and then I get transformed into this world of almost I shouldn’t say this dinosaur like, waterfall like approach to the world where none of this is really happening, so it was interesting when you said you guys work with you know 15 or the top one 100 or do you also find that there are just stark cultural differences between these businesses where some get it and some you are just, you are looking at these people thinking I can’t even start to explaining this to you because you don’t understand the first principles, I mean I pull my hair out over this stuff.

 Theo Schlossnagle:  I think that all of the organizations actually understand the value of data.  They all understand, when you go in with a value proposition around understanding how your systems and your business are working I think you basically would be asinine not to understand that right like that’s clear.  When it comes to the tactics for are actually running the organization, but you know as it relates to technology how technology is deployed, how it's developed, how it's managed, I think there is an enormous cultural divide but I think that it's reinforced with a vernacular that the Velocity group will now let go off to this idea of move quickly and fail fast.

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  Enterprise, you will let, you know, to say failing is never an option, it's just reality, right.  So they don’t want to embrace the fact that they are failing right they want to avoid it.  But that doesn’t preclude very similar tactics by this whole waterfall model, we were consultants, when we go into a place we don’t try to disrupt their language, if they are with Linux , we stay with Linux; if they were with Solaris, we stay with Solaris, you know, Java, Python; they are using waterfall, if they are using AdRoll, we fit in and try to add value inside the framework that they have established.  So we have worked in a lot of you know large contracts that have been waterfall driven and all of it comes down to how well a project manager or scrum master understands when to apply the model and when to not and that’s the success of project, it's not really AdRoll is better than waterfall.  It's that leaders, some leaders are awesome, some leaders aren’t. 

In the waterfall load you can actually orchestrate it so that you do a deployment everyday right, we will see this idea of dark features for example right.  Waterfall says I am not going to release my product until a certain date well I didn’t release it, I mean at the point it's not released, right.  So now it's all mincing words and when you maintain that exact same facet that the enterprise expects you have a lot of flexibility in the way you implement things.  So the problem is, is that in the velocity group when we talk about our implementation we try to push those tactics in that mentality all the way up into management and they don’t want to use that mentality.

 Joshua Bixby:  Yeah but as you say the tactics can be very similar almost indistinguishable in many ways.

 Theo Schlossnagle:  Absolutely, but there are distinct advantages too, for example launching code every day or every hour all right so if I am forced to develop in a model, even if that model – you know even if the approach to management is waterfall, but I am forced to develop code that’s not allowed to break the existing system right I am not adding features, even if they are not released I am putting things in and I am not allowed to break the current regression tests.  That means I can deploy the code, no one is going to run it.

 Joshua Bixby:  Yeah.

 Theo Schlossnagle:  Right, at the end of the day when you are ready to launch your product you have removed that, you know that two months death cycle and death march of deploying that code that has never been deployed before that’s a typical problem in enterprises that they don’t embrace operational mechanics of the release until the very end when it's suppose to be launched, which is disastrous.

Joshua Bixby:  Yeah and then they wake up and think and wake up and realize that software is you know made by humans and errors and when an error when you know because it, because of errors they are in trouble and they haven’t got you know they have got a long cycle to get that stuff out.  I am interested on the data side the last time when we were in Beijing last year, was it Beijing?  No it was Berlin last year you were trying to drink a different beer every day or your goal was to have if I remember correctly three hundred and sixty five different beers drunk am I remembering that goal properly.

 Theo Schlossnagle:  That is correct.

 Joshua Bixby:  And can you tell me what you achieved, that was last year right so 20, end of 2011 as a goal.

 Theo Schlossnagle:  Yeah I measured those things, Thanks Giving to Thanks Giving so my dead line is tomorrow.

 Joshua Bixby:  Okay so what were you last year, set the table for me what were your - what last year? 

Theo Schlossnagle:  I think I hit three seventy eight or something like that.

 Joshua Bixby:  Amazing, so for those who didn’t spend time in -.

 Theo Schlossnagle:  But I haven’t achieved this year.

 Joshua Bixby:  You have.

 Theo Schlossnagle:  I am at 715 total, so whatever that – that is.

 Joshua Bixby:  Okay and those are 715 unique beers.

 Theo Schlossnagle:  Unique beers.

 Joshua Bixby:  So can you find any left, I am just thinking as you were talking about data and the process I am thinking if there is a data project that we talked about are you at the end or do they import more varieties?

 Theo Schlossnagle:  They make more varieties every year and there is usually you know some of it's been a 100 and 250 new beers, new recipes that come out every year, the micro brew scene in the United States is incredibly vibrant, but there is probably about 8,000 to 10,000 beers, unique beers on the market today that I could go buy.

 Joshua Bixby:  Okay so you got, you still have ways to go, you could still hit, you still hit six or more, six or seven more years with this?

 Theo Schlossnagle:  Absolutely, I have enjoyed every minute of my weight gain.

 Joshua Bixby:  That’s brilliant and what is the – not only weight gain but you are also in the business of changing what hair and facial hair quite readily, what is your current, what's the current configuration?

 Theo Schlossnagle:  I would say that I am in unkempt San Diego style.

 Joshua Bixby:  Okay describe that?

 Theo Schlossnagle:  I had the, I had been told I look like the person who does the voice of doofenshmirtz.

 Joshua Bixby:  Okay so for those who don’t specialize in such arcane references, can you describe that for us?

 Theo Schlossnagle:  I have the goatee with no mustache and a head of hair.

  Joshua Bixby:  And color, color of hair?

 Theo Schlossnagle:  Oh yeah brown.

 Joshua Bixby:  Okay, I think the last time you saw you were blonde were you?

 Theo Schlossnagle:  I had dyed my beard, I had -.

 Joshua Bixby:  Oh that was it yes.

 Theo Schlossnagle:  I had an enormous beard and it was that black, I was going to go as Brian Wilson for Halloween.

 Joshua Bixby:  Oh nice, nice, for those who don’t know Theo there is a constant change, is this, there is just a testing, there is a multi variant testing process that you undertake with your looks.

 Theo Schlossnagle:  I wish I had the discipline to you know take a picture of myself with my laptop every day. 

Joshua Bixby:  You should.

 Theo Schlossnagle:  Because that would have been brilliant but now alas I haven't.

 Joshua Bixby:  Two more things before we end.  First we are at the end of the year getting close to the end of the year and this is we are getting close to prediction time, so I am not going to ask you for a super bowl pick but I am going to ask you to tell me about two things that will surprise me next year?

 Theo Schlossnagle:  Two things that will surprise you next year.

 Joshua Bixby:  Ideally tech but you know I am open.

 Theo Schlossnagle:  Okay that’s a great question.  I think that the convergence on distributed database technology, people feel like it's coming and they will be surprised that it did arrive.

 Joshua Bixby:  Okay.

 Theo Schlossnagle:  Yet again -.

 Joshua Bixby:  Yeah that, I have heard that predicted a few time so okay but the negative side of that prediction is probably a much safer that okay that’s a good one, what else?

 Theo Schlossnagle:  Yes, let's see I think that the android marketplace, the you know the devices running android, the android app market, the ecosystem around the android has hit the you know the corner of the hockey stick and I think that it will absolutely explode next year.

 Joshua Bixby:  That’s fascinating because we -.

 Theo Schlossnagle:  That’s been steady this year but I think that the tides will shift.

 Joshua Bixby:  So I use strangeloop as an example we have you know fifty, sixty people here, we went from the Blackberry to deep, deep iPhone in penetration and where I am not seeing my dedicated iPhone users all move to android and it's happening on mass it's amazing, it's like a virus that’s taken over.

 Theo Schlossnagle:  Yeah and I think the trends that we see in the tech community like that always you know they immediately pressure end consumer activity that that was in with that so.  I think that, I don’t think it will happen this year and we have less yet but and I don’t actually think it's going to be the nexus four seven ten series but that really does it and I think that will raise the awareness necessary to make that you know that tidal shift.

 Joshua Bixby:  And lastly Velocity Beijing you know I had a chance to be there with you last year, I understand you are running it this year, so tell us about quickly about running it and finding who is listening why they should go?

 Theo Schlossnagle:  Beijing is awesome, sure opportunities see the great wall, it's a forbidden city, Beijing is a fantastic town, you can combine technology with good site seeing.  I would say that if you are not fluent in Chinese that there are probably other velocity conferences that you will extract more value from.  Most of the talks are given in Chinese and the translation there is always challenging because almost live translation and technical topic from English to Chinese is difficult.  Planning here has been a very interesting trying to recruit talent, it's a very different demographic that I normally talk to so being in the United States and finding conference in the United States it's kind of assumed that everyone speaks English and they do and so everyone who works at an American company basically speaks English. 

 So the criteria for finding speakers at the companies that I would like hear from so my usual approach to finding speakers is I find that you know some really interesting architectural problem that’s been solved identify the company that encountered it and then I try to find some member of that technical staff, management staff that that really understands what that solution look like, what were the challenges involved with, there were personnel challenges or technology challenges or what have you.  We are limiting that to people who are not limited, we are finding our good number that were fluent in mandarin, posed on interesting challenge though I had a lot of help.  But it worked out great so now we have a speaker from Google, a speaker from Facebook, a speaker from Twitter, one speaker from Etsy and there is me.

 Joshua Bixby:  Who is coming from Etsy?

 Theo Schlossnagle:  Sam Haskins.

 Joshua Bixby:  Okay I don’t know.

 Theo Schlossnagle:  He is going to be talking about a continuous deployment.

 Joshua Bixby:  Oh great well those guys know that stuff.

 Theo Schlossnagle:  They know that stuff.

 Joshua Bixby:  It is, thank you for taking the time the day before Thanksgiving to chat, I appreciate it, enjoy your meat and bird over the next two days and let's connect, I guess we will see each other definitely in June at Velocity, but hopefully before that.

 Theo Schlossnagle:  Yeah excellent.

 Joshua Bixby:  Thanks man.

 Theo Schlossnagle:  Looking forward to it.

 Joshua Bixby:  Good take care.

 Theo Schlossnagle:  Alright thanks.

 Joshua Bixby:  Thanks for joining me today and thanks again to Theo for making the time out of his incredibly busy schedule to chat.  Be sure to check out my blog for all the links what we talked about today.  If you have a suggestions for a future podcast fire me a note at, hope to see you again thank you

Where to find Theo:

Mentioned in this podcast:

Subscribe in iTunes: