Data Driven Decisions Ben Curren

hopefully I'm gonna share some experiences that I've had since being an entrepreneur and share a lot of the learnings that I've had over the last five six years and hopefully we'll help you guys be more successful with your startup so I I am Ben Curran I'm the CTO and the founder of outright comm I started my journey as an entrepreneur about six years ago I was at Intuit I was working on QuickBooks in payroll software a software engineer writing C C++ C sharp code and then at some point I said hey I am tired of working for the man and I have some ideas of my own and I think that I could step out and do a good job here so I left Intuit the same year I bought a house and had my first kid to start a business had very little income and I started a business doing web development mainly as a consulting chop and my idea was to say okay I'm going to find all of these projects and make tons of money and then I'll be able to take that money and build a bunch of my own products it took a little bit longer than I thought it would take first it's hard to find customers so after a bit of time I actually had that rolling and I could spend about 30 to 40 hours a week of doing consulting and then we started spending that other time on building our own products and we built a total of five products fairly quickly like three to three to four weeks like on each idea tried it out on small basis and just see if we got traction one of those ideas at the time was called go bootstrap which is now out right and what how that came about is when I started East East Omni which was my consulting company I brought on a partner and I used QuickBooks desktop software and it's a real pain to share your books with a partner so I said that's that's that's fine I'll just go find a product that was my original ideas I'll just use QuickBooks Online or or find another product well two things kind of happen there one QuickBooks Online at the time only worked on ie6 Windows I literally have have to run parallels just to run in a website essentially and it was really convoluted and complicated so I kind of said hey I know a lot about this I worked on QuickBooks myself I know a lot about accounting I even long time ago was a consultant so I've actually done some of this before and that's kind of the original idea of go bootstrap was hey build a very simple accounting and bookkeeping solution that I can share with my my partner in my accountant in to keep track of my business so that is how we got started that happened about four years ago now we got we gained some traction enough traction where we were able to raise a seed round and now we've raised a series a we're about 25 employees we'll be growing to 50 this year we have 200,000 customers and continuing to grow and trying to grow to about 800,000 customers this year so first I'm gonna go right into what is data-driven decision making so to me what it is is you're you're tying your decision to the objective and the outcome and you make that connection extremely clear so I'm sure many of you have been on projects or building a product or designs and you get into a room and and you're like oh you know I think this this workflow should be these four steps and then someone else goes oh I think it should be one page and you guys argue and you have really no data to back it up it's just at that point kind of opinion on why you think one version is better than the other and we have a lot of that especially at the beginning you know when you're hashing out ideas so what what this is is when you're making those when you're making those arguments you really step back and say hey what is the objective of this thing is there a way that we can actually try out one or two versions or really cheaply figure it out and see which one achieves that objective better so I thought to get us started I'm gonna start with some actual real-world examples that we have done it outright and then I'll go back to the basics of like how to run this on your own so here is kind of a typical ad type example and I'm sure you've seen this before so hey we want to increase our registration rates from 12% to 18% so that's like that's the core objective the next thing we do is we start looking at our hypotheses so we generate a bunch of these and then you're like okay which one do I believe is going to have the highest chance of moving from 12% to 18% so we run a lot of experiments and this experiment we actually ran in last week and we decided to add our partner names on our homepage let me step back for one second explain outright and what this is to understand the context here so what I'll write does we've evolved a lot since our original idea so it's still the basis bookkeeping and accounting and help you run your business finances but it we do it in an automated way so the way the way it works is we have various data partners like eBay's and Etsy's and PayPal and bank accounts and credit card accounts and we attached if you're a business we hook up to all of these accounts so you just make you sell invoice get paid use your credit card just like normal and our product will suck in all of that data in every transaction and as it does that it will figure out your tax implications and how your business is doing what are your most profitable products who are your best customers and things like that so it's like doing analytics on on your actual financial data so a partner here is what we would call like a data source so a partner would be PayPal or Etsy because we support those that did those data flows and so our current homepage didn't really have anything about that and our office is hey if we told people that our our registration rate would actually go up so you'll actually see here the two variants hopefully guys can read that so there's a slight text change here that's literally the only change between the a/b variants and you'll see the bottom one is it says hey you can import your data from eBay and Etsy in PayPal so just kind of maybe as a show of hands I'm curious who who would think that top-1 would win the bottom will win or it actually makes no difference whatsoever so let's first start with the top who would guess the top one would have a better conversion rate okay a couple people what about the bottom one okay more people and then it what about you don't think would make a difference alright okay so yeah so it's kind of tight between the last one and it does make a difference so obviously we thought this one we do better and after we ran the experiment we saw that it actually decreased by 19% this would be like this is a very classic example of intuition just being dead wrong like right you're like hey this seems obvious our product does this thing and this is a no-brainer change so most people just make this change and just move on with their life but by doing this we can we're now attaching like the actual outcomes to the decision and we can say hey that decision was actually not that wasn't a that wasn't very good decision and and then we can get some learnings from that so we're actually in the process like every time we run an experiment we then go and try to learn and get some new learning like why aren't people why does this turn people off so we have hypotheses there and we're currently working to actually find out why the second one didn't win so then we can learn and then the next time we come up with a hypothesis we'll have this additional knowledge so I know that's a pretty simple example but all I thought I'd take it one step further so homepage testing is something like is very like that's a part of the funnel that's really important and people always work on it so we have this concept called champion versus Challenger and I think it's kind of a powerful idea whenever you want experiments time is really against is against you right so it's how many experiments can you run in any given time is about is like how many times you can learn so it's concept of champion versus challenger especially for small tweaks like this like changing button colors changing text what we do is we just create a humongous list of every idea we have to test on our homepage and every single week because that's about how long it takes for us to get statistical significance on our homepage we we would run a variant against the champion so the champion in this case is the top one the Challenger was the bottom one in every single week we do that and then so the Challenger lost okay we'll get some learnings but now we're actually right if you go to our site we're actually running a different test and we're learning something new and I think for these simple these simple type experiments it's important to just continue just get a process and a rhythm going to just go ahead and be testing every single week and then and you'll see that you'll get a lot of learnings very quickly next I want to start on another example it's actually the same objective increasing registration rate from 12% to 18% except in in this can in this example we did something slightly different that I think you would find interesting so our hypothesis here and it's it's best practice like every time I go to a board meeting they're always telling me about another portfolio company that put Facebook Connect on their page and and you know like you know rainbows and unicorns came out and everyone's super happy and that and that's great but we we are in the small business space in finances and we were like well I I could believe that could happen for us but I have some doubts some of the doubts are are people really going to use Facebook Connect to connect to their financial data I I don't know the answer to that and because yeah so so we could have built Facebook Connect tried it and then said oh yes it worked succeeded or no it didn't but what we what we try to do is push our thoughts to say hey what's the absolute cheapest way in the world to test this out so what we ended up doing is we built this experiment so it looks like we have Facebook Connect there however if you actually clicked on that button all it does is we're measuring the people who click on that button and it goes to the page it says hey we actually don't have facebook connect yet however you can register this old way and thank you for the valuable feedback and what's great about that is now we don't actually have to you know to put a button up there is very very cheap and we don't have to then go and build Facebook Connect do the support for managing passwords and all the other headaches to make this learning and once we did this what we actually found is that it did not perform to what we needed so I don't go through the detailed math of how he came up with these goals but essentially what we did is we took the we took the current traffic in its registration rate and said hey how many of these people have to move to Facebook Connect and what conversion rate to make a difference that's how he came with these goals but what you'll actually see is we've we proved here that our goal is three percent we actually got one percent our goal is 10 percent they have two percent so if we built this it wouldn't have performed enough to even justify the investment so you you're gay that's all for anything see you're dealing with security else you're dealing the security we do so we measure like we measure a lot of things and so experiments we have various hypotheses and we look at these but we also look at like other core numbers as well to see if they affect other business metrics I think that's what you're asking but perfect perception of security and I'm not sure oh I see yes and yeah I don't was there quiet it was that a question sorry oh I mean yes in it kind of depends on each experiment I guess but in this one yeah we would look at so you know sometimes I'm doing these fake well actually I'll just go to the next slide real quick cuz you'll see okay so I call this like a fake feature so it's kind of like hey we want to build this feature because we believe it'll be really great and it's gonna change all these metrics well before you do that especially if it's expensive and you're really unsure you know build a fake feature first and measure it and then if that works then move move forward to actually start building the feature and figure out the rest of the gaps there essentially and yeah you have to look at lots of different things there and you have to be careful that these two because you can really annoy people like in this in this case I don't like where we were hey if someone clicked on Facebook Connect and got our our original signup form are they going to like hate outright forever and we just didn't think it would matter and we got no complaints for example but you know stuff that you have to watch and then let's see so the third example I have is is you had a and yet another way of experimenting so what we wanted to do is we wanted to increase customer satisfaction by making it easier to manage and categorize your transactions so first let me explain the product a little bit more so it's very important in our product to once you connect those data sources and this data is flowing through is we do our best to do categorization and tax roll-ups and all these things but hey Natalie people have to go in and they're gonna say like oh did this invoice get paid and they're gonna want to search their data they're gonna want to categorize things to say oh you thought that was a meal but it was actually an office supply things like that so we have heard customer complaints that they're frustrated with the way they have to manage that so we want to say okay let's revamp that and the way we we measure satisfaction is something called NPS the Net Promoter Score let me explain that real quick so Net Promoter Score is what you're doing is you asking people to say what is will you recommend our right to a friend on a scale of one to ten if they say a nine out of ten you call that a promoter a 1 to 6 would be a detractor and a a 7 & 8 would be neutrals essentially and you take that you take the promoters minus the detractors and you create an average and that's your your Net Promoter Score and it's a good way to measure customer satisfaction and we care about it so much that we automate it so every person who signs up for our site like after two weeks after four weeks after one after three months and after a year they get Net Promoter surveys and we attached it to those customers so we actually have a customer how they think of our product we can roll that up across like eBay people versus Etsy sellers so we can actually see that so four changes like this why it's great is because you can actually make these changes in the core product and you can say we did that Net Promoter Score change in a positive or a negative direction so in this case we said okay let's let's update let's do like let's go crazy let's let's make like almost a new account center so we almost started totally from scratch here because we've heard so many problems here but we still wanted to validate our hypotheses and these are actually the two variants so the right side is the new version and the left side is the old version so as you can probably see it's drastically different the way you manage data is different and this is a pretty big change so I think a couple interesting is that we did is one is we created the the right side version like a really minimal version at first and it actually had a subset of the functionality that our current version had so if you gave that to current customers they would not be very happy because you're taking away functionality and no one likes when you do that so we kind of got creative we said all right well what could we do to test this but not to annoy our current customers so what we did is we actually gave it to 50% of new customers because they don't know that that feature existed in the other version so we built kind of the core of our idea and then all new customers for about two days were split be 50/50 between the new version and the old version and and then everything else and the product worked the same and then over time they got NPS scores is like everyone else so we waited 30 days and we got back our Net Promoter scores and what we found is that the new version actually has a 12 plus 12 NPS score which is huge it moved just from a 45 to a 60 what c7 oh sorry well anyways excuse my math and that brings us to I mean that's that's a very big deal in Net Promoter in the Net Promoter land it brought us from kind of better than QuickBooks to approaching Apple so this is this is this is a huge win for us and we actually just so after that data point we then heavily invested here spent another two months adding every single other feature continuing to iterate on it with new customers and then roll it out to our full customer base actually about two weeks ago and now across all all of our cohorts you can actually see increases on that promoter score so this is a really big win for us so I think one of the learnings here that I thought our best practice would be talking about local maximums and so what this is is like when you have a homepage and you're changing button colors and text you're somewhat assuming at that point that the general design of your homepage is reasonable and you're just tweaking it but if you think of it like you could have probably come up with 20 different ways of building out the home page you know it could be like just an image with the signup form it could be you know you'll the ones that scroll down kind of like the QC merge page okay so there's a lots of different ways you can attack this problem so by iterating these like incrementally on one version you're assuming that that that can reach the maximum potential but sometimes it's not you actually have to just completely switch designs like hey that poor homepage design can only get us to 18 percent conversion there's nothing you can do more than that but if you drastic they changed the product you can maybe move to a higher conversion rate and so that's kind of what happened here we didn't incrementally make changes on the old version we just said hey we're gonna scrap that come up with a brand new idea test it and I think it's important to do that when you're getting in this mode of incremental e testing and you're like oh I increase it by 0.1% maybe you should start thinking hey is there something bigger we can do to get you know so we don't have this local maximum and really break out of the gates rather than making these tiny tiny changes so with with some of these examples I'm going to go back into understanding like why does this matter like why are making decisions based on data important for your business so I'm not sure if any of you guys know Steve Blank but he's a yeah he's an entrepreneur and a professor at Berkeley and I mean he says the startup is an organization formed to search for a repeatable and scalable business model and I think some of the important things here is repeatable and scalable so we had this in the beginning about right where you would have like these traffic bursts and signups like you get on Digg and also to get all these customers so that's great but you can't build a business well at least the way we were doing it you can't build a business on it so this wasn't for us ever repeatable there are some businesses that have made that repeatable but for us that's not a repeatable way for us to gain customers so while it's interesting to like short-term to say okay great we got a couple thousand people we cannot build a business on top of that so we had to continue even after that searching for other ways to repeatedly get new customers and then scalable the important scalable is that hey if you actually threw more effort at that you could actually increase the return so if we write let's let's do an example of SEM is a really classic example so search engine marketing you're gonna pay for keywords you can actually a lot of research on keywords and you can see the search traffic so if you found like a keyword that for us let's say bookkeeping so bookkeeping delivers a thousand searches a day we need to acquire customers for less than two dollars so let's say I could actually run an ad campaign that would get 500 people a day from that search keyword for less than $2 the the problem with that is that the the search traffic is only a thousand so like it's not scalable no matter how much effort I throw at that like I can only get you know like 500 people a week or something there's some there's some level which it stops and you really when you're having a startup you want to make sure that you are continuing that when you put more effort at a channel you can continue to grow it over time so so when you have when you start your business you don't know any of this stuff that I'm talking about so like when I started out right I have no idea that our conversion rate would be 12% that like if someone was active they have a 40% chance of being or activate 40% chance of active and that means that they have X percent of pain you don't know any of those things you have these this grand vision of your product and you can make money but you need to somehow fill in all those gaps and figure out all those data points so I kind of think of it as when we started out right I'm not sure if you guys have played like real time strategy games and it's like a map and it's all blacked out and you don't see anything and then as you move it kind of like fills out and you can see the land and you can see your opponent's I feel like that's kind of what it's like when you first when you first start a startup is it's the worlds all blacked out you don't really know anything you're you're like oh I have this great product idea you have no idea if anyone's gonna buy it where you're gonna find these customers how much it costs to acquire them can you even just no idea and that's fine so but what that means is the key is to learn fast you want to of that black that blackened out map you want as much as possible start seeing the land underneath it so then you can actually say okay now I'm sorry understating the variables in my business and I can actually say yeah I actually believe this is possible why because I have the data to back it up not this I just have this belief so learning fast is extremely important for a startup because the faster you can learn the faster you can fail which is good right like so failing you're not how about failing but if you know this is not gonna work and you know that in one week versus like a year you just saved yourself you know a lot of time to go try something different so I think it's really important to learn fast both on failures and successes so and and that is why it's key to be able to make these data decisions and and because it helps you operate your business and know where to focus so with that being said it's kind of when you want to run these experiments where should you start I gave examples of of outright but we're fairly mature so we know certain variables but when you first start out where you should start really all comes from your business model so when you guys when you start a business it's important to understand the the key variables that make you win or lose and then and so this is kind of how we look at it at all right so if you are venture back company like there's this magic number of a hundred million run rate okay so so it's like how do you get to one hundred million run rate in five years that's 8.3 million per year in revenue that's like an important number so if you want to rate raise capital and figure this number out there's lots of different business models I won't go through all of them we happen to be like a freemium SAS model you can be just pure paid ads supported like Facebook where your actual customers are the are the marketers who are selling ads and then you have the users who are just using Facebook you're just create a platform with with traffic so you can sell them things you have enterprise sales where you're actually like pitching and selling to executives who don't even use the product so all of these different business models have different variables that matter for the success of your business so I mentioned Steve Blank earlier if you're really interested in this kind of stuff I would suggest reading the four steps to the Epiphany it's a great book described a lot of these things he's got a really good model on this so so okay so let me go over our how our model basically works when we started that's the basic way of looking at this is you look at registrations okay times how many what those people aren't gonna in a pain ARPU is average revenue per user multiply that your revenue then you say okay now take those registrations times the cost to acquire a user okay so you know so if you're gonna make five dollars on the customer and it cost you eight dollars to to acquire them it's not a very good business losing three dollars every customer so and then the greater the difference mean the revenue expenses hey the more profit you make per customer that's awesome and then you want to make sure you can get a lot of customers so that is like the core to our business and what's great is once you have that you can start looking at its risks so you know you have when you start it's like when I started out right was me and one other person so we don't we can't try like 100 different things at a time so when you start looking at that equation you can now start looking and say hey where are our biggest risks where are the things that we really have no clue about and so like for us it was well how do you acquire three million businesses at no cost actually no one's done that so so to me that's like okay how do you do that that's that's really really important so you'll see when we started we were like hammering that like crazy like tried sem we tried try viral things we tried affiliates and we try partners so partners for us ended up working that's a way we can actually scale we in a scalable way acquire customers and that's the strategy we currently have but that's why we chose to experiment on that number first cool so when you start it's important is to focus all of your energy on validating your business model the key assumptions there prove them out as successes or not and then if like if once you find out that number is not possible you need to pivot right change the way your business operates come up with a different idea something big and this this is where it comes to kind of the Lean Startup in that world so the Linc Stark was really talking he so Eric Ries took Steve blanks hi his book and he took kind of agile development and combine them together to create the the Lean Startup and he did he's done a really good job of taking this decision-making and figuring out your business model incrementally over time he did a really good job at this and so this is the way he looks at it where you have ideas you're gonna build those ideas and they turn into products when you build those products you're gonna measure it that's gonna turn into data which allows you to learn and then to make better better ideas and you just keep doing that until you prove out your business and what's great about this this loop is every time you go through that loop your intuition gets better so when when you first start you don't know a lot of these you don't know the variables or how customers are going to act with your product but every time you go through this loop and you test something you now have this new learning that you know so the next time you make a product feature you actually understand your customers better you understand your business better or you understand your partner's better and because of that you start making better decisions and then each time it's important to make sure you validate those decisions so you can get more learnings and get smarter at this so let's kind of jump into the idea phase so it's very important that when you come up with an idea that you can measure its success and you always want to connect it to your business so like every if you looked at every one of my previous examples you'll see that they're all tied directly to our business model so why do we care about increasing registration because if because for every customer that comes into our homepage we convert them to a customer we now have another customer who's activated who's more has a higher chance of pain very simple so we've connected it all the way back we've even done some interesting things like are people willing to pay for outright so we had that question because we were focusing so much on customer acquisition we weren't focusing on the percentage paid and we came up with an experiment they're very similar to the fake feature that we talked about earlier where when people came to that ride homepage we did we actually made it look paid and it would actually go through a signup form and ask for credit card numbers and odd stuff and what we did is we actually just threw it away wait it was a literally a blank form yeah we did some validate we did validation like we did a quick check to say is it somewhat valid because we were we were concerned that people put in fake numbers so we said fine we'll do quick validation on it but we literally just put it up said hey I'll write is paid people came to the product and then we just measured how many people would pay and are people willing to pay and they are that's what that's what we found out is no traffic and just go down I think our conversion went from 12% to about 4% however we now know that people can pay and we can use that data later on to improve our product and to improve premium tiers and things like that because we purposely set up the experiment to be the most harsh which is there's no free trial you have to give a credit card upfront so you know we were really really strict to see hey what is the propensity to pay here and I think I talked yeah I talked a lot about the fastest and cheapest way and and hopefully from to my examples you are starting to get what I mean by the fastest and cheapest way I think when people say fastest and cheapest they're thinking like oh maybe we won't unit test no we're talking like don't build so the next the next important piece is measuring so we spend a bit of time on when we're when we're measuring actually let me make I'm skipping the build in the product because that's what you guys do so there's lots of other talks on that piece so I'm just skipping those two assuming you can build build your product and I'm getting to the measuring phase so the measuring phase is you really want to capture as much user information as possible where on these experiments and the reason why is is when the experiment succeeds or fails you want that learning and sometimes you can't find the learning by just looking at the raw data so it's important to be able to to connect those data points with actual human beings and to be able to go back and and be able to do qualitative research on that I've listed a couple tools here Google Analytics KISSmetrics Optimizely optimizer and then custom beacons and and you'll see that they goes from like the really simple to more complex so as your as your business matures you're gonna kind of go down this chain and we are currently yeah we do custom beacon logs and a bunch of things there but we still actually use a lot of the tools above like KISSmetrics to run a be test and funnel because we want to be you know cheap and quick and I don't want to build all these funnel funnel tools however KISSmetrics we actually do get the identity of users and we sync it back up with our data warehouse and what that actually allows us to if a person if a cohort does when they go through a funnel if they perform different and we can actually go and call those customers and ask them why and to follow up with them and I think that's the important because yeah when you get to the data piece you really are going to want to look at the data and then you're going to find different trends and the fact the trends are different is interesting but what you want to do is then connect why is that data why did that trend change and that's where real learnings come from and it's important and that's why it's important to have those that user level information also so with data you're gonna do a lot of stuff in spreadsheets in pivot tables I don't think I've ever used a pivot table in my entire life until I started doing a lot of this so you're gonna go into your databases at the beginning so what I'll write let me give an example at the beginning we have this we were a rails app we have this application database I would just go in and like dump user tables and do crazy queries with like dates and I mean it's like ridiculous the amount of queries that I ran running through various scripts and then I put that into Excel and I would build a chart and that's pretty much how I did all of the analysis and but it works really well in at the beginning as you get more sophisticated you'll build things like a data warehouse and a Dayo warehouse takes all of those data sources and puts them together and allows you to do that analysis all in once all in one spot our data warehouse for example we we connect to KISSmetrics we use then desk so all customers like well we'll create tickets and we associate their user ID when they create tickets on kiss on Zendesk and our data warehouse will actually pull in all tickets created and put it into a central database and then our application will grab data and put it into a central area so we can actually mirror that up so we can start asking questions like when someone registers if they submit a ticket in their first week how does that correlate to active use over time you could actually answer those types of questions which in general is pretty tough to do one off but over time as you get more sophisticated you can keep doing that so I would suggest at the beginning you just use tools like KISSmetrics they'll give you high-level data you're gonna probably run a bunch of ad hoc queries and just throw whatever you can together and then over time as that comes more comfortable you'll start building more and more tools that enable this analysis and to do even more rich analysis over time which enables you to do even better decision-making and then lastly for the learn it's not only important to take the data in to do the analysis and to to understand the people behind the data once you have those learnings you need to build in this this cycle too for your organization to learn and make decisions so at this point we have alright has 25 employees we're trying to grow to 50 so I can't just keep all of these learnings in my head you want to encourage other folks and other departments to be able to create these learnings and to share them with the team and everyone gets smarter and everyone be able provide their own experiments so there's a couple things I think are really important one is just create like a Google spreadsheet that has like all your experiments just the list just lists all your various experience you've ever done what the hypothesis was successful or not and why very simple every time you want to experiment and something happens make sure you share it with your team so just send out an email with a summary saying hey this is what we learned because that is the next thing we're doing those two things are will greatly help your organization get better and more analytical thinking and then lastly what's important is iteration speed so what you as a start-up what you really want to optimize for is how fast can you get through that feedback loop because the faster you can get through the feedback loop is the faster you're learning which is the faster you're uncovering that map and the faster that you can find out is this business for real and can we scale it scale it and can we grow and it builds your intuition so you were making wiser decisions when people are arguing you actually have data to back things up like oh do it this way because we ran an experiment that way and it failed that ends an argument very quickly and you can show them the data in the spreadsheet see here there it is is yours different than that if so let's try it it's also nice to empower your team so we're building tools that out right for developers to like a beta section and what they can do is they can take any idea they have and they can put it under beta and then people can opt into that and when they opt into it we will collect data on those people versus the others so engineers can then say I have this idea they don't have to go through this management chain of approval they can put something out there they can collect the data once they've collected that data they can come back to the product team and say here here's a cool feature why because I thought it would do this and this is what it does and look at the core numbers here we did this with one of our engineers he built like a small Google Chrome plug-in to allow people to categorize and that was under our beta and even the ledger or sorry the account Center that's the public facing name when we built the account center we were giving it to new users but in the beta section we allowed people to opt in if they wanted to so even like existing customers could have opted in if they wanted to even though we were kind of forcing half the new people into them so with some of this knowledge hopefully it gives you enough context and a pointer to some tools to go and do some of this yourself so this is actually really easy to get started and I think once you get used to making decisions this way it's really empowering because you no longer have this kind of voodoo magic anymore so I really encourage everyone next week to just try an experiment just anything use Google Analytics use use KISSmetrics whatever you can yes Doug who are also intimidated by this and get to give some newly practical specific way yeah so whatever you're working on now just whatever you're working on now think of the one thing think of one small test you can do that's going to to increase a number so there's two areas there's there's one where you don't even know what numbers matter but we can all agree like registration rates in general are important so I the most concrete thing is what's the most important thing bothering you right now in your business or worrying you think about that number and say how can I prove that inexpensively and cheaply and quickly I can guarantee everyone here if you just got together with like one other person or by yourself and just wrote that down you come with a bunch of ideas and then how can you do that so you'll have all these ideas and then how can you test it cheaply that's like the two things right like this is I'm trying to change I'm trying to change this how could I do it brainstorm then each one of those brainstorming say how could I test that brainstorm so you might say like Oh build this whole community site and and if I did that then our viral coefficient will go crazy okay so think so that's great that's a great idea but how can you test that and we've actually had this we're actually talking about adding social features to the product and and what we did is we go okay well why do we want to add viral because that's going to help bring in registrations so you can kind of go through that workflow of I'll just go through my thought process of how we did this so it'll take you through a workflow of going so how if someone shared how many people see it what's the percent of people that will click through to our homepage and then how many people register and that's a register and that's a registration so if you did that math very quickly you could you can actually come up where this would only make this feature it only makes sense for us if it delivers X number of customers per customer coming in so if one person came in and recommended we need at least 0.5 customers coming back and if you did that you can actually follow through that chain and you can say oh so they would have to have at least a thousand Facebook followers to make it to make that happen and then you can just just prove that right away just by thought you know like because not many people have a thousand Facebook followers for example or you would say I'm not sure and we've done this before for example we were we were saying we want someone to share and to come back and register into facebook we were like well but we're small business so how many people like if I'm friends with people on Facebook how many of those are also small businesses right so games like the it's to everyone essentially so if you share that to you know to your hundred friends there's a much higher chance than saying oh it's this very specific thing for a small business because how many people are our small businesses and so that variable like made it made is concerned and we devised an experiment to prove that variable and the way we did it is if you think about it it doesn't matter if the share came from your product so what we did is we actually just created custom messages and emailed our customers and said hey will you help us post this on Facebook with this link and we've measured that link and we just literally had our real customers manually post it there and then we measured conversion and then we said okay is that conversion high enough that would make this equation work so like just something like that just like a thought experiment and it's really trying to change this number with this feature how does feature connect to that number build that out what are your concerns build an experiment on that thing kind of long-winded is that or are you more intimidated now after that sorry yeah early example where you show the Facebook the next day or actually show the the one thing that struck me there was at what point would you just go in instead of running on these numbers at what point would you just go in this Assam users like or would you people would you ever do that s the users why wouldn't you register so in that question is it after the experiment failed or before we ran the experiment yeah so after it failed that is what we would do is go talk to people but some things are kind some things you even they don't know why if you know so I think it why would you rush for the red button on a green I mean yeah so I spent like it into it for example we spent a lot of time with like focus groups and before building something doing UI mock-ups and showing them to people I think there's a lot of value there but over time I've put more on to quantitative and just experimenting and I the reason why is because as it becomes cheaper to experiment and to as it becomes cheaper to experiment and to build those experiments it's it real data is better than someone saying like we could go to customer and say would you pay for outright if it were $100 and they're gonna go yes or no but that you would see that that and then if I actually put a pager to say it's $100 you would vary greatly between the two yeah so you have to be a little bit careful of that so we do that with like tax messages for example so one we you know you split the traffic so at least you're not you know so definitely don't go the route of just putting a hundred percent one and going oh this metric is different than last week's that's don't even that's just a really bad idea you got to split it because yet things change on a daily basis and so if you split it you know 50/50 or whatever thirty seventy and your wafer statistical significance that like that's one way of getting around that but then you have to you do have to be aware of other things that could influence it especially untimely manners and that's what I was saying like our our headline on our page we test that and we've tested it each different quarters and during tax time of course tax messages work way better and during the beginning before tax time getting organized works better so you have to yeah in the lettuce if you kind of have to think through and you have pretty good good about it like oh this could you know this could change you know seasonality and then make sure you realize that was environmental thing that's changed like maybe a few sports yeah I mean that does happen to over time because when we first yeah if you were doing if we were doing Facebook Connect two years ago it compared to now it would be drastically different so yes I would but this experiment happened to rerun recently but yes oh I'm sorry yes so log beacons are it's like you see them all the time like Google Analytics it's like a Java it's a Java Script piece that says oh this event occurred what that is doing behind the scenes is just going to a to some site and it's hitting some URL and that thing gets logged when that thing it's it just creates a huge log of these events and then what our data warehouse is gonna do is going to grab those logs and then to go through them and turn them into some schema that you can then query like that's that's so that's one data source for us and we build that in Ruby because like our system we hit multiple systems and servers so you know user could hit this web server and then this web server then this web server they could also every request can span multiple like we have a server scenario architecture so it can hit multiple services and so we'll log in all those places and then at one point we grab all those logs process them and put them all back together essentially and that's what you're doing with Zendesk as well that's another big log of data and then your application database and you're pulling all that stuff together into a data warehouse find that you we read into the data warehouse kind of how to iterate through because maybe your hypotheses granularity change yeah so I mean yeah we build everything iteratively so yeah when the data warehouse was like one table or you know like one fact table like registrations with a couple dimensions and then over time we've built more and more and we added like we built a mobile product while sudden that's a new thing that you have to start adding that concept into the data warehouse and then yeah as you run certain experiments like maybe testing like we didn't have we didn't have like strict a be testing hooked into the data warehouse and so there's that as well like when once you did that you have to now have that flow through the day warehouse and be able to split data on that on that attribute as well so I think it just iteratively you have to you have to do that as you find new needs we just don't have that yeah big companies they would have a whole team of like eight ETL and data warehouse guys and analysts and they just do this all day long and for us this is you know just an engineer spending a part of a time on part time on this yeah Doug oh yeah I didn't want tomatoes thrown at me because yeah I know it's Cincinnati never once want to be a recruiter for us you can come up here and well yeah so I mean yeah we recruiting is a full-time job I had no idea when I started coming I was a to spend so much time just recruiting well so there's there's a bunch of Engineers but there's also a bunch of companies and so it's highly highly competitive high rates the best people like anything are taken so you have to have you have to be pretty creative to get the great best people because you don't want just someone who can program you want someone who can program and who's really smart and can be on their own and it really has the passion to help your business grow and to find those types of people is extremely challenging even though there's a lot of engineers there because they're tied up they have their own ideas or they're at like LinkedIn or Dropbox or all those guys are right in the exact same area so we're all competing against the same talent pool so I would say yeah well there are more engineers there's a lot more competition you know there's a lot of competition trying to grab the same pool of Engineers it's extremely competitive but I mean for an example I spend probably about 40% of my time recruiting and I have yeah I have office manager who dead eight who's dedicated to like candidate flow I've got to hiring managers that help and then our team when we do coding interviews and things like that so I mean we are and that's like something that we've gotten a lot better at that just like just like the Challenger champion versus Challenger you just get that process going that's the same thing like I'm here they're interviewing like that's happening right now I think you had a question as well yeah so the Lean Startup stuff I'm kind of getting into but it seems like all examples are my question is how much of this applies like your life like the other fitness is like let's say I want to own open the driving range downtown what kind of experiments I see interesting I guess I haven't thought about it much but I am from the okay so with that being said I am from the desktop world so before I was at Intuit and we had a yearly release cycle on QuickBooks so it makes it tough so definitely the the slower that feedback loop is the less this works and it's the same thing with kind of when we're talking about the UX and doing qualitative things upfront like oh here's the design a or design B which one do you like better I think the real why would you do it up front versus quantitatively after what content view always gives you better data but some it usually costs a lot to build two versions but if you can reduce the cost so much then it then why even ask upfront right just go oh here here's two versions and put it out there and then they'll people will tell you with real data so I think it's this kind of the same thing there just depends how fast you get that feedback loop and and but I don't know how you would do some of the fake stuff right like you come to your your range and you have a like a sign that says here you can rent this gun and then it doesn't exist I don't know you can either I don't know how but it's definitely that the fees that feedback loop yeah and that's why the web is so great right because we for example do continuous deployment so I can code something now and have a production of 5 minutes so if I want to know something I can just try it and that's really fast but if we had like a traditional softener moment process where it takes two weeks and then goes to QA and then and then it goes off four weeks later I would no longer are gonna go oh I'm not sure I'm just gonna try to in two seconds like you wouldn't do that anymore you're gonna go well now I'm gonna really think about it I'm gonna ask people and it changes your behavior but as soon as though and that's why I think the web is so powerful because you can get that really quick it's easy to split test things you know it just reduced the cost of all those types of activities so I think yeah once you start removing that yeah it may not be as worth it anymore no no oh there yeah that's like a good idea great idea actually I like that yeah thanks you guys [Applause]