← Back to Bookshelf ×
Run a Side-Hustle

Testing Business Ideas

By Alex Osterwalder

My Personal Takeaways →
Motivation for Reading & Implementing the Book

Summary

Great ideas often fail because they are executed before they are tested. Testing Business Ideas gives a disciplined method to reduce uncertainty before spending major time, money, and credibility. The core approach is evidence over opinion: turn assumptions into hypotheses, run targeted experiments, gather real signals, and adapt quickly.

Read this if you want to stop debating and start validating. Implement it by testing your riskiest assumptions first across desirability, feasibility, and viability. Use low-cost experiments early, define success criteria in advance, and run multiple tests before scaling. Build a team rhythm around testing, synthesis, and decision points so learning becomes systematic instead of accidental. This book helps entrepreneurs and innovation teams kill weak ideas early, strengthen promising ones, and move forward with confidence grounded in evidence rather than enthusiasm alone.

Direct Quotes & Excerpts From The Book

Testing Business Ideas

by David J. Bland and Alexander Osterwalder

  • This book will help you….

    • Learn how the testing process works
    • Design and run your first experiments
    • Reduce the risk of your business idea
    • Fine-tune your testing process
    • Discover an extensive experiment library that goes beyond interviews, surveys, and minimum viable products
    • Bullet-proof your business ideas with stronger evidence than you’ve ever gathered before
    • Learn about experimentation ceremonies.
    • Be able to share an extensive testing library with all your teams
    • Reduce risk and uncertainty of new ideas across your organization

  • This book was made for…

    • Corporate Innovator who is challenging the status quo and who is building new business ventures within the constraints of a large organization.
    • Startup Entrepreneur who wants to test the building blocks of your business model to avoid wasting the time, energy, and money of the team, cofounders, and investors.
    • Solopreneur who has a side hustle or an idea that isn’t quite yet a business.

  • Too many entrepreneurs and innovators execute ideas prematurely because they look great in presentations, make excellent sense in the spreadsheet, and look irresistible in the business plan… only to learn later that their vision turned out to be a hallucination.

  • Don’t make the mistake of executing business ideas without evidence: test your ideas thoroughly, regardless of how great they may seem in theory.

  • This book outlines the most extensive testing library on the market to help you make your ideas bulletproof with evidence. Test extensively to avoid wasting time, energy, and resources on ideas that won’t work.

  • To test a big business idea you break it down into smaller chunks of testable hypotheses. These hypotheses cover three types of risk. First, that customers aren’t interested in your idea (desirability). Second, that you can’t build and deliver your idea (feasibility). Third, that you can’t earn enough money from your idea (viability). You test your most important hypotheses with appropriate experiments. Each experiment generates evidence and insights that allow you to learn and decide. Based on the evidence and your insights you either adapt your idea, if you learn you were on the wrong path, or continue testing other aspects of your idea, if the evidence supports your direction.

SECTION 1 - DESIGN

DESIGN 1.1 - DESIGN THE TEAM

  • If you do not have all of the skills needed or are unable to partner with external team members, then evaluate technological tools to fill the void.

  • A cross-functional team has all the core abilities needed to ship the product and learn from customers. A common basic example of a cross-functional team consists of design, product, and engineering.

  • Commonly Required Skills to Test Business Ideas

    • Design, Product, Tech, Legal, Data, Sales, Marketing, Research, Finance

  • There are new tools coming on to the market every day that allow you to:

    • Create landing pages
    • Design logos
    • Run online ads

  • A lack of diverse experiences and perspectives on a team will result in baking your biases right into the business. When forming your team, keep diversity top of mind, rather than as an afterthought.

  • Successful Teams Exhibit Six Behaviors

      1. Data Influenced
      • You do not have to be data driven, but you need to be data influenced. Teams no longer have the luxury of burning down a product backlog of features. The insights generated from data shape the backlog and strategy.
      1. Experiment
      • Driven Teams are willing to be wrong and experiment. They are not only focused on the delivery of features, but also craft experiments to learn about their riskiest assumptions. Match experiments to what you are trying to learn over time.
      1. Customer Centric
      • To create new businesses today, teams have to know “the why” behind the work. This begins with being constantly connected to the customer. This should not be limited to the new customer experience, and expands to both inside and outside of the product.
      1. Entrepreneurial
      • Move fast and validate things. Teams have a sense of urgency and create momentum toward a viable outcome. This includes creative problem-solving at speed.
      1. Iterative Approach
      • Teams aim for a desired result by means of a repeated cycle of operations. The iterative approach assumes you may not know the solution, so you iterate through different tactics to achieve the outcome.
      1. Question Assumptions
      • Teams have to be willing to challenge the status quo and business as usual. They aren’t afraid to test out a disruptive business model that will lead to big results, as compared to always playing it safe.

  • The Team Needs to be…

    • Dedicated

      • Teams need an environment in which they can be dedicated to the work. Multitasking across several projects will silently kill any progress. Small teams who are dedicated to the work make more progress than large teams who are not dedicated.

    • Funded

      • It’s unrealistic to expect these teams to function without a budget or funding. Experiments cost money. Incrementally fund the teams using a venture-capital style approach, based on the learnings they share during stakeholder reviews.

    • Autonomous

      • Teams need to be given space to own the work. Do not micromanage them to the extent where it slows down their progress. Instead, give them space to give an accounting of how they are making progress toward the goal.

  • The Company Needs to Provide…

    • Support

      • Leadership
        • Teams need an environment that has the right type of leadership support. A facilitative leadership style is ideal here because you do not know the solution. Lead with questions, not answers, and be mindful that the bottleneck is always at the top of the bottle.
      • Coaching
        • Teams need coaching, especially if this is their first journey together. Coaches, either internal or external, can help guide the teams when they are stuck trying to find the next experiment to run. Teams that have only used interviews and surveys can benefit from coaches who’ve seen a wide range of experiments.

    • Access

      • Customers
        • Teams need access to customers. The trend over the years has been to isolate teams from the customer, but in order to solve customer problems, this can no longer be the case. If teams keep getting pushback on customer access, they’ll eventually just guess and build it anyway.
      • Resources
        • Teams need access to resources in order to be successful. Constraints are good, but starving a team will not yield results. They need enough resources to make progress and generate evidence. Resources can be physical or digital in nature, depending on the new business idea.

    • Direction

      • Strategy
        • Teams need a direction and strategy, or it’ll be very difficult to make informed pivot, persevere, or kill decisions on the new business idea. Without a clear coherent strategy, you’ll mistake being busy with making progress.
      • Guidance
        • Teams need constraints to focus their experimentation. Whether it’s an adjacent market or creating a new one, to unlock new revenue teams need direction on where they will play.
      • KPIs
        • Teams need key performance indicators (KPIs) to help everyone understand whether they are making progress toward a goal. Without signposts along the way, it may be challenging to know if you should invest in the new business.

  • Team Alignment

      1. Define the mission.
      1. Define the time box for the agreement.
      1. Create joint team objectives.
      • Joint Objectives: What do we intend to achieve together?
      1. Identify commitment levels for team members.
      • Joint Commitments: Who does what?
      1. Document joint resources needed to succeed.
      • Joint Resources: What resources do we need?
      1. Write down the biggest risks that could arise.
      • Joint Risks: What can prevent us from succeeding?
      1. Describe how to address the biggest risks by creating new objectives and commitments.
      1. Describe how to address resource constraints.
      1. Set joint dates and validate.

DEISGN 1.2 - SHAPE THE IDEA

  • The design loop has three steps.

      1. Ideate
      • In this first step you try to come up with as many alternative ways as possible to use your initial intuition or insights from testing to turn your idea into a strong business. Don’t fall in love with your first ideas.
      1. Business Prototype
      • In this second step you narrow down the alternatives from ideation with business prototypes. When you start out you might use rough prototypes like napkin sketches. Subsequently, use the Value Proposition Canvas and Business Model Canvas to make your ideas clear and tangible. In this book we use these two tools to break ideas into smaller testable chunks. You will constantly improve your business prototypes with insights from testing in future iterations.
      1. Assess
      • In this last step of the design loop you assess the design of your business prototypes. You ask questions like “Is this the best way to address our customers’ jobs, pains, and gains?,” or, “Is this the best way to monetize our idea?,” or, “Does this best take into account what we have learned from testing?” Once you are satisfied with the design of your business prototypes you start testing in the field or go back to testing, if you are working on subsequent iterations.

  • The Business Model Canvas

    • You don’t have to be a master of the Business Model Canvas to use this book, but you can use it to shape ideas into a business model so you can define, test, and manage risk. In this book, we use the Business Model Canvas to define the desirability, feasibility, and viability of an idea. If you’d like to go deeper than the synopsis of the Business Model Canvas, we recommend reading Business Model Generation or go online to learn more.

      • Customer Segments: Describe the different groups of people or organizations you aim to reach and serve.
      • Value Propositions: Describe the bundle of products and services that create value for a specific customer segment.
      • Channels: Describe how a company communicates with and reaches its customer segments to deliver a value proposition.
      • Customer Relationships: Describe the types of relationships a company establishes with specific customer segments.
      • Revenue Streams: Describe the cash a company generates from each customer segment.
      • Key Resources: Describe the most important assets required to make a business model work.
      • Key Activities: Describe the most important things a company must do to make its business model work.
      • Key Partners: Describe the network of suppliers and partners that make the business model work.
      • Cost Structure: Describe all costs incurred to operate a business model.

  • strategyzer.com/books/business-model-generation.

    • Value Map: Describes the features of a specific value proposition in your business model in a structured and detailed way.

      • Products and Services: List the products and services your value proposition is built around.
      • Gain Creators: Describe how your products and services create customer gains.
      • Pain Relievers: Describe how your products and services alleviate customer pains.

    • Customer Profile: Describes a specific customer segment in your business in a structured and detailed way.

      • Customer Jobs: Describe what customers are trying to get done in their work and in their lives.
      • Gains: Describe the outcomes customers want to achieve or the concrete benefits they are seeking.
      • Pains: Describe the bad outcomes, risk, and obstacles related to customer jobs.

  • strategyzer.com/books/value-proposition-design

SECTION 2 - TEST

TEST 2.1 - HYPOTHESIZE

  • Identify the Hypotheses Underlying Your Idea: To test a business idea you first have to make explicit all the risks that your idea won’t work. You need to turn the assumptions underlying your idea into clear hypotheses that you can test.

  • Prioritize Most Important Hypotheses: To identify the most important hypotheses to test first, you need to ask two questions. First, “What is the most important hypothesis that needs to be true for my idea to work? Second, “For which hypotheses do I lack concrete evidence from the field?

  • When creating hypotheses you believe to be true for your business idea, begin by writing the phrase “We believe that..” “We believe that millennial parents will subscribe to monthly educational science projects for their kids.” Be mindful that if you create all of your hypotheses in the “We believe that..” format, you can fall into a confirmation bias trap. You’ll be constantly trying to prove what you believe, instead of trying to refute it. In order to prevent this from occurring create a few hypotheses that try to disprove your assumptions. “We believe that millennial parents won’t subscribe to monthly educational science projects for their kids” You can even test these competing hypotheses at the same time. This is especially helpful when team members cannot agree on which hypothesis to test.

  • Characteristics of a good hypothesis:

    • Testable: Your hypothesis is testable when it can be shown true (validated) or false (invalidated), based on evidence (and guided by experience).
    • Precise: Your hypothesis is precise when you know what success looks like. Ideally, it describes the precise what, who, and when of your assumptions.
    • Discrete: Your hypothesis is discrete when it describes only one distinct, testable, and precise thing you want to investigate.

MARKET RISK (Desirability Hypotheses)

  • Customer Profile: We believe that we…

    • are addressing jobs that really matter to customers.
    • are focused on pains that really matter to customers.
    • are focused on gains that really matter to customers.

  • Value Map: We believe…

    • our products and services really solve for high-value customer jobs.
    • our products and services relieve top customer pains.
    • our products and services create important customer gains.

  • Customer Segments: We believe…

    • we are targeting the right customer segments.
    • the segments we are targeting actually exist.
    • the segments we are targeting are big enough.

  • Value Propositions: We believe…

    • we have the right value propositions for the customer segments we are targeting.
    • our value proposition is unique enough to replicate

  • Channels: We believe…

    • we have the right channels to reach and acquire our customers.
    • we can master the channels to deliver value.

  • Customer Relationships: We believe…

    • we can build the right relationships with customers.
    • it is difficult for customers to switch to a competitors product.
    • we can retain customers.

INFRASTRUCTURE RISK (Feasibility Hypotheses)

- Key Activities: We believe that we can perform all activities (at scale) and at the right quality level that is required to build our business model. 

- Key Resources: We believe that we can secure and manage all technologies and resources (at scale) that are required to build our business model, including intellectual property and human, financial, and other resources. 

- Key Partners: We believe that we can create the partnerships required to build our business.

FINANCIAL RISK (Viability Hypotheses)

- Revenue Streams: We believe that we can get customers to pay a specific price for our value propositions. can generate sufficient revenues. 

- Cost Structure: We believe that we can manage costs from our infrastructure and keep them under control.

- Profit: We believe that we can generate more revenues than costs in order to make a profit.
  • ASSUMPTIONS MAPPING
- Step 1 (Identify Hypotheses) 


    - Use a sticky note to write down each: 
        - desirability hypothesis and put it on your canvases. 
        - feasibility hypothesis and put it on your canvases. 
        - viability hypothesis and put it on your canvases. 

    - Best Practices 
        - Use different color sticky notes for desirability, feasibility, and viability hypotheses. 
        - Your hypotheses should be as specific as possible, to the best of your knowledge, based on what you know today. 
        - Every hypothesis should be a single sticky note. Don't use bullet points; that makes it easier to prioritize your hypotheses. 
        - Keep your hypotheses short and precise. No blah blah blah. 
        - Discuss and agree as a team when writing.



- Step 2 (Prioritize Hypotheses) 


    - Use the Assumptions Map to prioritize all your hypotheses in terms of importance and existence or absence of evidence that supports different types of hypotheses. 
        
    - x-Axis: Evidence: On the x-axis you place all your hypotheses positioned to show how much evidence you have or don't have to support or refute a specific hypothesis. You place a hypothesis on the left if you are able to produce relevant, observable, and recent evidence to support a hypothesis. You place a hypothesis on the right if you do not have evidence and therefore will need to generate it. 
    
    - y-Axis: Importance: On the y-axis you place all your hypotheses in terms of importance. Position a hypothesis at the top if it is absolutely critical for your business idea to succeed. In other words, if that hypothesis is proven wrong, your business idea will fail and all other hypotheses become irrelevant. You place a hypothesis at the bottom if it is not one of the first things you'd go out and test.


- Step 3 (Identify and Prioritize Riskiest Hypotheses) 


    - For the purposes of this book, the major focus will be on how to test the top right quadrant of your Assumptions Map: experiments with important hypotheses and with light evidence. These assumptions, if proven false, will cause your business to fail.

TEST 2.2 - EXPERIMENT

  • Design Experiment: To get started with testing your business idea, you turn your most important hypotheses into experiments. You should start with cheap and fast experiments to learn quickly. Every experiment will reduce the risk that you’ll spend time, energy, and money on ideas that won’t work.

  • Run Experiment: Every experiment has a specific run time to generate sufficient evidence that you can learn from. Make sure you run your experiments almost like a scientist, so that your evidence is clean and not misleading.

  • What is a good experiment?

    • A good experiment is precise enough so that team members can replicate it and generate usable and comparable data.
    • Defines the “who” precisely (test subject)
    • Defines the “where” precisely (test context)
    • Defines the “what” precisely (test elements)

  • What are the components of an experiment? A well-formed business experiment is made up of four components:

    • Hypothesis: The most critical hypothesis from the top right quadrant of your Assumptions Map.
    • Experiment: The description of the experiment you will run to support or refute the hypothesis.
    • Metrics: The data you will measure as part of the experiment.
    • Criteria: The success criteria for your experiment metrics.

  • Call-to-Action Experiment: A specific type of experiment that prompts a test subject to perform an observable action. Used in an experiment in order to test one or more hypotheses.

  • “Anyone who isn’t embarrassed by who they were last year probably isn’t learning enough.” - Alan de Botton (Philosopher)

TEST 2.3 - LEARN

  • Strength of Evidence: The strength of a piece of evidence determines how reliably the evidence helps support or refute a hypothesis. You can evaluate the strength of evidence by checking four areas. Is the evidence based on….

  • Weak Evidence

    • Opinions (beliefs): When people say things like “I would…,” “I think ____ is important,” “I believe…,” or “I like…”
    • What people say: What people say in an interview or survey is not necessarily what they do in real life or will do in the future.
    • Lab settings: When people are aware that you are testing something, they may behave differently than in a real world setting.
    • Small investments: Signing up by email to be informed about an upcoming product release is a small investment and relatively weak evidence of interest.

  • Stronger Evidence

    • Facts (events): When people say things like “Last week I ____,” “In that situation I usually ____,” or “I spent ____ on.”
    • What people do: Observable behavior is generally a good predictor of how people act and what people might do in the future.
    • Real world settings: The most reliable predictor of future behavior is what you observe people doing when they are not aware they are being tested.
    • Large investments: Pre-purchasing a product or putting one’s professional reputation on the line is an important investment and strong evidence of real interest.

  • For example, you might start with interviews to gain some first insights into your customers’ jobs, pains, and gains. Then you might run a survey to test your insights on a larger scale with more customers. Finally, you might continue with a simulated sale to generate the strongest type of evidence for customer interest.

  • Your confidence level should rise with the number of experiments you conduct to test the same hypothesis. Three interview series are better than one.

TEST 2.4 - DECIDE

  • Solopreneur: Solopreneurs benefit from Weekly Plan if you are not coordinating with external contractors. The ritual of planning your work every week will help you keep a cadence and establish a sense of accomplishment.

  • Daily Standups

    • Stay aligned and focus on your daily work. Many experiments require a series of tasks to complete, and Daily Standups help coordinate your day-to-day work.

    • Agenda

        1. What’s the Daily Goal? Create a daily goal. If your goal is to get an experiment out the door, then it’s important to align your tasks to achieve that goal. Remember that daily goals feed into your larger, more ambitious goals for the overall business.
        1. How to Achieve That Goal? Identify the tasks needed to achieve the daily goal and plan your day.
        1. What’s in the Way? Identify any blockers that would prevent you from completing experiment tasks for the day or achieving the goal. Some of these can be addressed within the standup if it is quick, otherwise meet after the standup to work through it.

  • Weekly Learning

    • Have a conversation to interpret the evidence and turn it into action. Remember that what you’ve learned from experiments should inform your overall strategy.

    • Agenda

        1. Gather Evidence: Gather up the evidence your experiments have generated. This includes both qualitative and quantitative types of evidence.
        1. Generate Insights: Look for patterns and insights from your evidence. Even qualitative evidence can be quickly themed using techniques such as affinity sorting. Try to keep an open mind. You may find unexpected insights that lead you to new paths to revenue.
        1. Revisit Your Strategy: Take the new insights you have and revisit your Business Model Canvas, Value Proposition Canvas, and Assumptions Map. Make any updates needed so that they reflect your current state of learning. This is a crucial step in using what you’ve learned to inform your strategy. If it feels awkward, don’t worry, it’s a normal part of being an entrepreneur.

  • Biweekly Retrospective

    • Take a step back, breathe, and talk about how you can improve the way you work. In our opinion, this is the most important ceremony. When you stop reflecting, you stop learning and improving

    • Agenda

        1. What’s Going Well: Take five minutes to silently write down what’s going well. This gets the retrospective off to a good start as people have space to speak positively about team members and how they are working together
        1. What Needs Improvement: Take five minutes to silently write down what needs improvement. These are things that aren’t going well or could be doing better. It’s important to frame these items as an opportunity to improve, rather than as a personal attack against a team member.
        1. What to Try Next: Come up with three things you’d like to try. It can be one of the items you’ve previously discussed, or some- thing completely new. This gives you a chance to try out a new way of working that isn’t simply rooted in what needs improvement.

  • Principles of Experiment Flow

    • PRINCIPLE #1 - Visualize Your Experiments: Make your work visible to yourself and others. We’ve found inspiration from the lean and kanban movements, particularly on this principle. If you keep all this work in your head, you’ll never be able to achieve flow. Not only are your teammates unable to read your mind, but much of flow requires you to visualize your work.

        1. Write down your experiments. We recommend only one experiment per sticky, to keep things organized. You don’t have to write down hundreds of experiments-only the ones you feel you’ll be running over the upcoming weeks.
        1. Draw a simple experiment board. This is one of the simplest forms of an experiment board you can create. We’ve been playing with this format for quite some time and used to like the Validate column, which we got originally from Eric Ries. Over time, we’ve started to back off a bit on that language because teams will set the bar so low on their hypotheses that they’ll artificially validate them and move on too quickly. We prefer “Learn” over “Validate.”
        1. Add your experiments to the Backlog column. Rank your experiments from top to bottom, where the top is the one you are going to do next. Pull them across as you begin to work on each, moving from Setup, to Run, to Learn.

- PRINCIPLE #2 - Limit Experiments in Progress: Multitasking too many experiments can often lead to trouble.


    - Define work in progress limits for your experiments: For example, start with a limit of 1 for Setup, Run, and Learn columns. This will prevent the team from pulling a second experiment over until the first is moved to the next column and finally archived. In this example, the team runs the customer interviews before the survey, instead of trying to do both at once (and slowing everything down). The experiments flow, using what you've learned to inform your next experiment.



- PRINCIPLE #3 - Continuous Experimentation: Continue to experiment over time.


    - Blocker Experiment: As an example, is that the team is trying to line up customer interviews, but the research department won't let them. They state that it's against company policy to talk to customers. That is a "blocker" that's preventing you from making progress on that experiment. It's a good idea to identify and visualize these, which will help you communicate progress to stakeholders as to why things are slowing down. It's hard to achieve flow when you are blocked.
    - Splitting Columns Experiment: Another example is that the team has outgrown the initial board and is frustrated that the Setup column doesn't capture the nuances of experimentation. There is work setting up an experiment, but then you have to run it, and if the team is at capacity the experiment may sit around for a long time waiting to be run. When we talk about the board, it would be great to see which ones are ready to be run and which experiments are still being set up.
  • Experiment Guidelines Sample
      1. Our customer segment is [blank]
      1. The total number of customers involved in our experiment is [blank]
    • estimated to be [blank]
      1. Our experiment will run from [blank]
      1. The information currency we are collecting is [blank]
      1. The branding we’ll use for the experiment is [blank]
      1. The financial exposure of the experiment is [blank]
      1. We can turn off the experiment by using [blank]

SECTION 3 - EXPERIMENTS

EXPERIMENT 3.1 - SELECT AN EXPERIMENT

  • Experiment Section

    • Pick the right experiment by asking these three questions:

        1. Type of hypothesis: What type of hypothesis are you testing? Pick experiments based on your major learning objective. Some experiments produce better evidence for desirability, some work better for feasibility, and some are more appropriate for viability.
        1. Level of uncertainty: How much evidence do you already have (for a specific hypothesis)? The less you know, the less you should waste time, energy, and money. When you know little, your only goal is to produce evidence that points you in the right direction. Quick and cheap experiments are most appropriate for that goal, despite the generally weak evidence. The more you know, the stronger the evidence should become, which is usually achieved by more costly and lengthier experiments.
        1. Urgency: How much time do you have until the next major decision point or until you run out of money? The selection of the right experiment may depend on the time and money you have available. If you have a major meeting with decision makers or investors coming up, you might need to use quick and cheap experiments to quickly generate evidence on multiple aspects of your idea. When you are running out of money, you need to pick the right experiments to convince decision-makers and investors to extend funding.

    • Rules of thumb:

        1. Go cheap and fast at the beginning. Early on, you generally know little. Stick to cheap and quick experiments to pinpoint the right direction. You can afford starting out with weaker evidence, because you will test more later. Ideally, you select an experiment that is cheap, fast, and still produces strong evidence.
        1. Increase the strength of evidence with multiple experiments for the same hypothesis. Run several experiments to support or refute a hypothesis. Try to learn about a hypothesis as fast as possible, then run more experiments to produce stronger evidence for confirmation. Don’t make important decisions based on one experiment or weak evidence.
        1. Always pick the experiment that produces the strongest evidence given your constraints. Always select and design the strongest experiment you can, while respecting the context. When uncertainty is high you should go fast and cheap, but that doesn’t necessarily mean you can’t produce strong evidence.
        1. Reduce uncertainty as much as you can before you build anything. People often think they need to build something to start testing an idea. Quite the contrary. The higher the costs to build something, the more you need to run multiple experiments to show that customers actually have the jobs, pains, and gains you think they have.

    • Ask these three questions

        1. What type of hypothesis are you testing?
        1. How much evidence do you already have (for a specific hypothesis)?
        1. How much time do you have until the next major decision point or until you run out of money?

    • Rules of thumb

        1. Go cheap and fast early on in your journey.
        1. Increase the strength of evidence with multiple experiments for the same hypothesis.
        1. Always pick the experiment that produce the strongest evidence, given your constraints.
        1. Reduce uncertainty as much as you can before you build anything

  • Experiment Sequences:
    • Go beyond pairing with experimentation sequences. Once you’ve turned your insights into action, it’s time to move on and throw the experiment away, correct? Well, not necessarily. As illustrated in the pairings for each experiment, there are experiments you can run before, during, and after. But what about a sequence of experiments? Great teams are able to gain momentum and build up stronger evidence over time with a series of experiments.

EXPERIMENT 3.2 - DISCOVERY

  • Customer Interview:

    • An interview that is focused on exploring customer jobs, pains gains, and willingness to pay.
    • Customer Interviews are ideal for gaining qualitative insights into the fit between your value proposition and the customer segment. It’s also a good starting point for price testing. Customer Interviews are not ideal as a substitute for what people will do.

  • Partner & Supplier Interviews:

    • Partner & Supplier Interviews are similar to Customer Interviews, but you are focused on whether you can feasibly run the business. You’ll be sourcing and interviewing Key Partners to supplement the Key Activities and Key Resources that you cannot do, or do not want to do, in-house.

  • Expert Stakeholder:

    • Interviews Stakeholder Interviews are similar to Customer Interviews, but are focused on getting “buy-in” from key players inside your organization.

  • A Day in the Life:

    • A method of qualitative research that uses customer ethnography to better understand customer jobs, pains, and gains.
    • A Day in the Life is relatively cheap. You may need to compensate people for their time if you decide to work alongside or observe them for the entire day.

  • Discovery Survey:

    • An open-ended questionnaire used in the collection of information from a sample of customers.
    • Discovery Survey is ideal for uncovering your value proposition and customer jobs, pains, and gains. Discovery Survey is not ideal for determining what people will do, only what they say they’ll do.

  • Search Trend Analysis:

    • The use of search data to investigate particular interactions among online searchers, the search engine, or the content during searching episodes.
    • Search Trend Analysis is ideal for performing your own market research, especially on newer trends, instead of relying on third party market research data.

  • Web Traffic Analysis:

    • The use of website data collection, reporting, and analysis to look for customer behavior patterns.

  • Discussion Forums:

    • The use of discussion forums to uncover unmet jobs, pains, and gains in a product or service.
    • Discussion Forums are ideal for finding unmet needs in your existing product or a competitor’s product.

  • Sales Force Feedback:

    • The use of sales force feedback to uncover unmet jobs, pains, and gains in your product or service.
    • Sales force feedback is ideal for businesses that use a group of people to conduct sales.

  • Customer Support Analysis:

    • The use of customer support data to uncover unmet jobs, pains, and gains in your product or service.
    • Customer support analysis is ideal for businesses that already have a substantial amount of existing customers.

  • Online Ad:

    • An online advertisement that clearly articulates a value proposition for a targeted customer segment with a simple call to action.
    • Online ads are ideal for quickly testing your value proposition at scale with customers online.

  • Link Tracking:

    • A unique, trackable hyperlink to gain more detailed information about your value proposition.
    • Link tracking is ideal for testing customer actions to gather quantitative data.

  • Feature Stub:

    • A small test of an upcoming feature that includes the very beginning of the experience, usually in the form of a button.
    • Feature Stub is ideal for rapidly testing the desirability of a new feature of an already existing offering. Feature Stub is not ideal for testing mission critical functionality for your product.

  • 404 Test:

    • Another faster, and somewhat riskier, variation of a Feature Stub is the 404 test. It is very similar, except you do not put anything behind the button or link whatsoever. Hence, the 404 test name, as it generates 404 errors each time it is clicked. To learn if a feature is desirable, you simply count the number of 404 errors generated.
    • When running a 404 test, do not run it for more than a few hours.

  • Email Campaign:

    • Email messages that are deployed across a specific period of time to customers.
    • Email campaigns are ideal for quickly testing your value proposition with a customer segment. Email campaigns are not ideal as a replacement for face-to-face customer interaction.

  • Social Media Campaign:

    • Social media messages that are deployed across a specific period of time to customers.
    • Social media campaigns are ideal for acquiring new customers, increasing brand loyalty, and driving sales.

  • Referral Program:

    • A method of promoting products or services to new customers through referrals, by word of mouth, or through digital codes.
    • Referral programs are ideal for testing with customers how to organically scale your business.

  • 3D Print:

    • Rapidly prototyping a physical object from a three-dimensional digital model by using a 3D printer.
    • 3D print is ideal for rapidly testing iterations of your physical solution with customers.

  • Paper Prototype:

    • Sketched interface on paper, manipulated by another person to represent the software’s reactions to the customer interaction.
    • Paper prototypes are ideal for rapidly testing the concept of your product quickly with customers. Paper prototypes are not ideal as a replacement for proper usability with customers.

  • Storyboard:

    • Illustrations displayed in sequence for the purpose of visualizing an interactive experience.
    • Storyboards are ideal for brainstorming scenarios of different value propositions and solutions with customers.

  • Data Sheet:

    • One page physical or digital sheet with specifications of your value proposition.
    • Data sheets are ideal for distilling down your specifications into a single page for testing with customers and key partners

  • Brochure:

    • Mocked up physical brochure of your imagined value proposition.
    • Physical brochures are ideal for testing your value proposition in person with customers who are difficult to find online.

  • Explainer Video:

    • A short video that focuses on explaining a business idea in a simple, engaging, and compelling way.
    • An Explainer Video is ideal for quickly explaining your value proposition at scale with customers.

  • Boomerang:

    • Performing a customer test on an existing competitor’s product to gather insights on the value proposition.
    • Boomerang is ideal for finding unmet needs with potential customers in an existing market, without building anything. Boomerang is not ideal for stripping away branding and testing a product as if it’s your own.

  • Pretend to Own:

    • Creating a nonfunctioning, low fidelity prototype of the solution to determine whether it fits into the day-to-day life of the customer. Sometimes called a Pinocchio experiment.
    • Pretend to Own is ideal for generating your own evidence on the potential usefulness of an idea.

  • Product Box:

    • A facilitation technique used with customers to visualize value propositions, main features, and key benefits in the physical form of a box.
    • Product Box is ideal for refining your value proposition and narrowing in on key features to your solution.

  • Speed Boat:

    • A visual game technique used with customers to identify what’s inhibiting progress.
    • Speed Boat is ideal for going beyond conversations and having a visual representation of what’s slowing your customers down and learning how it impacts feasibility.

  • Card Sorting:

    • A technique in user experience design in which a person uses cards with customers to generate insights.
    • Card sorting is ideal for getting insights into customer jobs pains, gains, and value propositions.

  • Buy a Feature:

    • A technique where people use pretend currency to buy the features that they would like to be available for a given product.
    • Buy a Feature is ideal for prioritizing features and refining customer jobs, pains, and gains.

EXPERIMENT 3.3 - VALIDATION

  • “Invention is not disruptive. Only customer adoption is disruptive.” Jeff Bezos

  • Clickable Prototype:

    • Digital interface representation with clickable zones to simulate the software’s reactions to customer interaction.
    • Clickable prototype is ideal for rapidly testing the concept of your product quickly with customers at a higher fidelity than paper. Clickable prototype is not ideal as a replacement for proper usability with customers.

  • Single Feature MVP:

    • A functioning minimum viable product with the single feature needed to test your assumption.
    • Single Feature MVP is ideal for learning if the core promise of your solution resonates with customers.

  • Mash-Up:

    • A functioning minimum viable product that consists of combining multiple existing services to deliver value.
    • Mash-Up is ideal for learning if the solution resonates with customers.

  • Concierge:

    • Creating a customer experience and delivering value manually, with people instead of using technology. Unlike Wizard of Oz, the people involved are obvious to the customer.
    • Concierge is ideal for learning firsthand about steps needed to create, capture, and deliver value to a customer. Concierge is not ideal for scaling a product or business.

  • Life-Sized Prototype:

    • Life-sized prototypes and real-world replicas of service experiences.
    • Life-sized prototypes are ideal for testing higher fidelity solutions with customers at a small sample size, before deciding to scale your solution.

  • Simple Landing Page:

    • A simple, digital web page that clearly illustrates your Value Proposition with a call to action.
    • A simple landing page is ideal for determining if your Value Proposition resonates with your customer segment.

  • Crowdfunding:

    • Funding a project or venture by raising many small amounts of money from a large number of people, typically via the Internet.
    • Crowdfunding is ideal for funding your new business venture with customers who believe in your Value Proposition. Crowdfunding is not ideal for determining whether your new business venture is feasible.

  • Split Test:

    • Split Test is a method of comparing two versions, control A against variant B, and determining which which one performs better.
    • Split Test is ideal for testing different versions of Value Propositions, prices, and features to see what resonates best with customers.

  • Presale:

    • A sale held before an item is made available for purchase. Unlike mock sale, you are processing a financial transaction when it ships.
    • Presale is ideal for gauging market demand at a smaller scale before you launch to the public.

  • Validation Survey:

    • A closed-ended questionnaire used in the collection of information from a sample of customers about a specific topic.
    • A validation survey is ideal for getting insights into whether customers will be disappointed if your product went away or if they’ll refer other customers.

  • Wizard of Oz:

    • Creating a customer experience and delivering value manually, with people instead of solely using technology. The name Wizard of Oz is derived from the movie, where you have a request that is handled by a person. Unlike Concierge, the people involved aren’t visible to the customer.
    • Wizard of Oz is ideal for learning manually, firsthand about steps needed to create, capture, and deliver value to a customer Wizard of Oz is not ideal for scaling a product or business.

  • Mock Sale:

    • Presenting a sale for your product without processing any payment information.
    • Mock sale is ideal for determining different price points for your product.

  • Letter of Intent:

    • Short, written contract that is simple to read and not legally binding.
    • Letter of intent is ideal for evaluating key partners and B2B customer segments. Letter of intent is not ideal for B2C customer segments.

  • Pop-Up Store:

    • A retail store that is opened temporarily to sell goods, usually a trendy or seasonal product.
    • A pop-up store is ideal for testing face-to-face interactions with customers to see if they’ll really make a purchase. A pop-up store is not ideal for B2B businesses: consider a booth at a conference instead.

  • Extreme Programming Spike:

    • A simple program to explore potential technical or design solutions. The term spike is derived from rock climbing and railroads. It’s a necessary task to stop and perform so that you can feasibly continue to make progress.
    • The Extreme Programming Spike is ideal for quickly evaluating whether or not your solution is feasible, usually with software. The Extreme Programming Spike is not ideal for scaling the solution, as it is typically thrown away and re-created afterwards.

SECTION 4 - MINDSET

  • “The more success you’ve had in the past, the less critically you examine your own assumptions.” Vinod Khosla Venture capitalist

MINDSET 4.1 - AVOID EXPERIMENT PITFALLS

  • Experiment Pitfalls

    • Time Trap: Not dedicating enough time.

      • You get what you invest. Teams that don’t put in enough time to test business ideas won’t get great results. Too often, teams underestimate what it takes to conduct multiple experiments and test ideas well.
      • Carve out dedicated time every week to test, learn, and adapt. Set weekly goals in regard to what you’d like to learn about your hypotheses. Visualize your work so that it becomes clear when tasks are stalled or blocked.

    • Analysis Paralysis: Overthinking things that you should just test and adapt.

      • Good ideas and concepts are important, but too many teams overthink and waste time, rather than get- ting out of the building to test and adapt their ideas.
      • Time box your analysis work. Differentiate between reversible and irreversible decisions. Act fast on the former. Take more time for the latter. Avoid debates of opinion. Conduct evidence-driven debates followed by decisions.

    • Incomparable Data/Evidence: Messy data that are not comparable.

      • Too many teams are sloppy in defining their exact hypothesis, experiment, and metrics. That leads to data that are not comparable (for example, not testing with the exact same customer segment or in wildly different contexts).
      • Use the Test Card. Make test subject, experiment context, and precise metrics explicit. Make sure everybody involved in running the experiment is part of the design

    • Weak Data/Evidence: Only measure what people say, not what they do.

      • Often teams are happy with running surveys and interviews and they fail to go deeper into how people act in real life situations.
      • Don’t just believe what people say. Run call-to-action experiments. Generate evidence that gets as close as possible to the real world situation you are trying to test.

    • Confirmation Bias: Only believing evidence that agrees with your hypothesis.

      • Sometimes teams discard or underplay evidence that conflicts with their hypothesis. They prefer the illusion of being correct in their prediction
      • Involve others in the data synthesis process to bring in different perspectives. Create competing hypotheses to challenge your beliefs. Conduct multiple experiments for each hypothesis.

    • Too Few Experiments: Conduct only one experiment for your most important hypothesis.

      • Few teams realize how many experiments they should conduct to validate a hypothesis. They make decisions on important hypotheses based on one experiment with weak evidence.
      • Conduct multiple experiments for important hypotheses. Differentiate between weak and strong evidence. Increase the strength of evidence with decreasing uncertainty.

    • Failure to Learn and Adapt: When you don’t take time to analyze the evidence to generate insights and action.

      • Some teams get so deep into testing that they forget to keep their eyes on the prize. The goal is not to test and learn. The goal is to decide, based on evidence and insights, to progress from idea to business.
      • Set aside time to synthesize your results, generate insights, and adapt your Idea. Always navigate between detailed testing process and big picture idea: which patterns that matter are you observing? Create rituals to keep your eyes on the prize: ask if you’re making progress rom idea to business.

    • Outsource Testing: When you outsource what you should be doing and learning yourself.

      • Outsourcing testing is rarely a wise idea. Testing is about rapid iterations between testing, learning, and adapting an idea. An agency can’t make those rapid decisions for you and you risk wasting time and energy by outsourcing.
      • Shift resources you reserved for an agency to internal team members. Build up a team of professional testers.

MINDSET 4.2 - LEAD THROUGH EXPERIMENTATION

  • Improving Business Models

    • Language: As you lead teams through experimenting on a known business model, be mindful of the fact that overuse of your words can unintentionally disempower the teams. They may feel as though their decision-making authority is taken away, even if you are merely giving your opinion. They’ll simply wait for you to assign them experiments, which is not ideal.

    • Accountability: Remember to focus on business outcomes, not just the features and dates. Your teams need the opportunity to give an account on how they are experimenting and making progress toward business outcomes. As a leader, it’s your job to create an environment for these opportunities to occur.

    • Facilitation: As you grow into a leader at higher levels of the organization, you’ll realize that facilitation skills are imperative. We recommend taking courses on facilitation to level up your leadership game.

    • Do:

      • “We, Us, Our”
      • How would you achieve this business outcome?
      • Can you think of 2-3 additional experiments?

    • Don’t:

      • “I, Me, Mine”
      • “Deliver this feature by release date”
      • “This is the only experiment we should run”

  • Inventing Business Models

    • Strong Opinions, Weakly Held: Inventing new business models requires experimentation and openness to the idea of being wrong. One way to think about this is from Paul Saffo’s “strong opinions, weakly held” approach. It means you start out with a hypothesis, but be open for it to be proven wrong. If you are merely trying to prove that you are right, then you become susceptible to your cognitive biases.

    • Do:

      • “What is your learning goal?”
      • “What obstacles can I remove to help you make progress?”
      • “How else might we approach this problem?”
      • “What learning has surprised you so far?”

    • Don’t:

      • “I don’t trust the data.”
      • “I still think it’s a good idea and we should build it anyway.”
      • “You need to talk to 1,000 customers before it means anything.”
      • “This has to be a $15 million dollar business by the end of next year.”

  • Steps Leaders Can Take

    • Create an Enabling Environment:

      • Leaders need to abolish business plans and establish appropriate testing processes and metrics that differ from execution processes and metrics. They need to give teams the autonomy to make decisions, move fast, and then get out of the way.

    • Remove Obstacles and Open Doors: Access to Customers, Brand, IP, and Other Resources

      • Leaders can remove obstacles when teams that are testing business ideas encounter internal roadblocks, like lack of access to internal expertise or specialized resources.

    • Make Sure Evidence Trumps Opinion: Change Decision-Making

      • Leaders are used to deciding based on their often deep experience and extensive track record. Yet, in innovation and entrepreneur ship, past experience might actually prevent an individual from seeing and adapting to the future. Here, evidence from testing trumps opinion. The leader’s role is to push a team to make a compelling case for an idea based on evidence, not based on the leader’s preferences.

    • Ask Questions Rather Than Provide Answers: Help Teams Grow and Adapt Their Ideas.

      • They need to relentlessly inquire about experiments, evidence, insights, and patterns on which teams build value Propositions and business model ideas.

  • Create More Leaders

    • Meet Your Teams One-Half Step Ahead

      • Think of where you eventually want team members to be, then look backward. How will they get here? What steps will they have to take? It’s a small cognitive trick but it works. Leaders need a sense of where their teams are today and how to nudge them down that path. Find opportunities to guide them to take that first step, whether it be in scheduled one-on-ones, retrospectives, or in hallway conversations.

    • Understand Context Before Giving Advice

      • Leaders need to actively listen and understand the context before giving advice to team members. Practice letting team members speak until they are finished. Once there is a pause in the conversation, ask clarifying questions to make sure you understand the context before giving advice.

    • Say “I Don’t Know.”

      • We strongly recommend that you practice saying these three words, “I don’t know,” when you are in a situation where you don’t know. It will help your teams begin to understand that you don’t have all of the answers, nor should you. Follow it up with “How would you approach this?” or “What do you think we should do?” Saying “I don’t know” will help you model the behavior the leaders you create will embrace.

MINDSET 4.3 - ORGANIZE FOR EXPERIMENTS

  • Silos vs Cross-Functional Teams

    • This is why there has been a shift from traditional, functionally siloed organization models to more agile, cross-functional team approaches. When testing out new business ideas, speed and agility are imperative. Cross-functional teams can adapt more quickly than functionally siloed teams. In many organizations, small, dedicated, cross-functional teams can outperform large, siloed project teams.

  • Innovation Portfolio

  • Investment Committees

    • Designing the Committee

      • 3-5 members: Keep the committee relatively small in size so that you can make decisions and run fast.
      • External member: Consider adding an external member or entrepreneur in residence (EIR) who can help bring a fresh perspective to the portfolio.
      • Decision-making authority: Include members who can make decisions with regard to approval and budget.
      • Entrepreneurial: While members do not necessarily have to have a history of entrepreneurship, they need to be willing to challenge the status quo. Too many conservative members will prematurely stunt the growth of new innovations.

    • Create a Working Agreement

      • Be on time: Members have busy schedules, but they have to prioritize the stakeholder review ceremonies, otherwise teams will be left wondering if their initiatives are important.
      • Make decisions in the meeting: Teams should not leave the review wondering if they can move forward. Decide with the teams present before adjourning.
      • Leave ego at the door: Have an opinion in the review but be willing to be swayed by evidence. The teams will be bringing what experiments they ran and how to move forward. It is your job to listen, not talk over them.

    • Foster an Environment

      • As a committee, have a plan to revisit how you are helping the team with obstacles centered on:
        • Time.
        • Multitasking.
        • Funding.
        • Support.
        • Access.
        • Direction.