Overstock-BB&B App JTBD Strategic Opportunity Case
Overstock-BB&B App JTBD Strategic Opportunity Case
Business Problem:
App is a strategic focus for the company. My task is to help the apps team determine strategy and next-steps.
Business Results:
Three strategic opportunities were identified as focus for the next two years which resulted in a 1.5x top-line growth for the year for the apps team.
Timeline:
3 weeks (later negotiated to add 3 more weeks)
Methodologies and processes used:
Secondary research, literature, and data
Stakeholder interviews
Jobs-to-be-done-theory
In-depth interviews
Survey
Artifacts created after the research:
PPT stakeholder presentation
Email documentation with highlights
Excel spreadsheet with quantitative usability results
My Work Flow
Research Process Step-by-Step
*A note: Due to the time constraints and the business dynamic, this research focused on opportunities of the current users instead of potential users. (also see section II)
I. Prior research, literature, and data review:
Reviewed prior literature to understand if any research was conducted and what the findings were.
There was no jobs-to-be-done research done on Overstock app before.
Connected with relevant stakeholders and used internal data tools to review the shoppers' and suppliers' current financial, demographic, attitudinal(e.x. site intercept), and behavioral data of the relevant features, pages.
Overstock used behavioral and attitudinal segmentation instead of demographic. Therefore, the demographic data is only served to double-check against the population. The demographic criteria will be kept relatively wide, except the gender distribution will be kept relatively even to reflect the population.
Behavioral data gave insight into whether any specific behavior attributes of this population would necessitate recruiting changes(in this case, no except for returning customers vs. new).
Behavioral data also showed that the majority of the app users were on IOS for the past 2 years. Therefore rest of the research will focus on former and current IOS users due to time constraints.
Overstock used a one-question simplified version to recruit participants with population-like attitudes. This question, however, is not comprehensive enough to be 100% accurate. When there is a concern with time or recruiting costs, this question was not used for initial screening; it will simply be added for post-analysis purposes. The following research did just this to adhere to the timeline and cost.
II. Stakeholder discussion:
Interviews were conducted with stakeholders to understand their visions for the Overstock app.
In discussing with stakeholders and adhering to Ulwick’s JTBD and ODI framework, the “job” will be focused on when users use the app instead of downloading the app.
Continued discussions with stakeholders during the prototyping stage, debriefing stage, and solutions stage.
Investigate stakeholder availability and invite them to the JTBD exploratory interviews.
Define and align on scope of the research. Due to the time constraints and the business dynamic, this research focused on opportunities of the current users instead of potential users.
III. In-depth remote interview
Goals of in-depth interviews:
Using a mix of Ulwick’s (2016) and Christensen’s (2016) to investigate IOS user’s functional, emotional/social goals, desired outcomes along with the current obstacles and alternative solutions.
Recruitment:
12 participants were sampled from the User Zoom panel using probability sampling to participate in a one-hour interview
Sample size was determined using Bunce et al.(2006)’s research.
6 participants “hired” the Overstock app and 6 participants “fired” the Overstock app.
All participants are on IOS devices due to current usage (see section I for explanations)
Attitudinal segmentation questions were added for post-analysis purposes but not for screening purposes.
Age of the participants was kept broad, with only gender distribution kept even to reflect the population.
Set up
An hour interview conducted remotely using User Zoom GO.
The interview script generally follows the University of Michigan UX Research guidelines for conducting interviews, starting from broad to narrow. The discussion started from generally talking about online shopping for their homes down to specific devices, usages, etc.
The interview focused on getting functional goals, emotional/social goals, desired outcomes, current obstacles if any, and alternative solutions.
For participants who “hired” the app, questions will focus on the occasions where they used the app (not downloaded)
For participants who “fired” the Overstock app, the discussion will focus on alternatives they are using and reasons for not using the Overstock app. Uninstall is not the most important topic for them due to the business question and Ulwick(2016)’s definition for “jobs.”
Analysis:
By themselves, each individual who attended the interviews came up with statements for functional goals, emotional/social goals, and desired outcomes.
The researcher(me), used existing coding to speed up the statement writing process.
The researcher(me) also prepared video clips and photos to support some of the statements.
Following Ulwick’s emphasis on context, the statements were initially grouped by participant to allow full showing of the context in which decisions were made.
Due to tool and compensation constraints, the debrief leader(me) ensured the number of statement options didn’t exceed 30(based on prior dropoff rate knowledge) causing potential survey fatigue. If it does, then this will be brought up for discussion in the next step.
IV. Stakeholder Discussion 2:
This discussion is to push for 3 more weeks in order to conduct the survey portion of the study.
Introduce the advantage of a survey, which is to quantify the opportunity size so the product teams can quantitatively plan roadmaps with financial estimates.
Discuss potential technical constraints based on the statements from the in-depth interview.
V. Survey:
Goal
Using surveys to understand opportunity size and areas where improvements are needed.
Recruiting
286 participants were recruited from User Zoom panel using randomized probability sampling.
Ulwick(2016) recommended at least 180 participants for opportunity score calculation.
To maintain commonly accepted and appropriate confidence level(80%), margin of error(around 5%), and estimated recruiting speed., additional participants were added.
Recruiting criteria: random probability sampling from Uzer Zoom panel.
Have used Overstock IOS app in the last 3 months.(see section I for explanations on why IOS)
Attitudinal segmentation questions were added for post-analysis purposes but not for screening purposes.
Age of the participants was kept broad, with only gender distribution kept even to reflect the population.
Set up:
Using the statements from exploratory interviews, the survey asks participants to rate the importance of the intended goal of the app and their satisfaction with accomplishing their functional, social, or emotional goals.
The statements were rated on a 5-point scale to reduce response biases (Garret et al., 2011)
Some statements may be repeated and written in reverse meaning to check for speeders and reduce survey fatigue.
Analysis:
The results for the perceived need and satisfaction were normalized to 10 to calculate the potential relative opportunity size using the ODI framework by Ulwick(2016).
The results were plotted on a scatter plot chart and the opportunity scores were calculated using Opportunity Score = Importance + max (Importance – Satisfaction, 0) (Ulwick, 2016).
The top-ranked statements by opportunity score were recommended as potential opportunities to go after.
Stakeholders were verbally notified of the results for early alignment purposes.
*Mock data
*Mock data
VI. Stakeholder presentation(PPT) and discussion:
The goal of this presentation is to officially present the final opprtunity results to the stakeholders. The results were communicated in pieces ahead of time already. This presentation is to bring all the information together.
I led the discussion afterwards focusing on driving the team towards narrowing down to specific areas of opprtunities for the business.
The discussion first went through what we are currently doing well, what are our technical capabilities and constraints, PM’s roadmap, and upper management directions.
The second part of the discussion narrowed down the potential strategic directions, keeping in mind the information from the research and the first part of the discussion.
The meeting lasted an hour with several follow-ups to finalize the decision.
VII. Tracking post-research:
Periodic monthly and quarterly tracking to verify the financial top-line impact of the change.
Post-launch user behavioral tracking using heat mapping capabilities(Quantum Metric).
If I had more time and other discussions:
On reflection, if ChatGPT was available, I could experiment with it to shorten the time to arrive at the first version of the persona after the exploratory interviews. That way, I can save some time and start the quantitative surveying early.
In the future, consider linking personas with JTBD goals to create more interconnected artifacts and to develop a more holistic understanding.
If I had more time, I would recruit more Android users as they potentially presented small but more untapped opportunities.
Citation:
Bunce, A., Guest, G., & Johnson, L. (2006). How many interviews are enough? Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822x05279903
Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Competing against luck: The story of innovation and customer choice. HarperBusiness, an imprint of HarperCollins Publishers.
Garratt, A. M., Helgeland, J., & Gulbrandsen, P. (2011). Five-point scales outperform 10-point scales in a randomized comparison of item scaling for the patient experiences questionnaire. Journal of Clinical Epidemiology, 64(2), 200–207. https://doi.org/10.1016/j.jclinepi.2010.02.016
Ulwick, A. W., & Osterwalder, A. (2016). Jobs to be done: Theory to practice. Idea Bite Press.