Overstock-BB&B SP Expansion
Overstock-Bed Bath & Beyond Sponsored Product Strategic Expansion
Business Results:
Overstock-Bed Bath & Beyond (BB&B) expansion is currently driving an annualized $2M contribution margin with more room to grow.
Business Problem:
Is increasing Overstock Sponsored Products or “featured” products a plausible financial opportunity for mobile app?
Timeline:
5 weeks
Methodologies and processes used:
Secondary research, literature, and data
Stakeholder interviews
Unmoderated interviews
Unmoderated prototype testing(including torture test)
Usability tests
A/B tests
Heat map and scroll depth behavior tracking
Artifacts created after the research:
Stakeholder presentation
Word documentation
Email documentation with highlights
Excel spreadsheet with quantitative usability results
My Work Flow
Research Process Step-by-Step
I. Prior research, literature, and data review:
Reviewed prior literature to understand if any research was conducted and what the findings were.
One of the findings was that suppliers don’t subjectively feel featuring their products is effective despite the proven ROI. Suppliers were unable to easily find the actual “featured” products on the site or app, creating an emotional barrier for further expanding the featured products.
Suppliers were unwilling to spend more to have their products featured more often due to this emotional barrier.
Cost of featuring many of the suppliers’ products more often is larger than what Overstock could provide as promotional free credits.
Connected with relevant stakeholders and used internal data tools to review the shoppers' and suppliers' current financial, demographic, attitudinal(e.x. site intercept), and behavioral data of the relevant features, pages.
Overstock used behavioral and attitudinal segmentation instead of demographic. Therefore, the demographic data is only served to double-check against the population. The demographic criteria will be kept relatively wide, except the gender distribution will be kept relatively even to reflect the population.
Behavioral data gave insight into whether any specific behavior attributes of this population would necessitate recruiting changes(in this case, no except for returning customers vs. new).
Overstock used a one-question simplified version to recruit participants with population-like attitudes. This question, however, is not comprehensive enough to be 100% accurate. When there is a concern with time or recruiting costs, this question is not used for initial screening; it will simply be added for post-analysis purposes. The following research did just this to adhere to the timeline and cost.
II. Stakeholder discussion:
Interviews were conducted with stakeholders to understand their visions for the Overstock-BB&B Sponsored Product platform, timelines, and constraints.
In this scenario, the ideal solution should be able to create value for the suppliers and Overstock while, at least, not impacting the shopping experience for the app users. This exploratory research will focus on exploring options that can achieve this goal.
Considering prior findings, stakeholders and I agreed that expanding the number of places “featured” products can be shown is the best way to easily create value for both supplier and Overstock.
Continued discussions with stakeholders during the prototyping stage, debriefing stage, and solutions stage.
III. Unmoderated interviews
Goals of unmoderated interviews:
Using moderated interviews to understand how the sponsored products currently impact shoppers, if any.
Understand if the labeling is attention-drawing in any way. If so, what are their reactions to these labels and related products?
Recruitment:
15 participants were sampled from the User Zoom panel using probability sampling.
Majority of participants were Overstock IOS app users due to the behavioral data of the population (see section I for explanations)
Attitudinal segmentation questions were added for post-analysis purposes but not for screening purposes.
Age of the participants was kept broad with only gender distribution kept even(53.3% vs. 46.7%) to reflect the population.
All participants are on IOS due to Overstock mobile app users primarily being on IOS devices.
Set up of the interviews & walk-throughs:
Each participant will be directed to the live shopping site and instructed to shop for an item of their choice and stop when they find the item.
Participants will be instructed to “think out loud” while they shop.
Participants will answer questions after the task to understand if they remember seeing any “featured” or sponsored product. The questions will also draw attention specifically to the "featured” product signage to understand their comprehension of the signage.
Debrief:
AI-assisted transcript qualitative analysis: using chatGPT to assist in analyzing the transcript to find common themes.
Stakeholder affinity mapping style discussion, focusing on getting stakeholder buy-ins for the findings and insights.
IV. Unmoderated prototype testing:
Goals:
Using cognitive walk-through to understand the potential impact of increasing the number of “feature” products, if any.
Different prototypes were created to represent the possible ordering of the increased “featured” product on the search results pages, including a full page “featured” product torture test, to take into account the possible impact of the different graphical arrangement of the “featured” products.
Sample unmoderated prototype testing
Recruitment:
60 participants were sampled from the User Zoom panel using probability sampling.
Majority of participants were Overstock IOS app users due to the behavioral data of the population. (see section I for explanations)
Attitudinal segmentation questions were added for post-analysis purposes but not for screening purposes.
Age of the participants was kept broad with only gender distribution kept even to reflect the population.
All participants are on IOS due to Overstock mobile app users primarily being on IOS devices.
Set up of the interviews & walk-throughs:
Each participant will be randomly shown only one of the prototyped variations(full page, randomized, 2 of selected full rows) of the increased “featured” product page.
Participants will be instructed to look for a product of their choice within the given category and click on it when they have made the choice.
Participants will be instructed to “think out loud” while they shop.
Participants will answer questions after the task to understand if they remember seeing any “featured” or sponsored product. The questions will also draw attention specifically to the "featured” product signage to understand their comprehension of the signage.
Sample questions: Action: click on the image above and look at the tile of a rug. You may have seen this on the search result page. Question: what does the word "Featured" on the tile mean to you here? Be sure to think-out-loud as well.
Sample IOS test version
Debrief:
AI-assisted transcript qualitative analysis
Using chatGPT to assist in analyzing the transcript to find common themes.
The batch of answers for each prototype variation will be analyzed separately to ensure no cross-contamination between each possible “featured” product arrangement.
Stakeholder affinity mapping style discussion
V. Stakeholder discussion 2:
A total of 7 key findings were shared with the stakeholders using presentation.
Top 4 Findings
Overall, the sponsored/“featured” products did not cause an observable negative impact in any test or control versions; some barely even remembered any products that were “featured.”
There were no attitudinal differences or observable user frictions between the variations of the test prototypes and compared to the control version.
The interpretations of the “featured” tags vary when drawing attention to them. Some were not sure what “featured” exactly meant.
When drawing attention to them, the interpretations of the “featured” tag were impacted by the other aspects of patterns they observed as they browsed(sales price, “best-selling tag,” top row positioning, having the tag on products of the same partner.)
Stakeholders and I decided to move forward with one of the solutions, considering technical constraints and potential ROI.
VI. Usability tests:
A usability test was conducted using the chosen solution's prototype and the current solution's screenshot to ensure that time on task for finding items in top visited categories isn’t impacted.
293 participants were recruited using probability sampling from the Uzer Zoom panel.
Recruiting criteria: random probability sampling from Uzer Zoom panel.
Majority of participants were Overstock IOS app users due to the behavioral data of the population. (see section I for explanations)
Attitudinal segmentation questions were added for post-analysis purposes but not for screening purposes.
Age of the participants was kept broad, with only gender distribution kept even to reflect the population.
Sample size was calculated using the company's commonly accepted and appropriate confidence level, margin of error(around 5%), and estimated recruiting speed.
Key metrics were: % of success in finding something they liked, time-on-task, ease of completing the task, satisfaction with the product choice they made, and overall experience when browsing this part of the app.
Set up of the test:
Each participant will be exposed to either control or test version
Participants will be asked to find and click on a product they would be highly interested in learning more
Analysis:
A z-test was performed on the KPIs using 80% CI to check if test KPIs were different than control’s.
There weren’t any significant differences between all the KPIs, which achieves the goal of not impacting customer’s shopping experience.
VII. Tracking post-research:
A/B site test: A site test was conducted to verify the conversion impact of the change.
Periodic monthly and quarterly tracking of the key financial metrics.
Post-launch user behavioral tracking using heat mapping capabilities(Quantum Metric).
Overstock-Bed Bath & Beyond (BB&B) expansion drove an annualized $2M contribution margin with more room to grow.
If I had more time:
I would verify if the brainstormed solution would reduce suppliers’ emotional barriers before proceeding to A/B site tests. This was not possible due to time constraints, the maturation of UX Research at the org, and the momentum of the business.