Build Times Survey
This research was conducted to measure how long it takes to build each section of a vacation rental listing. The project’s objective was to identify which sections take the most time, gauge the efficacy of our BPO partners, and pinpoint areas in need of further refinement.
Intro
Role
Lead Researcher
Google Forms
Google Sheets
Google Slides
Tools
January ‘26 - February ‘26
2 months
Duration
The Listing Optimization Team had undergone numerous process changes over the course of two years. We launched in-house AI tools, integrated with new BPO partners, developed new resources (while optimizing old ones), and simplified internal SOPs.
In addition to these changes, the team was also scoping automation opportunities. In order to pinpoint areas of the listing build process that required the most manual work, and make a case for why/how these areas could be automated, we needed more information on how long the team was spending on these sections.
Background
With a multitude of booking platforms and a vast array of rental properties available to consumers, it’s hard to create a stand-out vacation rental listing, especially in saturated markets. Keeping in mind the ever-evolving landscape of the short-term rental business, we want to ensure performance-boosting strategies will be effective before investing in their development.
Problem
Research Plan
How valuable and efficient are our BPO partners who support the listing build process?
Research Questions
Which resources are reps referencing the most?
How much time are reps spending on each section of the listing build?
Methodology
Google Forms Survey
Age 28-65
Annual Income: $40,000+
Device: Smartphone
N = 20
(5 per test)
Recruitment
Employment Status: Employed, retired, unable to work, stay-at-home parent
Analysis
Usertesting’s built-in analysis tools
Deliverables
Slide deck presentation
Research
Two usability tests were conducted on the Usertesting platform to evaluate two distinct variations of social proofing.
Both tests began with screener questions to ensure our test subjects had familiarity with booking vacation rentals.
Screening
The first test explored the effectiveness of scarcity messaging, a marketing strategy that highlights the limited availability of a specific product (or in this case, amenity) to manufacture a sense of desirability and urgency. For this study, we chose to highlight that our lakefront property is one of only a few other properties in the area with a private dock.
The second test was centered around a reviews spotlight – a short snippet of text that compiled some of the top themes highlighted throughout the property’s reviews. For this study, the verbiage used emphasized the home’s primary audience and welcoming atmosphere.
Testing
When you travel, what type of lodging do you primarily stay in?Accepted:
Vacation rentals
How many times do you book and stay in a vacation rental per year?Accepted:
2-4, 5+
Who is responsible for finding and booking properties for your travels?Accepted:
Self, shared responsibility
I utilized UserTesting’s built-in analysis tools to analyze the qualitative data collected from these test sessions. The sentiment analysis feature helped to easily identify intervals of friction the user experienced while walking through the prompts, and the smart tags feature helped to refine these intervals by labeling the different sentiments expressed by the user.
These tools, combined with an in-depth review of each session’s transcript, allowed me to easily identify trends and generate behavioral and attitudinal insights.
Analysis
Conclusion
Scarcity Messaging Test:
Results from the scarcity messaging test revealed users don’t engage with scarcity messaging and it’s not an effective way to boost interest in a vacation rental property. The scarcity messaging was viewed as informative rather than distinctive. Users also felt the messaging was redundant and that the information highlighted could easily be found elsewhere on the vacation rental listing.
Reviews Spotlight Test:
While users noticed and lightly engaged with the reviews spotlight, they questioned the validity of the highlighted insights and were skeptical of how they were generated. Users wanted to know if the language was AI-generated or handpicked by the homeowner.
Similar to the results of the scarcity messaging test, the reviews spotlight test showed that users found the information redundant. Users noted that, assuming the highlighted reviews were an accurate representation of the listing, they could identify these trends for themselves by reading the reviews.
Findings
Given the lack of engagement exhibited in both tests, the research findings led the team to abandon scarcity messaging and review spotlighting. The team continues to test the efficacy of other engagement-enhancing strategies and is also exploring other avenues of social proofing.
While the two strategies tested were never deployed, the tests still provided valuable insight into how potential travelers engage with vacation rental listings. They underscored the importance of 5-star reviews and how influential they are in a traveler's decision to book a vacation rental. The usability tests also gave us a glimpse into certain navigational behaviors, like how potential guests typically scroll straight down to the reviews before engaging with the listing description.
Outcomes
UserTesting is a great platform to generate quick insights at low costs, but the simplicity it provides can undermine the recruitment process. While users are screened and monetarily incentivized, there is still the risk of recruiting disengaged participants.
While conducting the unmoderated usability tests, I received a couple of sessions where the user either didn’t read the prompts or completed the tasks while engaged in other activities, causing them to miss certain segments. To combat these failed sessions, I added a couple of additional ones in the hopes of attracting more engaged users.
My experience with the UserTesting software has taught me that online research platforms require careful planning and oversight when it comes to recruitment. While not every usability session will be equally as productive, fine-tuning the screener questions may help eliminate less engaged participants.