User Testing Research of Google Flight (User Test Report)
Executive
Summary
Executive
Summary
The goal of this exercise was
to assess the usability of the google flight website. This user test was
conducted with an initial interview and screening process, followed by the
test, a debrief interview and an administration of the SUS (System Usability
Scale). This user test provided 5 key findings for the reader’s review. Most
tests were completed successfully, however, several factors and/or glitches in
the system created user frustration, confusion or added significant time to the
task-at-hand.
The problems, listed from most to least severe, along
with recommended fixes are:
- Key Finding: An initial search of hotel
pricing presumably shows the lowest prices found by the system. When taking the
next step to book, however, a new list of prices along with booking options is
generated and those prices do not necessarily include the price shown in the
first search. Recommendation: Do not show prices that are not actually
available to the user.
- Key Finding: Multi-city pre-set is hidden
in a drop-down menu and overlooked by users. Recommendation: Make this option
visible to users and prominently display, similar to Expedia’s design.
- Key Finding: Users cannot easily
save/print/share itineraries and must leave the system to achieve this goal.
Recommendation: add save/print/share feature.
- Key Finding: Search criteria are retained
from the previous search leading to unintended results when searching for new
trips. Recommendation: Provide users with a “New Search” option.
- Key Finding: Flight “Times” slider is
“clunky” and easy to overlook. Recommendation: Redesign this tool so it is
easier to find and gives the user more control over input.
- “Amenities” filter (when searching hotels) require a click selection of each item plus clicking the “Apply” button or else choices are lost. Recommendation: Redesign this feature to streamline and do away with the “Apply” button.
- Prices cannot be tracked unless a user is logged in to their Google account, in which case the option disappears entirely from the site. Recommendation: Show the “Track prices” tool even when a user is not logged in, prompt the user to either sign in or create a new account to take advantage of this otherwise useful feature.
- When a user opts to track a flight, there is no immediate feedback other than a colour change on the toggle. Recommendation: Have the system send an email to verify and include a link to the flight details.
- There are no labels indicating whether prices reflect totals for all passengers being booked on flights or multiple nights being booked at hotels. Recommendation: To keep pricing clear for the user, including small text labels with short price descriptors.
Introduction
Google Flights is an online
flight booking search service that facilitates the purchase of airline tickets
through third-party suppliers. An innovation of Google Flights is that it
allows open-ended searches based on criteria other than the destination; for
example, a user may search for flights within a range of times and a budget and
be offered various destination choices. Alternatively, a user can select a
destination, and Google Flights will calculate every price for each day of the
next 12 months, visualized in a graph or table. This allows users to easily
spot the cheapest date to fly to the destination. The service was immediately
compared to competitors such as Expedia, Orbitz, Kayak.com, and Bing.
This User Test Report examines
the basic functionality of the Google Flights search system for planning and
booking travel-related services. This report is limited in scope and focuses on
the basic flight and hotel searches conducted by average users who have used
other online booking systems (such as Expedia, Spirit Air) to book, at minimum,
1–3 or more trips in the past year.
The goals of this test are to
see if experienced users of on-line travel booking sites can use the site,
google.com/flights, to plan their trips and to discover what problems users
encounter when using the site. Google’s mission statement reads as follows:
“Our mission is to organize the world’s information and make it universally
accessible and useful.” Bearing this in mind, the goal is to support Google’s
mission by finding ways to streamline the travel booking process so that the
information it provides is ‘accessible and useful’ to all users.
Methodology
The goal of this formative user test is to answer the
following questions:
- Can experienced users of on-line travel booking sites use the site, Google.com/flights, to plan their trips?
- What problems do users encounter when trying to use the site to plan trips?
- Required no special flight preferences or accommodations.
- Booked 1–3 or more than 4 trips per year on-line using expedia.com, spirit.com and flyfrontier.com websites.
- Searched for cheap, round trip airfare then emailed the itinerary to the moderator.
- Searched for round trip airfare for 4 passengers.
- Searched for a multi-city trip in business class.
- Searched for round trip airfare with time constraints.
- Searched for hotel accommodations with rating, amenities, and price constraints.
- Set up an airfare alert for a future trip.
- experienced users of on-line travel booking sites but who had not previously used Google Flights
- participants who had booked on-line 1–3 times, or more than 4 times
- participants who had bought a ticket online in the past year
- participants who required no special accommodations
- participants with complex needs (for example, dietary restrictions, travelling with infants, or other special needs)
- First Participant (P-01) is an adult female living in the U.S. who has travelled mostly for pleasure, has used Spirit.com to book most of her domestic trips. She has also used Expedia.com and Airbnb.com to book multi-destination travel to Ireland and Croatia within the last year. Additionally, this participant has travelled with elderly passengers who have required special assistance when flying and also with young children.
- Second Participant (P-02) Is an adult male also living in the U.S. who has travelled for business and pleasure. In the past year he has booked flights and hotel stays on Expedia.com for domestic travel. He requires no special accommodations and his biggest concern when flying is getting the cheapest pricing available. He knows of Google Flights from coming across it during on-line research but has not used the site.
- Zoom recordings of the screen to capture the participant's actions, dialogue and facial expressions while taking the test; a Voice Memos recording of the screening process, testing process and debriefing.
- Logging sheet reviews.
- Analysis of critical incidents.
- Site reviews including Google Flights, Expedia.com, Spirit.com, Flyfrontier.com
Findings and Recommendations
Task 1:
Plan a round trip
from Detroit to Atlanta for under $113.00 (or the next cheapest price) from
July 16, 2020, to July 19, 2020. Average completion time: 6.75 min.
Summary: Test participants were able
to complete the task, however, the process caused frustration, users had to
leave the site to accomplish the task and ultimately required some assistance
from Moderator.
Key Finding: P-01 and P-02 were able to
search for and select the cheapest airfare, however, there is no
easy/streamlined way to save and/or print an itinerary prior to actually
booking the flight. Google Flights presents the User with multiple “Booking
options” after the User selects desired flights: “Book on Google with Spirit”
and “Book with Spirit” (see Figure 1a). P-02 chose “Book with Spirit” and was
taken to a new tab at the Spirit website where more options and selections were
presented—basically starting the whole selection process over again (see Figure
1b). P-02 then took a screengrab of the Spirit itinerary and emailed that to
the test email site. At this point, the moderator suggested going back to
Google to try the first booking option where they selected the bright blue
“Show itinerary” link (the bright colour was eye-catching to P-02 and easy to
spot). Because there was no save or print option once on the booking page (see
Figure 1c), P-02 resorted to a screengrab in order to email the itinerary.
Being fearful of actually booking a flight during the test, P-01 printed the
web page showing the itinerary, saved as a PDF then emailed it to the test
email address but leave the site in the process.
Figure 1a: After selecting flights, the
User is presented with multiple booking options.
Figure 1b: If User selects “Book with
Spirit” new choices need to be made consequently the itinerary is created
outside of the Google system.
Figure 1c: The Google itinerary window
offers no way of saving, sharing or printing
Level of Severity for this
issue: 3 (Major
Usability Problem which is important to fix)
Heuristic violated: #7 (Flexibility and
efficiency of use).
Reason: This issue created
frustration for both Users took extra time, multiple extra steps to complete
and caused the Users to leave Google to achieve the task. While the finding and
selecting the cheapest flight took an average of 1.5 minutes to complete, finding
the itinerary, saving to an emailable format and sending it took an additional
4–5 minutes.
Figure 1d: Adding functionality to the
Itinerary window would be quite helpful to Users
Recommendation: Oftentimes people need to
save or share flight information with family members, colleagues, or for their
own future reference while planning travel. Providing the option to print, save
and share an itinerary prior to actually booking their flights would be an
efficient, time-saving, and convenient function. Also, flight tracking could be
easily and logically integrated into an itinerary-save feature. (See Figure
1d).
Task 2:
4 people from the
Chicago office want to attend a conference in New York from July 8, 2020, to
July 10, 2020. What is the cheapest total price of the trip?
Summary: Test participants were able
to complete this task easily. The average time for completion was 1 minute.
There were two minor issues found:
Issue 1: When starting a new flight
search, some settings for the previous search were retained such as “nonstop”
option and “number of passengers” (Figure 2a). While this might be convenient
when searching anew for the same trip, Google could make it more obvious for
the user to start with a new search or retain settings from a previous search.
Level of Severity for this
issue: 2
(Minor Usability Problem, fix if there is time).
Heuristics violated: #5 (Error prevention)
Reason: Unintentionally retaining
settings from a previous search could cause the system to pull unintended flight
information and yield erroneous pricing options.
Recommendation: A “Start New Search” button
such as the Spirit Airlines example (Figure 2b), which clears any previous
settings or a warning that some options are pre-test from the previous search
would at least draw the User’s attention and give them the opportunity to make
adjustments before seeing the “wrong” flight information.
Figure 2a: “Nonstop” criteria were
picked up from the previous search.
Figure 2b: Spirit’s “NEW SEARCH” button.
Issue 2: P-02 did not add in the
number of passengers (4) travelling and calculated the total price by
multiplying the single passenger fare x 4. The rationale for this was that
Expedia, the system familiar to this User, only shows the per fare price until
the User has selected flights then the total for all passengers is shown in a
sidebar.
Level of Severity for this
issue: 1
(Cosmetic Problem; No real usability impact).
Heuristics violated: #4 (Consistency and
standards).
Reason: To keep the information and
settings as clear as possible, it would be helpful to simply bring more visual
importance to the number of passengers and label the price in the flight
listing as entered by the User.
Recommendation: Instead of a drop-down menu,
a device that is more visible and interactive for the user to enter the number
of passengers and/or a simple descriptor in small type, for instance: $X.XX
total for N passengers would clarify.
Figure 2c: Google Flights drop-down menu
for passengers and the total price for 4 passengers.
Figure 2d: Expedia shows single-price
fare until reaching the Trip Summary window.
Task 3:
Your manager
wants to join the Chicago team in New York (your office is in Detroit), but
then she wants to go to London for a week then return to Detroit. She plans to
fly business class for the entire trip. What is the cheapest price for her
trip? Average completion time: 5 min BUT at the short end of the spectrum the
task was completed in 2 minutes by P-01, 8 minutes at the long end by P-02.
Summary: Users were able to complete
this task but some users overlooked the important Multi-city setting.
Key Finding: “Multi-city” drop-down
selection is easy to over-look. P-01 discovered this functionality early in the
flight search process and completed the task in 2 minutes. P-02 began by
entering one-way destinations and used paper and pen to write down pricing for
each leg of the trip (3 flights) then used the computer’s calculator to tally
the flight cost. This, of course, left more room for human error because
additional steps were taken outside of the system as well as yielded a much
higher cost for the trip, $8,093, one-way airfare as opposed to P-01’s (and
moderator’s pre-test) cheaper price of $4,477.
Figure 3a: Google Flight’s drop-down
menu for passengers and total price for 4 passengers
Level of Severity for this
issue: 3
(Major Usability problem: important to fix).
Heuristics violated: #6 (Recognition rather than
recall).
Reason: The Multi-city menu choice is
a time-saving, efficient search tool that helps the user to find better-priced
airfare. Because it is such an important feature, it should not be difficult to
find and use (see Figure 3a) Expedia.com, for example, makes this option
visible and easily accessible (see Figure 3b). By contrast, Google Flights’
Multicity selection is hidden in a drop-down menu and can be overlooked by the
User.
Figure 3b: Expedia’s Round-trip, one way
and Multi-City buttons are
clear, prominent, easy to see
when activated. Recommendation: Make this feature visible and accessible to the User,
as in the Expedia example below (Figure 3b).
Task 4:
The L.A. office
manager has a meeting in New York on August 16, 2020, at noon. She wants to
leave on August 15 after 9 am, and can arrive any time before 9 am on the next
day. How many flight options do you have? Average completion time: Average
completion time: 5 minutes, unsuccessful use of “Time” filter.
Summary: Users overlooked the “Times”
filter which increased time to completion by 6 minutes or more.
Key Finding: Neither participant completed
this task successfully by using the “Times” filter. It was not until P-02 saw
that there were several hundred flight options and became frustrated trying to
count that he spotted the option. After attempting to use the sliders, however,
P-02 became frustrated again with the lack of control over the tool. P-01 never
discovered this filter and also manually counted but her search yielded fewer
options due to having selected the “Round Trip” option. P-02 did not reset from
the “One way” setting leftover from the previous task.
Level of Severity for this
issue: 3
(Major Usability problem: important to fix)
Heuristics violated: #2, #6 (Match between system
and the real world), (Recognition rather than recall).
Reason: The “Time” filter is clearly an important tool for filtering results when scheduling trips around business meetings or other time-sensitive travel plans. The filter was overlooked by test participants and therefore needs more visual weight or attention drawn to it. Furthermore, whereas the sliders are a nice idea, they are clunky to use and frustrate a User who wants more exacting control.
Figure 4a: Google Flights drop-down menu with sliders are “clunky” and does not allow for controlled input.
Figure 4b: Expedia’s Departure time
options can be toggled on/off and are visible to.
Recommendation: Make this feature visible and
accessible to the User, as in the Expedia example, redesign so users can input
specific time criteria.
Task 5:
Help your manager
book a place to stay from August 16-18. Find the top-rated hotel that has Wi-Fi
for under $350/night in New York City.
Summary: Both Participants completed
the task successfully. Average time 7 minutes. Multiple sort options helped
individual participants filter and sort to find the top-rated hotels in their
price range which was good— it caters to Users’ individual style and approach
to searching. P-01 found the pricing to be confusing in not knowing whether the
price listed was per night or for the duration of the stay and used the “View
prices” button which opened a new window before the results had been
efficiently narrowed down in the early search phase. P-02 was frustrated, as in
Task 4, by the lack of control over the slider tool for specifying pricing. In
addition, there were two glitches in the system:
1. Selecting Amenities: each item looks like a button and can be clicked and highlighted but unless the “Apply” button at the bottom right is clicked (see Figure 5a), these chosen amenities are lost upon exiting the window.
Figure 5a: Google’s Amenities filter
requires the selection of amenities buttons PLUS clicking the “Apply” button.
Level of Severity for this
issue: 2
(Minor Usability Problem, fix if there is time).
Heuristics violated: #4 (Consistency and
standards).
Reason: It is easy to click the
amenities desired and overlook the “Apply” button at the bottom which is needed
to filter the search results. It would streamline the process for the User if
the extraneous button was eliminated.
Recommendation 1: Do away with the need for the
“Apply” button. The amenities choices are already highlight when clicked so
they should filter the results as expected.
1. Key Finding: When clicking the “View prices” button from the search list, the choices presented do not reflect the lowest price is shown in the previous search list (Figures 5b and 5c). Additionally, choosing from the “Nightly price / Nightly total/ Stay total” options change the fee structure yet again, so the search results from the initial attempts are no longer realistic (Figure 5b).
Figure 5b: First price listing with the
“View prices” button.
Level of Severity for this
issue: 3
(Major Usability problem: important to fix).
Heuristics violated: #2, #4 (Match between system
and real-world), (Consistency and standards).
Reason: When Users are looking for
the best price for any product, a lower price can draw in the User only to
cause disappointment or frustration when the price being shown is not really
what is being offered. This seems like a bait and switch tactic.
Recommendation 2: The initial price listing
should only include the lowest price that is shown as an option on the “View
prices” list. It is misleading to offer a low price, have only to have the User
discover that the price really does not exist—especially when taxes and fees
may drive the final price up even further. Users invest a lot of time and
effort navigating a system like this and the results, ultimately, should be
consistent with expectations set by the system itself.
Figure 5c: View prices reveal
significantly higher pricing options.
Figure 5b and 5c.
1. Prices displayed in the
initial search do not include any indication of whether the price shown is for
a single night or the number of nights specified by the user.
m
Figure 5d: Add a simple price
descriptor.
Level of Severity for this
issue: 1
(Cosmetic Problem; No real usability impact).
Heuristic violated: #4 (Consistency and
standards).
Reason: Neither P-01 nor P-02 was
certain that the prices they were shown by the system included both nights
being booked.
Recommendation 3: Include a brief descriptor in
small type under the price, for instance: $ X.XX for “N nights” would clarify.
Task 6:
You want to
surprise your family with a visit over Christmas but money is tight. Set up a
fare alert for a trip from Detroit to Seattle from December 22, 2020, to
December 26, 2020.
Summary: Both Participants completed
the task successfully. Average time 4 minutes. Both participants found it
fairly easy to track flight pricing by tapping the convenient “Track prices”
toggle. The small line graph icon and bright blue colour are eye-catching and
easy to spot. A problem, however, is if the user is not logged into their
Google account, when not logged in, this feature disappears entirely from the
screen as was the case for P-02. Perhaps more importantly than that, there is
no immediate feedback for the User after toggling this feature other than the
colour change and position change of the toggle. An on-screen notification or
an immediate email confirming that tracking has begun would help the User to
know that the feature is active.
Figure 6a: If the User is not logged in
the “Track prices” feature is not visible.
Figure 6b: “Track prices, toggled “on”
but no additional feedback occurs until flight prices change which could take
days to happen.
Figure 6c: Instead of not showing the
flight track tool at all, show it greyed out to prompt the User to log in or
set up an account with Google.
Level of Severity for this
issue: 2
Minor Usability problem: fix if there is time.
Heuristic violated: #1 Visibility of system
status.
Reason: The “Track prices” feature is
great for users who have time to plan for and track flight pricing. The fact
that this feature is easy to spot and conveniently toggles on and off is a
helpful and time-saving feature for Users. A message from the system verifying
flight tracking information would indicate to the User what to expect, such as
emails or notifications will be received when prices change (Figures 6a, b, c).
Recommendation 1: The system should immediately
send the User an email confirming that prices are being tracked along with a
link to the original search criteria so that the User can return at any time to
modify, cancel tracking or even book their flight.
Recommendation 2: Gray out this feature if a
user is not logged in which would prompt a person with an account to log in or
to possibly set up a new Google account if they do not already have one (Figure
6a).
Limitations
There were two limitations
affecting this test:
1. The pandemic: It was difficult
to recruit people outside of my family to participate due to the restrictions
in our area. Although I could use Zoom to screen share and record, I couldn’t
also have in-person interaction with them.
2. Having more than 2
participants would have greatly expanded the findings and led to better
information so this study is quite limited in that regard.
Conclusion
The Google Flights website is
a great alternative to travel search and booking sites like Expedia.com,
Cheapoair.com, Travelocity.com, etc. It is a natural and convenient alternative
to these sites because of the Google search interface and the number of users
who search using Google. Google Flight’s goal is to enable users to “find cheap
flights in seconds, explore destinations on a map, and sign up for fare alerts”
per the site’s tagline. This test identified areas for improvement that would
go a long way to bring the Google Flight’s site closer to those goals.
Test participants who have booked travel with other sites, who were new to the Google Flights site, and who represented a diverse population of users exposed some positive and weak areas of the site. Additionally, these participants, through their experience using other sites were able to share information that could help strengthen the Google site. By working through the following six tasks:
Search for cheap, round-trip airfare then email the itinerary to the moderator.
search for round-trip airfare for 4 passengers
search for a multi-city trip in business class
search for round-trip airfare with time constraints
search for hotel accommodations with rating, amenities, and price constraints
set up an airfare alert for a future trip
This User Test helped to
identify the following findings and make several recommendations:
Key Finding: An initial search of hotel pricing shows the lowest prices found by the system. when clicking to choose one of those prices, a new list of prices along with booking options is generated and those prices do not necessarily include the price shown in the first search. Recommendation: Do not show prices that are not actually available to the user.
Key Finding: Multi-city pre-set is hidden in a drop-down menu and overlooked by users. Recommendation: Make this option visible to users and prominently display, similar to Expedia’s design.
Key Finding: Users cannot easily save/print/share itineraries and must leave the system to achieve this goal. Recommendation: add save/print/share feature
Key Finding: Search criteria are retained from the previous search leading to unintended results when searching for new trips. Recommendation: Provide users with a “New Search” option.
Key Finding: Flight “Times” slider is “clunky” and easy to overlook. Recommendation: Redesign this tool so it is easier to find and gives the user more control over input.
“Hotel Amenities” filter requires a click selection of each item plus clicking the “Apply” button or else choices are lost. Recommendation: Redesign this feature to streamline and do away with the “Apply” button.
Prices cannot be tracked unless a user is logged in to their Google account. Recommendation: Show the “Track prices” tool even when a user is not logged in, prompt the user to either sign in or create a new account to take advantage of this otherwise useful feature.
When a user opts to track a flight, there is no immediate feedback other than a colour change on the toggle. Recommendation: Have the system send an email to verify and include a link to the flight details.
There are no labels indicating whether prices reflect totals for all passengers being booked on flights or multiple nights being booked at hotels. Recommendation: To keep pricing clear for the user, including small text labels with short price descriptors
Recommended next steps would
be a more in-depth test involving many more participants to support the findings
in this Test and offer more quantitative data.
References
Nielsen, J. (1994) Heuristic
Evaluation. In J. Nielsen. & R. L. Mack (Eds.) Usability Inspection
Methods. New York, NY: John Wiley & Sons.
Comments
Post a Comment