Remote Usability Studies

Boxes and Arrows has published a thorough review of their experience using WebEx for
remote online usability testing. They review WebEx since it’s what they used. I’ve had some experience using Vividence so I thought it would be useful to add to the conversation.

Vividence offers online usability testing based around a downloaded Windows application which controls the user’s browser and thereby prompts the user through a series of desired actions and related questions. In the process, the download tracks all clickstreams and records time lags between clicks.

The core product of Vividence is the Vividence XMS Suite, a powerful set of tools which allows you to test up to 800 users, include conditional logic (e.g. if the user answers “Yes” to question 1, they can skip to question 3), and professional services. When I was evaluating Vividence for a corporate client I was given a very high price quote for XMS – I think it was $30,000, but I could be mistaken. This puts XMS out of the range for the vast majority of web projects.

The more affordable solution is Vividence Express, a scaled down version of the same technology with a considerably lower cost of $1,500. Express panels are limited to 50 users and two “tasks” for testing. I used Express for my project so that’s where my experience is coming from.

Caveat: The project took place about 6 months ago and I don’t have access to my original documents so some of the details are approximated. Also Vividence may have updated their product in the interim, Express was fairly new when my test was completed.

Panel Selection
Because Vividence’s methodology requires the use of a downloadable application, panel selection offers a difficult choice. They recommend use of their panel (who presumably have the download already), and for Express they include the recruitment cost in the fee. Many clients, however, would prefer to use their own customer list or intercept screens on their websites to recruit panelists. Although Vividence supports customer-generated lists, respondents have to download and install a program to take part in the testing, which is a considerable barrier to getting a clean sample group.

Vividence’s list is very convenient to use, with segmentation based on numerous criteria accomplished through the web interface. I was able to select up to 4 criteria for list restrictions, which I limited to:

  • Age range: 25 – 50
  • Career: Non-professional, no PhD’s
  • Internet Experience: Mid, no Advanced
  • Country: US

There were two problems with this list selection process. My client wanted a panel which represented the product’s audience which was roughly 75% female. Vividence Express only offered me the ability to select all female or all male, no percentage make-up. Not that big of a deal, but it illustrates the sort of compromises you have to make when using someone else’s list.

The more important problem with Vividence’s list was its accuracy. Since the criteria were provided by the panelists themselves on sign-up, there is always going to be a problem with inaccuracies. In particular, though, I found a big disparity between my desire for a panel of average Internet usage and the panel that was actually recruited. This is probably a self-selection problem since only advanced Internet users are likely to download an application and take part in interactive surveys. Whatever the reason, the panel recruited for my survey was much more advanced in terms of Internet usage than our client’s target market or probably the overall US average. When I asked the open-ended question:

“How does the design of the site make you feel?”

I got a response:

You should use more JavaScript mouse-overs.

Not exactly what I wanted from a user panel for a consumer product!

Vividence Express offers a very nice online interface for building your testing script. The system lets you create different types of survey questions (free response, radio buttons, check-boxes, etc) without any programming and offers nice extras such as the ability to randomize the order of the multiple-choice responses and include or exclude an “other” choice. Not much room for improvement here.

The tasks users are asked to complete, however, is another story. Express scripts are limited to two tasks. In our test we first asked users to

“Go to, and without clicking anywhere answer the following questions…”

The clickthru analysis provided by Vividence showed that more than half of the respondents actually clicked around quite a bit before answering the following questions. Users who don’t follow instructions are going to be a limitation to any online testing not guided by a human tester.

The second task went more smoothly, but I felt very limited by the two task limit. Yes, I was able to get some good insights into the usability of the front page and the key function of the site, but the lesser functions remained wholly untested.

Vividence’s reporting system was excellent, offering raw data, graphs, and cross-tabs as required. The click-thru analysis left a little to be desired as the system provided each user’s path information without any meaningful summaries. Clickthru data can be hard to analyze because users will come to the same pages from different routes and may back-up through previously-visited pages. It would be very helpful if they offered some better visualization tools to help give better insights in this area.

Vividence Express is probably the cheapest online usability option available but it comes with a number of fatal flaws. The panel recruitment process requires the use of a downloadable application, their internal panel has serious bias problems, and the restriction to two tasks limits the testing usefulness to very simple websites or applications.

I’d be very interested to hear from anyone else with experiences with Vividence.