Using mixed methods in online research

I’ve been thinking about online ethnographic methods a bit recently and I chanced across an interesting article entitled Combining ethnographic and clickstream data to identify user Web browsing strategies which essentially explores how you can draw together quantitative analysis of the statistics which are generated by user behaviour with qualitative investigations of the how’s and whys of that method.

The paper is interested in the ergonomics of websites. How do users interact with websites and how can we improve this esperience for them. As anyone who has ever had any level of responsibility for a website will be aware it is possible to access a vast range of statistics about people’s behaviour online. We can find out what pages people have visited, how long they have spent on a particular page, where they came from and where they are going. The problem with this kind of data is two fold, firstly it is really difficult to get hold of any meaningful benchmarking data. How long should people spend on a web page? Even if you can source an average or typical figure from somewhere, your page is different from those on other sites in its content, complexity, purpose and so on. Secondly it is difficult to use these figures to answer the why questions. Why are people on that page so long and that page so briefly? What is it that makes one popular and another unpopular. Web statistics raise issues, but they don’t really give you the tools to answer them.

The paper notes that it is also possible to gain insights into users experiences of websites by watching them and asking them questions (the paper refers to this process as ethnography, which I fear would be questionable by hardcore doctrinaire ethnographers). The study describes a three component methodology:
1. 86 students were asked to interact with a website that supported their course. The web statistics for that site were then analysed.
2. Surveys were administered to the entire cohort in week two and week eight of the course. 74 completed in week 2 and 60 in week 8.
3. Six participants were interviewed and observed during their second week of using the website in the University computer lab. Two of these were then observed in week eight using their own laptops to access the site from home.

What was found from this particular study is actually pretty interesting, even if you don’t care about web usability. One of the major findings was that the web statistics were not giving an accurate picture of the usage of the site. Students use of the “back” button was not being measured and recorded and so the web statistics were giving a misleading picture. This only became apparent when students were observed and observations compared with the data that was produced. In additon to highlighting some of the technical limitations of the web statistics, the observational data also enabled new analysis of results. So “habits of walking away from the computer during study to prepare food, visit people, or simply take a break from reading, meant that time spent on any one page became virtually meaningless as a determinant in assessing behaviour”. Previously researchers had felt that time on a page would correlate with the level of interest in that page, but observational findings suggested alternative analyses that the web statistics were not able to answer. In the main however the study found that qualitative and quantitative observations supported each other and provided a useful triangulation of results.

The article finds the mixed methods approach to be extremely powerful both in helping to analyse and explain the data that is gathered, but also in allowing for methodological development. It is only once we know the problems with the data we have been collecting , that we can begin to adjust our methodology and collect the data differently.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s