Can user and task characteristics be used as predictors of success in health information retrieval sessions?

Introduction

Information on health topics started to appear on the Web during the 1990s (Pallen, 1995). With the increased availability of health information, health search on the Web started to have an impact on people’s health care routines (Fox and Rainie, 2002; Fox, 2006; Fox, 2011; Espanha and Villanueva, 2008). Regarding health information, it has been observed that the Internet is a popular source of information (Kim, 2009; Savolainen, 2008). A survey reported that, in 2012, 72% of all adults looked online for health information (Fox and Duggan, 2013). Users’ habits concerning their Web and health search have been studied in several surveys (Espanha and Villanueva, 2008; Fox and Rainie, 2002; Fox, 2006; Fox, 2011; Fox and Duggan, 2013).

Relevance represents a key concept for evaluating the effectiveness of information retrieval. Although there are several types of relevance, most of the attention has been given to topical relevance, that is, the ‘relation between the topic expressed in a query, and topic covered by retrieved texts’ (Saracevic, 1996, p. 12). Fewer studies have focused on motivational relevance, defined by Saracevic as the ‘relation between the intents, goals, and motivations of a user, and texts retrieved by a system or in the file of a system, or even in existence; satisfaction, success, accomplishment, and the like being criteria for inferring motivational relevance’ (Saracevic, 1996, p. 12).

The importance of motivational relevance is visible in the definition above. Since the aim of any retrieval system is to fulfil the user’s goals, we think this kind of relevance should not be disregarded. The objective of the present study is to analyse which characteristics influence the satisfaction of users during health information-seeking, with the help of a set of annotated Web pages that were assessed by users in a user stud (Lopes, 2013) during which they were assigned a set of search tasks. We want to address the following major research question: How can motivational relevance, that is whether a user feels successful after the search tasks or not, be predicted? More specifically, we want to determine whether (1) it can be predicted through user characteristics such as their experience with Web search, how frequently they conduct health searches or their habits regarding the terminology used during health query formulation; (2) it can be predicted though task features such as the level of clarity, simplicity and familiarity of the tasks for the users or by previous searches conducted by the user about the given tasks. Our goal is to find good descriptors and potential predictors of motivational relevance.

Literature review

The notion of relevance has been studied for decades, and the retrieval of relevant information has become the main concern of any information retrieval system (Manning, et al., 2009). Relevance is estimated by search engines and is considered ‘a measure of the effectiveness of an interaction between a source and a destination’ (Saracevic, 1975, p. 321).

The exhaustive literature review by Mizzaro (1997) introduces and discusses the concepts, types and history of relevance in information retrieval, summarising the research studies from 1959 to 1996. He gives references to the origins of the problem of finding relevant information before 1958. Regarding motivational relevance, there are references to user satisfaction and the success of the search as judged by the user, as considered measures for the evaluation of an information retrieval system (Su, 1991; 1992; 1994). O’Connor (1968) discusses the types of relevance in terms of satisfying users’ information needs. Sandore’s (1990) discovery of a correlation between user satisfaction and the precision of their search is also pointed out in Mizzaro’s study (1997), as well as other features that affect user satisfaction such as interaction with the intermediary and the library’s location (Tessier et al., 1977). Schamber (1994) also studies user satisfaction in her literature review from 1983 to 1994. Reviews the literature on the themes of behaviour, measurement and terminology, she proposes three different views of relevance (system, information, and situation views) on the basis of a classical IR interaction model; discusses recall, precision, utility, and satisfaction; describes the factors affecting relevance judgments; reports recent results of the criteria identified by the users; sketches the interdisciplinary models and the theoretical approaches to relevance, and discusses some methodological problems.

Saracevic’s first literature review of relevance from 1975 (Saracevic, 1975), was followed by two more of his research works focusing on the nature and manifestations of relevance (Saracevic, 2007) and its behaviour and effects (Saracevic, 2007). These works distinguish between user and system relevance. System (or algorithmic) relevance is defined as the ‘relation between a query and information objects in a given system, where the aim is to retrieve a set of information objects that the system inferred as being relevant to a query’ (Saracevic, 1996, p. 12). User relevance is subjective and dependent on the user and context. It is divided into four major categories: topical, cognitive, situational and motivational (Saracevic, 1996). Topical (or subject) relevance is defined as the ‘relation between the topic expressed in a query, and topic covered by retrieved texts’ (Saracevic, 1996, p. 12). Cognitive relevance (or pertinence) is the ‘relation between the state of knowledge and cognitive information need of a user, and texts retrieved’ (Saracevic, 1996, p. 12). Situational relevance (or utility) is the ‘relation between the task and texts retrieved by a system’ (Saracevic, 1996, p. 12). Finally, motivational (or affective) relevance is defined as the relation between the intents, goals, and motivations of a user, and texts retrieved by a system or in the file of a system, or even in existence; satisfaction, success, accomplishment, and the like being criteria for inferring motivational relevance (Saracevic, 1996, p. 12).

Dumais (2012) summarises various studies on the evaluation of interactive information retrieval, referring to Cole et al. (2009) with respect to motivational relevance. The authors, focusing on the evaluation of interactive information retrieval systems, propose usefulness as a measure of system performance. Belkin (2010) also suggests that usefulness could be an appropriate criterion for evaluating interactive information retrieval systems. In a previous study, we showed that user and task characteristics are good descriptors and possible predictors of motivational relevance (Oroszlányová, 2015). Relevance assessment was shown to be influenced by query, document characteristics, user and task (e.g., age, health search experience, task clarity) (Lopes, 2010).

Method

This work studies both how and to what extent task and user characteristics are useful in predicting motivational relevance. With the help of a multivariate prediction model for the motivational relevance of search sessions, we analyse which user and task characteristics influence the motivational relevance of health Web documents. A detailed description of the user and task characteristics is provided in the following subsection.

Description of the dataset

The present study is based on an existing dataset composed by a sample of 4533 annotated health Web documents. This set of documents was initially collected for a user study (Lopes, 2013), where forty participants performed eight tasks, associated with different health information-seeking situations, based on questions submitted to the health category of the Yahoo! Answers service. From the list of open questions of this category, starting with the most popular, eight questions about treatments for a symptom/disease were selected. For each question, four different search queries were defined, two in English and two in the participant’s native language. In each language, the two queries were formulated by using lay and medico-scientific terminology, respectively. Queries were built by concatenating the eight symptoms or diseases (painful urination/dysuria; head itching/head pruritus; high uric acid/hyperuricaemia; mouth inflammation/stomatitis; bone infection/osteomyelitis; heartburn/pyrosis; hair loss/alopecia; joint pain/arthralgia) with the word treatment, with different medical terminology (lay/medico-scientific). To reduce the risk of Google learning from the previously-submitted queries, it was ensured that returned links were never clicked. Further, to prevent changes in the search engine, all queries were submitted within a very short time span. For each query, the top-thirty results were collected. The documents were assessed by a set of Information Science students in terms of relevance and comprehension using a three-valued scale.

For these documents, a metadata scheme was defined and used for a latter annotation with manual and automatic approaches (Sousa, 2011). For instance, the specificity of the vocabulary (related to technical and scientific terms of the health area) was one of the document characteristics, and it was evaluated by annotating a value according to the defined scale: 1: barely perceptible; 2: perceptible; 3: completely perceptible. To evaluate the quality of the annotation, 10% of the documents were also assessed by an external health professional (Sousa, 2011). The agreement rate between both assessments was very good (93%), thus, the way the characteristics were evaluated/annotated was, in general, well defined.

Our study includes only user and task variables plus six aggregated variables containing the number of documents in each session assessed as non-relevant (nrel0), partially relevant (nrel1), totally relevant (nrel2), not understood (ncomp0), partially understood (ncomp1) and completely understood (ncomp2). In the present work, motivational relevance is assessed by a question posed at the end of the search session. In this question, users were asked to evaluate their feeling of success with the task in a five-level scale (1-extremely unsuccessful; 5-extremely successful). The metadata scheme that was used to annotate the dataset contains specific characteristics of tasks and users, listed in Table 1. Task-related characteristics include users’ feedback on the tasks’ clarity, simplicity and familiarity. User characteristics describe the user in terms of their health literacy, Web search and health search experience.

Statistical analysis

In the subsequent section Multivariate analysis of motivational relevance, we analyse multiple variables from our data collection in relation to motivational relevance. First, we selected the variables to produce a model that best fits our data, using the least absolute shrinkage and selection operator (lasso). The model selects the best subset of predictors by shrinking the regression coefficients towards zero, and estimates their coefficients based on logistic regression (James, et al., 2013). The logistic regression models the probability of motivational relevance given the task and user characteristics/variables. We can write it as probability (relevance = yes|characteristics), where the probability values p(characteristics) range between 0 and 1.

Originally, our model had a multinomial distribution with five relevance levels (0, 1, 2, 3, 4 and 5). Here we merged motivational relevance levels 1, 2, 3 and 4, 5, inducing a binomial distribution of the model.

After the lasso variable selection, we included the chosen characteristics in the multiple logistic regression model and used leave-one-out cross-validation (LOOCV) to estimate the accuracy of the model (James, et al., 2013). The LOOCV error rate is estimated by averaging the misclassified observations over the total number of observations.

The LOOCV approach splits the set of observations into a single observation used for the validation set, and into the remaining observations which form the training set, where the prediction is made for the former observation. We also built a second model, which we call reduced model. It contains only the variables which were significant in the full model. Finally, we compared the LOOCV estimates of prediction (or test) errors for the two models.

Results

Multivariate analysis of motivational relevance

This section describes how we built the prediction models, the models themselves, and their evaluation, with the aim of predicting whether or not a user feels successful after the search tasks, based on user and task characteristics. Following the statistical strategy defined in the previous section (Statistical analysis), after classifying tasks as successful (fourth and fifth level of the scale) or not (first, second and third levels), we built and evaluated two logistic regression models for motivational relevance. We built the reduced model to analyse whether we could reach similar results using a lower number of features, which would be advantageous for practical reasons.

Full model

The first model considers all variables. The analysis began by fitting a lasso model on the training set. In the next step we chose the best tuning parameter using cross-validation and used it to fit the lasso model on the full dataset (section Model definition process). Finally, with the variables selected by the lasso model, we fitted a multiple logistic regression model (section Logistic regression model) and evaluated the results (section Evaluation).

Model definition process

Applying the lasso to our dataset and using the potential predictor variables discussed in the section Method, we built a model predicting the motivational relevance. The lasso, with the minimal tuning parameter chosen by cross-validation, yielded a prediction model containing candidate variables to be analysed with the multiple logistic regression model.

Logistic regression model

The lasso helped with variable selection, and we continued the analysis with model selection using logistic regression. The resulting variables from the lasso model were added to the multiple logistic regression model which is summarized in Table 2. The first column contains the category of the feature, where T and U refer to task and user, respectively. The variables are listed in the second column, where the numbers in the parentheses indicate the levels of the variables (according to the scales defined in Table 1). The variables’ corresponding estimated coefficients are in the third column. The fourth column lists the standard error when assessing the accuracy of the coefficient estimates. In the fifth column, the z-statistics are listed, and their large (absolute) value indicates evidence against the null hypothesis of the coefficients being equal to zero. The last column contains the corresponding p-values. Those marked in bold relate to the variables that are statistically significant at α = 0.05 in the full regression model.

Evaluation

Our regression model was verified by leave-one-out cross-validation, and its results are reported in the last row of Table 2. The p-values associated with the variables, marked with bold in Table 2, are statistically significant at α = 0.05. The negative coefficients indicate that users with the corresponding features/variables are less likely to feel successful after the search task than the users without these characteristics, for fixed values of the remaining variables. Variables with large coefficient estimates highlight the importance of such variables (e.g. high level of success in Web search, and in health information search on social networks and chats, or clearly given/defined search tasks) for motivational relevance. To assess the accuracy of the model, we have fitted the model using half of the data (a training dataset), and then examined how well it predicts the held-out data (a test dataset) as explained by James et al. (2013). Using the test dataset, we then computed the probabilities of the document being relevant, allowing us to compute the accuracy, sensitivity and specificity of the model. Given these predictions, we determined how many observations were correctly or incorrectly classified. Our logistic regression has an accuracy of 91.94%, a specificity (true negative rate) of 89.71% and sensitivity (true positive rate) of 93.17%. The LOOCV estimate of prediction error from Table 2 is very low (0.067), meaning that the regression model is of high accuracy.

Reduced model

We built a second model, including only the statistically significant variables from the full model. In this second model, all variables remained significant except the one pertaining to the second level of health search in social networks. Table 3 shows the coefficient estimates for the reduced logistic regression model, i.e., for a logistic regression model that uses the selected 34 variables to predict the probability of a user feeling successful or not after completing the search task. The p-values marked with bold indicate that the corresponding variables are associated with motivational relevance and are statistically significant at α = 0.05 in the reduced regression model. The model's accuracy was assessed using leave-one-out cross-validation, with estimated prediction error of 0.0913. The logistic regression has an accuracy of 89.03%, a specificity of 86.27% and sensitivity of 90.53%. The LOOCV estimate of prediction error for this model is higher than the error estimate for the full regression model in Table 2.

Discussion

As expected, the best model to predict motivational relevance is the one containing all variables suggested by lasso, although the reduced model was very close in terms of error rates and has the advantage of not requiring as much information. In Table 4 we summarize the evaluation metrics of both models. The first row contains the number of variables included in each model. Looking at the second row we can observe that the full model has the lowest prediction error estimate (LOOCV error). The higher values of accuracy, specificity and sensitivity in the full model also support this finding.

The suggested models let us answer our research questions and were useful in understanding which characteristics are more relevant when estimating whether a user feels successful after the search tasks. The characteristics that significantly contribute to the prediction of motivational relevance, either positively or negatively, might be important for this purpose. For example, search engines might use this information to improve their performance.

The production and analysis of these models allowed us to identify important user and task features to estimate motivational relevance. We found that users who frequently conduct health search on chats or social networks feel more successful after completing the search tasks. Users who consider themselves healthy succeeded better in their search task. We ponder whether healthier people are less demanding than less-healthy people on health information seeking. The more successful a user is in Web search, the more successful s/he feels in completing the search tasks.

As expected, users feel more successful when they find more totally-relevant documents. This is reasonable, because when the user encounters useful information, s/he will feel more satisfied with the completion of his search task. It did not surprise us to discover that having an initial idea about what to search for also contributes in increasing the users’ feeling of success with the search task. Users who gave appropriate or somewhat appropriate answers to the information need after the search session are considered more successful in completing the health search tasks. This shows that users are somehow aware of the accuracy of the obtained knowledge.

On the other hand, the analyses show that users who generally conduct less frequent health search in RSS feeds, newsletters, blogs, and Web pages tend to feel less successful in completing their search tasks. We discovered that the more frequently users use medico-scientific terminology during their Web searches on health subjects, the less they will feel satisfied with their search task. This result is aligned with the findings of Saracevic (2007, p. 2136) which state that ’lesser subject expertise seems to lead to more lenient and relatively higher relevance ratings’. Considering the number of years users have been searching the Web, we also hypothesize that older users are less successful with their health search tasks. This is aligned with our suspicion that people with more health problems might be more demanding.

The findings of the present study revealed that, besides the above user characteristics, some of the task features such as the clarity and simplicity of the search task are also useful for estimating motivational relevance.

Clear and easy tasks positively contribute to the success of users within the search session. These results support our previous findings, where we observed significant positive association between the clarity and simplicity of the search tasks, user familiarity with them, their information idea, success in Web search, respectively, and motivational relevance (Oroszlányová, 2015).

There are many variables with multiple levels, which might suggest that regardless of the high values of estimates, some of the features included in the model might be less useful in real situations. Therefore, in practice, it might be useful to consider only one level at once. For instance, we might prefer the fourth level to the third one for the variable health search on chats, because its estimate is higher or because we want to make predictions for users who use chats more frequently, or we might prefer to exclude the variables with no available response (Does the user frequently conduct health search? (NULL))

Conclusions

We conducted a multivariate analysis focused on how and to what extent user and task characteristics are useful in predicting motivational relevance. For this purpose, we built two regression models. Our best model, the full model, had a LOOCV estimate of prediction error of 6.70%, a sensitivity of 93.17%, an accuracy of 91.94%, and a specificity of 89.71%. Several of the features used to estimate motivational relevance are related to users’ search habits, and to characteristics of the task such as their clarity and simplicity.

Among the variables which were identified to predict motivational relevance, a high level of success in Web search, and in health information search on social networks and chats are some of the most influencing user variables. This might show that users with higher abilities in both Web and health search might feel more satisfied and successful after completing the search tasks. In terms of tasks features, our findings suggest that users need help with performing the search tasks and with clarifying their goals and retrieved information to feel more satisfied with the search tasks.

The present study resulted in several potential predictors of motivational relevance. These features can be useful in improving the estimation of motivational relevance, particularly in the domain of health Web search. The application of our models to other datasets might be also interesting for the generalisation of our results. As future work, we aim to use the findings of this study to improve information retrieval in the health domain, for example making the search tasks clearer and easier. Since some of the features included in these models can be assessed automatically, they can be useful to improve the estimation of motivational relevance by search engines, particularly on health Web documents. Having found that the more successful a user is in Web search, the more successful he feels in completing the search tasks, we also would like to study the relationship between personality, experience and motivational relevance in more detail. For instance, will positive and confident people naturally feel more successful in Web search or in completing a health search task? We will also work on the development of methods to automatically detect these features.

Acknowledgments

This work was supported by Project “NORTE-01-0145- FEDER-000016” (NanoSTIMA), financed by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, and through the European Regional Development Fund (ERDF).

About the authors

Melinda Oroszlányová - Department of Informatics Engineering (DEI), Faculty of Engineering of the University of Porto (FEUP), Porto, Portugal - melinda@fe.up.pt
Carla Teixeira Lopes - Department of Informatics Engineering (DEI), Faculty of Engineering of the University of Porto (FEUP), Porto, Portugal; INESC TEC, Porto, Portugal - ctl@fe.up.pt
Sérgio Nunes - Department of Informatics Engineering (DEI), Faculty of Engineering of the University of Porto (FEUP), Porto, Portugal; INESC TEC, Porto, Portugal - ssn@fe.up.pt
Cristina Ribeiro - Department of Informatics Engineering (DEI), Faculty of Engineering of the University of Porto (FEUP), Porto, Portugal; INESC TEC, Porto, Portugal - mcr@fe.up.pt

How to cite this paper