Appendix 7. Papers submitted to journals

Appendix 7.3 - Information seeking and mediated searching. Part 3. Successive searching

A. Spink
School of Information Science and Technology
Pennsylvania State University, USA

T.D. Wilson, N.J. Ford, A.E. Foster, D. Ellis^*,
Department of Information Studies
University of Sheffield, UK

Our project has investigated the processes of mediated information retrieval (IR) searching during human information-seeking processes to characterize progressive changes and shifts that occur during an information seeking process. This has included information seekers' situational contexts; information problems; uncertainty reduction; cognitive styles; and cognitive and affective states. We have also sought to characterize related changes over time, and examine changes in information seekers' relevance judgments and criteria, and characterize their differences. Few studies have investigated these issues. The research has involved observational, longitudinal data collection in the U.S. and U.K. Three questionnaires were used for pre-and post-search interviews: reference interview, information seeker post-search and search intermediary post-search questionnaires. In addition, the Sheffield team employed a fourth set of instruments in a follow-up interview some two months after the search. Related search episodes, with a professional search intermediary using the Dialog Information Service, were audio taped and search transaction logs recorded. The findings are presented in four parts. Part 1 presents the background, theoretical framework, models, and research design used during the research. Part 2 is devoted to results related to uncertainty. Part 3 provides results related to successive searching. Part 4 reports findings related to cognitive styles, individual differences, age and gender. Further papers will discuss findings from this complex research project.

INTRODUCTION

As the Web, digital libraries and IR systems become more prolific for information access for many people worldwide, we need to learn more about users' interactions with IR technologies during their information-seeking behavior. The study reported in this paper seeks to contribute to the investigation of human information-seeking behavior in interactive environments and improve our understanding of users' behavior when seeking information from IR systems. In other words, users' interactions with IR systems are studied within the context of their information-seeking behaviors and the context of their search for information (Spink, 1996, 1999; Spink, Bateman & Greisdorf, 1999; Vakkari, 1999; Wilson, 1999). This paper provides data on characteristics of mediated successive online searches conducted by intermediaries using the Dialog online information service for 8 information-seekers present during the online search interaction. The reasons for successive searches, the frequency of successive searches, and characteristics of successive searches are identified. This study extends previous research that shows information-seekers often conduct successive mediated or unmediated searches over time on the same or evolving information problem (Spink, 1996; Spink, Bateman & Greisdorf, 1999).

Successive search episodes then become units for observation and analysis. A search episode is a user interaction with either a single or multiple digital information systems, i.e., CD-ROM databases, online databases, digital libraries, Web search engines, or OPACs separated by a time (hours, days, weeks, months) for evaluation of the previous search episode before embarking upon a new search episode. The modeling of users in successive searches is then successive user modeling.

Spink, Wilson, Ford, Foster and Ellis (2002) propose a theoretical framework for understanding IR interactions within an information-seeking context. The theoretical model depicts a user's situated actions within IR interactions over time. Time is represented in four categories: (1) interaction time, (2) successive searching time, (3) information-seeking time, and (4) problem-solving time. Successive searching currently receives little, if any, support from present IR interfaces and procedures, or from Web search engines. Largely, IR systems are built following a single search paradigm, i.e., they are designed and operate on the assumption that every search is an unrelated search to any previous or future searches by the user on the same or evolving topic. Some systems (such as Dialog or Lexis/Nexis) support saving searches for successive searching. However, research to improve support features for successive searching is in its formative stage.

Lin and Belkin (2000) propose a multi-dimensional conceptual model (MISE) for successive searching, including episodes, information seeking processes, information problem and problematic situation. Based on work by Schutz and Luckman (1973) they propose reasons for renewal of information seeking episodes or successive searches, including:

1. problem transmutes: the original information problem is modified and re-initiated the problematic situation;
2. problem spawns sub-problems: new concepts emerge that generate new sub-problems. Spink, Greisdorf and Bateman (1998) point to the role of partially relevant documents in a user's identification of new concepts related to their information problem;
3. problem transits problem: the original information problem transits into a different problem;
4. problem rolls back: the user is unable to clarify their information problem and the user returns to a previous information seeking stage;
5. problem with answer lost: a diversion from the original information problem that cannot be resolved;
6. problem unanswered: an interruption in the information seeking process due to lack of domain knowledge, ill-defined information problem, etc.;
7. problem cultivated: process of repeatedly searching to keep up to date in changes in a topic;
8. problem envisioned: an information seeking process is re-engaged due to external or internal pressures.

Different factors lead to the initiation, sustaining, halting and re-engagement of information seeking and searching processes. This line of research is significant as it goes beyond the one search approach generally adopted by IR researchers. The one search approach is limited by recent research that shows information-seekers with a broader problem-at-hand often seek information in stages over extended periods and use a variety of information resources. As time progresses, information-seekers' often search the same or different IR systems for answers to the same or evolving problem-at-hand. As they learn or progress in their work, or as they clarify a problem and/or question, or as their situational context changes, users come back to various IR systems for further related searches. The process of repeated, related searches over time in relation to a given, possibly evolving, information problem (including changes or shifts in beliefs, cognitive, affective, and/or situational states), is called a successive searching.

The studies reported in this paper are part of an ongoing project that seeks to investigate successive searching (Spink, Wilson, Ellis & Ford, 1998; Spink, Wilson, Ford, Foster & Ellis, 2002). The next section of the paper provides theoretical background and discusses research in IR interaction, information seeking and Web studies.

RELATED STUDIES

Successive Searching Studies

Recent studies highlight the weakness of research based on the single search approach and the need for studies that classify and categorize successive searching behavior. Studies show many IR system users conduct successive searches when seeking information related to a particular information problem.

Early Studies

Some early studies, exploring other issues related to IR systems interaction, noted that users were conducting more than one online search on a topic.

• Saracevic, Mokros, Su and Spink (1991) found that 45% academic users in their study had a previous mediated online search on the same topic, frequently with the same search intermediary.
• Huang (1992) found 19 of 44 end-users conducted successive searches.
• Robertson and Hancock-Beaulieu (1992) identified successive searches by users of the Okapi online catalog using identical or closely related search strategies.

These studies, in the early 1990's, identified a phenomenon that had not been explored previously in IR research. The common approach in IR research at this time was to examine only one search conducted by an information seeker.

Successive Searching Studies

As the 1990's progressed, accompanied by the development of information-seeking studies and models, specific studies were conducted to investigate successive searching behavior by IR system users and later by Web users.

IR Systems Interaction

• Spink (1996) conducted the first specific study of successive searching. She interviewed 200 academic CD-ROM and Online Public Access Catalog (OPAC) end-users working on a search topic and found an average of two searches per end-user. Many end-users reported conducting successive searches at different stages of their information-seeking process related to a particular information problem.
• Bateman's (1998) findings confirmed Spink's (1996) findings. She investigated the searches over time conducted by 35 end-users and found that 33 (94%) end-users conducted more than one search during their information-seeking process related to a particular information problem with an average of between 3-4 searches on their topic over time.
• In 1999, Spink, Bateman and Greisdorf studied forty-seven (47) mediated searches. They found a mean of two searches with successive searches requested by information-seekers to extend or expand, or refine the results of previous searches. Most successive searches involved changes in search terms and databases from the previous search. Precision did not necessarily increase over successive searches and the percentage of partially relevant items decreased significantly after three searches.

Web Searching

In the context of Web searching, Spink, Bateman and Jansen (1999) found that one-third of respondents to an interactive survey of Excite users were first time users, conducting their first search of Excite on their current topic; two-thirds reported a pattern of successive searches of between one Excite searches on their current topic; many reported more than five Excite searches on their topic; and 38 reported conducting more than 20 searches on their topic. Those who were beginning their information seeking reported mostly single searches.

Modeling the Successive Searching Process

Following the growth in successive searching studies, an NSF funded study by Amanda Spink (Spink, Wilson, Ellis & Ford, 1998), began a deeper exploration of the successive searching process reported in this paper. In addition to this study, various researchers began to conduct studies related to modeling successive searching.

• Vakkari, et al., (1999, 2000) and Hakala and Vakkari (2000) asked 11 Finnish users to conduct three searches each over time on their own topic. Vakkari (2000) found that users problem stages during an information seeking and searching process are connected to their choice of search term and tactics. As the searches progressed, they were characterized by more and increasingly specific search terms, more tactics and more use of operators. Vakkari and Hakala (2000) suggest that a user's relevance criteria depend on their stage of their task performance.

In summary, a growing number of studies have begun to identify the characteristics of successive searches and model the successive searching process in the end-user and mediated search context. Previous studies by Vakkari, et al., imposed a requirement of three searches on study subjects during an information seeking process. The study discussed below collected data on mediated searches conducted at the request of an information seeker to further explore and model the successive search process.

RESEARCH QUESTIONS

The research questions we addressed during this study were:

1. How frequently were successive searches are requested during an information-seeking process?
2. Why do information-seekers request successive mediated searches?
3. What are some characteristics of successive mediated searches related to a particular information problem, e.g., changes in search terms, databases?

Such research is the logical next step for research to further model the successive searching process; improve IR and Web system and interface design, and user education.

RESEARCH DESIGN

Data Collection

Three search intermediaries were recruited from University of North Texas graduate students at the School of Library and Information Sciences during the semesters from Fall Semester 1998 to Spring Semester 1999. Each intermediary had been trained in the Dialog Information Service. Each intermediary then worked with volunteer information-seekers to conduct as many Dialog searches as necessary to assist them to resolve their information problem. Many of the information-seekers were students, staff or graduate students.

The data collected during our research included: (i) search transaction logs, (ii) numerical data and responses to given questionnaires, (iii) texts retrieved and assessed relevance judgments. The research was conducted for eighteen months in the U.S. Clients were classified by broad discipline, i.e., humanities; 'pure' social sciences, such as economics, political science, sociology, etc.; applied social sciences, such as social welfare and social administration; pure science; medicine; and engineering. The numbers of humanities and medical clients were rather small and the former were incorporated into the pure social sciences group, while the latter were included in the pure science group. This gave four discipline categories.

Procedures

Pre-Search Interview: In this first interview, a detailed description of the participant's problem was obtained, together with responses to interview questions and responses to a questionnaire, which covered, for example, problem stage, Kuhlthau's stages, feelings about the progress of the work, other information seeking activities, and uncertainty.

On-line Search and Post-Search Interview: During the search, computer logs were kept, together with audiotapes of the interaction between information seeker and the search intermediary. After the search, the participants completed another questionnaire on aspects of the search and, again, on their certainty/uncertainty with regard to different stages of problem resolution. The search intermediary also completed a search assessment instrument.

Questionnaires

Three questionnaires were used to record various aspects of context that are connected to context and not record able in transactions: an information seeker pre-search (reference interview) and post-search, and search intermediary post search. The aim of the pre- and post-search questionnaires was to capture the information seeker's state in a number of areas before and after their search. This allowed the measurement of changes or shifts by information seekers resulting from their search.

RESULTS

We provide results related to the frequency of successive searches, the reasons for successive searches and characteristics of successive searching.

Frequency of Successive Searches

Table 1 lists the search topics, search frequency and reasons for the successive searches conducted for 8 information-seekers. Search topics ranged across the physical science, social sciences, humanities and medical issues.

The data in Table 1 show that 18 mediated searches were conducted, including:

• 2 in 5 information-seekers requested 2 searches
• 1 in 5 information-seekers requested 3 searches,
• Information-seekers most frequently requested 2 searches

Successive searches were generally spaced over time with some information seekers requesting a second or third search within a week and some within a month. Many reasons were identified for requesting successive searches.

Reasons for Successive Searches

Table 2 takes the data from Table 1 to show the frequency of reasons cited for successive searches.

The major reason reported by the intermediaries for conducting successive searches was the information seeker's need to refine or extend the first search based on their evaluation of the previous search results or due to changes in their information problem - including the need to search different databases or use different search terms to find more information. In some cases, there were multiple reasons for conducting more than one search.

Refine and Enhance Search Using Results From a Previous Search

All but one information seeker requested a successive search to refine or enhance the results from the previous search. This may include the use of new search terms.

Information Seeker Requested More Information

In six cases information seekers requested another search to seek more information. This reason was often related to the need to refine or enhance the results from the previous search.

Search Different Databases

In four cases information seekers requested a search using the same or modified search terms on different databases.

Refine the Search - Too Much Data Retrieved in a Previous Search

In three cases the need to refine the next search resulted from too much data retrieved during the first search that was difficult for the information seeker to evaluate.

Refine the Search Due to Increased Problem Complexity Due to Previous Search Results

In two cases the information seeker reported that the results of the previous search increased the complexity of their information problem and necessitated another search.

First Search Only Exploratory

In one case an information seeker reported that they regarded the first search as exploratory and they wanted a more refined successive search.

In four cases information seekers requested successive searches either to:

• Refine the search to print abstracts
• Intermediary suggested another search
• Lost data from previous search
• Secure more valuable information

Refining and enhancing previous search results relates to Lin and Belkin's (2000) process of "problem transmuting" or the modification of an information problem that necessitates a successive search. Further characteristics of successive searches were identified.

Characteristics of Successive Searches

Some characteristics of the successive searches were investigated, including: information-seeking stage of the information-seeker, sources of the search terms, and changes in the search terms and databases searched.

Table 3 provides data on the number of searches requested by the 8 information-seekers.

Number of Searches

All information seekers requested a second search on their topic, but only two information seekers requested a third search. Information seeker six requested three searches. The second search refined the first search strategy with new search terms and databases. A third search used new terms selected from the second search results to refine the search strategy. A similar situation emerged with information seeker seven.

Items Retrieved & Search Cycles

The mean number of terms used in second searches was significantly higher than for first searches, but higher than third searches. Second searches were characterized by more search items retrieved and more search cycles than first searches. Third searches were the longest in terms of search cycles, but lower items retrieved. First searches were often exploratory and their results were used to identify new search terms and identify new areas and databases. Even though second searches were described as "refining searches", they were more extensive and interactive in search cycles.

Search Terms

Table 4 provides results related to the search terms used during successive searches.

The mean number of search terms per search (with overlap) did not change significantly between first, second or third searches. With no overlap, the number of search terms used in second searches was significantly lower than for first searches. Second searches were characterized by more refined search terms within more search cycles and more items retrieved. The mediated search situation is obviously not too different from the end-users search situation is this respect. Vakkari, et al., (2000) found that search terms became more specific over end-users successive searches.

Changed Search Terms Over Successive Searches

Table 5 shows how the number of search terms changed over successive searches.

Overall, there was limited overlap in search terms used in first and second searches. About one in five search terms appeared in both searches, reflecting the major search term changes between first and second searches. By the third search, there was no overlap in search terms used from the second search. Information seeker six identified a number of terms from the previous results that were used in the third search.

Sequential Order of Search Terms Use Classified by Source

Table 6 provides the sequential order of search terms that were classified by the source taxonomy developed by Spink and Saracevic (1997):

• QS (Question Statement): terms appearing on the written question statement completed by the information seeker.
• UI (User-Intermediary Interaction): terms resulting from the conversation between the information seeker and search intermediary.
• I (Intermediary): terms suggested by the search intermediary
• TH (Thesaurus): terms identified in a printed thesaurus
• TRF (Term Relevance Feedback): terms identified in the retrieved items

Interestingly, none of the search terms were sourced from the search intermediary. All first searches and most searches included terms largely from the question statement and conversation between the information seeker and search intermediary. Spink and Saracevic (1997) found that most search terms during mediated searching were sourced from these sources. In two searches, a thesaurus was used to identify search terms. In two searches, term relevance feedback was used predominantly during a second and third search. Information seeker six identified many terms from the results of the second search that were used during the third search.

Information-seekers were the main source of search terms during the successive search process, although intermediaries did contribute search terms in more than 50% of searches. Spink and Saracevic (1997) also found that information-seekers were the major and most effective source of search terms during mediated online searching. Table 6 shows that most successive searches involved changes in search terms from the initial search.

Interestingly, in nearly one third of cases, successive searches involved the use of the same search terms, with no additions and deletions of terms from previous searches. This finding supports previous studies (Spink, 1996: Robertson & Hancock-Beaulieu, 1992) of OPAC end-users who frequently changed their search terms between successive searches. Spink, Bateman and Jansen (1999) also found a similar result with end-users conducting successive searches of the Web.

Search Operators

Table 7 shows how search operators were used across successive searches.

More search operators were used during first searches than later searches. The only exception was the NOT operator that was primarily used during second searches as part of the refining process. The mediated search situation is obviously different from the end-users search situation is this respect. Vakkari, et al., (2000) found that search operators were used by end-users later in their later searches. Search intermediaries are distinguished by their training in Boolean searching and seem more likely to use Boolean operators from the initial search.

Search Commands

Table 8 and Table 9 provide a summary of the commands analysis.

During mediated searches, the second search was characterised by greater use of more commands and tactics, except for the select command. The mediated search situation is similar to the end-users search situation is this respect. Vakkari, et al., (2000) found that more commands and tactics were used by end-users in their later searches.

Databases

Table 10 shows the number of successive mediated searches that involved a change in databases.

In six cases the databases used changed over successive searches. Interestingly the same databases were often repeatedly searched (with either the same or different search terms) over successive searches. Successive searches often included a change in both search terms and databases. From the data we can see that successive searches involved changes, refinements or extensions from the initial search. How the changes evolved depended upon the nature of the information problem and the information-seeking stages and changes experienced by the information-seeker due to the results from previous searches.

Information-Seeking and Problem Solving

Information-seekers were asked to indicate if their problem solving and information seeking processes before and after each search (Table 11).

The data show that information seekers often experienced shifts in their information seeking and problem solving stages during and between successive searches. For example, after Search 1, most information seekers reported being in the same problem solving stage as before their search. Concurrently, more information seekers reported a change in the information seeking stage as a result of the first mediated search. Some information seekers reported shifting back to a previous stage or "problem rolls back" (Lin & Belkin, 2000). Other studies (Spink, 1996; Spink, forthcoming; Spink & Wilson, 1999) show that information seekers experience shifts and changes in their information seeking and problem solving processes due to their interactions with IR systems and subsequent changes in their information problems.

Spink, Greisdorf and Bateman (1999) also identify relevance judgments as an important element in successive searching.

Relevance Judgments

Table 12 shows data concerning the relevance of the retrieved items during successive searches.

An analysis of the data in Table 12 shows that:

DISCUSSION

The results of the study reported in this paper extend previous research by identifying additional characteristics of successive mediated searches. On average, information-seekers requested two searches, and some information-seekers requested three mediated searches. Successive searches often involve a refinement or extension of previous searches, with new databases searched or search terms changed, as the information-seekers' understanding and evaluation of results evolved over time from one successive search to the next. Some successive searches involved no change in databases.

Theoretically, most interactive IR models and studies of IR system use should take account of the reality that users not only iterate their queries, but also their searches over time. The integration of interactive IR models with human information behavior models is presented in Spink (1999) and can be extended by adding successive searching processes.

In practice, information-seekers and intermediaries should be trained to understand that many information problems are not resolved with one IR system search. The picture is more complex. On average, information-seekers may need to conduct more than one search or possibly 2-3 searches during their information-seeking process - just to provide focus and clarity to their information needs. Modifications to information-seeker and intermediary training will be required to account for the reality of this need.

CONCLUSION & FURTHER RESEARCH

This study highlights many issues that need further research in interactive information retrieval. We need to examine more characteristics and processes associated with successive searching. Does the stage of an information-seeking process that the seeker has reached, or the time elapsed, have an impact on the number of successive searches undertaken? Further research is also required to examine the factors more deeply that compare and characterize information problems that satisfy the user with a single search as opposed to those information problems that lead to successive searches. Currently, most IR systems and interfaces do not greatly assist users during successive search episodes.

ACKNOWLEDGEMENTS

We thank referees for their useful comments on the paper.

This is a draft of a paper published in Journal of the American Society for Information Science and Technology, Volume 53, No. 9, 2002, 716-727

Note

David Ellis is now Professor, University of Wales, Aberystwyth

REFERENCES

• Bateman, J. (1998). Changes in users' relevance criteria: an information seeking study. Unpublished Doctoral Dissertation. School of Library and Information Sciences, University of North Texas.
• Huang, M. H. (1992). Pausing behavior of end-users in online searching, Unpublished Doctoral Dissertation. University of Maryland.
• Ingwersen, P. (1996). Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory. Journal of Documentation, 52(1), 3-50.
• Komlodi, A. (2000). Search history for user support in information seeking interfaces. Doctoral Consortium and Poster. Extended Abstract of ACM CHI 2000: Human Factors in Computing Systems Conference. The Hague, The Netherlands, April 1-6,2000.
• Lin, S-J. & Belkin, N. J. (2000). Modeling multiple information seeking episodes. Proceedings of the Annual Meeting of the American Society for Information Science, November, 2000, Chicago, IL (pp. 133-147).
• Robertson, S. E., & Hancock-Beaulieu, M. M. (1992). On the evaluation of IR systems. Information Processing and Management, 28(4), 457-466.
• Robins, D. (2000). Shifts in focus on various aspects of user information problems during interactive information retrieval. Journal of the American Society for Information Science, 51(10), 913-928
• Saracevic, T., Mokros, H., Su, L., & Spink, A. (1991). Interaction between users and intermediaries during online searching. Proceedings of the 12th Annual National Online Meeting, 12, 329-341.
• Schutz, A., & Luckman, T. (1973). The structures of the life-world. Translated by R. M. Zaner & H. T. Englehadt, Jr. Evanston, IL: Northwestern university Press.
• Spink, A. (1996). A multiple search session model of end-user behavior: an exploratory study. Journal of the American Society for Information Science, 46(8), 603-609.
• Spink, A. (1999). Toward a theoretical framework for information retrieval within an information- seeking context. Proceedings of the 2nd International Information-seeking in Context Conference, August 12-15, 1998. Department of Information Studies, University of Sheffield. Sheffield, UK.
• Spink, A. (Forthcoming). An evaluation study of the Inquirus web meta-search tool. Information Processing and Management.
• Spink, A., Greisdorf, H., & Bateman, J. (1999). A study of mediated successive searching during information seeking. Journal of Information Science, 25(6), 477-487.
• Spink, A., Bateman, J., & Jansen, B. J. (1999). Searching the Web: Survey of Excite users. Internet Research: Electronic Networking Applications and Policy, 9(2), 117-128.
• Spink, A., Greisdorf, H., & Bateman, J. (1998). From highly relevant to not relevant: Examining different regions of relevance. Information Processing and Management, 34(5), 599-622.
• Spink, A., & Saracevic, T. (1997). Interaction in information retrieval: Selection and effectiveness of search terms. Journal of the American Society for Information Science, 48(8), 728-740.
• Spink, A., & Wilson, T. D. (1999). Toward a theoretical framework for information retrieval (IR) evaluation in an information seeking context. Proceedings of MIRA 99: Evaluation Frameworks for Multimedia Information Retrieval Applications, Department of Computing Sciences, University of Glasgow. Scotland, April 14-16, 1999 (pp.75-92).
• Spink, A., Wilson, T. D., Ellis, D., & Ford, N. (1998). Modeling users' successive searches: A National Science Foundation/British Library Study. D-Lib Magazine (April 1998).
• Spink, A., Wilson, T. D., Ford, N., Foster, D., & Ellis, D. (2002). Information seeking and searching study. Part 1. Theoretical framework and research design. Journal of the American Society for Information Science and Technology, 53(9) 695-703 [Electronic copy available at http://informationr.net/tdw/publ/unis/app7.1.html (11 Nov 2002)]
• Vakkari, P. (2000). Cognition and changes of search terms and tactics during task performance: A longitudinal case study. Proceedings of RIAO 2000 Conference, Paris, April 12-14.
• Vakkari, P., Hakala, N. (2000). Changes in relevance criteria and problem stages in task performance. Journal of Documentation.
• Vakkari, P. (1999). Information source use as a function of organizational information requirements, task complexity and information types: Integrating theories of media richness, task complexity and uncertainty of information-seeking. Proceedings of the 2nd International Information-seeking in Context Conference, August 12-15, 1998. Department of Information Studies, University of Sheffield. Sheffield, UK (pp. 35-54).
• Wilson, T. D. (1999). Models of information-seeking behavior research. Proceedings of the 2nd International Information-seeking in Context Conference, August 12-15, 1998. Department of Information Studies, University of Sheffield. Sheffield, UK (pp. 55-66).
• Xie, H. (2000). Shifts in interactive intentions and information seeking strategies in interactive information retrieval. Journal of the American Society for Information Science, 51(9), 841-857.