Information Research, Vol. 6 No. 2, January 2001
Information Research, Vol. 6 No. 2, January 2001 | ||||
|
|
|
|
|
This paper discusses some of the challenges encountered whilst researching and writing a thesis that explores individual understandings of "relevance" and "topic." It is based upon a discussion paper and presentation prepared as part of the Doctoral Workshop held during the ISIC 2000 Conference in Borås, Sweden. The focus of this paper is the doing of qualitative research. To provide a framework for this discussion, the key assumptions that have shaped the author's thesis are presented in the first section of this paper. The paper then focuses on some of the dilemmas of qualitative research encountered during the research and writing of this thesis, giving particular attention to the notion of "context" and the writing of qualitative research. Forthcoming results from the thesis are mentioned in the closing section. A Thesis Summary is also provided at the end of this paper.
While networked information systems can potentially perform tasks quickly and easily, it is not always easy to find the information we need using these systems. Relevance is a pivotal concept in this information research process, inseparable from the context in which judgements are made. However, we are still trying to understand what "relevance" means for individuals and how they make such judgements. As a contribution to understanding relevance, my thesis is an ethnographic exploration of the relevance judgements of individuals using networked information retrieval systems for task-oriented searching. The information seeking behaviour and relevance judgements of two academic researchers were observed over a period of two years. A brief thesis summary is provided at the end of this paper. To observe relevance in context, I explored their individual understandings of the problem at hand and search practices during the course of their successive searching.
Applying qualitative research methods to explorations of information behaviour requires familiarity with the tools and techniques of the qualitative researcher. The dilemmas I have encountered during my research involve issues that any doctoral student undertaking qualitative research with extensive fieldwork is likely to encounter:
As this paper will demonstrate, my attempts to resolve such issues often led to even further questions. Facing these challenges, however, has led me down a wonderful road of discovery. In this paper, I wish to share my personal experiences and reflections of this research journey.
In the next section, the key assumptions shaping this thesis are presented. Subsequent sections deal with some of the dilemmas of qualitative research that I have encountered during the research and writing of my thesis. Particular attention is given to the ways that I addressed the notion of "context" and the writing of qualitative research. Finally, while the focus of this paper is the doing of qualitative research, forthcoming results from the thesis will also be mentioned in the closing section.
Earlier research has demonstrated that there is a need to explore information seeking concepts in the broadest possible context. Process-oriented approaches to relevance assessment (Barry, 1994; Bateman, 1998; Park, 1992; Schamber, 1991; Tang & Solomon, 1998; Wang & Soergel, 1998) help us explore the varied factors contributing to searchers' judgements. The key assumptions shaping my own research include the following:
In summary, my thesis [see Thesis Summary] builds on the need to explore the broad context of searching and relevance judgements over time. Relevance judgements evolve during the process of information seeking and retrieval. To connect the varied contextual factors, I took an ethnographic approach to explore individual understandings of relevance by searchers using "networked information resources" (Klobas, 1997) for personal academic research.
Qualitative research strives to understand how people make sense out of their lives. To understand a particular social setting over time, the researcher must become the research instrument (Bogdan & Biklen, 1998). In my own work, this translates into wanting to know:
These concerns led to the research questions discussed in my Thesis Summary. /p>
I describe my research framework as naturalistic, in a manner consistent with discussions by Lincoln & Guba (1985) and Erlandson et al. (1993) of this holistic and open-ended approach:
The naturalistic researcher... does not attempt to insulate him or herself from the setting but seeks to establish relationships through which the mutual shaping of constructions is a collaborative exercise in which researcher and respondents voluntarily participate (Erlandson et al., 1993: 26).
At the same time, my intensive involvement with my informants and the way that I have been collecting data in their individual social settings can also be described as ethnographic.
At various times during my research, I have tried to make sense of distinctions and relationships between research strategies with labels like: naturalistic, ethnographic, case study, symbolic interactionist, and interpretist. Denzin (1994) contends that simplistic classifications do not work in qualitative research. Instead, he writes that:
Any given qualitative researcher as bricoloeur can be more than one thing at the same time… (p512).
Many qualitative researchers share this view. Nevertheless, as an apprentice researcher, it has not been easy to make sense of the varied labels used to describe different frameworks, strategies and approaches. Even now, when I feel more confident about moving beyond these boxes and distinctions, I am aware that my choices will influence the shape of my inquiry. For this reason, I feel it is important to continue to discuss the issue.
From the very beginning, a naturalistic researcher
…struggles to infer from the context an overall, though tentative, design that will provide direction for subsequent data collection and analysis (Erlandson et al., 1993: 39).
This is an apt description of my own experience. Taking this approach to explore my research questions compelled me to maintain a flexible approach to my research design. I was thus very encouraged to encounter Janiseck's (1994) use of the metaphor of dance to describe qualitative research:
Qualitative research design has an elastic quality, much like the elasticity of the dancer"s spine. Just as dance mirrors and adapts to life, qualitative design is adapted, changed, and redesigned as the study proceeds, because of the social realities of doing research among and with the living (p218).
Constantly asking questions and recording reflections on the process has helped shape my inquiry. My attempt to apply naturalistic principles has been both a significant challenge and a source of rich understanding about my research questions. These reflections have also made me aware of the challenge of collecting the broad range of data that emerges in a naturalistic approach whilst trying to maintain a focus on the original motivation for my research – exploring individual understandings of relevance and topic.
My observations and analysis are integrated activities – a typical situation in qualitative research. Management of such research is therefore also a common problem. When I moved towards the final stages of observation, I began to focus more on analysis of the material collected during the course of my research. It was a challenge to strike the appropriate balance between what at times seemed to be conflicting goals: remaining open to the experiences of my research participants as they unfolded, while also moving toward the timely completion of a thesis!
I am conscious of the richness of the observations I have made and the material collected during this period. However, I am equally aware of the need to manage this data so that I do not "drown" in this diversity and variety. Research has demonstrated that a variety of experiences shape the process of learning and interpreting that is associated with searching and relevance assessment (Hert, 1997; Kuhlthau, 1999; Park, 1992). An interest in relevance assessment informs my exploration, but I want to avoid forcing the data to fit with my own expectations. How can I be sure that I am allowing findings to emerge from my observations and analysis?
In the early months of struggling with these issues, I found Denzin's (1989a & b) view on the methodological principles of symbolic interaction useful. The observation of human behaviour, he explains, involves analysis which tries to capture reflections of symbols (verbal, non-verbal, intended, unintended), images, and conceptions of self. To adhere to such principles, symbols and interaction must be brought together with a primary emphasis on the process (Denzin, 1989b). This assertion suggested that I must explore the whole process of searching rather than just examine snapshots of single interactions with electronic databases. Such methodological principles were encouraging for my exploration of context.
When I began to wrestle with the notion of context as it related to my research questions [see Thesis Summary], I imagined myself attempting to "capture" the essential elements that would enable me to achieve a workable definition of the search context of the academics participating in my research. However, this approach seemed a direct contradiction to a key understanding of qualitative research when addressing context – the view that context is not reducible (Dervin, 1997). The researcher cannot assume that a summation of parts will reveal a total or complete context. It was also clear that identifying the full context of electronic searching would be an elusive, and unrealistic, goal.
Dervin (1997) discusses the methodological tools available for a contextual inquiry. She reviews social science and humanities literatures and observes that the exploration of context is a “hot” topic. Reading about this recognition of context's importance for exploring the process by which humans make sense of their worlds was inspiring. However Dervin goes on to say:
The bad news is that the very question turns out to be almost embarrassing, and certainly a question leading to a quest that demands extraordinary tolerance of chaos (1997: 13).
Clearly, I had my work cut out for me as I tried to identify the context within which I should explore my research questions.
Trying to look at the search process from the perspective of individual searchers seemed the most appropriate way to begin addressing the context of relevance assessment. To be consistent with the principles of a naturalistic inquiry, I had to find a way to collect as much information as possible. This collection needed to include all the elements comprising individual searchers' interactions with the databases they selected and the information retrieved and used as a result of their searches. If I was to adhere to the principles suggested by Denzin and Dervin, I needed to find a way to record the entire search process. I needed to find a way to describe what the searcher looks for, how they decide they have "found" it, what they look at on the computer screen, and what they collect as a result of searching. And yet it meant much more than that as well. Understanding the context of the search I would be observing required an understanding of the searcher's motivations, desired outcomes and intentions. To understand what a person looks for during a database search, I had to understand what prompts database use in the first place. I had to learn about the project on which they were working as well as what they hoped to find during a search. I found myself questioning what "natural context" meant. Where would a search process begin and end, for the purposes of my research?
Understanding human-computer interaction requires exploration of socially situated interactions. (Blomberg, 1995) Thus, it is important to study search activities in the "natural" typical settings in which they occur and develop detailed descriptions of lived experience. The focus should be on what people do and not simply on their accounts of behaviour (Blomberg, 1995; Silverman, 1998). To apply my research framework I had to define the context of the phenomenon I was trying to understand.
To analyse the dynamism of searcher requests for information in database interactions, I was going to have to look at more than relevance assessment or the request put before a system. I had to explore the whole information retrieval experience. I was really looking at how a searcher communicated with a system to find information on their topic. Yin (1994) suggests that “how” questions need to be traced over time. Once again, such methodological reasoning related well to other information seeking research. Tang & Solomon (1998), for instance, investigate how people judge the usefulness or relevance of information during a database search.
I realised that addressing context in conceptual terms was one challenge, but developing a strategy for the operational exploration of context was a very different matter. To describe the information retrieval experiences of my participants, I wanted to find a way to record all activity surrounding the search: search requests and system responses as well as the searcher's feelings, expectations and reactions. Difficulties experienced in preliminary testing highlighted the problems involved in putting the approach into practice. How could I hope to collect enough of the searchers' experiences in a "naturalistic" manner? How would I know when I had "enough"?
I decided that the most workable solution would be using a human-computer interaction (HCI) laboratory, traditionally used for software testing. Detailed observation of experienced researchers who regularly conduct searches in electronic environments is made much easier because the whole search session can be recorded from the moment a searcher enters the room until they leave. While not a "completely natural" search environment, the computer search is observed in the broader context of the search process.
As I tried to reconcile the pursuit of a natural contextual study with the constraints of such a testing environment, new questions about identifying and recording the context of a search were raised. What parts of a person's context was I actually recording during data collection? What elements were lost? What limits could I set for my research? What were the consequences of such decisions? Given that any action on my part would be intrusive, I was again left wondering what the "natural context" meant.
Equally problematic was the construction of search boundaries. Limited to reporting only what I could observe and what the searchers could tell me, I could not really claim to know where a search started or stopped. The search for information is a dynamic process (Kuhlthau, 1993) as is the judgement of the relevance of information (Schamber et al., 1990). A searcher's problem context extends well beyond the article or database entry being read at any given moment (Park, 1994). In light of this, aiming to observe the full context of searcher-system interaction seemed futile. Notions of "before", "during" and "after" could not be assumed to be inherent in the process, but rather had to be viewed as constraints imposed by me, the observer of that behaviour.
Another important question was how I should report information about the searchers I was observing. In early developmental testing, I realised that I felt very strongly about trying to provide holistic descriptions of the searchers' experiences. Previous research about relevance assessment that focused on information retrieval, in my view, reported only part of the story. Thinking about these search experiences as "stories" led me to start thinking about my research as an ethnographic work. Getting closer to human factors in information retrieval involves much more than a dialogue between system and searcher. It must include a description of the encounter that conveys the mood and feeling of the situation under observation.
The management of the rich collection of "data" described in my Thesis Summary involves a heavy emphasis on writing. Ethnographic writing contains not only the details of the observed experiences, but
…also conveys an argument and an informing context as to how these details and facts interweave (Van Maanen, 1988: 30).
Van Maanen contends that there is no "sovereign method" for writing up fieldwork, but he notes that:
The crucial problem of what we so cavalierly call "writing it up" is to balance, harmonize, mediate, or otherwise negotiate a tale of two cultures (the fieldworkers' and the others').... [T]he tacking back and forth between two cultures (or systems of meaning) will always characterize fieldwork writings (Van Maanen, 1988: 138).
Producing an ethnography requires decisions about, "...what to tell and how to tell it" (Van Maanen, 1988: 25). I have sought to find a way to effectively tell both the story of the researchers I have observed and the story of myself as a researcher. Stories relating to my research will be contained in two chapters:
I am now at the stage where writing these stories is my major concern.
Van Maanen describes one form of ethnographic writing as impressionist tales. It is a description that sits well with my attempt to provide tales of information retrieval using an individual's expressions of "topic" as the unifying theme of the descriptions. I am also exploring the searcher's experience with electronic information retrieval. The narratives developed will contain observations of what I hear, see, sense, but they are stories based on the experiences of participants. Vignettes about these experiences, e-mail correspondence, descriptions of audio and video recordings will be combined with reproductions of screen images, abstract reviews and retrieved articles. Feedback from my informants on these tales will also be encouraged.
The confessional tale [another of Van Maanen's descriptors] chronicles my own journey through the research – it is a chapter discussing my process of methodological resolving that discusses the dilemmas faced whilst conducting my research and the pathways I have tried to follow in this process.
The descriptions of the information retrieval experiences of participants are not a personal account of my own search, but that of the researchers I am observing. The research theme (relevance and expressions of topic) informs the writing of the impressionist tales. It seems essential to present these expressions of topical understandings within the context in which they are uttered – that is, within the research process each researcher participant is undertaking. Nevertheless, I am aware that I am devising an interpretive account of their experiences containing selective observations. Thus, as I work on the story of my research process and the stories of my participants, a major concern has been how to appropriately convey my interpretations of each of these participants' process of searching and evaluating information they pursue?
Despite these challenges – or perhaps because of them—my intense involvement with the two academics participating in my research provided me with a rich collection of experiences revealing aspects of user-determined relevance. The aspects of relevance described in my thesis were only possible because of the ethnographic approach that emerged from responding to searcher behaviour. Laboratory settings and document evaluations alone were not enough for a truly user-centred exploration of relevance assessment over time. Being entirely consistent with a naturalistic framework meant not only ensuring each participant determined when such searches were required but how they would search. Being true to the approach also required total flexibility and openness to the situation as it unfolded (Erlandson et al. , 1993; Lincoln & Guba, 1985).
Kuhlthau (1999) writes:
Only through sustained research do patterns and concepts emerge that allow the researcher to move beyond a particular situation to develop theory for research and practice.
Search stories developed over months of observation and participation can enrich our understanding of the nature of the information retrieval experience. Discussions that are sometimes nothing more than incidental meetings in the library or local coffee shop become a rich source of information about the individual understandings and reactions to searching and research, enhancing the data derived from observed searches and retrieved documents. User-oriented approaches must continue to examine the "situatedness" of information retrieval interactions and the ways that people work with information systems (Hert, 1997; Kuhlthau, 1999; Silverman, 1998).
Trying to "capture" context I now realise is like "taming an unruly beast," to borrow a phrase from Dervin (1997). Just like the trainer who learns to work with a wild animal, I too have learned to work with context. Trying to rein it in and collect everything about the searcher, the search and the system proved impossible. The unruly beast cannot be bent to my will. Instead, starting with a focal point and embracing the context of that situation as it continues to unfold during my research has led to understandings of relevance that are conceivable only because I am trying to understand the broader context. The longer I explore the context of the searchers participating in my research and the more I learn about the situations in which they find themselves, the more questions and uncertainties emerge. I now have a diverse collection of data: words and images in many forms and formats; database records; field notes; stories describing the experiences of searcher and researcher. All of these components are valuable for describing the processes of searching and information retrieval. If, as Dervin suggests, chaos is an inevitable feature when contending (or trying to contend) with context, I guess I must be doing something right! This chaos seems a small price to pay for the richness and diversity of data that a researcher can be privileged to collect.
Choosing a qualitative path offers the researcher a fabulous opportunity to explore the richness that is revealed in the process of doing such research. Dealing with the challenges that emerge when taking qualitative approaches like ethnography can tell us a great deal about information seeking in context. This exploration in turn can contribute a deeper understanding of the information issues examined.
In my case, addressing the issue of context meant striving to maintain a user-determined perspective on relevance. Relating expressions of a topic and search goals with system output involves looking at the process by which meaning is communicated. To explore the variety of experiences that shape this learning and interpreting (Kuhlthau, 1999), I attempted to describe as much of the contextual factors surrounding searching as possible. This persistence in turn led to rich understandings about individual assessments of "topic" and boundaries and dimensions of relevance. These observations are described in greater detail elsewhere (Anderson, 2000).
What is worthwhile noting here is the impact of sticking with my approach. Striving to maintain a user-determined approach to my exploration moved the exploration of relevance judgements away from document evaluation and closer towards a dialogue about individual understandings of the topic pursued during information seeking. These observations suggest how we might address Vakkari's (1999) concern that information retrieval research has focused more on document representations and relevance assessment of retrieved documents than on the representation of a searcher's information needs. Furthermore, focusing on an individual's research project and not just their interaction with a specific database separates relevance from retrieval and shows how that person really uses citations, abstracts and documents to build up a picture of the area they are investigating.
Writing qualitative research is a process. To stay in that process requires concentration and, according to Ely and her colleagues (1997) a "Zen-like" discipline. It requires:
…an awareness that writing will help move understanding forward (p8).
While there may be diverse motivations for writing and conflicting views on the notion of interpretation, Denzin's (1994) words are encouraging:
In the end it is a matter of storytelling and the stories we tell each other (p512).
Sharing experiences and discussing dilemmas with other researchers has helped me move through this final stage of my own thesis. I hope that telling these stories can also contribute to the learning process of my colleagues.
My thesis builds on the user-centred relevance findings of researchers like Barry (1994), Bateman (1998), Bruce (1994), Park (1992) and Schamber (1991). It is exploratory and descriptive, focusing on the searchers' interactions with information retrieval systems (human and mechanical) and the language used by them to request information and to articulate relevance assessment. Relevance is conceptualised for this study in terms of individual searchers' perceptions of the potential of particular items or document representations to solve a research problem. I sought to use a qualitative framework which is consistent with naturalistic inquiry to explore evolving understandings of "topic" as expressed by each searcher during the course of their research. This decision ultimately led to an ethnographic approach to explore the phenomena influencing individual relevance assessment.
Four research questions provided a starting point for my observations of users of networked information systems, reflecting the need to understand each searcher"s context:
However, it quickly became apparent in the study design that implementing the naturalistic framework to explore these questions required an more open-ended line of questioning than I had established initially. Thus, my questions became:
"What are you looking for?" and "How do you know when you have found it?"
Discussion and observation moved on from there indirections determined by the participants' responses and the research activities they were undertaking at the time.
My goal was to describe the process of relevance assessment and evolving understandings of a topic. Consequently, to consider the information retrieval experiences from the informants' [that is, the researchers I observe] perspective, I decided not to define relevance or topicality. Instead, I decided the individual's definition of "topic" should emerge during the inquiry. Underlying this decision was a recognition that my informants' constructs and expectations of topic are very much situated in their own experience and evolving research process. To understand these individual views, it was important to avoid imposing my own expectations – or at least try to minimise them.
Since searcher judgements were observed in the context of each individual's own information seeking and use, collecting intensive and highly detailed data at different intervals during the course of their own research projects was necessary. In addition to exploring their database and Internet searching, items used by each academic to prepare papers and presentations for their projects during this period were collected and assessed.
A large, industry-standard usability laboratory at the School of Computing Sciences at the University of Technology, Sydney (UTS) was adapted for detailed observation of searches. Such human-computer interaction (HCI) laboratories are traditionally employed for usability evaluation of hardware accessories, software programs and interfaces (Bevan & Macleod, 1994; Maguire, 1996; Nielsen, 1993). During the course of each academic's individual research project, audio and video tools recorded successive search sessions and explored decisions to reject items or take them away for further review. Using the laboratory made it possible to take advantage of detailed data capture equipment like video cameras, sound and image mixing boards and diagnostic recording tools. In this way it was easier to simultaneously observe and record all computer screen activity, think-aloud narratives, facial expressions, body language and keyboard use.
The HCI laboratory was originally intended to be the main venue for data collection, along with semi-structured interviews before and after each session. However, as indicated above, it became evident that taking an ethnographic approach to relevance assessment and information seeking behaviour needed an more open-ended approach. This resulted in the collection of a far more diverse assortment of words and images including:
These sources depict the processes of searching and information retrieval with a focus on the searchers' own understandings and language. Discussions with each academic about the evaluation and use of these items explored further their relevance assessment and led to document histories on selected material. I have also been preparing ethnographic stories similar to Van Maanen's (1988) "impressionist tales" to incorporate these searchers' own words and experiences. This combination of data has been used to build each case study of information retrieval experiences.
Using these detailed descriptions, I have been able to map the dynamism of the searchers' experiences, which has in contributed new understandings about the dimensions of relevance and the nuances of the topic being explored by each participant. Adapting usability evaluation techniques and tools to such a naturalistic framework has enabled the collection of a broad range of contextual factors associated with each individual's experiences.
I am using a qualitative framework to explore the information retrieval experiences of two individuals during the course of their own research projects. I am currently working with my transcripts, field records and other materials collected over the past two years to begin conceptualising the data. This process involves creating codes and labels for select elements of the varied components. It also involves chronicling the observed experiences of participants in cameos that are based upon my interaction with each of them and the data collected during their individual research projects. Analysis emerges from creating these ethnographic descriptions of their individual experiences.
How to cite this paper:
Anderson, Theresa Dirndorfer (2001) "Situating relevance: exploring individual relevance assessments in context". Information Research, 6(2) Available at: http://InformationR.net/ir/6-2/paper97.html
© the author, 2001. Updated: 27th December 2000
Contents |
|
Home |