Vol. 11 No. 4, July 2006
| Vol. 11 No. 4, July 2006 | ||||
Introduction. Reports findings from research in progress investigating judgment and decision making during information seeking in the workplace, in particular, the assessment of enough information. Characteristics of this judgment and the role of context in shaping it are framed against theories of human judgment and decision making.
Method. Thirty-three semi-structured interviews were conducted with public sector policy workers in Australia. Two interviews were carried out, the first with individual participants and the second, a joint interview with two participants. Interviews were taped and transcribed and inductive data analysis carried out.
Findings. Findings discussed in this paper focus on contextual factors that frame policy workers' judgment and decision making while information seeking, factors including ill-structured problems, shifting goals, time stress and action-feedback loops. Also revealed was the importance of developing a framework, against which the judgment of enough information can be made, and the fluid and iterative nature of these judgments.
Conclusion. The contextual factors reported show similarities with those identified by naturalistic decision making researchers, suggesting this new field of decision theory has much to offer researchers into information seeking in context.
This paper reports on ongoing research aimed at developing a deeper understanding of the concept of enough information during information seeking. Although often depicted as a linear sequence, researchers (e.g., Kuhlthau 2004; Wilson, et al. 2002 ) acknowledge that information seeking is not experienced as a neat, single-pass process and recently others (e.g., Foster 2004) have developed non-linear models to emphasise this attribute. Search closure or, to use Foster's terminology, consolidation, has received little focused research attention. While some common themes have emerged in recent empirical studies, there is much we still need to understand about the nature of the phenomenon of enough information and its relationship to what some researchers term the stopping rules which govern to decision to stop seeking further information.
Several themes related to the experience of decision making during information seeking emerged during preliminary data analysis, themes that included the relationship between the deadline and the assessment of enough information and the fluid and iterative way in which that assessment is made. Implications are discussed for an expanded understanding of how judgements and decisions are made during information seeking. Similarities with decisions studied in the field of naturalistic decision making are examined and the suggestion is made that this field of decision theory offers a new perspective from which to investigate judgements and decisions during information seeking.
Framed by a theoretical perspective situating information behaviour in context, the research conceptualised human information behaviour in a series of nested relationships (Wilson 1999). The focus of the research was information seeking behaviour, the purposive looking for information to meet a need, although it has drawn on findings from a broad range of human information behaviour literature. While judgements are made throughout information seeking, this study focused on the assessment of enough information made during the closing stages of information seeking, as a precursor to putting together a finished written work product.
Understanding that people face a series of choices throughout the information search process (Kuhlthau 2004) suggested that the field of human judgement and decision making would offer insights into the relationship between enough information and information seeking and stopping. The classical definition of a decision is choosing between two options (Hardman and Macchi 2003), with the acts of judging the situation and making a decision seen as 'parts of a multistage cognitive process' (Jungermann 2000: 587) and the decision itself a commitment to action following ongoing evaluation of options (Harrison 1999: 5). The following section frames findings from human information behaviour research which illuminate judgements and decisions made during information seeking against the major theories of human judgement and decision making.
Stopping behaviour, that is, user termination of a search for information, was investigated during the 1970s and 1980s in theoretical and experimental studies, often with a particular focus on information retrieval from databases and usually framed within theories of rational or classical decision making (e.g., Kantor 1987; Kraft and Lee 1979; Kraft and Waller 1981; Morehead and Rouse 1982) Within this classical decision making tradition, theorists developed normative models which prescribed optimal decision making behaviour (Kerstholt and Ayton 2001 ).
Researchers sought to determine rules for the 'optimal stopping point' of an information retrieval search and investigated how stopping rules of satiation, disgust or a combination of the two affected that search length (Kraft and Waller 1981: 349). Other stopping rules investigated included time constraints, diminishing returns and frustration (Morehead and Rouse 1982). Using Bayesian probability theory, Kantor demonstrated 'a very simple cutoff criterion' to end the search (1987: 211) employing both the searcher's prediction of success and their confidence in that prediction.
While these studies offered insights into the decision to stop searching for further information, the laboratory-based experiments and mathematic modelling were far removed from the real world context in which humans make judgements and decisions during search closure thus limiting their applicability to real world situations (Case 2002). As well, many of the studies focused solely on information retrieval and were based on a closed decision model.
The work of the classical decision theorists generated a reaction as researchers broadly grouped within the behavioural school of decision making recognised and studied where and how people deviated from the optimal decision making model (Newell 2005). Particularly influential were Simon's theories of bounded rationality. Although he theorised about sequential decision making, Simon's (1997) ideas on the inter-related constructs of satisficing and levels of aspiration provided insights into judgement and decision making during information seeking. Satisficing sees an individual accepting an alternative good enough to allow them to achieve the outcome they seek rather than seeking the optimal outcome. In an information context, satisficing suggests individuals stop searching at that point, even while possibly recognising that further searching may well yield additional or even better information. Importantly, satisficing occurs against pre-existing levels of aspiration, shaped by personal experience and contextual factors (March 1994; Simon 1997).
Two recent studies by Agosto and Zach have drawn explicitly on Simon's theories of bounded rationality and satisficing to understand the choices being made during information seeking. Agosto reported the anticipated time and cognitive constraints operating during web-based searching and grouped 'satisficing behaviours' (Agosto 2001: 23) into two: reduction methods to limit and organize the amount of information being dealt with, and termination methods which included satisficing (stopping when an 'acceptable' site was found, physical discomfort, boredom, self-imposed time limits) and 'snowballing' (that is 'repetition' or redundancy of information). Zach also reported time as an issue although she positioned time - together with a sense of 'comfort' (Zach 2005: 31) with the amount of information found - as a 'stopping rule'.
Other human information behaviour researchers have illuminated the experience of search/seeking closure without explicitly drawing on human judgement and decision making theories. Time available, positioned as one of the major constraints by behavioural decision theorists, is a recurrent theme (e.g., Kuhlthau 2004; Limberg 1999) 'Task, interest, information available and time' (Kuhlthau 2004: 101) all play a part in influencing the decision to end information seeking activity and, in Kuhlthau's studies of the information seeking behaviour of professionals, the context in which the work product was to be used was a factor in determining enough information. Additionally, increasing certainty signals search closure, certainty that the most important issues have been addressed (Kuhlthau 2004; Wilson, et al. 2002) and a sense that the puzzle is looking completed (Kuhlthau 2004). Increased certainty was evident too in Zach's arts administrators' 'arbitrary level of comfort' with the amount of information they had gathered and the sense of building a complete picture found parallels in empirical work by Cole (1997) and Foster (2004).
As well as time constraints and confidence, a personal investment in the quality of the assignment being completed was a further way in which students described their experiences of assessing enough information (Limberg 1999; Parker 2004).
This review reveals attempts to understand search closure have approached the phenomenon from various research perspectives and investigated different aspects of the phenomenon. Major themes running through empirical findings about the closing stages of information seeking included time and cognitive limitations in the form of deadlines and information overload; increasing certainty and confidence in being able to present a case; increasing redundancy in the information located; and a sense of having put together a complete picture. Framing the information behaviour research findings to date against theories of human judgement and decision making, we can differentiate between the action (the decision to stop or continue) and the evaluations (including the judgement of enough information) that precede that action.
To examine some of the questions arising out of the findings reported above, this study sought to understand, first, how people in the workplace made an assessment they have enough information and secondly, what influences were apparent that might shape those assessments.
Public sector policy workers of state government organizations in Australia were the participants in this research. This group and setting were selected for sound methodological reasons, as they provided a site where the phenomenon being studied could be found in abundance and where the setting was familiar ( Morse and Richards 2002; Patton 2002).
Public sector workers operate in a volatile environment 1 in which the ambiguous and complex nature of policy work (Considine 1994; Gualtieri 1999) is characterised by 'continuous work on persistent issues' (Considine 1994: 189) in a process which is convoluted and unpredictable. Policy workers are players in this process, seeking and using information throughout, both as a resource (or input) and as an output (Kirk 1999).
Diversity was apparent in both length of participants' experience in policy (from nine months to thirty years), and in the level at which they worked (from a recent first appointment to a senior executive). Diversity was also apparent in the nature of the work tasks in which their information seeking was embedded. These tasks ranged from the frequent and routine preparation of briefing papers for Ministerial meetings, albeit often on topics of which the policy workers had little existing knowledge, to projects described as once-in-a-career experiences such as the development of major ground-breaking legislation. Although some of the tasks were described by participants as routine, none fell into the categories of 'automatic' or 'normal' information processing tasks (Bystrom and Jarvelin 1995: 194). Rather the tasks, characterised often by their unexpected and unstructured nature, appeared to have more in common with what Bystrom and Jarvelin identify as 'genuine decision tasks' (1995: 195), even though the policy workers themselves were not decision makers.
Interpreting and understanding a social phenomenon - in this case, how individual workers assess they have enough information and decide to stop seeking more -required an approach that facilitated an in-depth analysis of the context and the people under investigation and mandated a research approach situated within the naturalistic paradigm (Lincoln and Guba 1985). In keeping with this naturalistic posture, qualitative data collection methods were used (Morse and Richards 2002). The research was designed as a multiple case study, an approach appropriate for exploratory, descriptive and interpretive studies (Lincoln and Guba 1985; Yin 2002). A case may be an individual or an organization, a program or some 'event or entity [including] decisions' (Yin 2002: 23); in this study, the case was the judgement of enough information.
Given that most workers do not see information seeking as a discrete and immediately identifiable activity (Kuhlthau 2004; Solomon 1997), it was important to develop a research design and techniques that would help overcome the 'hidden' nature of information seeking at work. As well, the exploratory nature of the research suggested a holistic approach was important (Yin 2002), that is, an approach that would allow participants to bring to the surface those factors important to them when assessing whether they had enough information. Much research into information seeking behaviour has focused on a small number of factors identified before the investigation. However, variables that, potentially, may influence information seeking behaviour exist on several dimensions (Fidel and Pejtersen 2004), suggesting a focus on just two or three may constrain a researcher's understanding of the phenomenon, or worse, mislead (Fidel and Pejtersen 2004). A further compelling reason for a holistic approach is that this is how individuals themselves view the activity of information seeking (Kuhlthau 2004).
Twenty one policy workers participated in the study, in a total of thirty three interviews. In all, two semi-structured interviews were conducted with each participant. The first interview was with the individual participant, focusing on their role and nature of the work being done. After Fidel and Green (2004), at the conclusion of the first interview, participants were asked to recall and describe a particular piece of work during which they had carried out information seeking activity. Also, the purpose of the second interview was explained and questions for the second interview asked, without answers being sought. The second interview was conducted with participants in pairs and focused on the closing stages of the information seeking experience. Joint interviews provided a way to increase the richness of the data gathered, with the sharing of information expected to trigger thoughts and ideas about information behaviour for both participants. These features of the research design were designed to assist participants in reconstructing the information seeking incident (Fidel and Green 2004) and overcoming the 'hidden' nature of information seeking at work.
Interviews were taped and transcribed. Analysis began when the tapes from the first interviews were listened to before the second ones took place. Following this first step in analysis, as the tapes were being transcribed, a fuller thematic analysis was undertaken., in the form of reading the transcripts and listening to the tapes to understand the concepts being discussed and identify salient concepts and their inter-relationships (Ezzy 2002). A comprehensive case analysis (Patton 2002) is still being carried out. However, the following discussion has been drawn from the initial thematic analysis.
As anticipated and foreshadowed in the earlier discussion of the research design, participants found it extremely difficult to distinguish the information seeking subtask from the broader work task (Byström and Hansen 2005). This confirmed findings from a number of earlier empirical studies (e.g. Kuhlthau 2004; Solomon 1997) and supported Vakkari's (2002: 433) findings that information seeking is 'systematically connected' to the task being carried out. Because participants often spoke of the two interchangeably, the following discussion only differentiates task from information seeking activity when participants did so.
Not surprisingly, the influence of time as a constraint was supported, with participants reporting that time constraints in the form of deadlines were very important in the decision to stop searching for more information. Deadlines drove these participants and caused them great stress: 'I was beside myself', 'I wanted to leave the country' (8) 2 ; the work is 'nerve-wracking [because] the timelines are so short now' (3). However, in spite of the importance of the deadlines, they were seen as flexible and negotiable, and often could be extended if necessary: 'But also, I think surprisingly even though you do have a deadline, often you can actually extend that deadline. It's often negotiable.'(15)
Paradoxically, the deadlines appeared to be less important in the assessment of enough information. The deadline was there and it had to be met, but sometimes, a less-that-complete product was submitted, a product that did not have enough information: 'sometimes the briefings we send over aren't very good' (3); 'you find something a few days later and you think, 'gee, it would have been nice to have known about that'' (3)
And participants expressed concern that 'we were going to have to cut corners in terms of quality in order to get it done on time' (12). So while the deadline is part of nature of the business, whether or not the product contains enough information may be assessed independently of the deadline.
Participants were conscious of the dynamic and complex environment in which they operated: 'the political climate... that can change from hour to hour' (9) and 'but the mood has changed now' (3). Collaboration appeared as a consistent theme in scoping both the task and the information seeking. In some instances, this was predetermined, with several participants involved in projects that crossed jurisdictions, increasing the number of stakeholders who held differing views about the purpose and nature of the task and adding to the complexity of what information was needed. The involvement of these multiple players resulted in experiences of information seeking that were 'coloured a lot by the processes that were going on around that [collaboration]' (12). However, even when participants had been asked to complete the task alone, they worked collaboratively, brainstorming with colleagues to create an initial framework for the task and relying on advice from colleagues and supervisors: 'I can't do this by myself' (21), and 'grabbing anyone around', to use as a 'sounding board' (21).
The tasks participants were dealing with were often vague and un-structured. So often, 'we didn't know basically what the scope of the work that needed to be done'(sic) (12). Both the tasks and the related information seeking was difficult because 'we're second guessing our Minister, the government has a public position that we don't support the initiative, but doesn't want to always be negative' (3). Further, goals were often unclear. Thus, 'I was feeling a little bit unsure about exactly what is was I was being set to do' (15) and sometimes, 'there was miscommunication about what was needed' (18) and it was difficult to ascertain 'what counts - and what doesn't' (18).
Adding to the uncertainty, participants often found themselves working in unfamiliar fields: 'it can be a topic that I personally know nothing about' (3). And even when they had knowledge of the broad field, the first important step was the scoping work needed 'to get the big picture' (21). Another characteristic of participants' experiences was the iterative nature of both task and information seeking: 'you can still, you can always reshape that, you can always shape that target' (12).
Within this dynamic, complex and fluid environment, how did policy workers assess they had enough information? How did they make the decision to stop seeking more information?
One striking theme to emerge from the data was the need for a framework against which to make the assessment of enough information. At the outset, often 'you don't have any reference points' (21) and rarely was a framework provided or suggested by the supervisor requesting the task: 'there was not a lot of guidance' (15). Participants' understanding of the purpose of the task provided some guidance to them: 'It always depends on what's driving the collection of information' (12). Also, the organization's goals provided clues when no explicit framework was provided. Tasks were approached with a 'general understanding' (21) of the agency's organizational view of policy work helping to sort through what would be important and what not.
But once that framework was established, the task became easier: 'There's a certain calming effect of actually getting the structure of whatever it is... and actually filling in the bits you can do already' (3); 'I had my framework [so it was] did I have enough information to write that bit of it, and could I substantiate why I put those bits in?' (15).
As participants worked on gathering information and working it into a draft document, they 'start[ed] to anticipate how it's going to be received... how they're going to react to it' (6). Towards the end of the task, when information had been gathered and an initial draft prepared, participants often used colleagues, supervisors and even formal reference groups as a feedback mechanism and, in some instances, relied on senior people to identify any gaps in their own assessment of enough information. Thus, participants 'gave it to someone else to read' to highlight gaps and 'if they change it, that's good in a way because... you know you're getting an idea for next time' (3).
While establishing a framework is a necessary first step and an essential part of the assessment of enough information, that framework is also fluid and changeable: 'your actual information needs have evolved throughout that whole experience' (6); 'I don't think you ever know [that you have enough]... it's a fluid thing' (9).
Much in this analysis of experiences of the closing stages in information seeking activity supported existing research findings. For example, time as a constraint was a major theme and the tactic of drawing in colleagues as sounding boards when putting the written product together was reported by Foster ( 2004) and the creation and use of a framework against which to assess enough information echoed the concept of solving a puzzle (Kuhlthau 2004) and integrating new information into existing knowledge (Cole 1997).
However, additional findings discussed below further extend our understanding of judgement and decision making during information seeking. Policy workers did not appear to experience the decision to stop seeking more information as a simple choice between two alternatives. Further, it was far from clear which, if any, stopping rules were being used.
The experience of emerging empirical findings apparently at odds with the economic and mathematical models of normative decision theory, paralleled that of researchers into naturalistic decision making, a field of decision theory research developed during the 1990s. Naturalistic decision making research arose out of dissatisfaction with the theories of both optimal and behavioural decision making as explanations for how people make decisions (Klein 1998). In the following section, key features only of the research approach are described; a full discussion of naturalistic decision making can be found in Montgomerery, et al. (2005) and Zsambok and Klein (1997).
Naturalistic decision making researchers investigate and describe how experienced people make decisions under constraints that are characteristic of real-world problem solving and decision making. Rather than conceptualising decision making as a single choice between two available options, naturalistic decision making researchers found that expert decision makers experience decision making as a process, and have increasingly focused on the role of judgement in decision making (Kerstholt and Ayton 2001). Characteristics of decision making in real world settings include contextual factors such as:
Similarities were apparent between these characteristics and the experiences described by the policy workers participating in the research, similarities such as the complex changing environment and the time stress. A summary of these similarities appears in Table 1.
| Factors | Descriptions of context by policy workers |
|---|---|
| Ill-structured problems | 'we didn't know basically what the scope of the work that needed to be done' (sic) |
| Uncertain and dynamic environments | 'the political climate... that can change from hour to hour'; 'endless briefs, some of which weren't used because events were moving so fast' |
| Shifting, ill-defined or competing goals | 'I was feeling a little bit unsure about exactly what is was I was being set to do' it was difficult to ascertain 'what counts - and what doesn't' |
| Action, feedback loops | I 'start[ed] to anticipate how it's going to be received… how they're going to react to it'; 'if they change it, that's good in a way because… you know you're getting an idea for next time' |
| Time stress | The work is 'nerve-wracking [because] the timelines are so short now' |
| High stakes | 'It's about really fundamental things'; 'It's a bit terrifying sometimes' |
| Multiple players | 'I can't do this by myself', 'grabbing anyone around', to use 'as a sounding board' |
| Organisational goals and norms | 'I suppose you do get to understand what's expected' |
In these volatile contexts, how do the experts studied by naturalistic decision making researchers make decisions? They are continually assessing and updating their understanding of a situation as part of an iterative process towards determining appropriate action (Meso, et al. 2002). By constructing and drawing on mental models of the situation, experts continually assess if that model is workable through a process of mental simulation, checking the plausibility of the model as the situation develops and changes around them and taking appropriate action (Lipshitz, et al. 2001). The experts do not evaluate and choose between a range of options before making a decision but rather take action on the first feasible model that presents itself (Lipshitz, et al. 2001). While this behaviour has parallels with satisficing behaviour, it does not appear to represent the classical definition of decision: a choice between two options.
Parallels were apparent with the experiences reported by the research participants. Policy workers reported an essential first step was scoping or assessing the nature of the work task ahead of them: 'I didn't know what the parameters was that I was working in (sic)' (27). This scoping helped them develop a framework which they felt was necessary both for getting started on the task itself and as a guide to the information needed. This framework can be seen as the mental model identified by naturalistic decision making researchers. The framework is fluid and will change over time as feedback from supervisors or colleagues is received and incorporated: 'we were working so fast and the goalposts were moving at such a rapid rate' (27), a process of continuing assessment and updating of their understanding of what was required of them. Additional inputs to this understanding of the situation come from their understanding of organizational norms: 'I suppose you do get to understand what's expected' (15). Their assessment of enough information was carried out by answering the question: Enough for what? with what being delineated by the framework they had developed and modified.
Research findings support the criticality of developing a structure for a task (Vakkari 2001) and a focus for the associated information seeking. Further, the findings suggest that this structure is also important during the closing stages of information seeking by providing that framework. It is against this framework that assessments of enough information are made.
Importantly, the similarities identified between the experiences of policy workers and the findings reported by naturalistic decision making researchers suggest their approach and the emerging theory has much to offer information seeking in context researchers. As noted earlier, where research has drawn on decision theory to understand judgement and decision making during information seeking, investigators have used classical or behavioural decision theory. However, drawing on these schools alone may limit our understanding of what is happening during search closure. For example, while much of the earlier research into stopping focused on modelling one-off sequential searches, with a choice to be made between two options, empirical findings suggest that this is not how people experience judgement and decision making while information seeking in real-world settings.
The naturalistic decision making approach offers a different perspective on human judgement and decision making. Although not without limitations, for example, the role of affect in naturalistic decision making to date remains under-researched, this perspective concentrates attention on judgement and decision making as a multi-stage and fluid process rather than a linear and one-off choice. The importance and nature of the contextual factors which shape the processes involved are also emphas
Findings from the study reported here suggest nonetheless that this emerging field can expand our understanding of how judgements and decisions are made when seeking information in dynamic and complex environments, such as the public sector workplace. Analysis of the study data continues and it is expected that findings will enrich our knowledge of an important judgement made during information seeking.
1. After Sonnenwald (1998), the fundamental concepts of context and situation are seen in an interdependent relationship, with environment, the totality of surrounding factors, not all of which may be contextual factors related to particular situations.
2. The numbers in parentheses (8) refer to the number of the interview from which the quotation is taken.
| Find other papers on this subject. |
|||
© the author, 2006. Last updated: 4 June, 2006 |
|
