Information Research, Vol. 7 No. 4, July 2002,
Information Research, Vol. 7 No. 4, July 2002, | ||||
|
|
|
|
|
The paper reports a novel approach to studying user-system interaction that captures a complete record of the searcher's actions, the system responses and synchronised talk-aloud comments from the searcher. The data is recorded unobtrusively and is available for later analysis. The approach is set in context by a discussion of transaction logging and protocol analysis and examples of the search logging in operation are presented.
The investigation of how people use electronic information resources is important as a means of understanding how to make systems more usable and as a means of understanding information-seeking behaviour. Important developments have been the use of transaction logging data and of protocol analysis. However a trend in understanding information-seeking behaviour and in information system evaluation has been the move from controlled experiments to the study of information seeking in a natural setting. The collection of transaction logging data of end users searching full text electronic information sources in a natural setting poses numerous challenges and raises methodological concerns. This paper explores the challenges and describes a novel approach that was developed to address the limitations of traditional quantitative methodologies as applied to CD-ROM databases. The transaction logs for the research project were required to provide more qualitative data, capturing the search from start to finish as experienced and seen by end users.
The paper discusses transaction log analysis and protocol analysis as methods of data collection before outlining the novel methodology in detail, illustrating its operation with some examples of the data collected and concluding with a brief discussion on its use in a number of projects. The intention of the paper is to concentrate on the methodology itself rather than report results of specific research. Appendices are included which provide system requirements/technical detail and contact information of suppliers.
A useful definition of Transaction Log Analysis has been provided by Peters et al. (1993) who characterise it as:
...the study of electronically recorded interactions between on-line information retrieval systems and the persons who search for the information found in those systems.
Furthermore noting that its use as a research methodology is relatively new, they have divided its development into three phases:
Peters et al. have identified a variety of uses of transaction log data namely:
Transaction-logging systems are usually a feature of automated library systems, which can be classified into two broad categories:
Whilst systems of the first type may provide useful management information to librarians, it is only systems of the latter type which provide useful data for researchers because they capture and store users' entries and provide for retrieval of the stored data. Additionally some systems will also record information about the response returned by the system. Flaherty identifies typical transaction log data elements as:
Blecic et al. (1998) define transaction log analysis as "the detailed and systematic examination of each search command or query by a user and the following database result or output by the OPAC" (p.40). According to this definition transaction log systems which do not record the system response to the user input are not true transaction logging systems.
Transaction logging systems have limitations. Hancock-Beaulieu et al. (1990) noted that whilst transaction logs could provide useful diagnostic evidence of procedural and conceptual problems, they also had specific limitations. These limitations include a lack of "information or the right type of data to inform adequately on how or why users searched in the way they did" (p.11).
Other restrictions include:
Having identified the limitations of existing systems, Beaulieu et al. (1990) developed a PC-based transaction logging software package, OLIVE. This system logged the user-host dialogue, condensed and summarised the data, played back the logon screen, acted as an agent in a three-way exchange by intercepting the user-host dialogue, and was able to include pre- and post-search questionnaires. This could be adapted for use across different OPACs. This system concentrated more on "screen logging rather than a transaction logging focus" (p.73).
Kurth (1993) noted that,
...students of transaction log data have begun to ask as many questions about what the transaction logs cannot reveal as they have asked about what transaction logs can reveal. (Kurth, 1993: 98).
According to Kurth these limitations fall into four categories:
System factors, which impede the collection of transaction log data, are divided by Kurth into:
Mechanical limitations include such elements as complex programming requirements and immense data size. Other system factors are determined by the fact that "on-line systems are unique entities which exist in time and space" (p.99). Thus, changes to a bibliographic database, modifications to commands, indexing and displays will impact on the researcher's ability to replicate searches, and the same system on the same day will behave differently over a network. Intersystem factors are concerned with the development of transaction logging subsystems which vary widely, making comparisons across different on-line systems difficult.
A major "user/search process" limitation identified by Kinsella and Bryant (1987) is the inability to isolate and characterise individual users of on-line systems in order to describe the pattern of their use. Users' perceptions of their searches are not recorded, transaction logs cannot measure the information needs that users' are unable to express in their search statements (input), and they cannot reflect users' satisfaction with search results (output). As Kurth states, "[the fact] that transaction logs are unable to address such cognitive aspects of on-line searching behaviour is a true limit of the methodology" (Kurth, 1993: 100). Supplementary research, such as questionnaires, protocol analysis and interviews, must be undertaken in order to build a fuller picture of searching behaviour, success and satisfaction. Kurth identifies data analysis issues as being threefold:
Execution creates collection, storage and analysis problems related to the large size and complexity of the data set. Conceiving a method of analysis that is appropriate to both the data analysed and the research question asked is also crucial to the success of the study. The limitations discussed above combine to create a large number of variables to control when developing a methodology for analysing transaction log data. Communication problems arise when researchers:
Whilst transaction log analysis has considerable value as a data collection method, it has its limitations and it is best used in conjunction with a method which captures data regarding users' real information needs, comments and reactions whilst using a system and satisfaction with the system.
Protocol analysis has been widely used across many research areas and has stimulated considerable debate about its merits and limitations as a method of eliciting information from the participant. There has been significant development and use of verbal protocol analysis in educational psychology (for example Ericsson & Simon, 1980; Long & Bourg, 1996) and as such it is relevant to consider lessons from this area. However, think-aloud protocols have been used increasingly in information retrieval research (Flaherty, 1993; Keen & Wheatley, 1978; Markey, 1984; Hancock, 1987; Manglano et al. , 1998).
Hancock-Beaulieu et al. have argued that Ericsson and Simon's (1980) paper on verbal reports as data "has led to the use of protocols or spoken thoughts in task analysis and decision making processes in a number of ... areas" (Hancock-Beaulieu et al.1990: 6). Ericsson and Simon's central proposal is that verbal reports are data and, as such, require explication of the mechanisms by which the reports are generated, and the ways in which they are sensitive to experimental factors. As a method for understanding the cognitive processes of subjects, doubts had been raised about the validity of verbal reports (Lashley, 1923; Nisbett & Wilson, 1977). Ericsson and Simon sought to evolve a methodology for verbal reporting which would address the limitations that had discredited it by proposing a move away from informal analysis of verbalised information toward objective procedures for collection and analysis. The model they propose for the verbalisation processes of subjects is based upon the theoretical framework of human information processing theory (Newell and Simon, 1972; Anderson and Bower, 1973; Simon, 1979). Ericsson and Simon conclude that,
...verbal reports, elicited with care and interpreted with full understanding of the circumstances under which they were obtained, are a valuable and thoroughly reliable source of information about cognitive processes. (Ericsson & Simon, 1980: 247).
Long and Bourg report on three areas of research where verbal protocol analysis provides unique information:
Long and Bourg's major premise is that.
readers do not provide a veridical report of their underlying mental processes when they 'think aloud' ... rather, they construct a text representation and then use it to 'tell a story' about their understanding" (Long & Bourg, 1996: 329).
Further, Long and Bourg in discussing Magliano and Graesser's work (1991) state that "verbal protocols are an important component of the three-pronged method [for studying inferential processing during comprehension]" (p.330). They put forward that verbal protocols provide a rich and detailed database that can be used to generate hypotheses about what knowledge-based inferences occur during reading and that verbal protocols can be used to identify the knowledge structures that supply inferences. In the opinion of Long and Bourg:
verbal protocol analysis is ideal for identifying those inferences that involve access to world knowledge and those that involve access to prior text information. We are somewhat less convinced that verbal protocols provide valuable information about the memory operations that underlie these inferences" ()
The second area of study often utilising protocol analysis is the study of individual differences. Here, verbal protocols are used to investigate individual differences in readers' text representations and comprehension strategies. Long and Bourg examined several studies in this area and concluded that verbal protocol analysis led to interesting hypotheses about differences in the text representations constructed by individuals, who vary in reading skill and memory capacity. Further they noted that verbal protocols also lead to hypotheses about individual differences in learning strategies and production skills. Research into the affect of learning styles (Ford et al., 1993a and 1993b) and cognitive styles (Griffiths, 1996) on successful information retrieval may be informed by this second approach. The third research area discussed by Long and Bourg is that of conversational discourse. Long and Bourg posit that "readers use their text representation to tell a story about their understanding of a story" (Long & Bourg, 1996: 335) and, as such, verbal protocols may provide useful information about some of the principles involved in conversational discourse.
As long ago as 1978, protocol analysis was used to describe the searching behaviour of a group of experimental searchers. Keen and Wheatley conducted an experiment in which five different kinds of printed subject indexes were searched and, from transcripts of the verbalised tape-recordings, evidence of the way the entries were processed by the user was found. The test focused on one user and was designed to contrast this with two other methods for capturing a comprehensive record of the search. The two other methods under investigation were the use of detailed record sheets and marking the index copies. Keen and Wheatley concluded that "as an analogy to recording searching, verbalised recorded indexing be used as a tool for exploring the indexing process and studying indexing aboutness" (Keen and Wheatley, 1978: 430). Keen has used the think-aloud methodology in subsequent studies (Keen, 1995).
Markey (1984) utilised protocol analysis to study manual searching of a library catalogue. With minimal prompting, users were encouraged to talk aloud as they searched. Their statements were recorded onto an audio tape and transcribed to produce a flow chart of the different steps of the complete search. From these protocols, patterns of searching behaviour emerged and models of different types of searches were identified. Transcription and coding was, however, a time consuming task.
Hancock-Beaulieu et al. (1990) identified the main limitations of protocol analysis as being two-fold:
To minimise these problems Hancock (1987) devised a combined observation and talk aloud technique to study searchers using a microfiche catalogue. Subjects were encouraged to talk aloud as they searched to give an indication of what they were looking for and to confirm whether or not they succeeded. The direct observation of searchers' actions provided Hancock with the framework of the activity and in this way the searchers' actions confirmed what they said they were doing. The verbal protocol data provided details which could not be easily observed or which were not observable.
The foregoing analysis indicates that whilst there has been considerable methodological innovations in the study of user system interaction, there is scope for further improvement. Ideally we believe that transaction log analysis ought to have the following features:
We believe that the method outlined here meets these criteria and therefore constitutes a considerable step forward. The approach can utilise either of two types of technology. The first consists of a PC to video converter, called Grand Art, produced by Imagine Graphics. This equipment consists of hardware and software that was connected between a PC and monitor and a standard VHS video recorder. Once the software is installed, the screen output is unobtrusively re-routed to the video and recorded, and simultaneously displayed on the monitor to the end user (see Appendix 1 for system requirements). The second is Lotus ScreenCam software that, once loaded, records all screen activity onto disc in the form of a film. Lotus ScreenCam is the authors preferred option as it records directly to PC (see Appendix 2 for system requirements).
With both of these systems the end result is a real time film of the searching session. Each of these technologies is able to record the search session from start to finish in real time, showing pauses, movements made by mouse, input by participants, system responses (for example, error messages, retrieved results lists, instructions and prompts etc.) as well as a user's comments and as they work through their search session. Importantly, use of different electronic resources can be logged without any changes to the system. These films can be played back for analysis at a later date through a standard video player and television set or viewed directly on a PC. This method of recording the end users' interactions with electronic resources can also be described as "screen logging" (Flaherty, 1993:.73) and provides a more qualitative observational approach to transaction logging.
The benefits of using Grand Art/ScreenCam as a method of transaction logging are threefold:
1. Screen logging considerations
2. Researcher considerations
3. Practical considerations
The main limitation is that the data has to be transcribed by the researcher before analysis is possible although this allows for an in-depth knowledge of each search. The data may be qualitatively or quantitatively analysed, for example, entered into an analytical software package such as SPSS, Atlas.ti etc. Transcription may take a variety of forms dependent on the requirements of the research. For example, linear pathways may be drawn showing each action of the user and the system, thus:
Opening screen 
types "export
licence" selects F1 [help]
selects ESCAPE selects 
selects SPACEBAR [to change operator from AND to OR]
selects types "export
licences" deletes "export licences"
selects 
[to go up]
selects SPACEBAR [to change to AND] selects
[to go up] types "*" [at end of
"export licence*" ] selects F5 [search]
"SEARCHING" [2 ARTICLES
found] selects F9 [to view headlines]
selects F2 [to go back to search screen] deletes "licence*"
selects
selects types "licence*"
selects F5 [search] "SEARCHING"
[7 ARTICLES found] selects F9
[to view headlines] selects 
[to
move down to 5th article] selects
[to move to 1st article] selects F9 [to view text]
selects [to move down text] selects
F4 [to view headlines] selects F10 [to exit]
"DO YOU WANT TO FINISH? PRESS Y TO QUIT OR N TO CANCEL"
selects Y
In this example a user is searching a newspaper on CD-ROM for information to satisfy their own need. User input is denoted by "italics", user selections of system features is denoted by CAPITALS, system response by "CAPITALS" and researcher notes are enclosed by [ …]. This example is a real search by a user participating in a study currently being undertaken by Griffiths. Searches may also be presented graphically in the form of visual maps.
With Grand Art the participants' spoken comments can be recorded directly onto the same videotape as the search is being conducted by connecting a microphone to the video recorder. Recording via ScreenCam requires connection of a microphone to the PC. Thus a complete record of the search can be captured with a synchronised voice-over of the participant. Hancock (1987) observed searchers' actions in order to build a framework of activity and then used these actions to confirm what participants' said they were doing. Grand Art and ScreenCam captures all user and system activity along with participants' spoken aloud thoughts.
The following screen shots are presented to give an indication of the system in use and the manner in which transaction log data and verbal protocol data are brought together, although such a static example does not provide true picture of the benefits of the film version. The examples are taken from a recording of a user's searching session on a newspaper on CD-ROM.
The following presents an abridged example of a user's search on Chadwyck-Healey's The Guardian on CD-ROM (including The Observer) and the verbalisations made during the search. Actions taken by the user are denoted by CAPITALS and verbalisations by italics. Observational notes by the researcher are indicated by the use of square brackets.
[Start of search]
It's telling me the special keys are F1 and Escape but it doesn't tell me how to move around the page, so I'm going to guess that I have to use the arrow key. USES DOWN ARROW TO MOVE TO THE OPTION 'SECTION OF PAPER'
So which section of the paper do I want to choose? This is asking hard questions at this stage. I don't know. But it defaults to All Sections, but I actually want substantial articles. So let me, I was wondering, if I press Return. Again it's not telling me what to do. PRESSES ENTER.
Oh, so it's saying Sections means whether I want The Guardian or The Observer, not the sections within it. This time at least it's telling me to Enter to select. PRESSES ENTER. So that takes me back to [user anticipates the next screen].
Oh! [user shows surprise as the screen appears, as they incorrectly anticipated where the system would take them to]. Well now it's going on, telling me the options. Home, World, Features. Now, will Features? Again it's not clear whether Features will be both Home and Foreign/World or Features is some other category. I'm going to press F1. PRESSES F1. For the first time I feel I need to ask for Help.
Now the Help has come up [reads from screen]. The database has been organised into familiar sections - well they aren't, that's why I'm using Help [continues to read out loud and then pauses]. So I can search one or more selected sections [continues to read out loud]. I think that's so fundamental that that instruction ["You may select a section by highlighting with the up and down arrow keys, then pressing <SPACEBAR>. When you have finished your selections, press <ENTER>"] should have been displayed on screen without me having to search Help to find it. OK, so I'll press Escape to clear the message. It tells me I can read more but I think I've got the message. PRESSES ESCAPE. [The user goes on to select Home News, Foreign/World, Features and Leaders as sections of the newspaper to be searched].
So I've chosen Home, Foreign, Features and Leaders. I'm happy with those dates [pause] and now it's saying do I want to search Headline, Byline or Text. Well I want all of those [pause]. Well , where do I put my search terms in? PRESSES DOWN ARROW FOR TEXT SEARCH. Text search? PRESSES UP ARROW FOR HEADLINE SEARCH. I wanted to search Text and Headline, it's not clear, again. Help? F1 Help. PRESSES F1.
[reads Help] And all that's telling me is how to select the options, up and down arrow keys or using the first letter of the option. That's bugger all use I think. PRESSES ENTER FOR MORE HELP.
It's implying that I can only have one main menu choice, so I'll Escape. PRESSES ESCAPE.
Well it will have to be a Text Search then. PRESSES DOWN ARROW FOR TEXT SEARCH. It's not telling me whether a Text Search includes Headline searching. So, I press, [pause] it's telling me to press Spacebar to view Headlines. No, but I'm doing a Text Search. I guess I'll just have to hit Enter. Again there's nothing on screen that I can see. PRESSES ENTER.
[reads from screen] Type in a word or phrase plus Enter to search the Story Text Index. So again it, [pause] I'm not, [pause] proximity? [pause] So, I want to type privacy [user's first term] TYPES PRIVACY.
[pause] Exact Match, or Word Between, no I don't want an Exact Match. In Same Paragraph? No, is there another option? [pause] So, it's either going to be Exact Match or In Same Paragraph. Well, I just want it in the same article [pause]. F2 is giving me an explanation but that's [pause]. Am I going to have to use Help again? PRESSES F1.
[reads] It looks like I can't do what I want to do so I'll have to Escape. PRESSES ESCAPE.
I'll just have to choose. TYPES INTERNET.
Privacy space internet, In Same Paragraph. PRESSES DOWN ARROW FOR 'IN SAME PARAGRAPH'. [pause] Hit Return. PRESSES ENTER. So now it's searching.
And now it's found three stories. Press Spacebar, but then it's also saying on screen privacy and Internet in story it's found eleven stories. But it's selected three, which I'm guessing is telling me that it's only found three where they are in the same paragraph, but I've got to remember that from the screen before. I'm not very happy about that. It raises a lot of questions.
[the user continued by examining the articles retrieved from the search and then quitting the application].
The use of Grand Art or ScreenCam enables transaction logging of end user interactions and system responses across any number of different interfaces and as such it is unlike those traditional transaction logging systems which are part of library automated systems. The equipment is robust and easy to operate, all relevant data is captured on a single media format and users may be individually identified. This approach fulfils the definition of transaction logging as proposed by Blecic et al. (1998) and overcomes many of the problems of more traditional methods of logging data. A user's interaction with an information retrieval system can be viewed more fully and in greater detail than traditional transaction logging systems.
This method of data collection and analysis can be used in a variety of situations including the study of information seeking behaviour, information retrieval, IR, research (use and evaluation of systems), human computer interaction, HCI, (design, development and usability aspects) on any type of electronic resource. This approach was first developed for a research project which sought to identify end user information seeking behaviour in relation to full text CD-ROM (Griffiths, 1996). Further research (which developed from this initial project) is currently being undertaken and again utilises this transaction logging approach (Griffiths, 2002). This work aims to determine the success factors of end user searching in relation to cognitive styles, HCI and IR. Both of these projects have required extensive user testing where transaction logging has been critical and this methodology has been proven to provide rich data suitable for analysis in a variety of ways, both quantitative and qualitative. This methodological approach has also been used in research looking at web based resource use of sighted and blind/visually impaired users (Craven, 2002; Craven and Griffiths, 2002). It also has wider implications in, for example, presentation of reports, use in teaching, or pre-recorded examples of searching systems for demonstration to an audience.
http://www.keyzone.co.uk/frame_pc.html
Cost details can be obtained from the above website.
http://www.lotus.com/home.nsf/welcome/screencam
Lotus ScreenCam is released in a variety of formats for the Windows 95 and Windows NT4x platforms. Each release is designed to work for specific operating systems. There is currently no version for Windows 2000 or Windows XP. The Windows 95 release will work on Windows 98 (and variants) with some limitations (due to some video cards driver/chip set manufacturers that do not comply with Microsoft recommendations and standards).
ScreenCam for Win95 system requirements:
Systems requirements for ScreenCam are IBM or compatible PC certified for use with MS Windows 95. VGA 16-colour or higher graphics adapter and monitor. Parallel port or sound card-enabled device supported by Windows 95, speaker(s) or headphones, microphone and mouse. 10MB free space is recommended for recording.
ScreenCam for NT4.0 System Requirements:
Systems requirements for ScreenCam for WinNT4.0 are IBM or compatible PC certified for use with MS Windows NT4.0, S3 compatible graphics adapter. Parallel port or sound card-enabled device supported by Windows 85, speaker(s) or headphones, microphone and mouse. 10MB free space is recommended for recording.
Cost details can be obtained from the above website.
Contents |
|
Home |