Appendix 1: The data-collection instrumentsOutline
A: Pre-search instruments
Reference InterviewIntroduction: introduce yourself to the client, and explain that you will be wanting to know not only some further information on their topic, but also information on such things as the previous searches for information, if any, that they have conducted. 1. Client. s description of the problem and nature of the enquiryBefore we begin the reference interview I. d like you to tell me in your own words a little about your problem and where you are in relation to it. Interviewer to probe once the client has made a statement to gain information about the nature of the topic, how it fits into the person. s research generally, what the context is, how significant is the work to you - personally? What pressure, if any, are you under to carry out this work? what the purpose of the search is and whether the topic is based in a single discipline or is multi-disciplinary - In either case, which discipline(s) is/are involved? NB: Recap on this description of the problem at the end of the reference interview. [Space for response] 2. Problem-solving stage.To determine at what point the person is in their problem solving you should ask - 2.1 "Is this a new problem area for you?" oNo - please go on to question. 3.2 o Yes - Is your problem precisely defined or is it still fuzzy in any way? oPrecisely defined 2.2 What stage are you at in terms of defining or resolving the problem, or in presenting the answer? Please respond using one of the categories indicated below. If no single category sums up where you are in your work please indicate the approximate position. [Interviewer to probe if client doesn. t specify one of the categories below to determine problem phase] Problem recognition: I am trying to determine whether or not the topic I. m interested in is a real problem from the point of view of the discipline or area that interests me. I need a search so that I can discover whether others have identified the same issue as problematical. Problem definition: I have identified a real problem and now need to define it more closely or carefully so that I can determine how to approach the problem and how it relates to other problems in the field. I need a search to help me define my research objectives. Problem resolution: I am in the process of resolving the problem (engaged in laboratory experiments, field work, etc.) and now need a search to enable me to proceed with and complete that work. The question deals with a particular problem that I need to resolve in completing the work or with aspects of methodology, research approach or research methods. Solution statement: I have effectively finished the work I was doing and am either tying up loose ends, or finding out from related work how best to report my research or where best to report it. 3. Client. s indicators of information seeking contextOn the scale below, can you indicate how would you describe the degree to which your intended use of the search results was well defined to start with?
4. State of personal or internal knowledgeA person. s knowledge about the broad domain and/or the specific problem at hand and its various aspects may vary. The next three questions ask you in your own estimate, how much you know about the broad domain and the specific problem that prompted you to submit the question. 4.1 Broad conceptual knowledge of the domain On the scale below how would you rank the state of your knowledge in relation to the broader domain to which your problem is related?
4.2 Specific knowledge or expertise of the problem On the scale below please indicate how you would rank the specific knowledge or expertise you feel you possess in relation to the problem at hand?
4.3 Familiarity with the language or terminology used in the problem or domain A problem, as well as a domain, tends to use terminology specifically defined for its own purposes. On the scale below please indicate how familiar you are with the specific domain or problem-oriented terminology in current use?
5. Information Seeking Behaviour5.1 Looking again at the stage you are at in your work, could you say which of the following terms best defines what you are doing?
5.2 How would you describe your thinking about the problem at this stage?
5.3 How would you describe your level of interest in the problem at this stage?
5.4 And finally, in general, how do you feel about the progress of your work at this point?
6. Information Seeking ActivitiesWhich of the following activities is important in your information seeking at this stage? 6.1 Following chains of citations or other forms of referential connection between documents.
6.2 Browsing or semi-directed searching in an area of potential interest.
6.3 Differentiating sources of information on the basis of the nature and quality of the material examined.
6.4 Maintaining awareness of developments in relation to this topic through the monitoring of particular sources.
6.5 Systematically working through a particular source to locate material of interest.
6.6 Verifying or checking the accuracy of information.
6.7 At this point in your search which of the following have you actually engaged in? (Please tick as many as necessary)
7. Uncertainty.(Client to fill this in on Client Response Form - filename clientc.doc) "I. d now like to ask you about how sure you are about the problem you are investigating and about the probability of useful information being found. Could you reply to each of these points in the way shown on the attached card?" Please respond to each of the items below by marking a cross (X) on the line to indicate your level of uncertainty regarding the item. For example: Very uncertainVery certain will indicate that you feel rather uncertain about an item. How certain are you: a) that you have recognised a real issue to investigate? Very uncertainVery certain (Prompt for uncertainty question (a) a real problem is - A problem that you think needs to be solved) b) that you have defined the topic appropriately? Very uncertainVery certain c) that the issue can be resolved? Very uncertainVery certain d) that an effective way of presenting the results can be found? Very uncertainVery certain e) that relevant information is available and can be found? Very uncertainVery certain 8. Information seeking.8.1 "Can you tell me what you have done so far to find information on this problem? That is, what journals you have been scanning, what personal contacts you have been following up, whether you have done a search on BIDS or on the campus CD-ROMS or the World Wide Web yourself, etc." [Interviewer: Probe to get the names of key journals that the client is aware of, which other information sources s/he has used, or regularly uses, which authors work regularly in the field or who are frequently cited, etc.] Did the searching involve (checklist for interviewer - client can indicate as many as necessary):
8.2 If you have carried out previous searches, how successful was the previous search or searches in finding desired information? (Client to fill this in on Client Response Form - filename clientc.doc)
8.3 Did any of the above previous searches contribute to the formulation of your present question?
8.3 Did anything arise out of that previous search that would be of use to me in carrying out a search for you? [Space for response] 9. Comprehensiveness of the search:A search could be varied as to its comprehensiveness, from retrieving a few potentially relevant or representative items, to retrieving as many potentially relevant items in the database as possible. On the scale below, please indicate how comprehensive you would like the search to be:
10. Other aspects.Language: are you able to use any material I might find in languages other than English? Date: how far back in time would it be useful to take the search? Five years, ten years, longer? How recent would you like the material to be. Kinds of information: are there any particular kinds of information that will be of interest to you other than journal papers? For example, laws and statutes, standards, patents, unpublished reports, conference proceedings? Or should any of these be specifically avoided? Sources to avoid: are there any sources (such as particular) journals that you already know are unlikely to be of value to you? Authors: are there any authors or researchers to avoid because they take a different approach from yourself? Or that ought to be included for the same reason? Approaches: are there particular approaches to the problem that I should avoid because they are inappropriate for the approach you are taking yourself? 11. Finally:Are there any other aspects of your work, or of this problem that might be of help to me in carrying out a search? 12. Recapitulation of the search problem and desired solutions.(NB: At the end of the
reference interview read out statement made at the start. Demographics
4. Age: ___ 5. Gender: Male___ Female___ Previous research has shown that the age of the information seeker may have an effect on other variables in information seeking behaviour. So I. d like to ask you to state your age for me. (Age - has been shown to be positively related to knowledge within a field.) 6. Research Student Is the project you are working on one that you identified yourself or one that your supervisor identified? Yes ____ No____ (To be completed after the reference interview by the interviewer - ) Classification of the client. s questionThis form is to be completed by the searcher prior to going on-line to conduct the search. We assume that the question has certain attributes or characteristics above and beyond its subject content and that the searcher can recognise these on the basis of the written question statement. 1. Clarity How clear was the question to start with in terms of semantics (meaning of terms) and syntax (relationship or logic between terms)?
2. Specificity How specific in terms of the majority of terminology involved was the question?
3. Complexity How complex was the question in terms of the number of search concepts likely to be involved (where a search concept could be translated into any number of related, near-synonyms or synonymous terms)?
Return to the Top of the Page B: Post search instrumentsAssessment of the search
(Asked immediately after the search) (The interviewer will ask:) Could you identify your main criteria for judging the relevance of items presented to you during the search? Example In some questions you will be asked by the interviewer to indicate your answer by marking a cross on the line. An example of this type of question is provided below. Please respond to each of the items below by marking a cross (X) on the line to indicate your level of uncertainty regarding the item. For example: Very uncertainVery certain will indicate that you feel rather uncertain about an item. Assessment of the search In the following questions I would like you to give us a brief indication of the search in relation to the items on this questionnaire. 1. Effectiveness of communication 1.1 Your explanation (as the client) to the searcher On the scale below please mark on the scale the position that indicates how well you think that your original explanation of the problem was effective?
1.2 Searcher. s explanation Please indicate on the scale below how well the searcher explained to you what was going on in the search.
1.3 Understanding by you as a client On the scale below could you please mark how well you understood what was going on?
1.4 Understanding by the searcher On the scale below could you indicate to what extent you think the searcher gained an understanding of your problem? Can't judge _____
1.5 Non-verbal communication On the scale could you indicate how far the searcher. s non-verbal communication contributed to the effectiveness of your interaction with the searcher? (Prompts: nodding, smiling or lack of smile, silence, eye-contact, non-verbal sounds etc).
2. Changes due to interaction 2.1 Changes in perception of the problem On the scale below can you indicate any changes you have made in your perception of the problem from the outset of the search?
If yes, what were the changes? Were the changes brought about by the search? 2.2 Changes in the question Have you made any changes in your question from the outset of the search? (Changes may include but are not limited to clarification, changes in terminology or level of specificity etc.)
If yes, what were the changes in your question? 2.3 Changes in personal knowledge On the scale below please indicate if any changes occurred in your personal knowledge, of the specific problem at hand, due to the interaction and/or the feedback during the search?
If there were any changes, what were they? 2.4 Changes in relevance judgement Please indicate on the scale below if you made any changes in your criteria for relevance judgements of the items retrieved due to the interaction and /or the feedback during the ongoing search?
If you made some changes in your relevance judgement what were they? 3. Uncertainty When you were first interviewed you were asked about the certainty of your knowledge about certain aspects of the problem. I. d like to ask you again about the same issues now we. ve carried out a search. Could you reply to each of these points in the way shown below? Please respond to each of the items below by marking a cross (X) on the line to indicate your level of uncertainty regarding the item. For example: Very uncertainVery certain How certain are you: a) that you have recognised a real issue to investigate? Very uncertainVery certain b) that you have defined the topic appropriately? Very uncertainVery certain c) that the issue can be resolved? Very uncertainVery certain d) that an effective way of presenting the results can be found? Very uncertainVery certain e) that relevant information is available and can be found? Very uncertainVery certain Research preference questionnaire Example. In some questions you will be asked by the interviewer to indicate your answer by marking a cross on the line. An example of this type of question is provided below. Please respond to each of the items below by marking a cross (X) on the line to indicate your level of uncertainty regarding the item. For example:
1.
2.
3.
4.
5.
6.
7.
Return to the Top of the Page C: Searcher assessment of the search
Post search assessments by the searcher(This form is to be completed by the searcher after the conclusion of the interaction and the search.) 1. Problem definition How well do you think the problem was defined by the user to start with? On the scale below please show how you would describe the user's degree of clarity. Can't judge _____
2. Problem stage What stage was the client in terms of defining or resolving the problem, or in presenting the answer? Please respond using one of the categories indicated below: o Problem recognition: I am trying to determine whether or not the topic I. m interested in is a real problem from the point of view of the discipline or area that interests me. I need a search so that I can discover whether others have identified the same issue as problematical. o Problem definition: I have identified a real problem and now need to define it more closely or carefully so that I can determine how to approach the problem and how it relates to other problems in the field. I need a search to help me define my research objectives. o Problem resolution: I am in the process of resolving the problem (engaged in laboratory experiments, field work, etc.) and now need a search to enable me to proceed with and complete that work. The question deals with a particular problem that I need to resolve in completing the work or with aspects of methodology, research approach or research methods. o Solution statement: I have effectively finished the work I was doing and am either tying up loose ends, or finding out from related work how best to report my research or where best to report it. 3. Intent or purpose for use of obtained information Please indicate on the scale below how you would describe the degree to which the user's intended use was well defined or envisaged to start with? Can't judge _____
4. Familiarity with problem or domain language or terminology Can't judge _____
5. Searcher's language familiarity
6. Searcher's measures on the interaction processes We are asking here for your impression or opinion as a searcher on a number of aspects and results dealing with the search of the above numbered question. The first set of measures deals with various aspects associated with effectiveness of communication between you and the user. The second set of measures deals with the effect of the search environment or the search setting Finally, the third set of measures deals with a series of possible changes that the user and you may have had during and because of the interaction. 6.1. Effectiveness of communication 6.1.1 User's explanation toward you as the searcher Please indicate on the scale below how you perceive the degree of effectiveness in the user's explanation to you about the problem and question at hand?
6.1.2 Your explanation as the searcher toward the user Could you indicate on the scale below how well you explained to the user what was going on in the search?
6.1.3 Understanding by the user On the scale below could you please mark how well you feel the user understood what was going on? Can't judge _____
6.1.4 Understanding by you as a searcher On the scale below could you indicate to what extent you as the searcher gained an understanding of the user's problem?
6.1.5 User language - 'user talk'. On the scale below indicate to what degree did you understand the user's specific language (or 'user talk') related to the problem at hand?
6.1.6 Non-verbal communication On the scale could you indicate how far non-verbalcommunication contributed to the effectiveness of your interaction with the user? (Prompts: nodding, smiling or lack of smile, silence, eye-contact, non-verbal sounds etc.
7. Changes due to interaction 7.1 User's and searcher's changes in perception of the problem 7.1.1 By the user Please indicate on the scale below whether you feel that any changes occurred in the user's perception or definition of the problem at hand, as the result of interaction with you and the feedback from the ongoing search? Can't judge _____
7.1.2 By the searcher Please indicate on the scale below whether you feel that any changes occurred in your perception of the user's problem at hand as the result of interaction and feedback from ongoing search?
7.2 User's changes in the question Please indicate on the scale below whether you feel that as a result of the interaction and the feedback from ongoing search, the user changed the question from the version stated in writing at the outset of the search? (Changes may include but are not limited to clarification, changes in terminology or level of specificity etc.) Can't judge _____
7.3 User's and searcher's changes in personal knowledge 7.3.1 By the user Indicate on the scale below, whether, as far as you can tell, any changes occur in the user's personal or internal knowledge, either broad in the domain, or the specific in the problem at hand, due to the interaction and/or the feedback during the ongoing search? Can't judge _____
7.3.2 By the searcher Indicate on the scale below whether any changes occur in your personal or internal knowledge, either broad in the domain, or specific in the problem at hand due to the interaction and/or feedback during the ongoing search?
7.4 Changes in relevance judgement On the scale below please indicate whether you feel that the user made any changes in his/her criteria for relevance judgement of items retrieved due to the interaction and/or the feedback during the ongoing search? Can't judge _____
D: Follow-up interview
1. Worth in relation to your time Could you rate your participation and interaction in this on-line search, and the information you got as the result of it as (circle one): 5 Worth much more than the time it has taken 4 Worth somewhat more than the time it has taken 3 Worth about as much as the time it has taken 2 Worth less than the time it has taken 1 Practically worthless 2. Time and difficulty in evaluation 2.1 How much time did you spend in evaluating the total set of answers: __________ (hrs. mins) 2.2 On the scale below please indicate how difficult it was for you to evaluate the answers as to the information conveyed?
2.3 Please comment on the most useful results. Which have been the most interesting, or look the most promising, what is it about them that looks interesting? (3 or 4 results should be considered) 3. Focus and comprehensiveness of retrieved information 3.1 Focus or targeting On the scale below please indicate how focused or precisely on target was the retrieved information in relation to the problem at hand and the question you asked?
3.2 Extraneous or non-relevant information On the scale below please indicate how much extraneous or non-relevant (unwanted) information was included in answers in addition to relevant ones?
3.3 Distraction caused by non-relevant information If you considered that a great deal of non-relevant information was included, please indicate on the scale below, how bothersome or distracting this was to you in relation to relevant information found?
3.4 Completeness of retrieval On the scale below please estimate how much of the information about your problem that is out there in the literature was retrieved, or conversely how much of it was missed by this search? Can't judge _____
4. Contribution to the resolution of your problem On the scale below please indicate what contribution has the information retrieved made toward the resolution of your problem?
5. Problem stage Now that you have looked at the results of the search and accessed some literature what stage are you in terms of defining or resolving the problem, or in presenting the answer? Please respond using one of the categories shown below. o Problem recognition: I am trying to determine whether or not the topic I. m interested in is a real problem from the point of view of the discipline or area that interests me. I need a search so that I can discover whether others have identified the same issue as problematical. o Problem definition: I have identified a real problem and now need to define it more closely or carefully so that I can determine how to approach the problem and how it relates to other problems in the field. I need a search to help me define my research objectives. o Problem resolution: I am in the process of resolving the problem (engaged in laboratory experiments, field work, etc.) and now need a search to enable me to proceed with and complete that work. The question deals with a particular problem that I need to resolve in completing the work or with aspects of methodology, research approach or research methods. o Solution statement: I have effectively finished the work I was doing and am either tying up loose ends, or finding out from related work how best to report my research or where best to report it. 6. Uncertainty When you were first interviewed you were asked about the certainty of your knowledge about certain aspects of the problem. I. d like to ask you again about the same issues now we've carried out a search.
a) that you have recognised a real problem to investigate? Very uncertain Very certain b) that you have defined the problem appropriately? Very uncertainVery certain c) that the problem can be resolved? Very uncertainVery certain d) that an effective way of presenting the results can be found? Very uncertainVery certain e) that relevant information is available and can be found? Very uncertainVery certain 7. Expectancy, novelty, serendipity 7.1 Expectancy Indicate on the line below to what degree were your expectations fulfilled or exceeded in respect to the retrieved information?
7.2 Novelty Indicate on the line below to what degree the information retrieved (which was related to your topic) new or novel to you?
7.3 Serendipity Indicate on the line below to what degree was the retrieved information serendipitous for you, i.e. finding unexpected, interesting information while looking for something else?
8. Overall satisfaction 8.1 On the scale below please indicate how satisfied in general were you with the results of the search?
8.2 Thinking about the whole search process how well do you think you could have carried out the search yourself independently?
8.3 How would you have carried out the search for information on this topic if this opportunity had not been available 9. About changes to your search topic 9.1 Changes in perception of the problem On the scale below can you indicate any changes you have made in your perception of the problem from the outset of the search?
If yes, what were the changes? Were the changes brought about by the search? 9.2 Changes in your question Have you made any changes in your question from the outset of the search? (changes may include but are not limited to clarification, changes in terminology or level of specificity etc.)
If yes, what were the changes in your question? 9.3 Changes in personal knowledge On the scale below please indicate if any changes occurred in your personal knowledge, of the specific problem at hand, due to the interaction and/or the feedback during the ongoing search?
If there were any changes, what were they? 9.4 Changes in relevance judgement Please indicate on the scale below if you made any
changes in your criteria for relevance judgements of the items retrieved
due to the interaction and /or the feedback during the ongoing search?
If you made some changes in your relevance judgement what were they? 10. Information Seeking Behaviour 10.1 Looking again at the stage you are at in your work, could you say which of the following terms best defines what you are doing?
10.2 How would you describe your thinking about the problem at this stage?
10.3 How would you describe your level of interest in the problem at this stage?
10.4 And finally, in general, how do you feel about the progress of your work at this point?
11. Information Seeking Activities Which of the following activities is important in your information seeking at this stage? 11.1 Following chains of citations or other forms of referential connection between documents.
11.2 Browsing or semi-directed searching in an area of potential interest.
11.3 Differentiating sources of information on the basis of the nature and quality of the material examined.
11.4 Maintaining awareness of developments in relation to this topic through the monitoring of particular sources.
11.5 Systematically working through a particular source to locate material of interest.
11.6 Verifying or checking the accuracy of information.
11.7 At this point in your search which of the following have you actually engaged in? (Please tick as many as necessary)
12. Further searches and General Comments In the time since the search have you conducted any further literature searches? If you have: 12.1 What were the search criteria you used? [Space for response] 12.2 Did the original search offer a basis for further investigation of the problem? [Space for response] 12.3 Would you like to take this topic forward through further searches as your project progresses? [Space for response] 12.4 Do you have any comments on the search, the interaction or the results? (Please use the back of the page if necessary) [Space for response]
Uncertainty in information seeking, by Professor
Tom Wilson, Dr. David Ellis, Nigel Ford, and Allen Foster
Library and Information Commission Research Report 59 ISBN 1 902394 31 3 ISSN 1466-2949 Grant number LIC/RE/019 |