Questionnaire Design in the Context of Information Research
T. D. WILSON
PROJECT INISS:1 A STUDY OF
|Frequency of need|
|Information Type||Not at all||Less than once a month||Monthly||Weekly||Daily|
|1. Legal, e.g., act of Parliament, DHSS Circular||1||2||3||4||5|
|2. Procedural, e.g., departmental procedure note/manual||1||2||3||4||5|
|3. Names, addresses, telephone numbers, i.e., 'directory information'||1||2||3||4||5|
|4. Training, e.g., courses, information syllabuses, course materials||1||2||3||4||5|
|5. Central Government statistical information, e.g., DHSS statistics||1||2||3||4||5|
|6. Internal statistical information||1||2||3||4||5|
|7. Records relating to clients, foster parents, adopters||1||2||3||4||5|
|8. Internal personnel and/or financial records, e.g., staff lists, budgets||1||2||3||4||5|
|9. News of developments in social work, including internal changes, whether written or not||1||2||3||4||5|
|10. Research in social work||1||2||3||4||5|
|11. Evaluations of experience or ideas in social work||1||2||3||4||5|
|12. Other—please specify ______________________||1||2||3||4||5|
Observation had shown that various difficulties were experienced in getting various kinds of information and, therefore, follow-up questions sought information on the problems and on access to more general documentary materials. These questions proved useful, as 80% of respondents perceived a daily need for names, addresses and telephone numbers, and as 13% experienced difficulty in getting such information; 53% needed access to client records daily and 21% experienced difficulty in getting access. In other words, two kinds of information of a common nature presented problems for significant minorities of those who needed access.
4. Formal communication systems. In this section, information was sought on the use of various specialized stores and files of information, such as correspondence files, financial records, case record indexes, and departmental or team libraries. The interviews confirmed the experience of observation, that is, that files of information relating to clients and to the availability of foster parents and adopters, as well as "message" books were highly used, whereas the use of team and departmental libraries was more sporadic. Given the nature of social work, this was to be expected.
5. Personal information habits. Observation had shown that social services staff maintained various personal files of information and other information aids, such as diaries and address books. The interview results showed that the most frequently used of these sources was the diary, used daily by 95% of respondents. Address books and notebooks were used daily by 72% and 58%, respectively, of respondents. Again, observation suggested that some personal information files were maintained because of shortcomings in officially provided systems, and 32% of respondents claimed that this was indeed the case.
6. Organizational climate and the structure of the department. Previous work (see Olson, 1977) had suggested that information-seeking behaviour would be related to respondents' perceptions of the "climate" of the organization. Accordingly, a short form of the Litwin and Stringer (1968) climate questionnaire was devised, selecting those items which observation had suggested would be most relevant to information use. Table 1 lists the items selected. However, the only significant relationship was that between climate and work role: fieldwork staff had more negative attitudes towards management than other levels and no relationship was found between climate and any of the information-use variables.
|1. The jobs in this Department are clearly defined and logically structured.|
|2. Our management is not so concerned about formal organization and authority, but concentrates instead on getting the right people together to do the job.|
|3. Supervision in this department is mainly a matter of setting guidelines for subordinates; they then take responsibility for the job.|
|4. People in this department don't really trust each other enough.|
|5. In this department it is sometimes unclear who has the formal authority to make a decision.|
|6. This department is characterised by a relaxed, easy-going working climate.|
|7. When I am on a difficult assignment I can usually count on getting assistance from my colleagues.|
|8. In this department people are encouraged to speak their minds, even if it means disagreeing with their superiors.|
|9. The policies and structure of the department have been clearly explained.|
|10. One of the problems in this department is that individuals won't take responsibility.|
|11. It's hard to get to know people in this department.|
|12. I feel that I am a member of a well-functioning team.|
7. Experience and training. These were questions covering years of experience in social services departments, in the present job, and professional qualifications.
In addition to the direct relationships between observation and questionnaire development noted above, there were two further relationships resulting from observational experience.
First, scenarios were prepared for the five work-role categories on the basis of the narrative accounts prepared following the observation. An example of a scenario is shown in Figure 4. These were sent out to respondents prior to the interviews, and respondents were asked to show whether the account was "very close," "close," or "not at all close" to their "information behaviour" in a particular week. They were also asked to modify the accounts to make them more representative of their experience. Analysis of these accounts was rather difficult and time consuming, but enabled the production of accurate portraits of the relevant categories of staff.
|SCENARIO (13 Information behaviour and contacts)|
|1) Please tick the appropriate box below:|
|This description is:
|to my normal working week|
|Please modify the account to make it a closer fit by changing various words or figures:|
|James Joyce's main source of information about his work is his supervisor (often during team meetings), but he also obtains information from colleagues in the department (including other social workers) and, to a lesser extent, from external contacts. Mr. Joyce never uses the department library, seldom reads journals (apart from glancing at the department's newsletter) and rarely reads other publications. Files on matters of current importance to Mr. Joyce are usually kept in his office filing cabiner, but most other records are held in the section filing system. He is a member of a department working party and occasionally attends external conferences or courses on professional topics.|
Second, the whole process of design was informed by the team's experience of observation. Individual team members assumed responsibility for particular sections of the questionnaire and prepared draft questions. These were discussed in team meetings in the course of which the researchers played the role of the people they had observed in order to test the propriety, the wording, and the sequencing of questions. The process was quite straightforward: the draft questionnaire was worked through, question by question, while each member of the team tried to respond in terms of the work roles and information-seeking behaviour of the people they had observed. As a consequence, differing perceptions of the meanings of questions emerged, and the influence of differing organizational structures was revealed. In a sense, therefore, the piloting of the questionnaire was done "in house," with the researchers representing respondents. In this way, questions were altered in their wording, resequenced, or dropped from the draft schedule. The final draft version of the questionnaire was then sent to members of the Project's advisory committee and further changes were made as a consequence. The final version of the questionnaire (by then, its sixth incarnation) was used in an interviewer training session before being used in the field. It may have been preferable to pilot the questionnaire in the field but, given the available resources, this was not possible, and the results obtained using the questionnaire suggest that role-playing on the basis of observation, coupled with informed comment from outside, was a satisfactory alternative.
In summary, therefore, the advantages of basing our questionnaire design on observation in the work settings of the respondents were considerable; the observation period was also essential, since without it the relevance of the questions to the respondents would have been problematic.
As noted previously, 151 interviews were carried out in the interview phase of the Project. The five interviewers (that is, the four research workers and the Project Head) all received interviewer training, conducted by Michael Brenner (an outline of the training procedure is given in his chapter in Part I). Training is often neglected in academic research interviewing (judging from the paucity of references to the subject in published research), but in the case of Project INISS, its worth was proved time and again.
One researcher, for example, was faced with a particularly "difficult" respondent on her first interview. The difficulty in the respondent's behaviour was not related to the questions, but to the interviewing relationship, as from time to time he would switch off the tape recorder, leap from his chair, and begin to hurl abuse against his section head and various other colleagues. He would then sit down again, switch on the recorder, and carry on with the interview as though nothing had happened. Without the training she had received, the interviewer might have been unable to cope with a situation such as this. As it happened, she carried on skillfully with the interview: the resulting tape recording gives no indication whatsoever that the respondent was behaving in a curious manner and no indication that the interviewer was disturbed by his activities.
Most of the problems encountered in the interviewing phase were anticipated and none proved insuperable. In fact, the main difficulty was in making contact with busy people (in some cases more than 250 miles away from Sheffield, where the research team was based) to set up the interviews. In 53 cases out of the 151, more than two attempts were needed before contact was made, and in 31 cases, four or more attempts were needed.
A very high level of cooperation was obtained from all respondents. Based on comments made to the interviewers, it is known that the cooperation was at least partly due to the team's having been observing in all four of the departments in which the interviewing was carried out. In other words, the staff in departments had gained the impression that the Project was non-threatening and seriously intentioned and that its work was worth supporting.
As might be expected, in spite of careful design and pretesting, difficulties were experienced with some of the questions. Questions that sought generalizations about the frequency of behaviour or specific statements about the amount of time spent in particular activities presented some problems. For example, some respondents were reluctant to specify a particular frequency in answer to the question, "in general, how often do other people in the department ask you for advice or information about these areas'?-" (areas of personal expertise). Also, the question: "can you estimate about what proportion of your time is spent on these activities^" (referring to a response card), caused considerable difficulty, particularly with fieldwork staff, and frequently disrupted the flow of the interview. The main problems with this question seemed to result from difficulty on the part of the respondents in being able to describe a typical week because of the unpredictable nature of client-oriented work, or being able to generalize about the use of time spent in particular activities, possibly because the separate components of work were not usually identified as such.
The work-role scenarios for social workers, senior social workers, and specialists also caused problems. For social workers, the different demands of work on long-term care teams and intake teams, as well as demands connected with differences between urban and rural social service problems appeared to be at the root of the difficulties. For specialists, two categories of problems arose: First, the scenarios had been prepared with the work of specialists responsible for advisory work in relation to particular client groups (for example, the blind, the mentally handicapped, and the elderly) in mind. Naturally enough, specialists of a different kind, namely training officers, research officers, and information officers, found the scenario less satisfactory. Second, several advisers who appeared in the sample were concerned with residential services, whereas the observational phase had been concerned with fieldwork services only.
Two rather unusual instances of failure to complete response sheets relating to types of information or relevance to the respondents occurred: one respondent was partially sighted and the other chose to breast-feed her child during the interview leaving only one hand free to hold the sheet. In the first case the interviewer read out the categories and recorded the responses; in the second, the interviewer passed the sheets to the respondent in such a way that she did not have to move in accepting them. He then recorded the responses. (The fact that the mother was feeding her child provides unusual evidence of the relaxed manner in which the interview was carried out!)
Finally, the items related to organizational climate caused some problems: frequent hesitation was encountered with some statements containing two elements that were not always seen as naturally associated. For example: "The jobs in this Department are clearly defined and logically structured." Also, some of the apparent underlying assumptions, such as that staff should "take responsibility for the job," were challenged, and items that referred to a "team" almost certainly were understood to refer to the local, social-worker team rather than to staff more generally in the department.
These latter points are all the more disturbing when it is recalled that the Litwin-Stringer list is usually employed in self-administered questionnaire studies, without the presence of an interviewer.
Effective interviewing demands (for details, see Brenner's chapter in Part I):
1. Trained interviewers.
2. A questionnaire designed to meet the research objectives of a study as well as the requirements of the interview situation.
3. Respondents who are cooperative.
The observational phase of Project INISS contributed to all of these aspects:
1. The interviewers were able to employ their training effectively in the context of the organizations and staff of the departments that constituted the research setting.
2. The questionnaire was relevant; that is, it enabled the gathering of the perceptions of information use and communication held by the respondents.
3. We obtained the cooperation of respondents, at least in part, because they were acquainted with the team's earlier work in the departments.
I do not intend to emphasize that prior observation of a research setting is necessarily the best prerequisite for questionnaire design. Clearly, methods suggested by other authors, such as informal, intensive interviews (see Part II), have their advantages, given that prolonged observation is certainly more expensive and time-consuming than interviewing. However, in a field like information science, where there is relatively little experience of complex, multimethod, social science research, we obtained intimate familiarity with the social life under study, which, in turn, provided an adequate foundation for questionnaire design.
Project on information needs and information services in local authority social services departments ( Wilson, et al., 1978 ).
Wilson, T.D. (1985) Questionnaire design in the context of information research, In: M. Brenner, J. Brown and D. Canter, eds. The research interview: uses and approaches. (pp. 65-77) London: Academic Press. [Available at http://informationr.net/tdw/publ/papers/1985qdesign.html]