The evaluation of current awareness services in local government

T.D. WILSON

Project LOGI was funded by the British Library R&D Department to develop 'self-help' manuals for those wishing to assess information needs in local government and to evaluate the effectiveness of current awareness services. The data obtained in pilot tests of the research Instruments point to conclusions of relevance for the design of information systems. In particular, the data on the value of current awareness services, and on the uses to which information is put, help to define further the idea of 'benefit' and to limit the scope of 'benefit measurement'.

INTRODUCTION

From June 1978 to February 1980 the British Library R&D Department supported a project at the University of Sheffield which was designed to produce 'self-help' manuals for the managers of information services which would accomplish two ends:

• provide, in the first manual, ready-made, but adaptable, questionnaires and interview schedules, together with instructions for their use, for the assessment of information needs in local government (Mullings et al. forthcoming); and,
• provide, in the second manual, similar instruments for the evaluation of current awareness services, together with a package of computer programs for the analysis of data resulting from demands made by the users of such services (Francis et al. 1981).

This article is concerned with data resulting from the pilot-testing of the instrument design for the user-evaluation of one specific service, the Social Work Information Bulletin.

THE BACKGROUND TO THE SOCIAL WORK INFORMATION BULLETIN

The Social Work Information Bulletin (SWIB) is a co-operatively produced, current awareness bulletin which covers some 300 journals in the fields of social welfare, medicine, psychology and psychiatry. It is produced by a public library service with considerable expertise in local government information service, Leicestershire Libraries and Information Service, in association with the University of Leicester, Derbyshire County Library Service and Coventry Social Services Department. It is probably unique in the UK as a current awareness bulletin in a limited field, produced co-operatively, although other, 'general purpose', co-operative bulletins do exist.

SWIB is produced every two weeks and is circulated to about 900 individuals in three departments of social services, one university school of social work and a number of other agencies. A number of subscriptions, in some cases with reproduction rights, are sold to other authorities.

From its inception in 1976 SWIB has been extremely successful, each issue resulting, on average, in a demand for 460 copies of articles. (Each issue carried, again on average, 85 abstracts.)

THE PILOT STUDIES

In 1978 and 1979 two interview surveys were carried out in Derbyshire and Leicestershire social services departments covering in all 66 recipients of SWIB. The aim of these surveys was to test the schedules and to prepare for their conversion to self-completed questionnaire form. A self-completed questionnaire was developed in 1979 and tested in Coventry Social Services Department. From a random sample of 121, 51 per cent responded after one follow-up letter. No further attempts to increase the follow-up were made as the aim of the work was to develop the instrument, rather than produce research data. Copies of the interview schedule and the questionnaire can be found in Francis et al. (1981).

THE USEFULNESS OF THE BULLETIN

In both the interviews and the questionnaire surveys, recipients were asked how useful they found the bulletin, and the results are shown in Table 1. As the table shows, the overwhelming majority of respondents found SWIB useful to some extent or other. When asked why they rated it in that way, the 87 'positive' respondents answered as shown in Table 2. In all cases these were unprompted answers and it is comforting to discover that users find a current awareness bulletin 'useful' for almost precisely the reasons that information workers use to justify its production.

Typical of the comments made on the usefulness of SWIB were the following:

They are very good pointers... even those little precis... you're learning all the time, even if you don't hunt out the article. It's quite good on that level.

I'm not very good at reading social work books. It brings to my attcntion articles that are up to date.

This is ideal. I can get what I'm interested in without having to read any of the bumf... There's no time to go thumbing through different things. It's really a big help.

It's very easy and quick. Social workers are bombarded with literature. The headings are good - that way you can eliminate a lot of irrelevant stuff.

I'm fairly isolated over here, so there's a tendency to get on with one's job without getting much stimulation from elsewhere.

Unless you get this type of service you sink into apathy. Stimulation, and for information on research and to broaden one's personal knowledge.

Gives people access to sources of information they don't normally come into contact with, such as nursing information. The wide variety of information it uses is very good.

I want a lot of things from it but don't have time to read all the things ordered. But it's useful to have them for reference in the future.

In general, the 'negative' responses on usefulness came from the 'administrative support' grades and the reasons given were the lack of SWIB's relevance to the respondent's work, or simply 'I suppose I'm too lazy to use it.'

Had we gone no further, it would have been possible to say that the users of SWIB perceived 'benefits' in the service: it keeps them up to date in their specialisms, it saves them time by sifting through journals to identify what is useful, its coverage is far greater than they could achieve by reading the journals that are circulated in the department, and it promotes ideas. For a 'blanket' information service to achieve this degree of support from its users is quite remarkable. We wished to take the investigation further, however, and did so by inquiring into the usefulness of the photocopies of articles obtained by users.

USEFULNESS OF DOCUMENTS RECEIVED

In the past, studies of information used have tended to be restricted to generalized questions such as those above. In this study it was decided to probe further to discover the uses to which specific documents had been put, in the expectation that this would cast some light on the notion of 'benefit'. This was done in the interviews by showing the respondent copies of a number of documents he or she had requested (usually three, but sometimes fewer, depending upon how much they used the service). (Nineteen items were supplied by the respondents themselves, either because no record of use was available, or because the person used the service through an intermediary.) Three questions were then asked:

1. Can you recall getting this item?
2. Why did you ask for it?
3. What use did it serve?

A total of 110 documents was examined in this way. Four turned out to have been erroneously ascribed to respondents by the library service records, 16 could not be recalled by the user, and 90 could be recalled.

Classification of the reasons for asking for the items proved to be rather difficult and more than one reason was given in a number of instances. The results are shown in Table 3. The 'relevance to work' category resulted from the manner in which people responded to the question: in general, they appeared to mean that the information was of direct relevance to their own job, whereas the use of something to help them train others could be expected to have a less immediate impact. However one categorizes the data, it is abundantly clear that by far the majority of items are requested for work-oriented reasons.

Rather more information was forthcoming when respondents were asked what use the document served: the results are shown in Table 4, excluding the 14 (16 per cent) items found not to be useful and a further two (2 per cent) the usefulness of which the respondents could not recall. Thus the 74 remaining documents received 93 'uses': a ratio of 1.26 'uses' to each document.

In the questionnaire study this aspect of information use had to be approached rather differently, as it was not possible to probe for full responses in the absence of an interviewer. Accordingly, a set of categories was developed from the interview responses and respondents were asked to indicate how frequently they used copies in the ways shown. The results of this exercise are presented in Table 5.

ON 'BENEFIT'

Did the users whom we surveyed 'benefit' from the service they received? This question, and the associated problem of attaching money values to benefits to produce 'cost-benefit' equations, has been a cause of much heart searching among information scientists in recent years as the economic climate has put increasing pressure on information managers.

If we take the responses at their face value, then an overwhelming proportion of users claim that SWIB is of use to them. The responses they give on the nature of that usefulness are clearly related to the idea of 'benefit'. Thus they 'benefit'

• by being kept up to date;
• by being kept in touch with developments in their work;
• by having the literature 'sifted' for them;
• by having a greater variety of documents made available to them than would otherwise be the case; and so on (see Table 2).

Similarly, the uses to which documents are put when they are received by users also point to benefits:

• generally being more informed;
• being aided in the performance of a given task;
• being assisted in training;
• being better able to inform or train others; and so on (see Tables 3 and 4).

These uses appear to be related, in general, to 'cognitive' needs, and it is possibly in this area of need that we generally think of information as benefiting people. Hints here and there in the data, however, point to another category of needs; that is, emotional or affective needs.

For example, the respondents in talking about the usefulness of SWIB in keeping them in touch made comments such as:

...I don’t feel isolated.

Unless you get this type of service you sink into apathy.

Again, in talking about the use to which specific articles were put, comments like the following were made:

It confirmed that I’d handled the case properly.

...it shores up any insecurity I may feel because of not having specialist knowledge...

...adding to security about non-specific areas, helping one to talk outside and to staff....

Another respondent said that he read articles on a particular topic because he could then appear to be interested in matters of concern to specialized subordinates and that, thereby, it left him less open to criticism that he wasn’t interested. This kind of use was noted by 25 of the respondents to the questionnaire survey.

There is some possibility that a carefully designed experimental study could determine the money value of information intended to satisfy cognitive needs. One would have to assess the degree of existing knowledge of control and experimental groups and cost alternative solutions to problems. Tom Allen’s work (1977) approaches this kind of experiment, and recently some work has been done on the value of ’insider information’ in stock market transactions. The idea of attaching money values to benefits, however, implies that money values can be attached to the outcome of a person’s acts and that money penalties are incurred if an act does not take place or if person A takes longer to perform an act than person B. This appears to be the case only in well-defined and very limited work roles: I would find it more difficult to operate a machine tool without being given the necessary information than would someone who had the information. Given a factory full of uninformed machine-tool operators one can easily envisage what would happen!

In many work roles, however, there is no such easily determined relationship between information inputs and the money values of outputs. It is even less likely to be the case when the afflectii,e needs which may be influenced by information may differ among people with the same cognitive needs. In such circumstances the quality of outputs is likely to vary.

A frequently used alternative to the money value of benefit is volume of ,use’; a recent example is to be found in Kent et al. (1979). But, as shown above, ’use’ in the sense of acquisition of a document may not lead to any useful inputs (approximately 16 per cent of the items received proved not to be useful), or it may lead to uses which, although they cannot be quantitatively assessed, are of qualitative significance to the user.

Another possibility is to ask a user to state how much he believes a service to be worth in money terms, or what money value it contributes to the performance of his job. Usually, however, there is no intention of charging and the user knows this; and often neither the user nor the information scientist knows how much the service costs anyway. It is difficult to see how the results from this kind of exercise can be regarded as properly ‘quantitative’.

It seems more realistic, and more honest, therefore, for the information manager to produce as cost-ef ficient a service as possible and to rely upon qualitative evaluation by users, of the kind reported here, to justify that service to institutional management. The service described here costs the co-operating libraries approximately £0.30 (c. $0.65) a copy, including the back-up photocopy service. That is, £7.80 ($17.00) for each user, a year. Bearing this cost in mind, if the authorities concerned are not prepared to believe the qualitative evidence of the service’s usefulness, there would appear to be little hope of them taking more notice of somewhat dubious ’quantitative’ evidence.

REFERENCES

• Allen, T. (1977). Managing the flow of technology. Cambridge, MA: MIT Press.
• Francis, G.M., Mullings, C., and Wilson, T.D. (1981). A manual for the evaluation of current awareness bulletins. London: BLR&D Department. (BLRDD Report 5584)
• Kent, A., Cohen, J., Montgomery, K.L., Williams, J.G., Bulick, S., Flynn, R.R., Sabor, W.N. & Mansfield, U. (1979). Use of library materials: the University of Pittsburgh study. New York, NY: Marcel Dekker.
• Mullings, C., Francis, G.M., and Wilson, T.D. (1981). A manual for the investigation of information needs in local government. London: BLR&D Department. (BLRDD Report 5585)

How to cite this paper

Wilson, T.D. (1982). The evaluation of current awareness services in local government. Journal of Librarianship, 14(4), 279-288

Web Counter