Appendix 4: Impact survey

Appendix 4: Impact survey

1. The survey

In order to explore the Project's impact within those institutions which had early involvement with it, a telephone interview survey was conducted over a period of four weeks in February and March 1997 with fifteen local NetLinkS co-ordinators or their successors. Each local co-ordinator (or their predecessor) had, in 1995 or 1996, organised a NetLinkS focus group meeting on the topic of networked learner support (NLS) in his or her institution, and it was hoped that these staff would take forward NLS activities in their institutions. Subsequently, updates and information were disseminated regularly to this group, and some co-ordinators remained in close contact with the Project throughout its life. However, the links between the Project and this group were essentially informal; co-ordinators had no official status or role within the Project, and the extent of their (or their successors') involvement with it following focus group meetings was determined by local factors. Awareness of both the continued work of the Project and of NLS activities within their institutions was therefore variable.

Fifteen interviews were carried out (from the group of nineteen institutions in which focus group meetings were held) and each lasted about forty-five minutes. Interviewees were all library or learning resource centre staff, typically senior library assistants and subject librarians, and employed in a cross-section of institutions, including one college of higher education. The interview schedule, which was used consistently with all interviewees, is reproduced below as Figure 1.

2. Personal involvement of interviewees

Interviewees were asked about the impact of the Project on their own professional practice. Eight indicated that they had changed their thinking or were more aware of, or interested in, NLS issues, as a result of the Project. Five judged that the Project had led to new initiatives, and that their institutions had moved beyond awareness-raising to changes in practice. For instance, one indicated that NetLinkS had informed the Library's strategic approach to innovation, another said that a Web resource had been created as a consequence of involvement in the Project, and another that online delivery modes were being adopted for information skills courses as a result of his involvement. Others confirmed that they were already engaged in developing NLS activities at the time of the start of the Project, and so it had dovetailed with this work. In contrast, two interviewees indicated that although they had heard of the project, it had had no impact on them or their practice, and others were unable to link the Project to specific initiatives although "background information" from the project had been valuable. One person commented on the value of the Project in providing an overview of the current national and international situation in NLS, and one that the fact that the project had been "well organised" had made an impact in itself.

The interviews shed light on the amount of personal change that many of the initial co-ordinators had experienced in the period after the focus group meetings, and which had affected their involvement with the Project and its aims. In some cases, events had resulted in curtailment of any personal NetLinkS-related activity , or had led to reduced participation. Changes included:

changing jobs or duties within the library, or leaving the institution;

increased workloads, sometimes due to colleagues leaving and not being replaced

management decision to prioritise other activities

changes in the physical environment, disrupting involvement in NLS

the co-ordinator role being passed to a colleague following the focus group meeting.

Other factors affecting the co-ordinators' role as regards the development of NLS were:

inability to take a leadership role in change due to relatively junior status (the task of being the co-ordinator was one that was given to them as a relatively junior member of staff and therefore perceived as a trivial task)

personal work responsibilities or interests having little to do with NLS or information and communications technology

local pressures meaning that there had been little time for any involvement in eLib projects

Changes and other factors were such that eleven of the fifteen interviewees indicated little opportunity or motivation to take forward local changes in respect of NLS. The remaining four had sustained their involvement with the Project and maintained their local development work; they were all personally interested in NLS and had welcomed the co-ordinator role rather than had the responsibility delegated to them. One of these four, in particular, was sufficiently senior to have a high level of self-determination.

3. Reflecting back on the focus group meetings

The composition of groups attending the meetings was commented upon by many interviewees. The importance of identifying the right mix of people from across institutions, to offer a better chance of follow-up activity, was emphasised. There had been considerable differences in the composition of groups. Some meetings consisted of staff solely from the library, and in these cases interviewees commented that on reflection, key people or representatives from outside the library had been missing. Some interviewees also indicated that those attending the meeting had not been the most appropriate participants; sometimes this was reported to have been a result of the co-ordinators' approach to organising the meeting. Some interviewees emphasised their limited knowledge of networked learning developments in their institution, and the problems this had caused for identifying those likely to be most interested in participating in the meeting. One person said s/he had 'taken the line of least resistance' and chosen 'those most accessible.' Another person said that s/he had conducted a 'random straw poll' of colleagues who might possibly be interested.

Nevertheless, some meetings were successful in bringing together a range of people from across the institution, including academics and support staff, and five had involved members of senior management in administrative or academic positions. In four cases, the meeting had been attended by people or groups already concerned with the development of learning technology and multimedia. In these cases, the meeting had provided another forum for continuing existing discussions. Three interviewees in particular judged that the mixed composition of their group had been successful.

Poor internal communications, impacting on the composition of initial meetings and the possibility of further developments, was a factor that was strongly emphasised by interviewees, ten of whom commented in some way about the poor level of cooperation and internal communication, especially across multiple sites and the limited circle of people in institutions who knew what was either going on or was being planned. Interviewees commented on the isolated way in which staff were developing innovative approaches to teaching, learning and learner support, and one commented that although there were attempts to disseminate good practice, this was done in very traditional - not necessarily effective - ways through committees.

There appeared to be considerable differences between institutions in terms of the impact of initial meetings; however, interviewees had difficulty in distinguishing impact directly attributable to a focus group meeting, and impact attributable to other factors and initiatives.

On the positive side, half the interview group still believed, two years on, that the meeting had raised awareness of NLS amongst colleagues, and several that it had fed in to new initiatives. In one case, the focus group meeting had stimulated an initiative to produce a policy document; this was judged to be largely the result of bringing a group of people together for the first time which included various members of senior management. In contrast, another interviewee commented that having achieved the task of bringing people together, it had been too difficult to organize another meeting.

3 people mentioned that this area was currently being addressed by existing special interest groups for learning technology and how that reduced the possible impact of the meeting or that any outcomes were being taken on by those groups.

One Contact remarked that although there were other special interest groups tackling learning technology issues, the focus group had provided a valuable opportunity for input from the Library which also resulted in a broader view of materials development.

One Contact said that, in a positive way, it made them realise how far behind they were in this area and an idea of the work still to be done.

Three interviewees reported trying to establish a special interest group of some kind as a result of the focus group meeting, but all had failed due to local circumstances, including staff involvement in other initiatives and the problem of getting people back together. One person said that s/he had intended to introduce a local mailing list in support of the group and ideas but it had never got off the ground. Another person said s/he had contacted everyone involved specifically to sustain the initial interest but the feedback had been against a further collaborative effort and in favour of exploring things individually.

Amongst the other twelve interviewees, the reasons offered for no further action being taken included existing initiatives and developments that were already in place; changes in personnel, either in terms of those attending or the organizer changing, the group being too small or unrepresentative to develop institution or Library level changes. Two interviewees felt that these kind of ideas were best considered under existing bodies such as learning technology initiatives and one said that these groups were actively addressing NLS issues. One of the contacts mentioned that they were too junior to organize and sustain a change initiative. Others felt they would be the wrong person due to their role either being less connected with networked information issues or because their role had changed since. Finally, one person thought it was the responsibility of the project to support further developments in the institutions involved as those most informed on NLS.

4. Project resources and activities

Seven interviewees said they used the project resources for keeping up to date and monitoring developments elsewhere, and several indicated that they passed on NetLinkS information to colleagues.

Web site. The large majority of the interviewees had used the Web site. Two thought they had used every resource on the site, including the "case base", bibliography, research reports and Project information. The general purpose of visits to the Web site appears to have been to keep in touch with developments in the Project and maintain current awareness. Two people said they had used the Web site and resources for giving demonstrations or talks to colleagues. The nls-forum discussion archive was highlighted by three interviewees, who said they were not able to keep up with the discussions in the mini-conferences in "real time." Three interviewees said they visited the Web site as a direct result of a prompt in an email message, and particularly when the Newsletter was sent out.

Discussion list. All interviewees had been a member of the list at some time during the life of the Project, and over half were still subscribed; four said they had been a member of the list since it opened. However, only one person had made a contribution to the list. The use of the list in general, and for "mini-conferences" in particular, was frequently commented upon as a model for the use of such tools. One person said s/he believed that the list had made a "huge impact on the library world" and thought they were "excellent." Ten interviewees commented that they had found the mini-conference discussions useful, enjoyable and practical.

Online course. Three interviewees had participated in the online course, and two indicated that they would have liked to participate. Another interviewee said s/he knew colleagues taking the course and had regular discussions with them to share thoughts and ideas. Those participating on the course said that it had very practical outcomes in terms of changing or developing practice locally. One person said they had developed "transferable technical skills" through the course, and another that it had informed her thinking and helped generate ideas.

NLS facilitator workshops. Only one person had made use of these workshops because the main aim was to support the organisation of review meetings in other institutions. The main outcome of this person's involvement was the goal of creating a "network of change agents" and that this issue was currently being addressed locally.

5. General

In the final part of the interview, participants were asked for any further views of the Project and what they had perceived to be particularly successful or unsuccessful. The following issues were raised:

Understanding of the project. Eight interviewees made reference to the aims or outcomes of the project being unclear or they said it was very "theoretical and five felt they needed more help in explaining the Project and NLS to colleagues. Someone said that s/he wanted "something more prescriptive" and felt that other projects in the same strand were more practical; another interviewee also said that other projects' materials were more usable and less theoretical. Six interviewees thought more attention should have been given to publicity and that the project had a relatively low profile. In contrast, four interviewees commented that whilst clarity of purpose had been an initial problem, this was later resolved.

Understanding of eLib Training and Awareness projects. Five commented that it was difficult to distinguish between projects in the Training and Awareness strand of eLib and suggested a need for greater co-operation between projects to help staff understand their different purposes and the relationship between them. For some interviewees, projects within the initiative appeared to overlap, and felt they should collaborate on common areas of interest and provide clearer points of access for the user-community.

Dissemination. Several interviewees commented on the need for projects to disseminate good practice, and felt that NetLinkS was already strong on this aspect; others felt that dissemination should continue and increase, for instance through publications and the continuing availability of the Web site.

Overall impact. Three interviewees felt that the impact of the Project was limited and one thought that there was a need for such a Project to address a wider audience, including more library colleagues, academic colleagues and senior managers. One person commented on the need for some realism in thinking about the anticipated impact of any project on a local institution but nonetheless welcomed the contribution made by the Project.

Continuation of the Project. Six interviewees raised the issue of the need for further work in the Project's area, to build upon the work carried out thus far and to cater for institutions at different stages of development. Suggestions for continuation of the activities included maintenance of the Web site, continued offerings of the online course, further focus groups.

6. Conclusions

The broader impact of the Project within their institutions or on the higher education sector cannot be determined from the feedback from this sample of interviewees. Nevertheless, their comments do suggest both strengths and weaknesses of the Project, as well as contextual factors which need to be taken into account by further cultural change initiatives in this field. The main issues are as follows:

External support for local change agents. Most local co-ordinators were unable, for a variety of reasons, to take the initiative to support the development of NLS in their institutions. It seems likely that more time, a more formal role, and closer involvement of the external Project would be a more successful strategy.

This suggests that the policy of working through individual change agents may not be enough to further develop change across an institution. Even where such a strategy is used, success depends very much on the level of individual interest and their level of self-determination. Other problems associated with this strategy include the need for a clear understanding of the concept of NLS amongst the Contacts and their colleagues.

It was often seen as difficult, for those contacted, to distinguish between the impact attributable to the Project (or the other related Projects) and that arising from internal developments. However, this was compounded where Contacts also had difficulty distinguishing between projects in the Training and Awareness strand. This suggests a need for clearer presentation and co-ordination from all those responsible.

There was general agreement, amongst those contacted, that where provision was practical in nature it was perceived as being of greater value. An example of this was the broad recognition, amongst those contacted, of the value of the nls-forum discussion list and particularly of the mini-conferences hosted on that list. Comparisons were made between projects in the Training and Awareness strand and there appeared to be a greater uptake by colleagues of interviewees where provision was felt to be practical.

Although the broader impact of the project cannot be determined through this survey, those in this sample group of Contacts appear to have felt a significant impact from the Project in terms of developing their awareness of NLS and changing their practice.
Figure 1 Interview Guide

Aim of interview: to identify the impact of the Project on those people and institutions involved in NetLinkS focus groups in 1995-6

Warm up questions and update

How much have you personally used the Projects resources or activities
and which ones have you used ?

For instance, have you used :
-the Web site
-the discussion list
-online mini-Conference
-Did you come to the Symposia we held
-or the culture change workshops
-Did you join the online course or does that have any appeal for the future
[probe on the value of particular resources]
How has your connection to NetLinkS affected your own job or your personal role ?

For instance, how has it changed the way you do things or the way you think about issues to do with networked learner support ? Or how much more aware are you of issues to do with networked learner support ?

Thinking back to the focus group we held at Strathclyde (or wherever)
back in September 1996 (or whenever)... What developments have there been since that meeting ?

For instance, were any further meetings planned or has the Teaching and Learning Strategy or Information Strategy changed as a result ?
[probe as to the possible reasons for either this level of activity]

Comments on anything to do with the NetLinkS that we haven't covered? Is there anything you'd like to say about the Project or your involvement ?

NB/PL May 1998.

[Contents] [Section 1] [Section 2] [Section 3] [Section 4] [Section 5] [Appendices]

Top of page

Authors/Graphics - Nabeel Nasser and Sarah Ashton
last updated 12th December 1998