Appendix 3: Phase 2 evaluation plan (overview)

Appendix 3: Phase 2 evaluation plan (overview)

The NetLinkS evaluation plan aims to follow the Guidelines for Elib Project Evaluation produced by the Tavistock Institute. We are very conscious of the need to develop a focused evaluation strategy and one which is realistic in terms of the overall scale of the NetLinkS project.
1. Purposes of the evaluation

to help achieve high relevance and quality of all project activities/products;

to provide evidence of the value of the NetLinkS activities/products to users, and of the impact of the project at both individual and institutional levels;

to contribute to overall learning in the eLib programme.

The project recognises the need for both formative and summative evaluation, and aims to place particular emphasis on gathering on-going feedback from participants in NetLinkS activities, and users of the project's resources.
2. Stakeholder groups

There are three main stakeholder groups for phase 2 of the project: JISC eLib programme management, and participating individuals/institutions in the broader professional community.

JISC eLib programme management will be provided with both formative and summative evaluation information during the life of the project, via the project's Steering Group and its Annual and Final Reports.

Likewise, individuals and institutions using the project resources and participating in project activities will be kept informed of the results of evaluations, via the Annual and Final Reports and, as an on-going procedure, via the project's Newsletter/WWW site.
3. Key evaluation questions

Each element of the project's activity will have a range of evaluation questions associated with it:

Discussion list (nls-forum) - We aim to identify the value of the list for supporting the development of NLS through an analysis of the server-traffic data and qualitative data gathering.

WWW environment and technical support -We aim to assess the usability of the online environment through an analysis of the server-traffic data and through qualitative data analysis.

Resource base and information support- We aim to identify which are the most valuable resources and which are most used through an analysis of server-traffic data and qualitative data gathering.

On-line course- We aim to assess whether or not the environment is usable and comprehensible and whether the content is appropriate. This will be achieved by piloting and formative analysis of the design and implementation processes.

Programme of on-line events- We aim to establish the value of these events and of email-based strategies for professional development. This will be done through a comparative view of list population-profiles together with qualitative data analysis.

Special interest groups -We aim to assess the value to other NLS-related groups, including conference-organisers, of adopting the NetLinkS Web-site as an on-line home for meetings and discussions.

Institution-based development activities (focus group training and follow-on activities) - We aim to assess the key issues in change-development as it relates to NLS and the further needs for supporting this process. We also aim to establish the value of our initial provision in supporting this process. This will be achieved through piloting, analysis of online discussion transcripts and use of qualitative evaluation methods.

The timing of evaluation activities will relate to the life-cycle of each strand of the project's activity. An impact assessment of the project as a whole will be undertaken towards the end of the funding period.
4. Data collection methods

Methods used will vary according to the product/activity being examined. However, we envisage that the following methods will be included:

surveys (user profiles and feedback)

electronic monitoring (usage patterns)

observation (nature of participation)

focus groups (feedback on pilot of on-line course, and summative impact assessment and examination of sustainability issues).

5. Steering Group

The Steering Group provides formative feedback for the project and aims to ensure that it maintains its objectives and progresses in the most appropriate way. The Group brings together advisors with interests and expertise in the different aspects of the project, as well as representation from eLib programme management.
6. External evaluation

Edna Blackie will provide external evaluation for the project. The details of her role are still to be determined, but it is anticipated that she will offer feedback on all on-going evaluations undertaken by the project, and be involved in the final impact assessment (for instance, by facilitating a small number of focus groups). We hope that she will participate regularly in our project review meetings.

7. Project review meetings

Formative evaluation is a key part of the NetLinkS evaluation strategy. Monthly team meetings to reflect back on what has been learned, and to discuss evaluation data when appropriate, are an important part of our evaluation plan. The meetings are intended as a way of sharing learning between team members, as a means of developing skills and understandings collectively. Minutes of the meetings are taken, to provide a record of the formative review process. For these meetings, team members report on:

an overview of what they have done over the previous month;

any changes made to plans, and why these changes have been made;

what has been learned in carrying out activities over the previous month (technical and other issues);

any evaluation data they have collected;

thoughts on implications for the direction of their activities and for the project as a whole.

8. NetLinkS evaluation co-ordinator

Nick Bowskill is responsible for co-ordinating evaluation activities and for collating evaluation documentation.

[Contents] [Section 1] [Section 2] [Section 3] [Section 4] [Section 5] [Appendices]

Top of page

Authors/Graphics - Nabeel Nasser and Sarah Ashton
last updated 12th December