Vol. 9 No. 2, January 2004 Contents | Author index | Subject index | Search | Home

Users and user study methodology: the JUBILEE project

Linda Banwell and Graham Coulson
Information Management Research Institute
School of Informatics
Northumbria University
Newcastle upon Tyne, U.K.

This paper presents both theoretical aspects and practical examples from the on-going and large-scale JUBILEE (JISC User Behaviour in Information seeking: Longitudinal Evaluation of Electronic Information Services) project, now in its fifth year. Particular emphasis will be placed on the importance of using robust theory and methods as the basis for reputable user studies, especially those undertaken by practitioners. Theory underlying the development of the JUBILEE project and Toolkit is outlined, followed by a demonstration of its practical use and impact during the project's on-going programme of monitoring and evaluating. Themes underlying the paper concern the importance to practitioners of the need to understand and adopt a well founded methodology and sound methods with which to conduct projects, the additional benefits they can derive by so doing, and the pitfalls and dangers of not so doing. JUBILEE is used throughout as the example project to demonstrate the points made.

Introduction: user studies methodology

In a paper written in 1980 and re-published in 2000, Wilson discusses the methods used in information research, which he sees as a particularly diverse field of study, more of an applied than a basic science (Wilson, 1980). The outputs of information science research are applied in a given managerial or organizational setting. Wilson stresses the importance of method. He sees that social survey methods are often misused to collect a great deal of data without the researcher having a theoretical (or even descriptive) framework into which to fit the data. He suggests that, in effect, collecting data becomes a substitute for thinking about the problem. This is a particular risk for practitioner-based research.

Studies are commonly seen as being either quantitative or qualitative. Quantitative data are seen as being objective, quantifiable, hard, generalisable, based on numbers, whereas qualitative data are seen as being subjective, socially constructed, soft and non generalisable, based on words. User studies are, by definition, about people, behaviour and contexts. They need both quantitative and qualitative approaches to be combined to produce the both the holistic view and the robust data needed to triangulate and thereby validate data collected. Indeed, Salomon, as described by Miles and Huberman (1994) sees the quantitative versus qualitative argument as ill-founded, seeing instead an "analytic" approach to understanding a few controlled variables, and a "systemic" approach to understanding the interaction of variables in a complex environment. Both approaches can coexist if required by the objectives of the particular study.

User studies cover a wide range of topics and foci, for example:

• Focus on the user will investigate users' wants, needs, contexts, motivations, expectations and tasks.
• Focus on use will investigate what a particular information source is actually used for, with the barriers and enablers to its use investigated.
• Focus on the information system or service will investigate aspects of technology, design and evaluation.
• Focus on the organization will concentrate on contextual aspects of the organizational setting, covering both internal and external factors such as resources, internal management procedures, internal and external strategies, which are all part of building up an holistic case study.

Given that user studies are context driven, and are typically multidimensional covering at least some of the topics and foci described above, they will generally require multidimensional methods with which to study their associated research questions.

Models describe and explain some aspect or aspects of a situation or behaviour, and can be used as the basis for prediction. Models typically focus on more limited problems than do theories, and sometimes may precede the development of formal theory (Case, 2002). User studies may well produce a model as part of their data interpretation process, which may form the basis for a project output such as a 'toolkit', or set of guidelines for practitioners to apply to their own institutions. JUBILEE is an example of a project which has developed a model (of electronic information service development in institutions), which underpins a toolkit for use by practitioners, and which uses a mix of methods in its data collection process.

In this paper particular emphasis is placed on the importance of using robust theory and methods as the basis for reputable user studies, especially those undertaken by practitioners. Theory underlying the development of the JUBILEE project and the Toolkit is outlined, followed by a demonstration of its practical use and impact during the project's on-going programme of monitoring and evaluation.

Themes underlying the paper concern the importance to practitioners of the need to understand and adopt a well founded methodology and sound methods with which to conduct projects, the additional benefits they can derive by so doing, and the pitfalls and dangers of not so doing. JUBILEE is used throughout as the example project.

The JUBILEE project

The Joint Information Systems Committee (JISC) of the Higher and Further Education Funding Councils in the United Kingdom funds a number on national information services for the use of research and learning communities in the post-16 education sector, which has recently been expanded from higher and further education to include Adult and Community Learning, and will also develop further to include school sixth forms. Information services provided by the JISC include online bibliographic databases, bulletin boards, software and data archives, socio-economic, scientific and digital map databases and electronic mail discussion lists. An aim of the JISC is to promote best practice and the use of agreed standards. The JISC strategy for 2001-5 outlines the vision of a single world-wide information environment that will support learners, teachers, researchers and administrators: it is designed to help the sector to manage change (JISC, 2000).

In 1999, the JISC established its Framework for Monitoring and Evaluating User Behaviour in Information Seeking and Use of Information Technology and Information Services in UK Higher Education to focus specifically on the development of a longitudinal profile of the use of electronic information services (EIS) and the development of an understanding of the triggers and barriers that affect such use. One of its great strengths is that the mixed qualitative and quantitative methodology generates a rich picture and a set of deliverables that have the potential to address the perspectives of, and be of value to, a variety of different audiences (Rowley, 2002).

JUBILEE (JISC User Behaviour in Information seeking: Longitudinal Evaluation of Electronic Information Services) is one of two large-scale projects funded under the JISC's MEUB Framework. It began in August 1999, and has since been extended to also include the further education sector, and is about to expand again to include Adult and Community Learning. JUBILEE is undertaking an holistic, longitudinal monitoring and evaluation, which seeks to predict, monitor and characterise information-seeking behaviour in relation to EIS, and is providing illuminative and contextualised pictures built up over time, of users and non-users in different disciplines.

Key questions for the Framework projects to answer are:

• Which electronic information services do users use?
• How do users use electronic information services?
• What are electronic information services used for?
• What influences the use of electronic information services?
• Are there variations in electronic information services use between different user groups (academics, students etc.) and disciplines?
• Trend spotting e.g. is electronic information services use changing? What influences electronic information services use over time? What will be the impact of changes in electronic information services use/availability on student learning? (Rowley, 2002).

In summary, JUBILEE is discipline-based, collecting longitudinal data on the basis of annual cycles of fieldwork in institutions, followed by analysis and development of JUBILEE outputs.

JUBILEE has twin aims:

• To improve the understanding of information behaviour in relation to electronic information services in a variety of disciplines and organizational contexts
• · To develop an evaluation toolkit for use by higher and further education managers to guide and benchmark their own institution's development in relation to the adoption of electronic information services.

Methodology and methods in JUBILEE

Methodology used in JUBILEE combines the approaches of social science, both qualitative and quantitative, with those more commonly used in an organizational setting, such as action research. The methodology must deliver a multidimensional insight into user behaviour in relation to electronic information services. JUBILEE is seeking to embed its findings in practice, and needs buy-in from practitioners in all stages of the research in order to realise this objective. A particular challenge for JUBILEE has been to use and then develop methods to specifically address the project aim of collecting and manipulating longitudinal data, with the outcome of providing a tool to promote good practice in institutions.

Methods for the JUBILEE project therefore use a mixed qualitative and quantitative approach including a literature review; questionnaires in paper and e-mail versions; face-to-face and telephone interviews; focus groups; and developmental workshops with practitioners.

Survey-based fieldwork activity has been undertaken in annual cycles of activity in Higher and Further Education institutions, and, over the four cycles of the project to date, has resulted in the following data being collected.

In-depth case studies are discipline-based, with a range of disciplines targeted for data collection in each cycle in a mix of large and small, and old and new institutions covering all geographical areas in the United Kingdom (England, Wales, Scotland and Northern Ireland). Academics, students and LIS staff are surveyed by questionnaire and interview in each institution. Snapshot case studies are shorter, conducted in institutions where deeper investigation is not feasible, with less data being collected. The breadth of fieldwork data is shown in the following table.

Small-scale action research projects within JUBILEE are providing close-to-practice data, which can be used to inform on the impact of electronic information services on institutional agenda items such as the impact of its use on student learning. Action research can obtain otherwise unavailable information, and is also a valuable dissemination mechanism to diffuse knowledge, both within institutions and more widely. It aids understanding, provides lived in experience, and is of demonstrable practical value to participating institutions. It increases practitioner acceptance of the research process, thereby reinforcing research outcomes in institutions. It should be of mutual benefit to researchers, and researched (Coulson et al., 2003).

Data collected through survey fieldwork are synthesised and interpreted to provide a longitudinal monitoring of changes in information behaviour in relation to electronic information services, within and between institutions and disciplines. This monitoring is aiding understanding and informing the development of policy by the JISC.

The synthesis and interpretation of the fieldwork baseline data collected in cycle 1 of JUBILEE permitted the modelling of critical success factors in relation to the development of EIS in institutions. This has formed the basis for the themes, which underlie the JUBILEE Toolkit, which is the second main aim of the project, under its evaluation remit.

Evaluation is the assessment of all data collected during the monitoring process, both quantitative and qualitative, with a view to determining the 'success', 'usefulness' or 'value' of the innovation (Wilson, 1980). For JUBILEE, the evaluation approach needed to be user-centred and holistic, encompassing the human and organizational aspects of a broadly defined system, that is, electronic information services..

An outcome of the project, as stated in the original project tender, is to produce:

...a benchmarking tool, in the form of an Action Plan for the use of HE managers, based on the characterisation of user-based success criteria in relation to EIS, as seen from the users' points of view. HE managers will be enabled to see how well positioned they are to exploit JISC resources and to support their decision making with respect to EIS.

In designing the JUBILEE evaluation, with the ultimate aim of producing a practical toolkit, various questions had to be answered, such as: what to include, and for whom? How to do it, and why? There are different approaches to evaluation, which suit different situations. The literature of evaluation suggests the importance of including qualitative as well as quantitative data as the basis for evaluation (Lancaster (1997), McClure (1999), Clarke (1999). Saracevic (2000) identifies the ethnographic, sociological, systems, economic and political approaches, which will be appropriate for differing audiences, with differing goals for their evaluation. The systems approach is the one most usually found in the context of electronic library projects, and indeed lies at the heart of the Tavistock model of evaluation used in the UK eLib projects (Kelleher, 1996). This is where aspects of information system performance lie at the heart of the evaluation, with data on the impact of the system on users, and data that synthesise impact more widely, also explicitly collected.

The key building blocks underpinning the evaluation are:

• the determination of critical success factors for EIS, as seen by users, and which therefore become a focus for the organization in developing its own delivery of EIS to its staff and students,
• the identification of development stages for the embedding of EIS in institutions, which can be used as the basis for benchmarking institutional practice,
• data collected through project fieldwork.

The interrelationship of these building blocks forms the basis for modelling the development of electronic information services in institutions, and forms the basis for the layout for the JUBILEE Toolkit as demonstrated in the following section.

Developing the JUBILEE Toolkit

The JUBILEE Toolkit is being constructed with the aim of improving the uptake and use of EIS in higher and further education institutions in the UK. Improvement resides in the fact that the Toolkit is designed for self application by individuals in relation to their own practice, and in relation to institutions. Individuals must identify their own strengths and weaknesses, and work towards solutions using the knowledge base and process model contained in the toolkit. The model of EIS development in institutions, which underpins the JUBILEE Toolkit characterises both the ingredients of information behaviour in relation to EIS, and a staged improvement process. The resulting framework is populated with fieldwork data, giving a knowledge base of practice based examples. It offers a template and indicative exemplars of good practice to guide practitioners in the their self-development and the development of their institutions.

Toolkits are decision making frameworks based on expert models (Oliver and Conole, 1999). The model underlying the JUBILEE Toolkit characterises both the ingredients of information behaviour in relation to EIS, and a staged improvement process:

The format of toolkits means that they can be used in a standard, linear fashion, or can be 'dipped into' by users whose level of expertise is stronger in some areas of the design process than others (Conole, 2000).

The JUBILEE Toolkit is constructed for user self-diagnosis and use. It permits flexible use giving several entry points to the data, by discipline, theme, level and stage of development, and user task (which is to be added in present cycle).

User critical success factors in JUBILEE

Critical success factors are those factors of critical significance, where effort should be concentrated, if the system is to be effective (Wilson and Huotari, 2001). Grounded theory can be used as the basis for deriving critical success factors, supported by documentary analysis and other methods used to determine the organizational context. They are useful in identifying organizational objectives, and in relating the information needs of various groups to those objectives. This is the approach used in JUBILEE. Themes were identified, which emerged from the year 1, cycle 1, fieldwork data. The focus was on optimising conditions which would result in the seamless and effective use of EIS. In year 2, cycle 2, of the project, the Toolkit was tested and refined, and was extended to the FE sector. No research of this type has previously been carried out in the FE sector, and it has been valuable in establishing a baseline of such work.

The rationale for the development of the Toolkit was described in the cycle 1 report as follows (Banwell et al., 2000):

At its most general level, the information provider's goal is to create a situation where users, academics and students, will use information seamlessly, irrespective of type or source, with confidence and trusting in its quality, in support of their learning and leisure. An individual's success criteria for information searching will be linked to achieving that goal personally. Evidence from cycle 1 fieldwork in JUBILEE suggests that the prerequisites for achieving this goal are for users to perceive they are in a situation where:
   —Access to information, including EIS, is easy for all users
   —The resource base (technical and human resources) itself is good to excellent
   —Users have the technical and evaluative skills to use information, including from EIS
   —EIS are embedded in course design and delivery, and in the research process
   —EIS are embedded in student learning
   —Quality assurance processes are in place for internet-based information
   —Seamlessness is achieved [in all aspects of service delivery to the user]

These user-based critical success factors became the basis for Toolkit themes, which provide the framework for driving change, around which a development path is being constructed as the project progresses. Barriers will be encountered by users, creating dilemmas for individuals and institutional managers. Enablers will provide solutions and will permit movement along the development path. Institutional and discipline contexts and constraints will determine the exact nature of the development path appropriate in each individual and institutional context.

Electronic information services development in institutions

In the cycle 2 report it was noted that the development of the JUBILEE toolkit was building on a recent study (Banwell et al., 1999) undertaken in the School of Information Studies at Northumbria University for the UK Office of Library Networking (UKOLN). In the 1999 project, Managing Organizational Change in the Hybrid Library, a development matrix for hybrid libraries in higher education was produced. In this project five development stages were identified (Baseline, Change, Congruence, Embedding, Full Integration) and were applied to themes to produce development paths. The themes had been identified through consultation with experts and through fieldwork and were:

1. The wider environment
2. Institutional context
3. Strategic management within the institution
4. Library service issues
5. User needs
6. Communication
7. Quality
8. Resources

Benchmarks were produced to indicate the point at which the Library and Information Service could move to the next level of development. These findings provided context for the JUBILEE project, and a similar analysis, undertaken with specific reference to EIS, helped form the basis for the JUBILEE Toolkit. Benchmarks to indicate the point at which EIS information behaviour in the organization moves from one level to the next would be increasingly characterised in subsequent cycles of JUBILEE. Generic characteristics were described for each development stage, derived from fieldwork evidence, and were applied to JUBILEE cycle 2 data. These stages are presented below in table 5.

By building up exemplars of good practice in different disciplines at different sites, evidence is being presented of baseline, intermediate and advanced development in the use of EIS. Each theme is being developed in detail to provide a development path showing the interaction between barriers to development, dilemmas for resolution and enablers, which could be employed at the local level to overcome the barrier.

The JUBILEE project is testing and developing this methodology in the context of EIS. Based on cycle 2 evidence collected in both higher and further education institutions, the project team was satisfied that the methodology from the UKOLN project is transferable to JUBILEE, in both types of institution.

JUBILEE's impact on practice

From Cycle 3 of JUBILEE on, the Toolkit has been developed in practice. In addition to the main, on-going fieldwork activity of the project as part of the longitudinal study of EIS, further longitudinal data are being collected by undertaking small-scale re-visits of case study sites from earlier cycles to determine whether evidence exists of any changes in EIS and information seeking over time. Interviewees are asked to reflect on the evolution of EIS provision and use by charting and analysing their past and current positions, using the Toolkit outline (see table 6) and a summary of their institution's case study key findings. Used together, these allow greater detail to emerge, highlighting examples of significant areas that sustain or hinder EIS development at the institution.

Another objective of the re-visits is to gain qualitative feedback and evaluation from academics and library staff as to the usefulness and validity of the Toolkit. There was a positive response from research respondents regarding the use of the Toolkit to measure changes in EIS attitudes, provision and use:

Involvement in the project is a good way of reflecting on these things for a little while, what we are doing, so it is good for us to be involved too. (Library staff member)

Fieldwork sites are providing detailed evidence of their involvement with JUBILEE resulting in positive changes in their institutions in areas such as:

• Development of an information literacy framework and action plans for developing IT skills.
• Increasing the volume and integration of EIS
• Increase in general awareness of EIS and improvement in student and staff usage.
• Improved relationships between LIS staff and academics.

The JUBILEE Toolkit is currently being populated with more data, from more disciplines and more institutions. By the autumn of 2003 it will contain data from around 25 higher and 40 further education institutions, with twelve broad disciplines included. It will have been validated both through re-visits to JUBILEE fieldwork institutions from earlier cycles, and through dissemination throughout both higher and further education. The original aim for the Toolkit was to be a tool for improvement. It is becoming clear that it is also a tool for the measurement of impact in institutions: the impact of EIS on student information behaviour, and the impact of the JUBILEE project itself through the promotion of EIS in institutions, with the use of the Toolkit as a staff development tool (Banwell et al., 2003).

The Toolkit is proving itself to be a discriminating method of monitoring change in institutions and in individuals' information behaviour in relation to electronic information services, and, in turn, use of the Toolkit is itself helping to improve practice in institutions and by individuals.

Summary

Themes underlying the paper and presentation are summarised as:

• An understanding of research methodology to inform the conduct of a project can bring the additional value of extending a "user" study to widen, deepen and extend an applied project close to practice.
• Robust and valid methods are essential to ensure replicable and comparative findings.
• A basis is provided for both monitoring, and for impact assessment in institutions.
• The pitfalls of not understanding and not using sound methodology and methods:
  • generalising from insufficient data;
  • invalid samples;
  • missed opportunities; and
  • lack of comparability between one-off projects, which constantly re-invent the wheel.
• Missed opportunities through not using robust methodology:
  • Improved outcomes for practice.

As JUBILEE matures as a project, now in its fifth year, with increasing amounts of longitudinal evidence available, the Monitoring and Evaluating User Behaviour Framework will increasingly be usable by the JISC to gauge the impact of electronic information services in the post-16 Education sector in the United Kingdom.

References

• Banwell, L., Day, J. & Ray, K. (1999) Managing organizational change in the hybrid library. London: Library Information Technology Centre.
• Banwell, L., Gannon-Leary, P. & Childs, S. (2000) JUBILEE JISC user behaviour in information seeking: longitudinal evaluation of EIS. First Annual Report, June 2000. Retrieved 13 January, 2003 fromhttp://is.northumbria.ac.uk/imri
• Banwell, L., Gannon-Leary, P. & Jackson, M. (2001) JUBILEE JISC user behaviour in information seeking: longitudinal evaluation of EIS. Second Annual Report, June 2001. Retrieved 13 January 2003 from http://is.northumbria.ac.uk/imri
• Banwell, L., Ray, K., Coulson, G. & Proud, D. (2003) Evaluation, impact and outcomes - the JUBILEE project. Performance Measures and Metrics, 4(2), 79-86.
• Case, D.O. (2002) Looking for information. A survey of research on information seeking, needs and behavior. San Diego, CA: Academic Press.
• Clarke, Z. (1999) EQUINOX: the development of performance indicators for the electronic library. In Proceedings of the Third Northumbria International Conference on Performance Measurement in Libraries and Information Services. DILM, University of Northumbria, 1999. Available at http://equinox.dcu.ie/reports/pilist.html (13 Jan. 2003)
• Conole, G. (2000) Resources for supporting decision making. First research seminar of the Learning Technology Theory Group, ALT-C 2000, Manchester, September 2000. http://www.ucl.ac.uk/~uczamao/theory/position/conole.htm (20 Dec. 2002)
• Coulson, G., Ray, K. & Banwell, L. (2003) Improving outcomes through the use of action research? Evidence from the JUBILEE project. Fifth Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, July 2003 (to be published)
• Joint Information Systems Committee (2000) JISC Strategy 2001-2005 http://www.jisc.ac.uk
• JUSTEIS website www.dil.aber.ac.uk/dils/research/justeis/jisctop.htm (1 Oct 2003)
• Kelleher (1996) Evaluation of the Electronic Libraries Programme. Guidelines for Elib project evaluation. London: The Tavistock Institute.
• Lancaster, F.W. (1997) Evaluating the digital library. In: Proceedings of the Second Northumbria International Conference on Performance Measurement in Libraries and Information Services, DILM University of Northumbria, 1997, p47-57.
• McClure, C.R. (1999) Issues and strategies for developing national statistics and performance measures for library networked services and resources. In: Proceedings of the Third Northumbria International Conference on Performance Measurement in Libraries and Information Services, DILM University of Northumbria, 1999.
• Miles, M.B. & Huberman, A.M. (1994) Qualitative data analysis: an expanded sourcebook. Sage.
• Oliver, M. & Conole, G. (1999) Assessing and enhancing quality using toolkits. http://www.unl.ac.uk/ltri/demos/media_adviser~files/Elt14.pdf (13 Jan. 2003)
• Rowley, J. (2002) Profiling and understanding user information behaviour: methodologies and meaning. ALT-C 2002 Research Paper
• Saracevic, T. (2000) Digital library evaluation: toward an evolution of concepts. Library Trends, 49(3), 350-369.
• Wilson, T.D. (2000) Recent trends in user studies: action research and qualitative methods. Information Research, 5(3) Retrieved 1 December 2003 from http://informationr.net/ir/5-3/paper76.html
• Wilson, T.D. & Huotari, M-L. (2001) Determining organizational information needs: the critical success factors approach. Information Research 6(3) Paper 108 Retrieved 13 January 2003 from http://InformationR.net/ir/6-3/paper108.html

Web Counter

© the author, 2004. Last updated: 1 December, 2003