Vol. 9 No. 2, January 2004
| Vol. 9 No. 2, January 2004 | ||||
The introduction of electronic journals into Higher Education institutions in the United Kingdom has been relatively well documented, in terms of their purchase, management and uptake. However, the impact on learning, other than trends in usage and some indications of students' and researchers' attitudes, has not been quantified. This paper evaluates a project designed with the primary aim of testing a hypothesis that learning can be enhanced by promoting the use of e-journals. It was run jointly by a member of the library staff and an academic within the Business School. A 'research quotient' was developed to measure a student's ability to carry out appropriate research to support their learning. Research quotient scores were analysed along with journal bibliographic citations in students' assignments. Analysis of the results indicated that effective collaboration between academic and library staff, the timely embedding of e-journal induction into the learning process and associating it with the assessment process, can significantly enhance the learning of students. It was also recognised that students need be encouraged to see beyond assignments and adopt an holistic approach to learning.
In 2002, the Business School at University College Worcester (UCW) decided to cancel most of their print journals in order to subscribe to two major e-journal packages: EBSCO's Business Source Elite service and the ACM Digital Library. Business Source Elite provides full text from over 1,090 journals covering business, management, economics, banking, finance and accounting. In addition to the full text, this database offers indexing and abstracts for over 1,600 journals. The ACM Digital Library provides access to over 102,500 full-text articles from journals, magazines, and conference proceedings as well as many citations and tables of contents. The rationale for this change was the belief that learning would be enhanced by the significant increase in titles available and by the easier access, especially for part-time and distance students. Some academic staff also hoped that electronic access would prove to be more attractive to students, many of whom did not make much use of journals.
Academic staff were disappointed when the inspection of assignments suggested that the introduction of e-journals was not affecting how students were carrying out research to support their learning. In order to encourage students to use the new resource and to exploit the benefits of electronic searching, it was decided to trial a strategy of actively promoting e-journals.
This paper evaluates a project that implemented this strategy, which was designed with the primary aim of testing a hypothesis that learning can be enhanced by promoting the use of e-journals. It was run jointly by a member of the library staff and an academic within the Business School and was funded by the Learning and Teaching Centre at UCW, to run over the academic year 2002-03.
The project was based on the assumption that an ability to carry out research is essential to a student's learning, a view that is held in the Higher Education community. For instance, the U.K.'s Quality Assurance Agency Subject Benchmark Statement for General Business and Management states that, "graduates are expected to be able to demonstrate... abilities to conduct research into business and management issues... and for such to inform the overall learning process." (Quality Assurance Agency, 2000, section 4.1k). Sconul (2000) refers to the ability as "a 'tool' for the 'job' of being a learner". Doherty. et al. cite a report on U.S. undergraduate education which:
..challenges universities to rethink their traditional instructional models, to move to a model of inquiry-based learning wherein the student is involved in research from the beginning.....The skills of analysis, evaluation, and synthesis will become the hallmarks of a good education, just as absorption of knowledge once was. (Doherty. et al.1999)
Other outcomes of the project were expected to be increased collaboration between library staff, teaching staff and students, improved learning autonomy and critical use of learning resources in students, and an increased awareness of the potential of library resources by academic staff.
A review of literature related to e-journals was undertaken. The vast majority of papers fall into two categories. The first category of article focus on the implications to library services, in terms of policies and strategies, working practices, funding issues, control, cataloguing and access management. These are not relevant in the context of impact on learning.
The second focus on the users and their attitude to e-journals, usually within a single institution. Many look from the point of view of academic staff and/or researchers, rather than undergraduates, and there is a general perception that postgraduates and researchers have accepted e-journals more enthusiastically than undergraduates (e.g. Wolf, 2001, Nelson, 2001, Lock, et al., 2001). These studies identify many common factors which affect acceptance or resistance to e-journals. Advantages are seen to be the convenience of desktop, twenty-four-hour access, the range of titles available, remote access, time saving and ease of searching. Barriers to use can be a lack of awareness or knowledge of how to access them, technical problems, lack of archives, password requirements and high user expectations.
Several major initiatives have been undertaken in the area of user behaviour by JISC (the UK's Joint Information Systems Committee of the Higher Education Funding Councils) of which the JUBILEE and JUSTEIS projects give valuable information on student use of electronic resources over several years (3rd Annual Report of the JUBILEE project, 2003). Of those students sampled, two-thirds claim to be dependent on Electronic Information Services, for similar reasons given in the earlier studies. As well as the technical difficulties etc., the time to search and to acquire competence emerge as issues.
The JUBILEE project has also identified that level of take up of services by students is dependent on factors such as training, awareness and promotion to and by academic staff. The importance of promoting the services, and the need to improve users' information skills also emerged from other papers. Aston University is one of the few institutions that reports very positive uptake of e-journals by undergraduates as well as postgraduates and staff (Mallett & Smith, 2001). They put this down to pro-active and careful marketing, and training which is embedded into undergraduate programmes of study. "However, the key to student acceptance..is the promotion of the sources to staff who then recommend these sources to the students." The need for skills training to be embedded in subject areas to maximize its effectiveness is also asserted by Roberts (2001), who argues that research such as the IMPEL2 project in the late 1990s makes it clear that, "the promotion or embedding of electronic information services into the learning process must begin with partnership".
It is notable that none of the papers reviewed attempts to quantify the effect upon learning, beyond measurements of uptake, and there is an implicit assumption that easier access to a greater range of resources must have a positive impact upon the student experience. The question must be asked whether the expense and activity being undertaken to introduce and manage e-journals is actually worthwhile in educational terms.
Several approaches were taken in order to test the hypothesis that promoting e-journals enhances learning. An e-journal induction was delivered to forty-eight students registered on a Level 2 Information Technology module [that is, a module taught to students in Years 2 and 3 of an undergraduate programme] and a 'research quotient', calculated from responses to a questionnaire, was developed. This quotient was designed to measure a student's ability to carry out appropriate research to support their learning. Prototypes of the e-journal induction and questionnaires were trialled on another group of students and consequently refined.
The process then employed was to:
In addition, the bibliographies from the students' module assignments were analysed with regard to the range of resources used, and compared with assignment bibliographies submitted by previous cohorts taking the same module.
Finally, focus groups were used as a means of obtaining qualitative data from the students by exploring their experiences and attitudes towards using journals and e-journals.
UCW undergraduate degree programmes currently have an undifferentiated Level 2, with students in their second and third years of study sharing modules, and with a diverse background in terms of entry routes and study pathways. Embedded library skills instruction is inconsistent, and the undifferentiated Level 2 makes it very difficult to track the acquisition of skills.
It could be assumed that there were almost certainly widely disparate information literacy levels and experience of e-resources amongst the students taking the module. This assumption was backed up by experience of dealing with students in the library (at the Enquiry desk for instance), anecdotal evidence and the questionnaires prototyped with an earlier group of students.
Experience suggested that students would be particularly lacking in the areas of advanced electronic search skills and evaluation skills, as well as having a limited experience of journal literature; what journals were and how they differed from other types of information. Feedback received in the training sessions and focus groups (see below) confirmed the validity of these assumptions.
The aims of the training session were therefore:
The content of the e-journal training session was based upon an analysis of the research requirements of second-year students, as defined in a paper by Godwin (2002). This identified appropriate benchmarks based upon the SCONUL headline Information Skills (SCONUL, 2000) and are listed below in relation to the Research Quotient.
The session started with an introduction to the nature of journals, with printed copies being handed out and the students encouraged to feedback their impressions. This was an interesting experience and it was clear that, as expected, many students were not familiar with journals. They successfully recognised the currency and authority of the content; some were daunted by the technical nature of some of the articles, and most were shocked by the price! They were then helped to identify the pieces of information (author, source etc.) that could be entered into different fields of a database, a concept with which they were familiar as IT students. It was explained how these could then be used to retrieve an electronic journal article.
The students were then shown how to access the various e-journal resources available through the library service, using a PowerPoint presentation. A practical session immediately afterwards allowed the students to gain experience in using the resources, with members of library staff on hand to answer questions and give help. The impression gained was that most students tried out the resources, with varying degrees of enthusiasm, but not many of them worked through the exercise sheet provided. This might have been because they felt confident as IT students to engage with the software. It meant however that they missed out on tips and examples specifically designed to help them research the assignments they had just been set, which would have reinforced the learning advantages of e-journals.
In order to test the hypothesis that learning can be enhanced by promoting the use of e-journals, a Research Quotient (RQ) was developed. This was designed to measure a student's ability to carry out appropriate research to support their learning; specifically which resources they used, how they located and selected them, and the electronic search techniques they used.
The students' answers from the questionnaires were compared against appropriate benchmarks from Godwin's paper (Godwin, 2002), as indicated in Tables 1- 4, below. A scoring system was devised with weighting designed to reflect any change towards attaining higher level benchmarks. All the benchmarks relate to Level 2 unless indicated otherwise.
[In the tables IS is 'Information Skill' (SCONUL, 2000) and L is 'Level']
| Skill and benchmarks | Weighting | |
|---|---|---|
| Q1-4: tick the boxes which describe the resource you normally use when doing research for an assignment | IS 2 Packaging of information and choosing suitable sources for research
| High score
|
| Skill and benchmarks | Weighting | |
|---|---|---|
| Q5-10 Tick the boxes which describe how you locate resources. | IS 2 Packaging of information and choosing suitable sources for research
| High score
|
| Skill and benchmarks | Weighting | |
|---|---|---|
| Q11-13 Tick the boxes which best describe how you select relevant resources | IS 5 Comparing and evaluating information
| High score
|
| Skill and benchmarks | Weighting | |
|---|---|---|
| Q14-17 Tick the boxes which describe the frequency that you use the following electronic search techniques... | IS 4 Locating and accessing information
| High score
|
The students who had received the e-journal induction were asked to complete the questionnaires on two separate occasions. Firstly, immediately before the delivery of the e-journal induction and then secondly, five weeks later. This second occasion took place after the students had had the opportunity to apply the lessons learnt from the e-journal induction to the completion of an assignment. Although more than forty questionnaires were completed by students on both occasions, only thirty pairs of these questionnaires could be matched and the remainder were discarded (for analysis purposes) - this lack of correspondence was not unexpected as not all students attended both lectures and other students intentionally or unintentionally failed to include identification on the completed questionnaires.
Data from the matched questionnaires were used to calculate the respective RQ. The result of these calculations was an average RQ of 64.3 out of 144 before the e-journal induction compared with an average score of 75.5 out of 144 in respect of the follow-up questionnaires. As yet no comparative benchmark data exist for the RQ.
Significance testing was used to establish whether the matched RQ scores suggested that promoting e-journals had affected how students were carrying out research to support their learning. It was assumed that the underlying data followed a normal distribution and so testing was performed using the standard test for comparing means, when matched samples are available. The null hypothesis, alternative hypothesis, significance level and formula for the calculated value of t are formally stated below
The calculated value of t is 3.82, compared to a t 5% (df=29) value of 1.699. This suggests that we must reject the null hypothesis at the 5% level and accept that promoting e-journals had affected how students were carrying out research to support their learning. The data are also significant at the 0.1% level.
The increase in the RQ was particularly influenced by scores in four of the seventeen questions. The scores for these four questions are considered below.
Question 2 was, "Tick the boxes which describe the resources you use when doing research for an assignment", using the four-point scale, "Always use—Normally use—Occasionally use—Never use" and Figure 1 aggregates the answers to this question.
The data indicate an encouraging increase in journal usage for assignment research. Students, with one exception, who, before the e-journal induction, never used journals to research assignments, now consider that they use journals occasionally. This is matched by occasional-use students, who now perceive themselves as normally using journals for assignment research. Inspection of bibliographies indicates that the majority of this enhanced journal usage is of e-journals. The high use of e-journals by these undergraduate students is markedly different to the behaviour of undergraduate students reported in Wise (2003), where only 5% of undergraduates used e-journals. As no data are available on printed journal use in the Wise survey it is perhaps sensible not to speculate on the markedly different usage pattern in the UCW project.
The authors suggest that increased utilisation of journals by these students, as a source of information, implies a maturing attitude to their learning. It can be argued that this maturity manifests itself in an awareness of the significance of the peer review process and of the need to evaluate information from a range of sources.
Questions 7, 16 and 17 query "the ability to locate and access information".
Question 7 was, "Tick the boxes which describe how you locate resources when doing research for an assignment" using the four-point scale, "Always use—Normally use—Occasionally use—Never use" and Figure 2 aggregates the answers to this question.
The increased use of journals (Question 2) has been accompanied by a corresponding increase in electronic database searching for journal location purposes. This is, perhaps, not surprising as the two major e-journal packages that were introduced include their own search engines and do not include a browse facility.
Questions 16 and 17 were subdivisions of the general question, "Tick the boxes which describe the frequency that you use the following search techniques to interrogate an electronic database", using the categories, "Boolean operators (i.e., and/or/not)" and "Phrase/proximity searching". The same four-point scale as in the previous question was used, and Figures 3 and 4 aggregate the answers to questions 16 and 17.
The data from these questions indicate that after the e-induction training, students are making more use of advanced search techniques. This increased use was confirmed in the focus groups (see below), when students also articulated their frustration in carrying out advanced search techniques. The authors suggest that advanced search techniques facilitate more efficient retrieval of relevant e-journal articles and it is reassuring to observe their acquisition and application. Even if we accept that students assess themselves as being more skilled than they actually are (Cmor and Lippold, 2001), the results indicate a pleasing relative increase in their skills.
The authors suggest that the increase of scores for questionnaire questions 2,7,16 and 17 are attributable to some or all of the following factors: the on-line access of e-journals and search engines, the collaboration between academic and library staff, the timeliness of the e-journal induction and the association of the e-journal with an assessment. These are considered below.
E-journals had been available for a semester prior to the project and on-line search engines for paper journals were well established. Furthermore, the e-journal search engines included filtering mechanism to identify held journal stock and this represented an improvement on the more generic search engines that had been previously used. Inspection of bibliographies submitted prior to the project did not indicate any change in the way that students were carrying out research and this suggests that availability is not a catalyst, in itself, for the improvement in the scores to questions 2,7,16 and 17.
Many students receive formal research inductions at the start of Year 1 that include content similar to that presented in the project. This approach is at odds with the progressive approach advocated by SCONUL (2000) A strategy of concentrating training in Year 1 induction can be criticised firstly, as it will be competing with many other academic and social activities that occur at that time, and secondly, as students are not always expected to use the full range of research skills in the first year of their studies. This lack of expectation will not encourage students to invest in the time required to become familiar with the advanced search techniques that the project team believe are essential for efficient identification of relevant articles. The timeliness of the e-journal induction (an assignment requiring research was issued on the same day) and of it being embedded in the Learning and Teaching process (as suggested in Roberts, 2001) ensured that the content was relevant and in the context of the immediate needs of students. Students were able to practice the lessons learnt at the same time as completing an assignment. The collaborative nature of the e-journal induction (delivery by academic and library staff) not only ensured that it was not viewed by students as a bolt-on, but also empowered the lecturer to champion the lessons in the weeks that followed.
It became clear in the focus groups that some students invested a considerable amount of time in mastering the lessons learnt in the e-journal induction. The authors suspect that this investment was driven predominately by the inclusion of a specific assessment criterion in the assignment. This focus on assessment, supporting the views expressed in Gibbs et al. (1998), paints a disappointing image of students not adopting a holistic approach to their learning.
The bibliographies submitted by the students with the assignment completed after the e-journal induction, were examined. Bibliographic entries were categorised as being from a book, journal or website. The same categorisation was also performed on the bibliographies of thirty students who completed an equivalent assignment in previous occurrences of the module, but would not have attended an e-journal induction. The data shown in Table 5 were obtained.
| Books | Journals | Web sites | Total | |
|---|---|---|---|---|
| Non-induction students | 4.0 | 0.8 | 2.0 | 6.7 |
| e-induction students | 4.7 | 2.5 | 1.8 | 9.0 |
Significance testing was used to establish whether the number of journal bibliographic entries suggested that promoting e-journals had affected how students were carrying out research to support their learning. It was assumed that the underlying data followed a normal distribution and so the difference in journal usage was tested using the standard test for the difference between means, when two small samples are available. The null hypothesis, alternative hypothesis, significance level and formula for the calculated value of t are formally stated below
The calculated value of t is 2.40, compared to a t 5% (df=58) value of 1.67. This suggests that we must reject the null hypothesis at the 5% level and accept that promoting e-journals had affected how students were carrying out research to support their learning. The data are also significant at the 1% level.
These data, the authors suggest, are consistent with the students changing the sources that they use for research, in a direction that was emphasized in the e-journal induction and reinforced by academic staff afterwards. It was expected that some overall increase in reading would result from the efforts of staff, but that this increase (34%) derives principally from a 217% increase in the use of journals is very encouraging. It is slightly disappointing that the students who had received the e-induction still included approximately the same number of Web site entries (1.8 per bibliography) as other students, as attention had been placed on the potential unreliability of un-refereed websites.
Focus groups were organised towards the end of the project to enable the exploration of issues that had arisen. Two focus groups, each including six students were chaired by staff from the library and the proceedings were recorded onto audio tapes.
Members of both focus groups highlighted the wish of students to receive effective support, to enable them to fully exploit the potential of e-journals. Although the consensus view of the e-journal introduction was that of "a good introduction", one student expressed the concern that "it took me another half day to learn how..". The contribution of this student to the discussion suggests that this had been an effective exploratory learning experience, in line with the findings of Rieman (1996). It had not been the intention to emphasise an exploratory learning style and if this approach is adopted in future it would be necessary to specifically engage students in this strategy. Students' opinions were polarized on whether e-journal induction should be delivered as traditional induction sessions or embedded in the learning process, an approach recommended in Roberts (2000). The anti-induction lobby argued that, as inductions are invariably included as part of Freshers' Week, they attract variable attendance, they are missed by both late entrants and direct entrants, and they present material that is more relevant in subsequent years of the course. Both groups of students agreed that an induction might be better placed in Year 2, when students' references and bibliographical entries count towards assessment. Students advocating the inclusion of learning support in lectures were split between those in favour of dedicated study skills modules and those in favour of embedding the learning in mandatory subject specific modules. It was suggested that "lecturers should embed e-journals in the teaching process to encourage interest" by including more e-journals in reading lists and providing keywords that can be used for search purposes and also suggested that "e-booklets might be introduced" to support exploratory learning.
The discussions offered students the opportunity to articulate the difficulties they had experienced. These included:
It became clear that time is a critical factor in selecting resources, in line with research in Roes (1999). The completion of assignments in a short time frame was perceived by students as being inconsistent with ordering paper articles using the inter-library loan system, even though such articles are normally delivered within one week. The benefits of instant access mean that the students are prepared to consider e-journals as a resource in learning, that is consistent with the "least effort research model" proposed by Chrzastowski (1999). Furthermore, the availability of search engines potentially contributes to meeting their criteria for acceptability. One student was enthusiastic at no longer having to "browse paper journals" The requirement for timeliness could explain the nature of the difficulties listed above.
It is also noted that, not surprisingly, some students considered some material "over my head", but, surprisingly, some students had not acquired the skill of evaluating an article by reading the abstract, but reverted to "skimming the article". Perhaps this last observation is an indication that instant on-line access diminishes the importance of abstracts.
The challenge the team overcame was not one of persuading students to move from paper journals to e-journals, but of encouraging students to use journals more generally.
The results supported the team's belief that success depended on the collaboration of library and academic staff and on the timely embedding of the e-journal induction in the learning process. The team did not see the initial e-journal induction session, presented by library staff, as a bolt-on session. Instead it was one part of a broader induction package that, critically, included academic staff participating in the initial e-journal induction session as well as reinforcing and championing the lessons learnt in subsequent lectures. The e-journal induction session was synchronised with the issue of an assignment that included a relevant assessment criterion. This was also crucial, in offering students the immediate incentive of using e-journals in a relevant assessment situation.
The project team believe that the project was successful in enhancing the learning of those students who received the e-journal induction. This belief results from the increase in Research Quotient scores recorded by these students, from the difference in research reading profile (compared to other students) and from some of the thoughtful comments made in the focus groups.
Further reinforcement will be needed to ensure that the improvement in learning and teaching is maintained, and so the team plan to embed the use of e-inductions in other mandatory modules. Hopefully, other academic departments within UCW can be encouraged to employ a similar strategy. The team is also mindful that it must address its concern that both the condensed nature of assignment completion and the focus placed by students on assessment paint a disappointing image of students not adopting a holistic approach to learning.
The need to review the delivery of the actual e-induction is anticipated. Perhaps, a revised initial e-induction session will require students to adopt an exploratory learning style and overcome the expectation of some that the induction is a complete learning experience in itself.
| Find other papers on this subject. |
| ||
© the authors, 2004. Last updated: 7 January, 2004 |
|