Information Research, Vol. 4 No. 1, July 1998
Information Research, Vol. 4 No. 1, July 1998 | ||||
|
|
|
|
|
Using the World Wide Web (WWW) as an interactive educative tool is still a relatively new concept, and little is known of its impact on learning when it is used as a dynamic learning tool. Despite this the use of educational internet sites, in the form of virtual classrooms and courses, appears to be increasing rapidly. Thus, it is important that their ability to facilitate learning is evaluated. We present the findings of a preliminary study which examined the amount, type and quality of leaning of an undergraduate indroductory history course when presented to three different groups of participants. All participants received four regularly spaced 30 minute study and repeated test sessions over an eight day period. A final test of new questions was also administered at the end of the study. Results showed that the amount of historical knowledge acquired by the end of the study was greatest for those paticipants who learnt using traditional methods, and that over the four test sessions this group consistently outperformed both computer groups. Moreover, the way in which knowledge was acquired was qualitatively different in the groups with the traditional group exhibiting more 'Know' responses while the Intranet group exhibited more 'Remember' responses. Finally, using useability questionnaires, we found that participants preferred learning via traditional methods to screen and Intranet presentations, and that participants who had learnt using computers felt that their learning experience had suffered. These findings have important implications for educators, and others who wish to use the Internet as a training tool, and we discuss our findings through the evaluation of the different presentational media used, specific Intranet design criteria and general usability factors, which, we suggest, are of paramount importance.
The concept of the Internet began in 1968, when the US Department of Defense connected four sites over a computer link. In 1980, Tim Berners-Lee designed a system to facilitate collaborative working within his organisation and it was the expansion of this which led to the birth of the WWW. This has developed enormously and largely unchecked, as a tool to share vast amounts of information between those connected to the Internet. Use of the Internet has increased rapidly from an estimated three million in 1990 (Nielson 1995) to approximately 50 million in 1998 (Phillips & Long 1998). Until very recently the WWW has been used primarily as an information resource. However, there is now a shift towards using it more dynamically, by allowing user interaction, participation and even in the auto-correction of multiple-choice questions. Thus, recent evidence suggests a move away from the use of the Internet as a database resource, towards more specific learning activities (e.g., The Computers in Teaching Initiative law and physics courses). Furthermore, Intranets, individually tailored packages, are becoming more prevalent as they are able to take advantage of the WWW platform while providing a structured content focused towards the aims of specific users. Many organisations, and in particular educational institutions, have implemented Intranets, because of the possible benefits for facilitating distance and student directed learning.
There has been little evaluation of either commercial off-the-shelf (COTS), or in-house software used in educational intranets, particularly with regard to their most important aspect, namely the extent to which they achieve their aim as learning tools. A brief search of the Social Science Citation Index, yielded only 228 entries under World Wide Web and less than 20% described educational 'impact'. Moreover, those that did, tended to concentrate on usability issues. It would appear therefore, that the fundamental issues of memory and learning and the integration of these factors within interface design have largely been ignored. The study reported here is part of a larger research programme which aims to address this imbalance by studying Intranet design, from both educational and usability perspectives.
Learning is a complex skill. Long gone are the days when it was seen just as an almost 'automatic' process stemming from behaviour reinforcement and conditioning (see, McGeoch & Irion, 1952). Learning is not just a matter of acquiring information, but also entails remembering and recollection and the application of acquired information to master and develop new skills. There are a number of generally accepted aspects to learning including, attention, practice, organization and consolidation (for a review see, Baddeley, 1997). Attention relates not only to the manner in which one attends, but also to the degree of motivation. The amount learnt can be expressed as a function of study time, however, equally important for effective learning, is the distribution of that time, the 'total time' and 'distribution of practice' hypotheses, respectively (Ebbinghaus, 1885). The material to be learnt needs to be organised, not only for ease of immediate understanding, but in order for it to match the existing organization within long term memory and for an appropriate link between these two to be created. Consolidation of the material learnt is required to facilitate its use and application. These differing aspects indicate the complexity of learning and the necessity to apply such knowledge to the construction and evaluation of Intranets.
More recently it has been found that different types of learning can occur depending on the nature of the to-be-learnt materials and the way in which those materials are presented, (Conway, Gardiner, Perfect, Anderson and Cohen, 1997). These different types of learning can be characterised by the type of recollective experience that occurs at recall and were originally described by Tulving (1985) as the " 'conscious state' or nature, in which a memory is recalled". Tulving originally described two such states: 'Remember' and 'Know'. Knowledge which is remembered is characterised as being recalled in association with related information about the actual learning episode. Thus, recall here can be best though of as taking place in conjunction with the recall of mental images, feelings and associated other memories that occurred at encoding. In contrast, known knowledge is recalled without such any conscious recollection and is thus best characterised as information which is simply based on a certain sense of familiarity, eg: one's own birth date.
Conway et al. (1997) used multiple-choice answers to test levels of learning in first year psychology students and also measured students ratings of their own awareness of their answers. They used four response categories: 'Remember', 'Know', 'Familiar' and 'Guess'. Familiar and Guess were included as these measure are sub-divisions of familiarity to an answer which allow finer classifications of responses to be measured. This prevents an artificial inflation of Know responses caused when only two categories are used. Responses to questions relating to both lecture and laboratory research methods courses, were obtained and these results were viewed against the students' final degree classification. It was found that the higher performing students gave more 'remember' responses for lecture courses and more 'know' responses for the more practical, research methods courses. A later retest for one of the lecture courses revealed, however, a shift from 'remembering' to 'knowing'. The authors concluded that these results suggested a change in the underlying representation of knowledge, depending on the academic level, retention interval and more importantly the nature of the material learned. This appears to implicate the importance of not only the material presented, but also the manner in which it is taught; traditional lectures versus the more practical, 'hands on' methods courses. Moreover, these findings suggest the quality of learning may vary depending on the medium of presentation. Thus, the implications for Intranet learning, with its highly novel and predominantly visual nature, could possibly be quite marked.
Presentation of a learning course via an Intranet, not only requires the material to be constructed to facilitate memory, but also needs to address the important consideration of the interaction between the learner and the computer. According to the International Standards Organisation: "The usability of a product is the degree to which specific users can achieve specific goals within a particular environment: effectively, efficiently, comfortably, and in an acceptable manner (1991)". To achieve this, there are a number of criteria which need to be met. According to Noyes and Mills (1998) these comprise: generic design criteria, namely learnability, effectiveness, attitude (with respect to a positive attitude from the end user), flexibility; and secondly, specific design issues, relating to various aspects of the structure and format of the presented material.
Nineteen volunteers, 12 women and seven men, were recruitted from the psychology department participant volunteer panel at Bristol University. Participants were selected such that none had a formal knowledge of or expressed interest in History. Participants were randomly allocated to the three conditions.
A short course, in early English history, was devised. This choice was made because the content of a history course shares many similarities with many undergraduate courses in other disciplines. The course was designed to contain not only text, but also pictures, other graphical representations and names and dates. The course was produced in identical content (text, colour and monochrome pictures and family trees) for each of three conditions. The first comprised a printed form presented in a small booklet for the 'pen and paper' group. The second contained the same material presented on computer screen, with the user solely able to direct their movement between pages of text. In contrast, the third group, received replicated material presented via an Intranet facility, that was ergonomically designed and which made use of hypertext links, navigation buttons, menus and other features typical to WWW sites.
Two sets of 36 questions were compiled; these varied in their nature, including some specifically relating to pictures or dates. Answers (three multichoice options), confidence ratings (guess, low confidence, moderate confidence, confident and certain) and awareness ratings (remember, know, familiar and guess) were required for all questions. The same questions were given to each group; however, the nature of the presentation was varied to match the presentation of the course material. The position in which the correct answers appeared was randomised.
A usability questionnaire was designed, based upon Brooke's (1996) system usability scale. This required responses to questions relating to the nature of the presentation for all conditions, and to the level of usability for the computer screen and intranet systems.
A pilot test indicated that 20 minutes reading time would be sufficient to complete the course material regardless of presentation media. Thus, each participant was given this amount of time in each session. After this period, they were immediately asked to complete their three choice multiple-choice answers along with two corresponding rating scales presented underneath each question. The first of these was Memory Awareness: 'Remember', 'Know', 'Familiar' and 'Guess' in which particpants rated their awareness of the answer they had just given. The second was Confidence of Answer which was rated on a 1 - 5 scale where 1=Guessing and 5=Certain. Instructions where standardised between groups.
This study was carried out over eight days; and comprised two parts. The first part comprised four sessions of reading time, followed in each case, by a set of 36 three choice A,B,C multiple-questions. These were constructed such that different questions measured learning of either text, picture / graphical content and names and dates. Testing was carried out at regular intervals over the study period and after each of the study phases particpants received the same 36 questions.
At the end of the last study phase and test participants were also given a second set of questions (again with rating scales) and were asked to complete the usability questionnaire on the final day. No additional time for re-reading the course material was allowed prior to testing on the second set of questions.
The raw data was recoded prior to analyses, to allow for the guessing and chance levels, using a high threshold model. This model assumes 12 out of 36 correct answers are obtainable by pure guessing, thus in order to score 50%, the participants would have to score 24 out of 36. As the numbers of observed familiar responses was very low it was decided to collapse the Familiar and Know responses together. Thus subsequent Memory Awareness analysis consists of three measures, Remember, Know and Guess.
ANOVA analysis of the percentage of correctly recalled answers in each group showed a reliable effect of test session (F(3,17)=61.898, MSe=802.891, p<0.01). means are summarised in figure 1.there were no group effects at test sessions 1, 2 or 3, however, by test session 4 differences were beginning to emerge (f(2,17)=3.588, mse=499.921, p=0.05). further analyses around planned comparisons showed a reliable difference in score between the paper and the intranet groups (f(1,17)=6.547, mse=912.071, p=0.03).
Two separate ANOVA analyses of memory awareness were carried out on the three ratings, Remember, Know and Guess. First, the quantity of the different awareness types produced in each group, at each test, was examined. The quantity was calculated as Hit (A) X Total number of Hits, where A equals each awareness type R,K and G
A three factor ANOVA was performed on the data. Factor 1: Between Participants Factor, Group with 3 levels: Web, Paper & Screen, Factor 2: Within Participants Factor, Time Period with 4 levels: Day 1, Day 5, Day 7,Day 10, Factor 3: Within Participants Factor, Awareness with 3 levels: Remember, Know and Guess. A reliable effect of awareness was shown over all three groups (F(2,18)=41.193, MSe=3.626, p<0.01) The means were Remember = 57%, Know = 29%, Guess = 14% . Thus, as expected, there is a general tendency to produce ‘Remember’ responses.There were no other significant main effects,(F’s <1)
Interaction effects: Goup x Test n.s. (F<1.3), Group x Awareness n.s. (F<1), Test x Awareness significant @ p < 0.01 and Group x Test x Awareness n.s. (F<1)
As shown above none of the interactions were significant with the exception of Test x Awareness, F(6,12)=37.647, MSe=0.691. The means of this interaction are shown in Figure 2.
As can be seen in Figure 2 over repeated test sessions the proportion of ‘Remember’ responses increase.There is a corresponding decrease in ‘Just Know’ and ‘Guess’ responses.
Second, the accuracy of the different awareness types produced in each group, at each test was examined. Accuracy here refers to calculations by which it can be ascertained whether a given type of Memory Awareness response is an actual genuine awareness response or just an indiscriminately selected response. An ANOVA analysis showed a reliable effect of awareness over all three groups (F(2,18)=41.193, MSe=3.626, p<0.01). Mean accuracy rates were: Remember = 88%, Know = 61% and Guess = 39%. These results show that ‘Remember’ responses are more accurately discriminated than 'Know’ and ‘Guess’ responses and that ‘Know’ responses are more accurate than ‘Guess’ responses.There were no other significant main effects, Group, (F<1) and Test, (F <2.5). There were no interaction effects. HoweverTest x Awareness did approach significance, F(6,12)=2.053, MSe=0.663, p=0.065. The means of this interaction are shown in Figure 3.
As can be seen in Figure 3 over repeated test sessions ‘Remember’ responses remain highly accurate. ‘Just Know’ responses also remain stable but are somewhat less accurate. However, ‘Guess’ responses show less accuracy and more fluctuation.
Finally, the same three factor ANOVA was performed on correct response data for the memory confidence ratings. Time period showed a reliable effect (F(3,17)=179.269, MSe=11.736, p<0.01). Mean confidence at: test 1 = 2.8, test 2 = 3.7, test 3 = 4.2, test 4 = 4.5. A significant group x test interaction effect was also observed (F(6,13), MSe=0.201, p<0.02). The means are summarised in Figure 4.
Thus, as expected, over repeated test sessions confidence of answering increases across all groups.
The following analyses are centered upon the new questions presented at the end of the study.
ANOVA analysis of the percentage of correctly recalled answers showed a reliable effect of test group (F(2,16)=5.433, MSe=551.76, p<0.02). Planned comparison around the means showed reliable differences in score between, paper and intranet groups (F(1,17)=5.130, MSe=520.964, p<0.04); and the paper the screen groups (F(1,170=10.385, MSe=1054.688, p<0.01). The means are shown in figure 5.
There was no significant difference between the screen and intranet groups.
ANOVA analysis was next performed on the memory awareness ratings. First in the quantity analysis a reliable effect of awareness over all three groups was shown (F(2,160=18.220, MSe=0.599, p<0.01). Means: Remember = 40%, Know = 46% and Guess = 14%. Planned comparisons around the means showed reliable differences in awareness between, 'Remember' and 'Guess' (F(1,17)=21.175, MSe=0.696, p<0.01); and between 'Know' and 'Guess' (F(1,17)=32.313, MSe=1.063, p<0.01). There was no significant difference between 'Remember" and 'Know', (F<1.2). Thus, as expected, there is now a tendency to produce ‘Remember’ and ‘Just Know’ responses equally. No effect of group x awareness was observed, (F<1.5). Mean quantity proportions are summarised in Figure 6.
It can be seen that there is now a tendency in the Paper and Screen Groups to produce more ‘Know’ than ‘Remember’ answers. However, in the Intranet group slightly more ‘Remember’ than ‘Know’ items are produced. More importantly, guessing is higher in the Intranet group.
Once again ANOVA analysis of accuracy levels were computed. Accuracy showed a reliable effect of group (F(2,16)=4.304, MSe=0.084, p<0.04). Mean accuracy: Pen & Paper Group = 53%, Intranet Group = 57%, Screen = 44%. Thus, awareness accuracy is the same for the Paper and Intranet groups, but the Screen group are less accurate.
A reliable effect for awareness was also found (F(2,16)=15.552, MSe=0.668, p<0.01). Mean accuracy: Remember = 68%, Know = 55% and Guess = 32%. Planned comparisons around means showed reliable differences in awareness between, 'Remember' and 'Guess' (F(1,17)=30.385, MSe=1.305, p<0.01, 'Know' and 'Guess', (F(1,17)=12.187, MSe=0.542, p<0.01, and 'Remember' and 'Know', (F(1,17)=4.085, MSe=0.175, p=0.055. No effect of group x awareness was observed.
Analysis of memory confidence found no effect of group.
Analysis of responses to the usability questionnaire were also conducted. It was found that all groups were consistent in their agreement for the future inclusion of three presentational factors to aid learning. Firstly, in response to the question, "Being able to make notes would have helped", 68% strongly agreed. Second, to the question "Feedback on each answer would have helped", 79% strongly agreed. Third in response to the "Questions throughout the text would have helped", 63% strongly agreed. The screen and intranet groups both found their individual systems easy to use (77% agreed or strongly agreed) and felt comfortable with their systems (69%). However, both groups disagreed (76%) with the crucial statement "I think I could learn more using this method than using a book".
An increase in learning over time was found across all groups. However, reliable differences were shown for a higher level of correct responses from the pen and paper over the intranet group. Initial answers showed the expected tendency for increasing remember responses as test sessions progressed. Later answers were found to indicate an overall shift from remember to know responses. This finding lends support to the Conway et. al. (1997) results. The implications are that the underlying representation of knowledge changed as learning was consolidated for later application. This trend was not, however, observed for the intranet group. There were no clear indications as to why this was so; and it will require further research to isolate possible variables. However, it seem certain that learning via the Intranet may well cause substantial qualitative differences in the ways in which knowledge is acquired. This then has considerable implications for the designs of such systems, and suggests that educators need to be made much more aware that teaching via these systems have such effects. Moreover, our findings clearly show that teaching should not rely on such methods. Rather our findings reinforce the notion that traditional teaching methods still have a vital role to play any educative process. This does not deny that WWW systems can play a useful role in education. However, our findings do suggest that much more careful exploration of their educative effects is essential.
Remember responses were found to be more accurate than know responses, which in turn were more accurate than guess responses. Both remember and know responses were stable, however, guess responses fluctuated in their accuracy. This provides confirmatory evidence, that awareness states are associated with degrees of consolidation and application of knowledge.
Analysis of confidence ratings showed a predicted trend, namely increasing positively as study periods increased. That there were no erratic variations in these levels, indicated that there were no confounds relating to confidence which may have unduly affected the overall findings. Alternative confounds were also possible due to repetition of the questions, with participants being able to focus their learning to match the presumed set of forthcoming questions. Although such a variable would have affected all conditions, the second set of questions was introduced to determine whether patterns from the first part of the study could be considered a fair reflection of learning progression. The findings relating to this second set of questions, did not suggest that the pattern of learning had been unequally influenced between conditions.
These results are considered important in their impact. The traditional method of presenting material to be learnt via printed text appears to create a higher level of learning than that achieved by text on a standard computer screen, or via an Intranet facility. Learning was achieved in both computer based groups, but it would appear that design features and their usability are influential in determining the extent of that learning. It could be argued that lack of practice for both computer systems could be a factor leading to the lower levels of correct responses. However, the operational requirements to allow both computer based groups to study the presented material were few and quickly mastered. Furthermore, the users themselves agreed that they were not only comfortable using the systems, but that the systems were easy to use.
The type of questions posed were predominantly ones requiring confirmation of a fact. Although there were some which could be deemed as requiring facts to be learnt within context to test less direct responses, the design did not give the option of narrative style answers. This type of response could well provide a better insight into the levels of learning from the different media, however, interpretation would undoubtedly be more complex due to the increased variance afforded by scoring such responses.
It is clear that further investigation is required to determine what factors lead to high quality learning through Intranet courses. The access to vast amounts of information available within the Internet, cannot be deemed a fair trade off against a lower level of learning, but must be harnessed to be beneficial and add to the potential of high quality learning through Intranet facilities.
Baddeley, A. D. (1997) Human Memory: Theory and Practice. Hove: Psychology Press.
Brooke, J. (1996) SUS: a "quick and dirty" usability scale. In , (eds) P W Jordan, B Thomas, B A Weerdmeester & A L McClelland (eds.) Usability Evaluation in Industry. London: Taylor and Francis.
Conway, M. A., Gardiner, J. M., Perfect, T. J., Anderson, S. J. & Cohen, G. M. (1997) Changes in Memory Awareness During Learning: The Acquisition of Knowledge by Psychology Graduates, Journal of Experimental Psychology: General, 4, 393-413.
Ebbinghaus, H. (1885) ?ber das Ged?chtnis. Leipzig: Dunker. (Translation by R. Ruyer & C. E. Bussenius (1913) Memory. New York: Teachers College Columbia University).
International Organization for Standardisation (1991) ISO 9241 (Draft International Standard) Ergonomic requirements for office work with visual display terminals. Geneva: ISO.
McGeoch, J. A. & Irion, A. L. (1952) The psychology of human learning. New York: Longmans.
Noyes, J. M. & Mills, (1998) In Press.
Phillips, T. & Long, C (1998) Get going on the Internet. Hemel Hemstead: Prentice Hall.
Tulving, E. (1985) How many memory systems are there?, American Psychologist, 40, 385-398.
How to cite this paper:
Garland, K.J., Anderson, S.J. & Noyes, J.M. (1998) "The Intranet as a learning tool: a preliminary study" Information Research, 4(1) Available at: http://informationr.net/ir/4-1/paper51.html
© the authors, 1998. Last updated: 4th August 1998
Contents |
Home |