vol. 15 no. 4, December, 2010 Contents | Author index | Subject index | Search | Home

Local versus global information relevance in Website use: a case study with the information literacy portal AlfinEEES

Aims and context

This study intends to explore the similarities and differences in the behaviour of the large audience of an e-learning Website, Alfin-EEES, on the one hand, and, on the other, a local subset of its main theoretical constituency of users, in this case, first-year university students.

In a more abstract sense, the study tries to offer insights into the relation between the macro-environment of a Website, its global audience, and the specific use that a well-defined group of target users would make and is making of the system: a micro-environment. Last but not least, it provides a model to study the relation between relevance and other variables like use, expectations and motivation in an e-learning site. Such studies can be interesting both for theoretical and practical reasons, as presented below.

Theoretical context

Relevance is one of the key aspects in information retrieval, and has been the object of extensive research. It could not be otherwise, because relevance is the ultimate measure of the effectiveness of an information system. Relevance has been the topic of an enormous corpus of research with very different approaches (see Saracevic 2007a, 2007b, for a recent review), some dealing with users' judgments and other more related to information systems architecture or document contents. Also, many constructs have been identified behind relevance that are useful to explain it, such as topicality, novelty and reliability. Eleven different factors were identified in a single study (Wang and Soergel 1998; Wang and White 1999).

Of the many approaches possible, this paper is interested in the relation between gross sociological measurements of the relevance of a Website (like the number of visits recorded in Web logs) and psychological measures of relevance provided by selected target audiences and that are related to their motivations and expectations.

In particular, this paper tries to research the relationships among three very different groups of measures related to the relevance of Website contents, with the peculiarity that the first is objective and the other two are subjective, and intend to capture a cognitive change in the user after being exposed to a Website.

• The first is the number of visits that each site, section or page receives. It is supposed that, under normal conditions, more visits imply that some content is considered more relevant, an approach that, of course, is widely used.
• The second is a user's judgement of the relevance of a section of the Website to their professional future and that of its peers.
• The third is a self-estimation of the user's knowledge of this topic.

The last two measures can be combined in different aggregations. When compared with the number of visits, only one of them offered consistent results: that consisting in the difference between the perception of competence before and after a preliminary but comprehensive use of the site, or, in other words, the change in the self-assessed competence after being exposed to a Website. This was an empirical relation found after reviewing different results regarding the relevance assigned by a group of students to the contents of the Website and a self-assessment of their relative competence in them, before and after being exposed to the Website.

The resulting theoretical model is expressed in Figure 1. The two user aspects that are explored regarding relevance are a) expectations as a result of previous knowledge and b) motivation that results form the perception of a knowledge gap.

Relevance is seen as eliciting the three different kinds of responses that are studied: relevance judgements, self-competence judgements and the use of the Website. It is supposed that if an item is judged more relevant after using the information, this is related with a growing sense of the previous knowledge gap, because previous knowledge remains constant and, therefore, the relevance assigned to this piece of information increases.

Applied research

This research was very interested in contributing to a model of how relevance judgments are affected by user behaviour, but also practical implications arise.

The relation between the global and local use of information is a very important question to develop fine models for Web site evaluation. This is a general problem in Web-based information systems and, consequently, also a key question for e-learning sites and, more specifically, for AlfinEEES, a portal that intends to offer relevant information in the specific field of generic and information competences.

It is a general problem because a Website is a project subjected to the laws of economy. It has a demand and it is by fulfilling this demand in an effective and efficient way, that it will get the resources it needs to thrive.

But on the other side, a Website, by its very nature, has a world-wide audience and this opens a door to a potential paradox: it might be that a Website is being successfully visited by a theoretically sufficient number of users, and that, on the other hand, it is failing to address its real intended audience, the one that makes it a feasible project.

Organization of the paper

This paper explains the results obtained in the research experiment and discusses the main conclusions that were reached. It is organized in four parts: first, the case study Website is briefly described and explained, attending to its aims, philosophy and content architecture; secondly, the methodology of the study is presented, which is summarily based on the comparison between Weblogs, on one side, and some relevance-related tests with a small survey of first year students; thirdly, the results about this relationship are offered; and, finally, the results are discussed, the main conclusions are summarized; and some perspectives for future research are offered.

The target Website AlfinEEES: a brief explanation

Alfin-EEES (Pinto 2005, 2006) stands for Alfabetización Informacional en el Espacio Europeo de Educación Superior, that is, Information Literacy in the Higher Education European Space. It is a Website developed by a team directed by Maria Pinto with a research grant from the Spanish Ministry of Education.

The portal was designed to provide online training in information competences for higher education students with a wider scope than usual, inserting information competencies into the frame of the development of generic competences, a key aspect of the Tuning project and the Bologna process (González and Wagenaar 2003).

Alfin-EEES is organized hierarchically in competencies and sub-competencies. From the home page the user can also access the presentation of the project, the mention of credits and authorship and the support tools. The presentation explains the underlying philosophy, objectives, methodology and contents. The support tools are of different types: a) information seeking tools—the map of the site and the search engine; b) help in using the site—help and contact; c) study support materials—glossary, list of general resources (the specific resources appear in each of the sub-competencies) and in the future, FAQs; and d) for communication—forums, e-mail and chat rooms.

The main part of the content architecture is the taxonomy of the six competencies, divided into their sub-competencies, which are:

• Learning to learn: the concept of learning to learn is explained, how to learn to learn, how to be autonomous and in charge of our own learning process.
• Learning to seek and assess information.
• Learning to analyse, synthesize, and communicate: how to read better, how to segment the information to subsequently reorganize it using the techniques of outlining, graphic representation and summary and how to communicate the new knowledge in writing and using graphic presentations while respecting the contribution ofauthors whose ideas have been used.
• Learning to generate knowledge: in-depth study of the processes of creation and innovation, of the principles of scientific thought and of the techniques for organizing projects, to train the student in the principle phases of knowledge creation: creation, research and development.
• Learning to work with others: the ethical bases for co-existence and teamwork are posed, as well as how to recognize and approach conflict, using win-win negotiation techniques.
• Using technology to learn: operational systems, office computer applications, communication tools and the e-learning environments most often used in universities are introduced.

The competencies and sub-competencies are related to the knowledge production cycle, documentation, analysis, synthesis, communication and to its sub-contexts: working within a community and often as part of a group, contemplated in turn within the objective of educational intervention, which is teaching to learn to learn at the university level (Figure 2). The screen with the whole set of categories is presented in Figure 3 and its translation is available in Table 2.

The information offered for each competence and sub-competence is standardized. A basic chart outline for each compentence-subcompetence is shown in Table 1, and an example is presented in Figure 4, specifically a screen corresponding to the entry page of the sub-competence 'Reading to learn'.

Methods

Fortunately the highly structured nature of the portal content allows for a precise recounting of the use of the Website and, subsequently, an easy correspondence between this information and the data regarding the users' attitudes and self-assessment results on their information literacy sub-competences. This one of the reasons for AlfinEEES serving as a test bed for this study.

First, the global interest for the Website was researched using the site traffic logs, with Google Analytics. As Alfin-EEES is highly structured, each page or group of pages corresponds to each competence or sub-competence, as explained above. So, it is very easy to compare the interest of the global audience in each competence and use this data to make comparisons.

Secondly, two surveys were carried out on a group of nineteen students of the first year of the undergraduate library and information science programme of the University of Saragossa during November 2009, with the aim of evaluating their interest in each competence and their statements about their subjective level of competence, both individually and in relation to their group. The first survey was carried out on 11^th November before being exposed to the Alfin-EEES Website, as a pre-test condition. The second survey, using exactly the same items, was carried out on 29^th November after they have extensively used the site, as a post-test condition.

The students were asked about the importance of each competence and sub-competence for their professional future and their personal opinion about their level of competence in each topic and about the level of competence of their whole group. They were asked also about their overall competence in information technologies and information management. All the questions were measured with a five-point scale. A reproduction of the survey Webpage is offered in Figure 5 and is available at the Alfin-EEES Website. The topics have been translated to English in Table 2.

To simplify, trying not to confuse them, the students were asked only for the sub-competences, assuming that the competences could be calculated as a mean of their sub-competences. This was proved partially untrue when a control question was introduced on 'Learning to analyse information'. This is a very interesting finding and a prospective hypothesis to explain it can be advanced: it is probably so because the competences are complex constructs that do not match exactly the sub-competences for all the population, so that, in practice, the designers of the systems and the users might have different concepts about them.

The research model and method

Figure 6 summarizes the logic of the research model. The intended aim of the Alfin-EEES Website is to improve the information competences of university students. It is used by a wide global audience, as showed by Google Analytics, that cannot be stratified accordingly to the initial target audience. It is supposed that the more a Web page is used, the more relevant is for a community or various communities of users; and that this measure has significance inside the site, that is, the more a page is visited in relation to the rest, the more relevant it is, and vice versa.

To study the intended audience, a survey was applied to a group of university students. The survey intends to establish the interest of the students in each competence (and sub-competence) by asking them about the importance of each item for their future. So, an indicator of relevance was obtained for each competence and sub-competence. Besides that, it asks the students about their level of knowledge on each competence, trying to establish if it is related to their interests. This survey is done before they know Alfin-EEES. This information serves as an estimator of the existing gap in their knowledge.

Later, they were exposed to Alfin-EEES and the test was applied again to get information about how their new knowledge on information and generic competences had affected their perception of the importance of each item.

As may be seen in Figure 6, the studied variables are considered expressive of the basic constructs that are studied: user motivation, which increases or decreases the assignment of relevance to an information item (e.g., a Web page on an information literacy competence), and eliciting a using behaviour. As a result of this behaviour, users re-evaluate both their interest in the information and their competence and the relevance of this item changes accordingly.

In the future, a measurement of the actual competences would be needed to complement data based on the students' self-reports. In this way, we will be able to obtain information in both directions: a) how the objective IL tools may diverge from students' opinions on their competence; and b) how students' self-perception differs from the standard consensus among information specialists. The research team has developed a tool for the online assessment of information competences and is conducting the first tests.

Finally, it would be interesting to research a black point in the model, that is, how student interests affect their exposition to information literacy information. This could be done, for example, measuring the time that students spend in each page of the Website or how many times they access the different Web pages.

Macro-analysis: using visit statistics to understand the users' interest

To get a glimpse of the topics of interest to the Alfin-EEES audience, the Google Analytics data were normalized, obtaining the percentage of visits to each competence and their sub-competences since 2005, and also an index that compares the visits per page in each item with the mean of visits per page for the whole site (Table 2).

Figure 7 shows the mean of the visits a page for each whole competence. The category that rises more interest is 'Learning to learn', which is understandable, as it is the most interdisciplinary, so with a bigger potential target audience, and a key topic of interest in pedagogy. The same can be said of the second and the third, 'Learning to analyse, synthesize and communicate information', and 'Learning to create knowledge'. Very near is the fourth category, which is the most specific of the library and information science field: 'Learning to search for and assess information'. The one that is less retrieved is 'Learning to work together'.

The sub-competence level offers a more granular perspective (Table 2). The topics that attract more interest are 'Autonomous learning', 'Learning to segment', 'Learning to analyse' and 'Learning to learn'. On a page-by-page basis, 'Learning to analyse' is, by far, the most visited; and the least visited is 'Ethics'.

Micro-analysis: assessing the motivations of a group of freshmen

The results of the two surveys with the freshmen from the University of Saragossa are provided in Table 3. The importance and subjective assessments of their competence and that of their group's are offered before (pre-test) and after (post-test) working with Alfin-EEES. The differences between their competence and the importance they give to each sub-competence have been calculated and also the differences before and after the test.

The students seem to think that all the sub-competences proposed are very important for their future, as they give them scores of more than '4'. It is curious, but not surprising, in a world that is becoming more social and interconnected, that they give the maximum punctuation to learning 'Decision-taking and negotiating'. The least punctuation is for 'Learning to segment', which they probably see as very peripheral or do not even understand.

In general, the students have a very positive impression of their competences in information management and other related generic competences, and data does not show great differences between their own assessment and the one they do of their peers'. They show a great confidence in the materials that they are offered, and they think that they have improved quickly in their competence after reading and using them, as it can be inferred from the punctuation.

Results could be very conditioned by local experiences. For example, the significant difference in 'Learning to search for information' could be affected by strong motivation and fresh new knowledge on the topic, because this is a subject they were just trying to pass at the moment of the survey. The same can be said of their struggling with digital 'Learning environments'.

After their exposition to Alfin-EEES, 'Learning to write' and 'Learning to work in group' become more important. This is probably also a local effect, as these aspects are very important in the teaching methodology that is followed in the library and information science undergraduate programme of the University of Saragossa.

Comparing both groups of data

After extensively reviewing line graphics for the different sets of data, it can be affirmed that there is a general lack of correlation in the information offered by page visits and the relevance and competence data offered by the group students. Only in the general competence related to autonomous learning and the general ability to learn, a great correlation can be observed. (A recent study, with a completely different methodology, found that simple 'user relevance rankings had little or no predictive power' about the actual use of a Web page to solve a problem (Coiera and Vickland 2008)

From this lack of connection, it seems clear that the Website is used by other audiences that give, for example, a great importance to learning to learn, autonomous learning, and segmenting and schematizing information.

There is, in any event, an important exception to this lack of correlations. It affects the comparison between the percentage of visited pages per category and the improvement in the perception of competence before and after the use of Alfin-EEES, as it is showed by the difference in 'My competence' after the exposition to the e-learning Website (Figure 8). If the number is positive, that means that the subjective sense of competence of the students has grown after using the system, and vice versa.

This difference of subjective competence before and after using the system can be considered a measure of the value of the materials offered in the Website (relevance), and it certainly correlates (with some exceptions) with the visits that the sub-competence Web pages are receiving. It is remarkable that this correlation is not working at all in the topics related to technology, which are not the stronger part of the contents (mainly because their rapid obsolescence), a fact that might be in fact strengthening the hypothesis.

Conclusions and future work

The strong relation between the shape of the percentage of visits, which was postulated as a measure of relevance at the macro-level, and the change in the subjective perception of personal competence after being exposed to the Website, a measure of the information gap that has been covered by the site, supports the hypothesis that the increase in the subjective level of competence in an e-learning site is a good predictor of the whole success of a site, when working with their primary intended audience. In this dimension, a strong and demonstrable relation between the macro/global and micro/local levels of analysis appears to exist.

From a theoretical point of view, this is just another confirmation of the relation between the information gap on the part of the user, measured throughout the process of knowledge improvement, and the information relevance on the part of the collection. From this perspective, relevance appears to be something connected with the perceived information gap, even a function of it.

Besides that, the study provides further insights in the relation between relevance and use of a Website, on the one hand, and the expectations and motivations of the users, on the other hand; and a model to study them.

Taking into account a more practical and applied perspective, the change in the subjective perception of personal competence indicator is interesting because it is not a direct or intrusive one, and, in consequence, it is less prone to be manipulated by the users. As a result, it could be used as a relatively inexpensive non-intrusive pre-test of the relevance of a site or as a complementary evaluation tool.

An important result of this study is that the correlation was clearly affected by the quality of the materials (technological pages get quickly out of date) and the existence of different potential audiences for a Website, in this case not only library and information science undergraduatess but probably also pedagogy and school and high school teachers and students. For these reasons, the model should be enriched in the future to deal with the quality of the collections and target group segmentation.

It is a clear finding also, that students are overly optimistic about their competence and that of their peers. So, an objective assessment of the information capabilities of the constituency is necessary also in the future, and should be incorporated to the model, too.

Acknowledgements

This article has been produced as a result of a project initially granted by the Spanish Ministry of Education (Studies and Analysis Programme, EA2005-43). We acknowledge the contributions of the anonymous referees for their motivation and insights.

About the authors

Francisco Javier García Marco achieved his Ph. Dr. in Philosophy and Arts in 1994 and has been Professor of Information and Library Science at the University of Zaragoza since 1996. He has been director of the Department of Library and Information Science, organizes annually Ibersid, an international conference in information and documentation Systems (1996-), and is the director of the journal Scire and referee for several Spanish and Brazilian journals. He can be contacted at jgarcia@unizar.es.

María Pinto is full professor in the University of Granada, Spain. She gained her PhD in 1984 in Philosophy and Arts, where she is a professor from 1981 and gained her full professorship in 1955. She has written several academic handbooks, many research articles and directed more than fifty projects in the field of library and information science, abstracting and indexing and information literacy. She can be contacted at mpinto@ugr.es.

Bookmark This Page

© the authors, 2010. Last updated: 12 December, 2010

online Website analytics