Vol. 9 No. 3, April 2004
| Vol. 9 No. 3, April 2004 | ||||
The paper presents the results of a user evaluation study exploring the usefulness of personalized access to scholarly journals and services with regard to the information behaviour of scientists. The aim was to investigate what factors would be critical to personalization, what personalization features would be relevant and to what extent profile and behaviour based personalization would be acceptable. Three providers of personalized access to scholarly journals and services formed the basis of the study. The user group was fourteen doctoral students. Personalization was found to be associated with expectations of increased efficiency compared to traditional searching, emphasizing the need for functionality oriented features such as search alerts. Profile based personalization was to some extent positively associated with reduction of information overload. Various quality indicators were found to affect the perceived reliability of a service, making trust a critical theme when establishing a personalized user experience. The study suggests implications for the digital library when designing for a personalized information environment.
The Internet and the World Wide Web have introduced new and powerful ways of finding and sharing information. In the scientific arena, the proliferation of electronic journals (E-journals) has revolutionized the way scientists conduct research and communicate their results; for example browsing and searching for information is facilitated far beyond what is possible in a print environment. As a new way to add value for the end users and to improve their ability to cope with information in a digital library setting, many publishers and providers of scholarly journals and services have recently introduced personalized access to their portals.
By personalized access is generally meant the ability to customize the user interface, the information channels and the services provided according to the individual user's needs, personal interests and preferences. In this way the user is given the opportunity to construct a personal information space with relevant information sources and services, and interact with the user interface in a personal manner. The service providers, on the other hand, hope to better serve the user or customer by anticipating needs, make the interaction usable, efficient and satisfying for both parties and build a relationship that encourages the user to return for subsequent use or purchase.
While personalization has become an important way to facilitate interaction, experiences gained so far indicate that many service providers have failed to create a personalized user experience (e.g., Bonnet, 2001; Manber et al., 2000; Reamy, 2001). According to Reamy (2001), some unsuccessful personalization implementations can be explained by a lack of knowledge structure that effectively supports the match between user and content. Another reason could be that personalization has been applied to domains that are by nature socially determined. To judge from a study of My Yahoo (Manber et al., 2000), other reasons might be that the personalization tool has been too difficult to use or that the default user interface is adequate, making personalization unnecessary - or simply that an inappropriate personalization strategy has been chosen.
Personalization means knowing who the user is, and for what purpose personalization should be implemented. This requires a thorough user model that also takes into account the user's information behaviour, e.g., needs, tasks and search behaviour. Originating from My Yahoo, the My-portal model is today a widely used approach to personalization. It includes specific features and software to build a personal environment in which the content and layout of the interface is completely controlled by the user. Thus, a static model of the user is maintained until changes are made by the user. On the other hand, to keep the user model up to date this approach requires a high level of user engagement and time investment.
To explore the usefulness of the My-portal model to the information seeking behaviour of scientists(1), a qualitative study of personalized access to scholarly articles and services was carried out in January 2003 at the Royal Agricultural and Veterinary University of Copenhagen (KVL). Based on results obtained from a pilot study on e-journals and personalization carried out in the spring 2002(2), the following research questions were investigated:
The theoretical section is based on an examination of literature on 1) personalization and 2) the information seeking behaviour of scientists, primarily focusing on the use of e-journals
Personalization reflects a new design philosophy which besides user orientation focus on the delivery of a contextual user experience. In recent years, the interest in personalization has increased as a way to filter information and reduce information overload, connect people and build relationships e.g., to facilitate knowledge exchange and sharing (Chia, 2002; Reamy, 2001). According to Bonnet (2001) personalization '…involves a process of gathering user-information during interaction with the user, which is then used to deliver appropriate content and services, tailor-made to the user's needs'. However, personalization may take many forms, depending on the characteristics of the target group and task, the information technology available and the personalization approach chosen. From a technological point of view, two types of personalization approaches, generally applied in portals, websites and intranets, are relevant here, namely profile based personalization and behaviour based personalization, the latter also known as collaborative information filtering.
Profile based personalization, is based on a match between profiles of users (user attributes) and profiles of content (content attributes) (Chia, 2002). This approach may be divided into customization and rule based personalization.
Customization refers to the user's ability to customize the user interface and choose what information channels and services that should be provided by the system according to the user's needs, personal interests and preferences. Thus, the control of content and the look-and-feel of the interface is user driven. A profile addressing both user attributes (e.g., name, address, gender and age) and content attributes is filled out by the user, often by 'checking off' lists of categorized preferences. In a digital library setting this could be content such as databases, e-journals, news, links and reference materials (Figure 1). The layout of the interface is often customized by choosing style and colour as well as which portal elements and applications that should be implemented and how these features should appear in the interface. Succesful customization is dependent on the number of relevant options and selection criteria matching the user´s needs. Furthermore, customization requires a high degree of user involvement to motivate the user to spend time maintaining the profile; otherwise, a static model of the user is kept. Customization is the approach generally associated with personalization, commonly reflected in expressions such as 'My portal', 'My site' and 'My E-journal'. In this paper this personalization approach is referred to as the 'My portal-model'.
Rule-based personalization refers to the personalization resulting from a match of user profiles with content profiles based on inference rules reflecting various relevance criteria (Instone, 2000), thus determining the quality of the personalization experience accordingly. Besides personal information, the user profile may include contextual attributes, such as geographic location, the type of device and access mechanism in use and the user's current role, interests and preferences. The content profile may include attributes, such as descriptive, structural and administrative metadata, varying with the type of content. This could be news, a person, a book and an article.
After the user has filled out the profile, it is used pro-actively to match the user with appropriate content, according to a given context. In this way, the control of the personalization is both user and system driven.
Rule based personalization is an important way to expose relevant content to the user, e.g., news about new articles, links or events, matching the user's interest or role, instead of waiting for the user to accidentally find it. However, to be useful, rules as well as the user- and content-attributes must continually reflect and adapt to changing needs of the user.
Behaviour based personalization refers to the personalization based on a given user behaviour in the system, e.g., purchases, search terms and clicks (Chia, 2002). Based on a simple user profile, addressing personal information and main preferences, the system dynamically creates a model of the user or a community based on his or other user's behaviour in the system. The model is then used to automatically filter out, recommend and match users with information and services. An example of this is the internet bookstore Amazon.com which recommends information based on the user's previous purchases, or by matching the user's interests with the interests of other users showing similar behaviour. Behaviour based personalization is primarily system-driven, leaving the user with less control over the content, though the quality of this form of personalization is proportional with the user activity in the system.
The characteristics of the different types of personalization are summarized in Table 1.
| Approach | Customization | Rule-based | Behaviour based |
| Input data |
Profile
(user and content) |
Profile
(user and content) |
Behaviour
(user or other users) |
| Output data | Layout, content, services | Exposed context dependent content and services | Recommendations |
| Initiation | User | User /system | System |
| User control | High | Middle | Low |
| User model | Static | Static / dynamic | Dynamic |
| Success criteria | Number of options of relevant content and services, selection criteria, updating | Differentiated user- and content-profiles, matching criteria, updating | Behaviour inference algorithms |
Personalization is not restricted to either type of personalization, but may be mixed, although one approach is often dominating.
The success of personalization is, however, not a question of the personalization technology and its features, but whether the personalization technology is usable and relevant to a given user group. Thus, a simple user interface may be more useful than a site which has invested a lot of resources in personalization.
Scientists, being a primary user group of academic and research libraries, have been the subjects of many user studies throughout the years. One of the parameters that have been investigated is the use of various types of sources with the purpose of optimizing library holdings to the user's needs.
For the scientist, a digital library should provide optimal access to the sources deemed most pertinent for a given research field. With the development of the Internet as a global communication structure and the adoption of web technology as the preferred access method, a vast range of sources are potentially accessible, only a few mouse clicks away. However, studies of scientists´ information seeking behavior have shown that many aspects dictate the strategy employed. For example, a study by Brown (1999) showed that the characteristics of the scientific field affected scientists´ preferences for information sources. In this study, the mathematics faculty tended to prefer personal communications and monographs to other sources, the physics-astronomy faculty prefered freely accessible pre-print archives over the Internet while the chemistry-biochemistry faculty tended to prefer abstracting and indexing services. Other studies exist that support this notion of field differences, e. g. the study by Kling & McKim (2000) showing that while high energy physicists exhibit a positive attitude towards preprint servers and experimental publishing models, scientists in molecular biology can be viewed as a somewhat conservative community, relying heavily on scholarly journals. This view of molecular biologists is supported by a citation study, in which journal usage was a measure to gauge the information needs of this group of scientists and researchers (Hurd, 1999).
Based on longitudinal studies, the role of the scholarly journal as the primary source of information for university scientists can be shown to be undiminished (Tenopir & King, 2002). However, a shift is seen in the format preferences of the scientists, implying new reading habits. Though print sources are still very important to most scientists, the availability of electronic versions have resulted in an increased number of articles being read by more scientists. Evidence for this impact of the electronic media can be found in a study comparing readership surveys from traditional settings with a survey of astronomers and astrophysicists having access to a comprehensive complex of full text journals, astronomical databases and core literature going back more than 150 years (Tenopir et al., 2003). It is shown that users of this advanced information system tend to read more articles than users obtaining their articles otherwise. Furthermore, a trend towards reading from a broader range of journals, attributable to the rise of electronic journals, has also been recognized. It can be argued that the availability of many new titles due to the popularity of licensing agreements, in which libraries or other institutions gain access to all titles of a given publisher, have enabled a more serendipitous reading practice (Mackie-Mason et al., 1999). However, another explanation might be that the increased interdisciplinarity of new research topics often blurs the traditional boundaries of scientific disciplines (Hurd, 1999; Lally, 2001)
The impact of e-journals on information seeking behaviour have been noted in many studies. Rusch-Feja & Siebeky (1999) have documented that scientists find e-journals advantageous due to the direct access from their workplace, the prompt availability and the ability to download a copy of relevant articles or doing full text searches. Interestingly, this may also vary according to the type of user. According to a study performed by King and Montgomery (2002) at Drexel University, doctoral students appeared to be more dependent on the library copies of journals than faculty members who had a greater number of personal subscriptions. Further, with regard to these preferences, it was found that faculty members often preferred to browse, whereas doctoral students in particular preferred to search. These findings are supported by similar observations made in other studies (Tenopir et al., 2003).
To summarize, scholarly journals are still regarded as the most important information sources in most scientific fields; and the availability of electronic versions have had a positive impact on the reading habits of scientists, increasing the number of articles read, apparently by reducing the need for additional information seeking accordingly. The use of e-journals also seem to have implied a shift towards a search based information seeking strategy. Further, with regard to the study reported here, doctoral students seem to be a group of users ready to take advantage of the electronic medium.
This section describes the research design and procedure of the qualitative study of personalized access to scholarly journals and services. The study was carried out in January 2003 at KVL.
Fourteen doctoral students recruited from various research fields at KVL, participated in the study. Eight of the participants were male, six were females, ranging from 23 to 51 years of age. All participants were experienced internet searchers, but did not have any prior experiences with personalization or the services used in the study. Their use of e-journals varied from 'several times a week' to 'once a year'. Most of the them, though, were using e-journals weekly or monthly.
Three providers of personalized access to scholarly journals and services formed the basis of the study: Science Direct, Ingenta and Community of Science. Each of them has implemented personalization facilities based on the 'My portal-model' addressed in this study.
ScienceDirect (www.sciencedirect.com), provided by the publisher Elsevier gives access to 1.9 mill. scientific articles in full text format within science, technology and medicine. Further, it provides services to support interdisciplinary needs in the academic, private and educational sector.
The My-facilities used in the study were: My Alerts and My Profile.
Ingenta (www.ingenta.com) is a global, interdisciplinary article service, giving access to 13 mill. scientific articles (bibliographic information) from 26,000 journals, of which 5,200 are in full text.
The My-facility used in the study was My Files.
Community of Science (www.cos.com) is an interdisciplinary portal providing services and tools to support academics in communicating, exchanging information, finding experts or software relevant to work or research. It provides access to a database containing information on 480,000 academics within research and development (R&D) and access to databases such as MEDLINE, AGRICOLA and GeoRef. The My-facility used in the study was the workbench, giving the user the opportunity to customize the interface according to his or hers needs and taste.
One week after the participants were recruited, they received an introduction to the study, also stating that their identity would not be disclosed to anybody but the two authors. After an introduction to each of the three services mentioned above, the participants were asked to perform a set of tasks involving several personalization features of each service and fill out a questionnaire afterwards. To facilitate this, all users were preregistered by the test leader and provided with temporary email addresses to allow for the creation of a customized user experience.
The questionnaire inquired about age, familiarity with the Internet and previous experiences with e-journals. To gain insight into the participant's use of information sources, each of them was asked to mark the importance of specific information sources with a number from 1 (low) to 5 (high). Further, they should indicate on a scale from 1 (low) to 5 (high) the perceived user friendliness, efficiency and relevance of each of the three services. User friendliness was defined in the questionnaire as being 'easy to use', 'intuitive' and 'user oriented'. Efficiency was defined as being 'fast' meaning 'not time-consuming' and relevance was defined as being 'relevant to a research related need'. Finally, they were asked to indicate their agreement to a number of statements about personalization (e.g., whether it was useful to customize the layout of the interface) on a 1 (disagree) to 5 (agree a lot) rating scale. The questionnaire data have been summarized in Table 2 and Table 3, and will be discussed in section 4.0 when reporting the findings.
The questionnaire was followed up by a focus group interview, two weeks after the questionnaires were handed in. A total of 9 students chose to participate in the two-hour focus group interview session.
The focus group interview was performed as a 'funnel' interview (Halkier, 2003), asking rather broad questions in the beginning, e.g., concerning user friendliness, then followed by more focused questions which addressed specific personalization issues according to the research question of this study. The focus group interview was recorded on minidisc. One of the authors acted as moderator, the other took detailed notes during the interview filling out a pre-designed formulary with headings that corresponded to the issues listed in an interview guide. Together with the questionnaire data these notes form the basis of the analysis.
All questionnaire data have been tabulated and median scores calculated to obtain central estimates for the responses. The focus group data have been analysed to identify themes and patterns reflecting critical factors affecting the usefulness of personalization and in support of factors already indicated by the questionnaire data.
Besides reporting the doctoral student's use of information sources, this section describes and discusses the findings according to the research questions of the study. Quotations have been translated by the authors.
The first part of the questionnaire, concerning personal information, experiences and use, was used to establish a profile of our respondents. As can be seen from data shown in Table 2, e-journals were deemed the most important source, closely followed by two other types of electronic sources, such as online bibliographies and catalogues, the library website, and the traditional printed sources. Personal communication was also prominent, whereas actual library usage or reliance on non-library websites were scant. Mailing lists were not in favour within this multidisciplinary group of doctoral students.
| Source | Median score |
| E-journals | 4.5 |
| Online bibliographies and catalogues | 4 |
| The library website | 4 |
| Printed sources | 4 |
| Personal communication | 3 |
| Campus library | 2 |
| Other scientific websites | 2 |
| Mail-lists | 1 |
These observations are in concurrence with the existing studies of the information seeking practice of scientists (Brown, 1999) and also supports the observation made by others that e-journals today are the primary information source for scientists (King and Montgomery, 2002; Tenopir et al., 2003; Tenopir & King, 2002). The low score with regard to the campus library may be explained by the fact that users have access to most of the library's journals as electronic journals directly through their personal computer, relieving the users from visiting the library (Rusch-Feja & Siebeky, 1999).
With regard to this study, the participants were then expected to be highly motivated to improve their abilities regarding information seeking in e-journals.
Regarding the research questions of the study, some important factors affecting the usefulness of personalization to scientists and their information seeking behaviour can be stated.
How important is user friendliness, efficiency and relevance to personalization, specifically what impact do they have on the user's acceptance of a service? Should other factors be taken into account
It appeared that the participants were very goal oriented when it concerned their research projects, meaning that they searched with a focused aim and information need in mind. When it concerned other subjects, browsing was preferred, also implying a willingness to spend more time on information seeking. This information seeking behaviour corresponds well to findings made by among others King & Montgomery (2002) and Tenopir et al. (2003).
Not surprisingly then, time was stressed by the participants as a very important factor when evaluating the usefulness of a search service. Accordingly, personalization was found to be highly motivated by expectations of efficiency, implying that in order to be attractive personalization must imply an increased efficiency compared to traditional searching. This also seemed to be reflected in their perceptions of 'user friendliness'. For example, when a service was accessed for the first time, the user friendliness of a personalized user interface was considered very important. In cases, though, where the content and the functionality of a site was already known as highly relevant, the participants indicated a willingness to reduce claims for user friendliness accordingly. This was supported by expressions such as '…what matters [to me] is what is behind the nice user interface; what it may be used for' …. 'if I know that a site is relevant to me, then I will experience it as user friendly, because my patience threshold will be much higher', also indicating that some of the users had difficulties in separating the access to information from the output and gain they expected to get. The difficulties in separating user friendliness, efficiency and relevance from the coherent user experience also seemed to be reflected in the questionnaire evaluations of the three services. As shown graphically in Figure 2, the scores registered for user friendliness (Figure 2a), efficiency (Figure 2b) and relevance (Figure 2c) follow the same pattern and almost have exactly the same median score values. It also appears that ScienceDirect and Ingenta are rated as the best services, whereas Community of Science elicits a more diversified response. The smaller number of answers for ScienceDirect was due to a glitch in the confirmation of user login credentials, in some cases denying access to users.
Other factors, relating to the perceived reliability of a service, also seemed to affect the personalized experience as well. For example, general quality indicators, such as site information, actuality, language and layout of the interface were often used by the participants as arguments for trusting or not trusting a personalized output, such as search results, filtered output or simply recommendations made by the system.
What type of personalization features will be relevant to support the user's information seeking behaviour?
Stressing the importance of efficiency, the participants focused on functionality issues when discussing requirements for a personalized user interface. Search alerts were by all participants regarded a very important, convenient and time saving personalization feature (Figure 3), but the envisioned usage varied. Some preferred to build complex search profiles in order to trawl for all relevant information, while others found it useful to monitor secondary subjects or simply as a way to be kept up to date during, for example, leave of abscence. Many of the participants, though, expressed doubts as to whether automated searches were 'safe', indicating a preference for searching interactively. It was also pointed out that relying on search alerts could lead to future searches based on outdated needs, if one neglected or forgot to keep the profiles up to date.
In general it was agreed that search alerts were important, but could not supplant periodic manual searches.
What type of personalization features will be just nice to have?
The ability to customize the layout and the functionality of the user interface recieved good scores in the questionnaire (Figure 3). But when addressing the customizability of CoS at the focus group interview, most participants downplayed the importance, in particular regarding the layout: '...if you feel like wasting time you could change the colours...' . However, some of the participants considered this feature a relevant way to reduce information overload and save time usually spent on searching.
To what extent will profile based personalization be acceptable?
The acceptance of customization based on user profiles, was strongly determined by factors related to relevance and trust. It was qualified by statements such as 'basically it is a question whether you trust those who operate [a site]' indicating that in order to establish a relationship, requiring the user to supply personal information, the service provider must appear reliable. This was also the case when one of the participants associated the 'find a colleague' service of CoS to a 'dating service' and consequently perceived it as less reliable.
The type of personal information required and the perceived risk implied, e.g., credit card abuse or unsolicitized email, was also dependent on 'who' asked and 'why', suggesting that this kind of information should be provided to reduce risk and contribute to trust. The questionnaire data also exposed a wide disagreement as to whether maintaining a profile was bothersome (Figure 3). For example, the amount of information that should be filled in was discussed. The very extensive profile in Cos, virtually a complete CV, was regarded as too time consuming when taking into account the expected outcome and relevance.
To what extent will behaviour based personalization be acceptable?
As already indicated, trust plays a critical role in the establishment of a personalized user experience. The noted scepticism towards a complete reliance on automated searches also came into play as a resistance against behaviour based personalization. Most of the participants liked the idea of getting filtered and recommended information as a way to cope with information overload. However, this was highly determined by whom the recommendation was made by and why. For example, most of the participants did not trust the recommendations made by Amazon.com '…you only know that people have bought a given book, not if they liked it after having read it'. And further 'when one of my a colleagues recommend an article, I will read it', touching here upon the issue of social navigation that addresses the social creation, guidance and use of information (Davenport, 2000). This was also the case when the participants were asked about their attitudes to recommendations made by intelligent agents; most of them were rather sceptical towards the concept, some of them explicitly stating that librarians were preferred to intelligent agents.
Other results
Besides the findings related to the research questions of this study, it was noted that a perceived lack of information seeking skills among some of the participants also seemed to be embedded in arguments for preferring one or another personalization approach. These participants were for example uncertain about how to perform a qualified search and how to determine which information sources would be most relevant to the subject of interest. Personalization was then regarded as a useful way to overcome this uncertainty. Statements such as 'I'm very focused when searching information and always use the same database, ...' and 'We have a list of relevant journals, I believe it is 27 journals...' could also be seen as a result of this phenomenon. Besides aspects of habit and convenience related to such statements, reliance on the same few sources independent of the information need given also seemed to result from the perceived uncertainty mentioned above.
These findings stress the importance of delivering not only content and services as a part of personalization, but also guidance and strategies to cope with information seeking the specific subject domain given.
This paper has reported the findings of a qualitative user evaluation study of personalized access to scholarly articles and services. Specifically, the appliance of the My-portal-model, implemented by three service providers, has been investigated in relation to the information seeking behaviour of fourteen doctoral students. Based on the findings, some conclusions can be stated.
The investigated services, built upon the My-portal model, have exemplified different ways of utilizing the information obtained through the user profile, thereby providing test cases for exploring the usefulness of personalization with regard to scholarly articles and services. The simple input required by ScienceDirect and Ingenta were sufficient to create profiles that enabled the participants to build a personalized system of preferred titles and alerts. The gain from this investment was sufficient for most of the participants to experience personalization as an instrument to obtain timely and relevant information.
On the other hand, there was a marked reluctance towards entering complex user profiles, and the establishment of virtual communities, as promoted by CoS, did not appear to hold much attraction. Furthermore, it was argued that the profile entered would keep a static model of the user, not necessarily reflecting his or her current information needs. Trust seemed to be the key issue, both in connection with profile based and behaviour based personalization, expressed in a critical attitude particularly directed towards the recommendations made by the system. Thus, the issue of trust should be addressed in advance of featuring proactive information filtering. Overall, the success criteria seemed to be simple profile enabled services with targeted functionality, e.g., search alerts, which will provide the user with a steady, filtered information flow from the sources deemed appropriate for the research field.
In a digital library setting, the My-portal model may serve as a foundation for providing scientists with timely and subject-specific access to content, given the context and user needs. This should also be seen as opposed to a strategy of developing and maintaining a range of subject specific portals. For example, in stead of requiring the user to access three different providers of e-journals, scientific articles and services, and to set up search alerts at each site, these should be integrated into the digital library, making research driven information seeking a one-stop experience. To support this further, other personalization approaches, such as rule based filtering, should also be taken into account.
Regarding new challenges and engagements, library and information professionals have a fundamental role to play in building personalized information environments, connecting people to information and people to people as well as helping users develop competence and confidence in handling information.
A personalized experience is not only about delivering content and services; personalization also means knowing the user, his preferences and behaviour to proactively guide the individual user when seeking information. If information professionals do not take this opportunity to maximize the appropriate use of resources, scientists -or other users- will continue to bypass these expensive and high quality mediums in favour of search engines - or to go directly to the information provider.
Personalization is still young as a design philosophy. Having been highly associated with technology, features and functionality, more research and development is now needed to understand when and why a user or group of users experience an e-journal, a website, a digital library as their personalized information environment.
Our users are gratefully acknowledged for participating and by sharing their impressions with us, contributing to our understanding of personalized access to scholarly articles and services. Peter Ulvskov, Biotechnology Group DIAS-KVL is gratefully acknowledged for hosting the focus group session. The Department of Information Studies at the Royal School of Library and Information Studies are acknowledged for supporting this study financially.
(1) Scientists are here defined as in King, W. & Tenopir, C. (1999) Using and Reading Scholarly Literature. Annual Review of Information Science and Technology vol. 34 p. 423.
(2) A small scale user study among a group of molecular biologists was performed by the authors in May 2002 to explore the importance of personalization to the information seeking behaviour of scientists.
| Find other papers on this subject |
© the authors, 2004. Last updated: 10 April, 2004 |
|