Information Research, Vol. 7 No. 2, January 2002,

Studying digital library users over time: a follow-up survey of Early Canadiana Online

Joan M. Cherry and Wendy M. Duff
Faculty of Information Studies
University of Toronto
Toronto, Canada

This paper reports the findings of a second user survey of a digital library collection of Early Canadiana materials. The main purpose of the study was to investigate whether the user group or the nature of use had changed since the first survey conducted a year earlier. As in the first survey we also wanted to gather feedback on satisfaction levels and suggestions for improvements to the Early Canadiana Online (ECO) site. A new section was added to the second survey to explore the use of ECO in teaching and research. Overall, findings showed that the user group and nature of use of the materials were remarkably similar to the first survey. Enthusiasm for ECO remained high but many of the requests for changes to existing features and the suggestions for enhancements to the site were the same as those in the first survey. The survey revealed that respondents who used ECO for teaching and research differed from other respondents in a number of ways. We close by discussing the implications of these findings for digital libraries in general and the value of studying digital library users over time.

Introduction

Early Canadiana Online/Notre Memoire En Ligne (ECO) is a digital collection of Canadiana¹ published before 1900. The collection consists of over 3,000 English and French language books and pamphlets. It is particularly strong in literature, women's history, native studies, travel and exploration, and the history of French Canada. This material is also available in rare book collections in the original paper format and in academic libraries on microfiche. ECO has full text of all material in a searchable ASCII file and provides individual GIF images for each page of an item. ECO was developed by the Canadian Institute for Historical Microreproductions (CIHM) in partnership with Laval University Library, the National Library of Canada and the University of Toronto Library. A grant from the Andrew W. Mellon Foundation and funds from numerous other agencies supported the development of the ECO site.

In April 1999 we conducted a user study of Early Canadiana Online. This study was carried out within the first few months of the site becoming available . The feedback gathered in the study was extremely positive (Duff & Cherry, 2000). However, users identified a number of areas for improvement and, based on these findings, we made 26 recommendations for changes to ECO. Some, but not all, of these changes have been made. We also suggested that a follow-up survey be conducted at a later date.

One year later we conducted the second user survey. Motivation for the second study was threefold. First, we wanted to see if there were any changes in the nature of use, or type of user². Second, we wanted to explore the use of ECO in teaching and research. Finally, we wanted to investigate satisfaction levels in more detail and to gather suggestions for further improvements to ECO.

Literature review

Few studies have evaluated digital library collections from the user’s perspective. The lack of user studies is surprising considering the increasing interest in, and number of, digital library projects. Saracevic and Covi (2000) point out that "a great many things are being done and explored, but evaluation is conspicuous by it absence in the majority of published work on digital libraries, be it research or practice" (p. 341).

The majority of digital library projects funded under the auspices of the Digital Library Initiative (DLI) have included research on the use of these collections. The studies have tended to take a fairly broad approach, using a variety of research methods including surveys, participant observation, usability testing, ethnographic studies, focus groups and transaction log analysis (Bishop, 1999; Bishop et al., 2000; Buttenfield, 1999; Hill et al., 2000) . Neuman and Bishop (2000) suggest that different methods of data collection or triangulation are crucial for one to obtain a holistic picture of the use of digital libraries.

Previous research suggests that the development of a good interface requires knowledge of how the collections will be used and the types of tasks the digital library supports (Bishop, 1999). The variety of tasks carried out by various communities often leads to different expectation of the system . The usefulness of a digital library depends on the existing beliefs of the users and their previous experience (Bishop et al., 2000) as well as the socio-technical environment in which the libraries are implemented (Kilker & Gay, 1998).

Research design

This study used a single method of data collection to gather information about the use of one digital collection. It employed a web-based questionnaire. We used Perseus Survey Solutions software to design the instrument and to administer the survey.

Design and testing of the questionnaire

The online questionnaire had 26 questions, grouped into five sections: (1) questions about the person's use of Early Canadiana materials in general; (2) detailed questions about how the person used an item from Early Canadiana Online in the current session; (3) use of the Early Canadiana Online web site as a whole; (4) use of Early Canadiana Online in teaching and scholarly research (if this did not apply, the user skipped to the next section); and (5) general questions about the person's use of computers and the Internet, and demographics. The questionnaire was available in English and French.

The questionnaire used in the first study included questions about use of Early Canadiana materials in original paper and microfiche formats in addition to the World Wide Web (WWW) format. In the second study we focussed on the WWW format only. Based on the responses to the first study, we clarified some of the wording and modified the choices available for some questions. We added a section on the use of ECO for two tasks (i.e., teaching and scholarly research) and another section on the use of the ECO web site as a whole which included satisfaction ratings for six features of the site. Finally, in the demographic section we added questions related to the size of the communities in which the respondents lived and their country of residence.

We pre-tested the revised questionnaire with five people and made minor changes in wording to four questions and dropped one of the choices in two questions.

Data collection

The questionnaire was available on the ECO web site from May 19, 2000 to June 12, 2000. Respondents in the survey were offered an entry for a draw with a chance to win one of five $100 cash prizes.

Data analysis

The response set from each user arrived as an email message. The email messages were uploaded to an Access database through Survey Solutions. A preliminary report of the data was generated through Survey Solutions. The data set was exported to SPSS 10.0 for more detailed analysis.

Findings

We present our findings in the context of the research questions which guided the collection and analysis of the data. The research questions were:

• Has the user group or the nature of use of ECO changed since the first user survey?
• How is ECO being used in teaching and research?
• How satisfied are users with features of ECO, and how can these features be improved?

User group and nature of use of ECO

User group

One hundred and fifty-nine people responded to the user survey³. Of the 159 respondents, 83% completed an English questionnaire and 17% completed a French questionnaire. Table 1 shows the profile of respondents in terms of gender, age, and area of interest.

Over half of the respondents were women (59%). In terms of age, over half of the respondents were forty or over (62%); 30% were in the age group 25-39; and 8% were under 25. In terms of primary area of interest, the two largest groups were history (33%) and genealogy (29%), followed by native studies (8%) and Canadian studies (7%).

The majority of respondents lived in Canada (72%); 22% lived in the United States. The majority lived in medium-sized cities or large metropolitan centres (75%). The majority of respondents (69%) were accessing the ECO site from a computer at home, while 23% were using a computer at work. The respondents were experienced Internet users with 79% having used the Internet for more than two years; 50% used the Internet 11 or more hours per week; 48% rated their knowledge of the Internet as excellent, while 42% rated their knowledge as fair.

Half of the respondents were regular users of Early Canadiana materials in some format with 29% reporting that they used the materials monthly, 18% weekly, and 3% daily. On the other hand, 14% used Early Canadiana materials rarely, and over one-third (36%) were first time users. It appears that these first time users had been introduced to Early Canadiana materials though the WWW. Figure 1 shows how frequently respondents used Early Canadiana materials in any format (original paper, microfiche, or WWW).

Nature of use of ECO

We explored the nature of use in terms of how people use ECO documents and how they used the ECO site in general.

One section of the questionnaire focussed on how the respondent used an item from Early Canadiana Online in the current session. The questionnaire asked how users found the item, their reason for using the item, how they used it, the parts of the item they used, whether they printed any portion of the item, and whether they would need to consult the original paper or microfiche format of the item.

Figure 2 shows how respondents found the Early Canadiana item. Almost three-quarters of the respondents found the item by using the ECO search function; 10% percent followed a link from another site; and 9% had used the item before and bookmarked it.

Approximately half (51%) were using the item for personal interest/hobby or genealogy. Twenty-one percent were using the item for scholarly research, 12% for professional purposes, and 6% for teaching. Only 5% read the entire item online; 29% read parts of the item online, while 30% read some of the item online and printed some, or all, of the item.

Over half (59%) of the respondents reported using the table of contents; almost one-third (31%) used the introduction; and about one-quarter (24%) used the index. Illustrations, graphs, tables/data, and citations were each used by slightly less than 20% of users. Less than ten percent (9%) of the respondents used the conclusion of the document. It is important to note that the collection contains many pamphlets and works of fiction and therefore many of the items used may not have contained some of these features. Almost half (49%) of the respondents printed some part of the item. For many (35%), this amounted to only a few pages. Well over three-quarters (82%) of respondents reported that they would not need to consult either the original paper or the microfiche format of the item just used. One user commented, "having access to facsimiles [sic] of each page obviates the necessity of consulting the original paper source."

Another section of the questionnaire focussed on the use of the Early Canadian Online site in general. It gathered information about how the respondents learned about the ECO site; how often they visited the site; how much time they had spent on the site on that day; and how satisfied they were with specific features. In addition, it asked how ECO might affect their work, research, education, and leisure activities.

Almost half (49%) of the respondents first learned about the Early Canadiana Online site by surfing the WWW. Eighteen percent learned about the site from an email discussion list or newsgroup, and 15% had a personal referral. Of the remainder, 9% learned about the site from a publication, 5% from promotional material, and 4% from other sources (Figure 3).

Almost half of the respondents are regular users of the site (monthly, weekly, or daily use). 17% of the respondents spent more than an hour on the site in the current session.

How is ECO being used in teaching and research?

A section of the questionnaire asked about use of Early Canadiana Online in teaching and scholarly research. If this did not apply, the user skipped to the next section. Sixty-two people responded to this section. As shown in Table 2, 38 respondents used ECO in their research; 24 respondents recommended ECO as a resource for students; 19 used ECO as a resource for their thesis or dissertation; 12 used page images from ECO in presentations and talks; and 11 used ECO to develop course material.

For teachers and researchers at institutions with small libraries ECO provides the means to do research or offer courses which require Early Canadiana materials. For example, one respondent stated, "I am at a university with a limited budget for microfilm acquisition, never mind the original documents. I might not be able to do specialist research from this university, were it not for the version of early canadiana online [sic] from the WWW database." A teacher commented that " it [ECO] will provide me with an additional place to search for Canadian content for my courses--i teach ata [sic] small community college so the importance of resources such as yours which supplement our relatively small library are invaluable to ourselves and to our students--and we are grateful for them…". ECO also provides source material for Canadian Studies courses taught outside Canada. "Being a Canadian who works in the United States, it's another search opportunity and another source to direct students to for class assignments and research."

ECO gathers source materials together and makes them available to users regardless of location. A researcher commented on the gathering function, "It will allow me to more easily pursue my research in material on my topic that is located all over the place, and usually difficult to access (rare book rooms with limited hours). It will also make it easier to share ‘finds’ and discuss with colleagues." One person commented on location in terms of convenience, "I can find things instantly, from my office. This is useful when I'm preparing lectures or advising students. Also, my students find it more interesting to access material on-line than on microfiche." Another commented on location in terms of geography, "ECO will allow me to continue my doctoral research next year even though I will be living in the UK where sources for 17C Canadian history are not readily available."

ECO facilitates discovery of new resources. As one graduate student stated, "It has been a fantastic resource for my Master's thesis research - I have found articles here I didn't even know existed. The search capabilities allow me to find information in sources I may otherwise have overlooked." ECO also helps users retrieve relevant information from material that has not been indexed. One respondent stated, "The capacity to key word search document speeds up the process of searching through books without indexes; prior to this I've had to scan page-by-page 19th-century titles."

Another question asked what would make ECO a more useful resource for teaching or research. In response, one person suggested links to "information about the texts and their authors". Another suggested that ECO provide access to the collection by genre, e.g., novels and poetry. Two respondents suggested that ECO include non-textual materials (e.g., illustrations). Several commented on aspects of the print function, with one suggesting "a print-on-demand, just-in-time delivery service at a reasonable cost. This should be partnered with one of the new e-publishing houses such as Trafford."

To explore how the tasks of teaching and research affect the use of ECO we compared the responses provided by those who used ECO for teaching or research with data from other users. We found significant differences in two areas: one related to participants' use of an item; the other related to the ECO site. In terms of use of an item, although only nine percent of respondents used the conclusion of the item, three-quarters of these respondents (9 of 12) used ECO for teaching and research. One question asked respondents, "Now that you have used this item in the World Wide Web format, will you need to consult it in the original paper or microfiche format?" There were three choices: Original paper, Microfiche, or Neither. Significantly fewer respondents who used ECO for teaching and research chose Neither (chi-sq = 3.904, df = 1, p = .048).

In terms of the ECO site, we found significant differences in how people learned about the ECO site, how frequently they used the site, and how satisfied they were with the ECO search function. Of those who used ECO for teaching and research, more than expected found out about the site by personal referral. Of the remaining respondents, more than expected found out about the site by surfing the WWW or an email discussion list (chi-sq = 21.190, df = 5, p = .001). Of those who used ECO for teaching and research, more than expected used the site on a frequent basis, i.e., monthly, weekly or daily (chi-sq = 21.321, df = 4, p < .001). Finally, although the levels of satisfaction with ECO features were high overall (as discussed in the next section), of those respondents who used ECO for teaching and research, more than expected were "Dissatisfied" or "Very Dissatisfied" with the ECO search function (chi-sq = 9.039, df = 3, p = .029).

How satisfied are users with features of ECO, and how can these features be improved?

Overall, users were satisfied with features of Early Canadiana Online. The survey asked users to indicate how satisfied they were with response time, browse capabilities, comprehensiveness of the collection, print function, search capabilities, and display of document pages. Over 90% of the respondents indicated that they were Very Satisfied or Satisfied with five of the six features.

For the sixth feature, the print function, 85% were Very Satisfied or Satisfied. Of the 57 respondents providing comments about the features, 15 people commented on the print function. People complained that printing was "very slow and cumbersome", "painfully slow", and that they wished to download or print multiple pages or the entire document. For example, one respondent commented that "when printing out a whole work it is very slow to do so page by page. It would be very useful to be able to download one whole work at once." Another stated, "To print a page at a time is very time consuming, especially when you want the PDF version. It would be nice if it were available in larger ‘chunks’." Two people said that they would like to be able to print the page without extraneous information such as headers and search criteria. ECO provides an option to view and print a PDF version of each page image (i.e., without any extraneous information) however it seems that some users are not aware of this option. The print function was also a focus of serious concern for respondents in the first survey. The recommendation which we rated as highest priority was that ECO should support printing of an entire document or multiple pages of a document. However, this recommendation has not yet been acted upon. Obviously the problems persist.

Some respondents commented on the navigability of the site, requesting the creation of links from the Table of Contents and back to the Table of Contents. For example, "Having a hyper-linked Table of Contents would be very helpful", and "I would like to see an option button at the bottom of every text page which allows me to return to the Table of Contents page for that document."

Many respondents expressed support for expanding the database, e.g., "I hope you are continuing to add more resources – the concept of Early Canadiana Online is excellent." Some respondents made specific suggestions for additions, e.g., "more regional resources to balance what is already in place", "more on Loyalists and the settlement of Upper Canada", and "more social history, settlement history". Some even recommended specific periodicals to add to the site, e.g., Canadian Magazine and the Bystander, and the addition of archival records such as the employee records of the Grand Trunk Company. Many respondents in the first survey made similar comments.

We asked users to comment on what they liked best about Early Canadiana Online. Most frequently mentioned themes were accessibility, searchability, and appreciation of the scanned images.

ECO provides ready access to materials that are otherwise difficult for some users to access because of distance and the restrictions of rare book collections. For example, one user liked, "Being able to read portions of these documents online, without having to go to the library, without having to read microfiche, without having to order and wait for receipt of items through interlibrary loan." Another user associated the improved access with saving time and money, "I like the fact that I am able to access material that has been previously available on[ly] in places such as the Thomas Fisher Rare Book Library. This cuts down on both research time and expenses." Many users explicitly commented on the Canadian content. For example, "Provides good access to historical materials on Canada that are not available otherwise" and "Excellent selection of first hand accounts of life in early Canada."

ECO provides full-text search on OCR text and many respondents commented on the search capability of the system. "The search service acts (miraculously) as a gigantic index to 3000 printed volumes. This is a major advance for historical researchers."

Users indicated that they liked the scanned images because using them is similar to using the original material. One user stated, "I love to look at the real pages for it is just like reading the original book. Having the picture of the whole book page by page is really fascinating." Another commented, "seeing original documents makes history ‘come alive’." Other users liked ECO because it provided "untampered source material".

The comments from the survey also revealed some unanticipated uses of the ECO collection, e.g., a toponymist uses ECO to pinpoint the date and origin of place names; one person uses ECO to help learn his native language; and another uses ECO as a source of knitting and crocheting patterns. These users may be some of the people who have been introduced to Early Canadiana through the WWW.

Discussion

Overall, the findings from the second user survey parallel those of the first survey (Duff & Cherry, 2000). The profiles of the respondents in the two surveys are remarkably similar except in the following areas. Fewer people completed a French questionnaire (17% in the second survey vs 26% in the first survey); there was a higher percentage of female respondents (59% in the second survey vs 47% in the first survey); and respondents were slightly less heavy users of the Internet (30% of respondents in the second survey used the Internet more than 15 hours per week compared to 46% in the first survey).

As in the first survey, we found that users continue to use the traditional methods to access the content of an item. For example, an increasing percentage of respondents reported use of the table of contents (59% in the second survey vs 26% in the first survey) and the index (24% in the second survey vs 17% in the first survey). In the second survey, a larger percentage of respondents reported that they printed some part of the item (49% in the second survey vs 30% in the first survey).

In the second survey we explored a new area, the use of ECO in teaching and research. The findings indicate that ECO can give teachers and students with limited budgets or at institutions with limited budgets access to rare, highly valued research material. The findings also reveal how respondents who use ECO for teaching and research differ from other respondents in how they used an item, how they learned about the ECO site, how frequently they used the site, and how satisfied they were with the ECO search function. These differences may be a result of the types of tasks they carry out or the culture of teaching and research.

In both surveys, users were very satisfied with ECO and very enthusiastic about it. They pointed out that ECO overcomes barriers of rare book libraries with limited hours and faulty or cumbersome microfiche equipment. Academics at small institutions noted that ECO provides valuable resources for Canadian history which their libraries, with limited budgets, could not afford. Other respondents noted the advantages of a single access point to materials that are held in rare book libraries across Canada and the advantages of ECO’s full-text search function which enables them to search the content of historical materials that are not indexed.

Our overall assessment based on the findings of the second user survey is that ECO is well received and has a great potential to attract new users and increase appreciation of Early Canadiana materials. However, despite the enthusiasm for ECO, some problems identified in the first user survey were noted again. For example, users continue to complain about the print function which does not allow printing of multiple pages or the entire document. Similarly, some enhancements suggested in the first survey were requested again in the second survey, e.g., the creation of links from the table of contents to the text of a document.

Implications for digital libraries

The study reported here addresses the use of one digital collection. However, its findings are relevant to research on the use of digital libraries in general. The findings suggest that digital library research should take into account the types of tasks carried out, as well as the culture, of different user groups. Furthermore, digital libraries of historical materials have the potential to attract new user groups who will use the collection to support unexpected tasks. The findings also indicate that it is important to study the use of digital library collections over time to reveal changes in both the users and their uses. Web-based surveys provide an efficient and effective way to gather such data. These surveys can be conducted on a regular basis (e.g., annually) or to gather feedback when changes are made to the collection or site. Comparing survey data across time and triangulating survey data with other sources of data (e.g., logs, interviews) will provide us with a more complete understanding of the use of digital libraries. Only with this fuller understanding will we be able to develop integrated digital environments for scholars, and assess how such environments might affect the scholarly process.

Acknowledgements

We would like to thank the following individuals for their contributions to this work: Clément Arsenault, Joan Bartlett, Zoran Piljevic, and Marlene van Ballegooie. We also wish to express our appreciation to Pam Bjornson, CIHM Executive Director, for supporting this research.

Notes

1  Canadiana includes documents published in Canada as well as documents published in other countries but written by Canadians or about Canada. 
[Return to main text]

2  First, the original survey covered a time period when publicity for the Early Canadiana Online site was heavy. We felt that this publicity might have affected the volume of use, the nature of use, and the type of users. The second survey was conducted one year later during a more typical period (i.e., publicity regarding the launch of ECO was over, and consequently there might be fewer users who were simply curious about the site).
[Return to main text]

3  We are unable to determine how representative the respondents in this survey are of the population of ECO users. The number of respondents is small compared to the number of hits that ECO receives, however we have no way of determining who uses ECO, or how often they visit the site. We note, though, that the number and types of respondents in this survey are similar to the respondents in the previous study.
[Return to main text]

References

• Bishop, A. P. (1999). Document structure and digital libraries: How researchers mobilize information in journal articles. Information Processing & Management, 35, 225-279.


• Bishop, A. P., Neumann, L. J., Star, S. L., Merkel, C., Ignacio, E., & Sandusky, R. J. (2000). Digital libraries: situating use in changing information infrastructure. Journal of the American Society for Information Science, 51(4), 394-413.


• Buttenfield, B. (1999). Usability evaluation of digital libraries. Science & Technology Libraries, 17(3/4), 39-59.


• Duff, W. M., & Cherry, J. M. (2000). Use of historical documents in a digital world: comparisons with original materials and microfiche. Information Research, 6(1), Available at: http://www.shef.ac.uk/uni/academic/I-M/is/publications/infres/paper86a.html [Accessed 12 September 2001].


• Hill, L. L., Carver, L., Larsgaard, M., Dolin, R., Smith, T. R., Frew, J., & Rae, M. A. (2000). Alexandria digital library: user evaluation studies and system design. Journal of the American Society for Information Science, 51(3), 246-259.


• Kilker, J., & Gay, G. (1998). The social construction of a digital library: a case study examining implications for evaluation. Information Technology and Libraries, 17(2), 60-70.


• Neumann, L. J., & Bishop, A. P. (2000). From usability to use: measuring success of testbeds in the real world. In S. Harum & M. Twidale, editors. Successes and failures of digital libraries. (pp. 58-69). Urbana-Champaign, IL: University of Illinois.


• Saracevic, T., & Covi, L. (2000). Challenges for digital library evaluation. In D. H. Kraft (Ed.), ASIS 2000: Proceedings of the 63rd ASIS Annual Meeting.Vol. 37 (pp. 341-350). Medford, NJ: Information Today.

How to cite this paper:

Cherry, Joan M. and Duff, Wendy M. (2002) Studying digital library users over time: a follow-up survey of Early Canadiana Online. Information Research, 7 (2)    Available at: http://InformationR.net/ir/paper123.html

© the authors, 2002.    Updated: 13th September 2001