Introduction

It is known that some significant portion of college research is done using commercial search engines on the Web. In this process, traditional gatekeepers of research have been replaced by search engine applications with unknown algorithms which return content of dubious quality. That college students today use the Internet for research is not unexpected. Most have had constant exposure to the Web and its vast sea of fragmented and unstructured information. Information literacy instruction attempts to address some of these issues. Librarians, however, often have difficulties scheduling any information literacy instruction and when they can perform the instruction, they have a limited amount of time to discuss evaluating sources.

Librarians and instructors strive to teach college students to evaluate sources of information for all their research projects. Critically evaluating information sources is a life skill and considered a core competency of information literacy highlighted by the Association of College and Research Libraries Information Literacy Competency Standard 3 (Association of College and Research Libraries, 2000). The standards have given librarians the support needed to advocate for information literacy, but librarians with just a one-time guest instructional session can do little more than suggest students evaluate Web sources and highlight the criteria to use, often in a checklist form.

It is unclear how effective information literacy instruction has been in teaching students to specifically evaluate Web information sources. A better understanding of how students perceive information sources on the Web and how they evaluate the suitability of those sources for their research could help improve information literacy instruction.

The intention of the research reported here was to evaluate the student perceptions of Web information sources, specifically their evaluation of various properties of those information sources in relation to traditional information literacy instruction. The study investigated student understanding of the evaluative criteria recommended in the Association of College and Research Libraries Information Literacy Competency Standard Performance Indicator 3.2a concerning "reliability, validity, accuracy, authority, timeliness, and point of view or bias" of sources (Association of College and Research Libraries, 2000). (See Appendix B for the Association of College and Research Libraries standards, performance indicator and outcomes that informed this research).

Literature review

Traditionally, it has been the librarian's role to ensure students have access to quality material. While most librarians and faculty direct students to quality information sources, students easily bypass the traditional method and use familiar search engines. A gap exists between what sources professors expect students to cite in their academic research papers and the sources students actually use. Grimes and Boening (2001) identify two reasons why the gap might exist. In their study they report that faculty provided instruction which directed students to use reliable resources. Results of their study indicated that faculty felt their advice on evaluating information sources was ignored, and students indicated that they did not know what was meant by a reliable resource. More confusion exists about what is acceptable to the instructor versus what librarians recommend, yet it is the librarians who are commonly tasked with ensuring students meet information literacy outcomes (Weiner, 2014).

Librarians lament when their information literacy efforts go to waste. Students show they have not retained knowledge when they cite unacceptable sources in their academic research papers. In attempts to improve information literacy instruction, librarians often get creative. Librarians might have an entire hour or just ten minutes to devote to teaching the evaluation of sources and strive to deliver effective instruction to leave students prepared to evaluate any information source they encounter. Evaluation of sources as an information literacy competency standard has become increasingly important as information on the Web continues to expand dramatically.

To address students' gaps in information literacy skills, many different approaches to Website evaluation have been taught by librarians. Librarians often teach to a checklist model with various acronyms and mnemonic devices such as RADCAB (relevancy, appropriateness, detail, currency, authority, bias), and CRAAP (currency, relevance, authority, accuracy, purpose) (Christennson, 2013; Blakeslee, 2004; Grimes and Boening, 2001; Shanahan, 2008). The goal of the checklists is to address the criteria defined in Association of College and Research Libraries information literacy competency performance outcome 3.2a, which stresses the importance of evaluating source research documents (Meola, 2004; Metzger, 2007). Librarians' information literacy instruction may also provide examples showcasing pre-selected Websites that might be good, bad, extremist or a hoax (Mathson and Lorezen, 2008). Other librarians' information literacy instruction may vary from the standard approach to information literacy instruction to teaching using various learning styles (Dahlen, 2012), teaching to various stages in cognitive development (Jackson, 2007; Benjes-Small, Archer, Tucker, Vassady and Resor , 2013), teaching using a contextual approach (Meola, 2004) or a constructivist approach (O'Connor et al., 2010), and engaging in new strategies such as active learning and redesigning learning spaces (Detlor, Booker, Serenko and Julien, 2013; Hsieh, Hofmann, Dawson and Titus, 2014; Julian, 2013).

Existing literature also discusses partnerships between librarians and faculty. Writing instructors and librarians partner in teaching writing information literacy as both have similar goals (O'Connor et al., 2010; Norgaard, 2004). There are many successful librarian and discipline faculty collaborations which can be found in the literature, for example, in engineering (Wang, 2011), nursing (Dee and Reynolds, 2013), business (Atwong and Heichman, 2008) and other domains (Weiner, 2014).

Ultimately it is the student who must apply the criteria and perform the evaluation of information sources. Studies however have reported a discrepancy between what students say they can do and how they actually perform. Students can confidently articulate what to do or what to look for in evaluating sources, but when they have to perform research, they either do not know how to do it, do not have the skills or just do not perform the work needed to evaluate sources correctly. Thus, the criteria students say they will use are not the same criteria they actually use (Currie et al., 2010; Grimes and Boening, 2001; Walraven, Brand-Gruwel and Boshuizen, 2009). A part of this discrepancy could be linked to students not really grasping how to apply the evaluative criteria or not understanding the need to apply the criteria while examining sources.

Other problems in the use of evaluative criteria are also evident. In Grimes and Boening (2001) some students assigned credibility based on how a page looked and students ranked the author as the most important feature of a quality Website yet failed to consider the author's qualifications (ibid, p. 18). Studies have also provided some indication that students claim to evaluate their sources much more than they do in reality (Flanagin and Metzger, 2008; Gross and Latham, 2011; Metzger, Flanagin and Zwarun, 2003). Gross and Latham (2011) recognise that this is part of a greater pedagogical problem in which students often believe they have skills to find, evaluate and use information, but teaching students to be more self-reflective of their need to learn without damaging their self-efficacy is challenging.

Previous research has also provided some indication that college-age searchers tend to skim sources and use just one or two pages of content to make a determination about the quality or suitability of a potential research source (Rowlands et al., 2008). Students also admit to selecting a source because their search keywords were in the title or in one of the lead paragraphs of a document (Currie et al., 2010). When probed, students disclose that their focus is on finding an answer and that the source of the information is of little concern (Van Deursen and Van Diepen, 2013). Research suggests that this light review of content may be related to disinterest or lack of concern for the evaluation of various characteristics of content among college-age searchers and Internet users in general (Williams and Rowlands, 2007; Hirsh, 1999; Grimes and Boening, 2001; Lorenzen, 2001; Carr, 2010). It may be that in viewing the search process as just another task, a consumerist approach may prevail whereby searchers simply view information as just another commodity to be consumed at the lowest possible cost, identifying cost in this case as ease-of-access with limited consideration for quality (Buczynski, 2005; Thompson, 2003; Young and Von Segern, 2001).

Some research has also focused specifically on the Web as a research source. Studies have suggested that students often prefer Google as their first choice for research (Head and Eisenberg, 2010) and that Web materials are easier for students to find (Biddix, Chung and Park, 2011; Currie, Devlin, Emde and Graves, 2010). Studies have also examined how students evaluate documents found through various information retrieval sources. In Taylor (2012), subjects appeared to have limited interest in source quality and appeared to procrastinate in their effort to find information as the deadline for the research project approached.

Other studies have identified additional information literacy problems specific to Web sources. Thirty-eight percent of students in a study reported by Tillotson (2002) did not recognise the need for evaluating resources, indicating that they rarely find incorrect information on the Web. Students are aware that not all information on the Web is reliable but appear to forget this while searching. Students report trusting institutional Web pages, Web pages free from spelling errors, Web pages with a list of good references, a professional layout, easy navigation, and no commercial purpose and/or a lot of detail (Lorenzen, 2001; Hilligoss and Rieh, 2008; Head and Eisenberg, 2010).

To understand how students evaluate sources, a clearer understanding of the criteria they use to evaluate those sources is needed. In a study conducted by Project Information Literacy, students rated how often evaluative criteria (currency, author's credentials and different viewpoints expressed) affected their selection of research sources. Questions on the survey instrument used inquired about which parts of the course-related research process were difficult. The survey included question categories for filtering irrelevant results, 'determining credibility', and 'evaluating sources', but questions did not specifically enquire about difficulties in determining page authorship or other evaluative criteria (Head and Eisenberg, 2010, p.25). Other surveys have examined social cues, social networks and students' use of instructors, classmates, friends/family, librarians and others for help with evaluation of information sources, but specific criteria used for evaluations were not examined (Metzger, Flanagin and Medders, 2010; Head and Eisenberg, 2010, pp. 13, 40).

In order the better to teach information literacy, librarians need to improve their understanding of students' decision processes when evaluating and selecting a source for their research assignments. If, as some studies suggest, the Web has become a primary source of information for student researchers, then a better understanding of how students specifically evaluate Web information sources is needed. Previous research has not specifically examined the criteria for source evaluation, such as source authorship, authority, accuracy, credibility, objectivity and quality, by the student information seeker. Research which examines these aspects of site evaluation for Web sources is needed. Such research will help clarify educators' understanding of these aspects of student information gathering and provide suggestions for improving information literacy instruction.

Research design

To understand better how searchers evaluate information sources and to examine this evaluation process in relation to current information literacy instructional strategies, the following research questions were examined in this study.

In relation to the source evaluation criteria identified by the information literacy competency standards (reliability, validity, accuracy, authority, timeliness and point of view or bias):

• Do searchers consider it possible to determine these criteria for documents being evaluated?
• How do searchers attempt to evaluate these criteria when evaluating information sources?

To gather information on the evaluation of sources during the information search process, a survey instrument was developed (see Appendix A). The survey questions were based on the Association of College and Research Libraries information literacy competency standards for higher education published in 2000. Questions were based primarily on standard three from this publication which indicates that 'the information literate student evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system' (Association of College and Research Libraries, 2000). A small number of questions were also used to gather demographic and other information about the subject. (See Appendix B for a specific description of how questions were mapped to the Association of College and Research Libraries information literacy competency standards.)

The survey instrument was designed to be delivered online, through the Web. Students were not monitored as they completed the survey and were allowed to take the survey at their convenience. Subjects took the survey before receiving information literacy instruction from library faculty at the university. Survey results were stored anonymously in a database for later analysis.

Subjects taking the survey were presented with a screen which indicated that the research was voluntary and that they could stop at any time. They were then presented with instructions on how to complete the survey (see Figure 1). Subjects were directed to answer questions about how they conducted and evaluated sources of information for their college research across all college courses.

Most subjects completed the twenty-seven questions in the survey in between five and ten minutes. The data collected was anonymous. Subjects were not directly compensated for participation, but some subjects were allowed to optionally be entered into a pool to win a $50 gift certificate.

The survey taken by the subjects contained twenty-seven questions (see Appendix A). (Note that question twenty-eight on the survey allowed subjects to optionally enter their email to be entered in a pool for a gift certificate and therefore was not analysed.) Respondents provided demographic and background information for questions one to eight. Respondents provided answers relating to how they searched for information and evaluated information sources for questions nine to twenty-seven. Survey questions nine to twenty-seven were related to Association of College and Research Libraries information literacy standards (2000) as detailed in Appendix B.

Results

The 389 subjects who participated were undergraduate students at an American university. Most were business majors (38%) with the remainder being a mix of liberal arts, education and communication majors. Most subjects were female (66%). Most subjects indicated they had a grade point average (GPA) of greater than 3.0 (74%).

(Note that for all survey questions presented in the tables that follow, the percentage column is calculated as the number of respondents who selected the item divided by the total number of respondents for the survey. Using this approach for single select questions, the sum of the percentages for a question response will be equal to or close to 100% with some deviations due to rounding errors. Using this same approach for multiple select questions, the sum of the percentages for a question response will not total 100% for a question.)

Information search habits

When asked about their search habits, approximately 90% of the subjects indicated they searched for information on the Web several times a day. Subjects also indicated they were most likely to search using Google, but suggested that other search engines and tools were also used in their research (see Table 1). Twenty-five percent of the subjects indicated they rarely use a search engine other than Google, 5% indicate they never use a search engine beyond Google, and 23% indicated they infrequently (less than 25% of the time) use a search engine beyond Google, so for 53% (25+23+5) of our sample, Google is the most commonly used information search tool for academic research.

When asked how they identify whether or not to use a source in their research, subjects provided a number of different answers from a set of criteria as shown in Table 2. The most common selection was that the Website was understandable and current, suggesting that Websites which were difficult to understand or did not appear to have current information were not used. It is interesting to note that the criteria for credentials of the author and the amount of information were selected less commonly than the criteria of understandability and currency of the information, suggesting that the credentials of the author were less of a concern for some of the subjects.

Evaluation of author qualifications and site authority

Subjects provided a range of answers concerning the authority of a Web page and the qualifications of the author. When asked how they determine the author of a Web page (see Table 3), almost half of the respondents indicated they could not make such a determination (45%) with a little over half indicating they could (55%). A similar conundrum exists with evaluating the qualifications of the author of the Web page. When asked if they could determine the qualifications of the author of a Web page, only 30% indicated they could, 63% of the respondents indicated they could not, and 7% indicated they did not understand what was meant by qualifications in the question. These results provide some indication of confusion concerning this information literacy instruction requirement and perhaps an issue with Web content in general.

A similar question asked subjects about evaluating whether or not a site is authoritative. A little over half indicated they could determine whether or not a site was authoritative (53%), almost a quarter (24%) indicated they specifically could not determine the authoritativeness of a site, and almost a quarter (23%) indicated they did not understand (perhaps hedging because they do not check or do not know how to check).

In examining how the subjects evaluated the authority of a site (see Table 4), subjects appeared to be confused by the general concept of authority, with 17% of respondents indicating they did not understand what was meant by that term in relation to Websites. Ten percent of respondents do not evaluate the authority of a site and 6% do not believe it is possible to examine the authority of a site. The total of these three selections represent one third (33%) of respondents who do not evaluate the authority of a site for some reason, whether it is lack of understanding or the belief that it is not possible to examine the authority of a site. Thirty-nine percent of respondents use the URL of the site to determine authority (perhaps not the best indicator of authority). Twelve percent confer with someone knowledgeable in the subject domain and 43% check to see if the author has published other content.

Evaluating the objectiveness of content

When asked about the objectiveness of a Website in a TRUE/FALSE question (see Table 3), most respondents indicated they believed it was possible to evaluate the objectiveness of a Web page (74%), while a quarter (25%) indicated they believed it was not possible, and 1% did not understand the concept of objectiveness in relation to a Web page.

Table 5 provides the results of a question concerning the evaluation of the objectiveness of an Internet source and reveals a more nuanced perspective on the part of the subjects. For this multiple-select question, 68% of respondents indicated that a fair and balanced discussion of the topic indicated the site was objective, a standard definition of the term, but an almost equal percentage (62%) indicated that they used the URL of the site and the domain to determine whether or not the source was objective. Seven percent of respondents do not check the objectiveness of a site and an almost equal number (5%) do not believe it is possible to check the objectiveness of a site. Interestingly, 8% of the respondents felt that all pages returned by a search engine are objective.

Evaluation of content credibility

When asked a TRUE/FALSE question about source credibility, approximately 80% of the subjects indicated that they felt it was possible to determine the credibility of a Web page (see Table 3).

When asked a multiple select question about how they determined the credibility of a document and determined whether or not the document was trustworthy, subjects responded as shown in Table 6. Most subjects (57%) reported that they looked at the URL of the site and an equal percentage (58%) looked at who published the site. Approximately 9% believed all pages returned by a search engine are credible, 5% do not evaluate the credibility of a site and another 2% do not understand what is meant by credibility, so a total of 16% (9+5+2) of subjects in this sample appear to not be evaluating credibility of Web content.

Evaluation of content accuracy

When asked a TRUE/FALSE question about the evaluation of the accuracy of Web content, over half (60 %) of the subjects indicated that they believed pages listed or returned by search engines usually contain accurate information (see Table 3).

When asked a multiple select question about how they evaluate the accuracy of information on a page, subjects appeared to take a more conservative and nuanced view of search engine accuracy (see Table 7). Only 6% of the subjects indicated that search engine results are accurate (the word 'usually' was not used here, implying either pages were accurate or were not accurate). Over half of the respondents (60%) look at the URL to evaluate accuracy and an almost equal percentage (64%) evaluate the content on the site against their own prior store of knowledge (what they know about the topic).

Evaluation of page quality

When asked about how they evaluate the quality of a Website, subjects responded as shown in Table 8. Over half of the respondents (51%) appear to consider spelling and grammar to be part of the evaluation, and an equal percentage (51%) feel the URL of the site has some bearing on quality. Most (66%) of the subjects based quality on a determination of whether the page is comprehensive and covers the topic in considerable depth and over a third (38%) chose Web pages which were interesting and gave clear and well-reasoned explanations.

Evaluation of information sources in relation to years in college

Evaluation of survey responses in relation to year in college provides some additional insights into the success or failure of current information literacy training efforts. Table 9 details the TRUE responses to a set of questions for early years college students (freshmen and sophomore year) and later years college students (junior and senior year). These results indicate a statistically significant change in students' perception of various properties of Internet sources in later years of college (Χ-squared = 19.3077, df = 11, p-value = 0.056). Subjects appeared to be more circumspect in their evaluation of Web sources, with more subjects in later years indicating they could not identify the author of a Web page and more subjects indicating they could not determine the credibility of a Web page.

A question concerning the choice of information search tool also provides some indication of a change in the student's perception of sources in relation to year in college as indicated in Table 10. When asked whether they use a search engine or library database, survey respondents in later years of college indicated a slightly greater preference for library databases and a slightly smaller preference for Google. These results are significant at the p <= 0.10 level (Χ-squared = 11.97, df = 7, p-value = 0.10).

Discussion

The major findings from this study can be summarised as follows.

• the Google search engine appears to be the most widely used source for academic research for subjects, though subjects did report using other sources
• subjects had a high degree of confidence in search engine results, with the majority (60%) believing that pages returned by search engines usually contain accurate information
• identification of the author of a Web page appeared to be difficult for subjects, with almost half (45%) indicating they could not determine the author of a Web page
• evaluation of the authoritative characteristics of a source was difficult for subjects, with almost a quarter reporting they could not determine the authoritative characteristics of a site and almost a quarter reporting they did not understand the concept in that context
• identification of the qualifications of the author were difficult for subjects, with a majority (63%) indicating they did not believe it was possible to evaluate author qualifications
• a quarter (25%) of respondents did not consider it possible to determine the objectiveness of a Web page and over half (62%) used the URL and domain of the site to determine objectivity
• subjects in later years of college (junior-senior) appear to be more discerning about Internet sources, less likely to use Google and more likely to use library databases

These findings and others are discussed in more detail in the following paragraphs.

Information search habits

Google was the most common information source for this search sample, though respondents did provide some indication that they also commonly use other information sources. Nevertheless, this provides some quantitative confirmation of other findings which indicate that the Web and specifically Web content delivered through the Google commercial search engine continues to be a primary source for college level research (Head and Eisenberg, 2010; Biddix, Chung and Park, 2011).

Evaluation of author qualifications and site authority

Information literacy instruction implores students to ask questions about the authority of content sources, specifically to learn about the authors of the source. Students are asked to determine the author's credentials, reputation, knowledge about the subject, education and affiliation (Doyle and Hammond, 2006). For this sample, almost half (45%) of the subjects indicated they could not identify the author of a Web page and thus would not be able to meet that information literacy competency standard performance outcome. This finding provides evidence to support Doyle and Hammond's (2006) proposal that identifying the source and determining authorship be supported through specific criteria (pp. 63-64).

Similarly, when asked about the qualifications of the author of a Web page, a majority of respondents (63%) indicated they could not evaluate the qualifications of an author, providing indications of further issues concerning the evaluation of Web content authorship. Furthermore, in evaluating the authority of a site, a third of the respondents indicated they do not evaluate site authority for some reason, and 39% of respondents use the URL of the site to determine authority (perhaps not the best indicator of authority). Twelve percent confer with someone knowledgeable in the subject domain and 43% check to see if the author has published other content. These results are an indication that subjects struggle with the identification of site authority using a variety of methods which may or may not provide good results.

Evaluating objectiveness of content

When teaching about objectivity, librarians expect that students will be able to identify if a source provides a fair discussion of all viewpoints. The majority of subjects in this study had confidence that they could evaluate the objectiveness of a Web page (74%). A majority of subjects (68%) attempted to determine if a site provides a fair discussion of all sides of a topic, an indication that they are making a valid effort to determine the objectiveness of content. Over half (62%) indicated they use the URL of the site to determine the objectivity of the source and a quarter of the subjects check with someone with more knowledge concerning a source. While a URL alone is not a good indicator of objectiveness, subjects were considering other criteria. That only 25% seek advice is potentially an indication that expediency may be more important to students than taking the time to perform a deeper evaluation of objectivity for a source.

Evaluation of content credibility

Most subjects indicated they could evaluate content credibility (80%), but 57% use the URL and 55% look at the author of the site, which almost half of the subjects indicated they could not identify. Five percent of subjects did not appear to be considering the credibility of a site at all. Additionally, about a third (36%) reviewed the site's content against what they already knew about the subject. This suggests that many students are not thinking critically about the information presented to them and evaluate credibility using a superficial and easy measure such as the site URL. This provides additional confirmation of a finding reported by other researchers that 'students value credibility but prefer efficiency' (Biddix, et al., 2011, p.180).

Evaluation of content accuracy

In response to a question about page selection, subjects indicated they used a somewhat balanced approach to selecting pages for their research, with a slight preference for understandability and currency of pages relative to other criteria. A majority of respondents believe the pages returned by search engines usually contain accurate information. The survey question asked if the results returned were usually accurate, so it is possible subjects allowed for inaccurate information sometimes being returned from search engines. Still this result appears to provide some indication that subjects were very confident in the accuracy of search engine results. This finding may be a disappointment to librarians who would prefer students not simply assume pages or articles are accurate. If students trust a search engine's results, giving credit to a source just because it appears in the first or second page of the results, as Head and Eisenberg (2010) also found, then there is little motivation to evaluate these sources.

An additional question provided more in-depth examination of how subjects evaluate the accuracy of Internet sources. Survey responses indicated that approximately 60% of the subjects look at the URL to evaluate accuracy and 64% evaluate the content on the site against their own prior store of knowledge. Only about a third of the subjects look for help in evaluating the accuracy of a source, indicating that they were fairly confident they could evaluate the accuracy of the source through the URL or their own knowledge.

Evaluation of content quality

Survey respondents provided a somewhat balanced set of responses when asked about the evaluation of content quality. Half (51%) of the respondents considered the URL and domain to be an indicator of quality and an equal percentage considered the spelling and grammar on the site to be an indicator of quality. Most (66%) considered the depth and comprehensiveness of the site to be an indicator of quality and 38% considered the well-reasoned presentation of the topic to be the standard for site quality. This finding provides some indication that subjects are evaluating quality and using a number of different criteria, though it is debatable whether or not the URL and domain alone provide a clear indication of Web page quality.

Evaluation of information sources in relation to years in college

Survey respondents in later years of college (junior-senior) indicated a slight but statistically significant change in evaluation of Internet sources. Questions concerning the examination of various properties of a Web source revealed that subjects in later years of college were more aware that it may not be possible to determine the author of a Web page and the qualifications of the author. Subjects also indicated that they were less certain about the objectivity, authoritativeness and credibility of Web sources in later years of college. Conversely, these same subjects also appeared to be more certain of the accuracy of Web sources.

This finding provides some indication that the junior and senior class students in this sample have learned some information literacy skills and report applying them in the evaluation of Web sources. However, there still appears to be a belief amongst these same students that Web sources are usually accurate sources. This certainty may reflect a personal confidence that they can determine the accuracy of a Web page or that they have been desensitised and falsely assume that this constant source of information (the Web) is truly accurate information.

Limitations of study

Evaluation of the demographic questions in the survey revealed a slight bias towards women and students with a high grade point average (above 3.0). Additionally, the survey sample was from a single U. S. university. A slightly more diversified sample from more universities, domestic and international, might produce different results.

Though subjects in this sample received information literacy instruction after taking the survey, it is possible that the subjects had previously received information literacy instruction in a previous year at the university or before coming to the university.

Subjects may not have clearly understood the terms and concepts addressed in the questions in relation to information seeking. Many questions provided a response which could be used to indicate confusion with the concept, but not all questions provided this response.

Conclusion

The findings in this study add quantitative evidence to the assertion that the Web has become the primary source for research for college students. Though subjects gave some indication that other sources were used, the Web, through the lens of the commercial Google search engine, appears to be their primary research source.

The goal of the study was to examine two research questions in relation to information literacy standards for evaluation of Web sources. Evaluation of the sources was based on the criteria of authority, reliability, validity, accuracy, timeliness and point of view or bias. The first research question was whether searchers considered it possible to examine these criteria for documents found on the Web. Survey questions fourteen to nineteen addressed this research question. Our findings indicate that for the evaluative criteria of author, authoritativeness, objectivity, credibility and accuracy, the majority of subjects felt they could determine these criteria for Web information sources. However, a majority of subjects felt they could not determine the qualifications of the author of a Web page. The second research question was to determine how searchers evaluate these criteria when examining information sources. Survey questions twenty to twenty-seven addressed this research question. Our findings indicate that most subjects reported using a number of different strategies to evaluate these criteria. While many of these strategies are logical and expected, careful examination of our results provides some indication of confusion amongst the subjects about how to perform these evaluations. These findings are discussed in more detail in the following paragraphs.

While most subjects (60%) responding to a TRUE/FALSE question concerning search engine accuracy indicated they believed search engine results are usually accurate, responses to more detailed questions concerning search engine results were more nuanced. The results of a series of questions asking survey respondents about credibility, quality and authority of search engine results indicated students examine a number of different criteria in evaluating sources. This could be an indication that a decade of information literacy instruction and expressions of concern by university faculty have had an impact on students' perceptions of search engine results. Still, the indication that most subjects considered search engine results usually accurate should be a concern. Given that students and instructors understand little of the unpublished algorithms which drive commercial search engines, it cannot be assumed that results returned by these applications meets any of the Association of College and Research Libraries information literacy competence evaluative criteria, including the criteria of accuracy. Though Google's search engine algorithm is proprietary and not publicly available, evidence available indicates that Google's search engine results are ordered largely by number of links and user selection counts, with no measure of accuracy or quality (Brin and Page, 1998). It is important that students understand this and evaluate the documents returned by Google accordingly.

Information literacy instruction emphasises identification of the author. This is crucial in understanding the authority or ability of the author to write about a particular topic. There is perhaps no other area where the distinction between traditional publishing and the Web creates difficulties for researchers using Web sources. The nature of the free-form publishing of the Web makes it very difficult to determine the true authorship of a page. The technology of the Web makes the development of collaborative documents extremely easy, leading to sites such as Wikipedia. While debate on the quality of the content of a collaborative encyclopaedic site such as Wikipedia is beyond the scope of this paper, the authorship of the content and the qualifications of the authors are clearly difficult to ascertain. Information literacy instruction and course instructors should address this issue.

While subjects provided various answers in response to the question about evaluating the objectiveness of a Web page, the indication that subjects use the URL or site domain to determine objectivity is problematic. Students should be more discerning and information literacy instruction should encourage more careful evaluation of objectivity of Web page sources.

Subjects in this study used a variety of techniques to evaluate Web content. In examining their evaluation in relation to Association of College and Research Libraries information literacy standards, subjects appeared to follow some standards but were confused about how to actually implement them. Specifically, identifying content authors, assigning authority for content and determining accuracy and objectivity appeared difficult for subjects. While subjects did indicate that they were attempting to perform these actions and did appear to have confidence in their ability to perform these actions, their selection of methods was inconsistent and problematic. These could be indications of problems with information literacy instruction standards or broader problems with assigning authorship and determining authority and accuracy of Web content in general.

The results of this study provide some indication of flaws in a Website evaluation checklist instructional model. Additionally, the broad proposals on evaluating information sources in the Association of College and Research Libraries information literacy standards may require context-sensitive strategies which are specific to various fields. Librarians alone cannot remedy this, since librarians cannot be experts in all fields. Faculty and librarian partnerships in course instruction may provide a solution. This can be accomplished using an embedded librarian within the course to accomplish more in helping students to identify authors, authority, accuracy, objectivity, point of view and other important criteria to determine which sources are really the most appropriate to use in the context of the course and assignment.

Future directions

The Association of College and Research Libraries has recognised that the Web and college students have changed since the release of the original information literacy competency standards in 2000 and they have been 'rethinking the standards' (Bell, 2013). Kuhlthau (2013) highlights how the current standards are not holistic and do not adequately prepare students to research reflectively in the 'dynamic global information environment' (p. 97).

The Association of College and Research Libraries Information Literacy Standards Task Force has proposed a Framework for Information Literacy. The framework explores information literacy criteria with threshold concepts (core understandings), a set of abilities or knowledge practices, and related metaliteracy objectives. These objectives expand evaluation criteria definitions and encourage evaluation of context referred to as the 'information need of the moment' (Gibson and Jacobson, 2014).

The results of this study demonstrate the need for course instructors to engage with information literacy objectives through their semester long courses. The new Association of College and Research Libraries information literacy framework provides a structure for doing this. Librarians need to partner with course instructors to design assignments that encourage students to think critically about the selection of information sources relating to course content.

Subject domains cannot be ignored given the nature of the information available on the Web, the lack of access mediation and the relative lack of peer or expert review of much of the content available. The evaluation of the results from these distinct domains may require a different set of standards and rules for each. Some subject-specific information literacy standards have been compiled by the American Library Association, but not all disciplines are currently represented (see Information Literacy Standards Committee, 2013 for the list of disciplines with subject specific standards).

About the Authors

Arthur Taylor is a Professor of Computer Information Systems at Rider University. Before joining Rider University, he worked for over 17 years in the field of information systems. His research interests include information literacy, relevance, and the information search process. He can be contacted at ataylor@rider.edu.
Heather Dalal is an Assistant Professor/Emerging Technologies Librarian at Rider University. Her research interests include instructional design and technologies, student research behaviour, and promotion and marketing of library tools and services. Heather serves on the Association of College & Research Libraries (ACRL) Distance Learning Section's Standards Committee, the New Jersey Library Association's (NJLA) Emerging Technologies Section, and the New Jersey Virtual Academic Library Environment's Shared Information Literacy Committee. Heather is the co-Chair of the ACRL-NJ/NJLA College and University Section User Education Committee. She can be contacted at hdalal@rider.edu.