vol. 17 no. 1, March 2012
vol. 17 no. 1, March 2012 |
||||
The generation born between 1982 and 2000 has been identified as the millennial generation (Strauss and Howe 1991; Howe and Strauss 2000). They have come of age in a digital world with ubiquitous information sources. The libraries with their strong mediated search support are no longer the primary sources of information for them. Consequently, existing information behaviour models may not adequately describe their approach to filling information needs. The goal of this research was to explore the information search behaviour of this generation in relation to commonly cited information search behaviour models.
The longitudinal study detailed in this paper evaluated the search behaviour of millennial generation students conducting information searches in a naturalistic environment. In the study participants provided detailed information on their search choices and judgments of document relevance over a five week period. The findings based on these empirical results provide some indication of how members of the millennial generation progress through a search process. The subjects in this study proceeded erratically through their search process and did not appear to validate sources or information gathered.
Information search process models provide a framework for the study and evaluation of information searches. Models such as Kuhlthau's (1993), Ellis's (1989) and Wilson's (1999) provide some guidance on the search process but have limitations in their application. These models were developed prior to the widespread use of the Internet as an information resource. Ellis's subjects were adult professionals, not millennial generation students. Kuhlthau's subjects were from a variety of age groups and may have included some young millennial generation subjects, but the research was done before the availability of the Internet.
Research has provided little insight into how information behaviour differs on the Internet, specifically with the millennial generation, and whether or not previously identified models are appropriate. millennial generation users are more likely to search for information on the Internet using commercial search engines (Abram 2006; Oblinger 2003). Research on how these searches are conducted could inform us on the current information behaviour of the millennial generation. Results could provide the basis for improved information search process models which better reflect current technology and the generation of information seekers raised with this technology.
In Generations Strauss and Howe (1991) present a broad historical panorama of generations and their influence on history. The authors identify the millennial generation as those born in or after 1982. In millennials Rising the authors identify the same start date for the generation and infer a cut-off date of 2000. The authors present theories concerning the influences of a generational cohort and the environment in which the cohort is raised and comes of age. The millennial generation is a group raised during a boom period. This prosperity created significant social and technological shifts leading to a distinct cultural identity.
The millennial generation is only familiar with a world where personal computers and information are easily accessible, and has distinct expectations concerning technology, communication, and access to information. Abram (2006) notes that millennial generation students expect instant access to information (cell phones, portable computers, Internet access), and tend to procrastinate with research and the choices it requires. They tend to be multi-taskers, often doing several tasks at once. They exist in a noisy, media-driven world, a condition that may lead to issues with filtering what is valid and important to their task. Constant socializing in a connected world leads to persistent distractions from any assigned task (Essinger 2006). The prevailing educational view is that students in undergraduate programs have incomplete cognitive thinking skills. This creates difficulty in discerning valid information from invalid information. In a mediated search environment, a librarian would help the student identify valid information sources and provide guidance on how to identify valid and useful information. This mediation is lacking in an environment where the student searches the Web unaided (Wieler 2004).
Gross and Latham (2011) conducted an investigation of experiences and perceptions of information with first year college students, members of the millennial generation. The researchers indicated that subjects viewed finding information as a product, not a process. Subjects in the study indicated that they did not perceive finding information as being difficult or requiring any particular skill set. One subject commented that 'the computer does all the work for you basically'. Subjects also felt the evaluation of information was subjective, not objective, suggesting a postmodernist view of information and knowledge in general. The focus on a search product and dismissal of a process for evaluating and verifying content by subjects is reflective of the millennial generation's neoliberal social view (Howe and Strauss 2000) where production and efficiency are important, and what appears to be a consumerist approach to education is a side effect of this view.
In Rowlands et al. (2008), the search behaviour of a subset of the millennial generation is examined in a broad, multi-method study. Researchers used a sample of those born after 1993, young members of the millennial generation. They noted that searchers of this generation tended to search horizontally rather than vertically, skimming content, viewing just one or two pages, and making quick relevance judgments based on this review. This thin, surface-level review of content is related to the lack of concern, or inability to discern, the authority of sources and the quality of content, as revealed in other studies of members of the millennial generation (Williams and Rowlands 2007; Hirsh 1999; Grimes and Boening 2001; Lorenzen 2001). The familiarity and comfort with the Web as an information search conduit creates other problems. The hypertext interface that is the foundation of the Web creates a fragmented view of information. The millennial generation is at ease in this environment, often not recognizing incomplete fragments as such. Each fragment of information appears to be as valid as any other. In Lorenzen (2001), subjects were clearly hesitant when asked how they evaluate the quality of a Website, some suggesting that a .com domain implied it was a reliable information source, and others indicating that a .com site was not a reliable source. Some subjects in the study felt that if a site was indexed by Yahoo! it had to be authoritative.
From a broader perspective, the culture within which the millennial generation has learned cognitive skills may be partly to blame for these problems. Philosophers have identified the postmodern condition as a rejection of the objective, scientific evaluation of knowledge, and the general acceptance of a an open, subjective, uncritical view of knowledge. There is growing concern that members of the millennial generation have adopted this uncritical view of information (Harley et al. 2001). Millennial generation information searchers appear to be more concerned with the time it takes to find and evaluate content than the overall validity or quality of that content, and generally prefer visual information to textual (Harley et al. 2001; Vondracek 2007; Wieler 2004). An additional cultural influence is the consumerist society within which the millennial generation has been raised. In Harley et al. (2001), the postmodern condition as applied to the dissemination of information is considered a disruptive force in the growth and distribution of knowledge in our culture. Consumerism, superficiality and knowledge fragmentation are a result. In the information economy, information is just another economic commodity which is consumed at the lowest cost. The cost in this context is effort, and when seeking information, the millennial generation individual often perceives the lowest cost as the most convenient, readily available information with limited consideration for quality (Buczynski 2005; Thompson 2003; Young and Von Segern 2001).
How millennial generation searchers work to fill their information need can be examined within the framework of an information search process model. Kuhlthau (1993, 1991) modelled information seeking as a series of stages. Her model was built on personal construct theory, Taylor's (1968) stages of need formation, Belkin et al.'s (1982) anomalous states of knowledge, and theories and models of expression and mood. Though developed in 1991, more current research continues to validate the basic tenets of the model (Kuhlthau et al. 2008). Kuhlthau's model is often interpreted as sequential, but the model does specifically allow backtracking and iteration. Ellis (1989, 1997) identified several search patterns used for gathering information to fulfil an information need. His model emphasized the subjective, iterative nature of the search process and avoids suggesting there is a consistent sequential process taking place. Wilson (1999) provided a synthesis and comparison of Ellis's and Kuhlthau's models which maps Ellis's behaviour into Kuhlthau's stages.
In order to better understand the information behaviour of the millennial generation, this research tracked millennial generation subjects as they progressed through a five week research project. The goal was to gather information to answer the broad question of how information seekers of the millennial generation progress through the process of finding information. Specifically, do they proceed through an orderly search process, or is their behaviour somewhat erratic and symptomatic of procrastination? Do millennial generation information seekers make some effort to discern the quality of documents reviewed on the Web, and if so, are they discerning throughout the search process? In general, is the search process used similar to previously identified search process models, or are there indications that their search behaviour has changed? These broad questions lead to the following specific research questions.
1. Do millennial generation information seekers progress through a search process similar to that of Wilson's (1999) consolidated model, or is their search behaviour different?
2. Do millennial generation information seekers evaluate the quality of Web resources? Are they discerning about quality-related attributes of the sources retrieved from the Web?
3. How do millennial generation information seekers make use of general information Websites such as Wikipedia? At what stage(s) in the search process are these pages used?
The study involved the collection of data based on user searches. Users were millennial generation students in an undergraduate business course. Subjects were given an assignment to create a presentation on a computer technology. The assignment was structured as a research report and subjects had to complete a bibliography of their sources. Most subjects had little knowledge of the research topic they were assigned. Data were collected on the results of each subject's information search. All searches were conducted on the Internet. Specifically information was collected on the document selected, the relevance judgment for the document (was the document relevant or not), the criteria used to judge document relevance, and the subject's stage in an information search process.
Data were collected using a Website with a modified search engine which executed searches using the Yahoo! search engine and then reformatted the results page to provide Web-based data collection for this study. Subjects were able to access this search engine interface over the Internet, thus allowing them to work in a naturalistic environment at their own pace. Subjects were not required to use the Website for searches, but they were required to use the Website to provide data on the results of their searches. Those who did not use the default search engine entered information on their sources using survey instruments identical to those used to enter the information from the default search engine. Approximately four source documents were entered by each individual using this mechanism. Almost all subjects chose to use the default search engine and almost all entries were entered using the survey instruments on those pages. It is worth noting that good sources of information could be found on the Web for the computer-related technical topics assigned to the subjects.
Approximately 80 subjects were drawn from a convenience sample of junior and senior undergraduate students at an American university. Data were collected in 2007 and subjects were on average 19 through 22 years of age, and were therefore born between 1985 and 1988, members of the millennial generation. Subjects signed an informed consent form which explained the purpose of the research and that the information they provided would be treated anonymously. Subjects were allowed to choose a research topic from a list of topics (see Appendix A). Topics were all of the same approximate level of difficulty and were related to course content. Subjects were given several weeks to complete their research assignment and were required to complete specific interim assignments during that time period. A project abstract was due the first week, a detailed outline was due next, the rough draft of the presentation slides due the next week, and the final presentation slides were due the final week.
Project data were collected anonymously using survey instruments integrated into the Web search engine interface (Figure 1).
Subjects examined the documents returned from the search engine and then used the Website to indicate the relevance of the documents they examined (relevant, not relevant, partially relevant/not sure about relevance), their current stage in an information search process selected from a predetermined list of search stages, and the criteria used to make that relevance judgment selected from a predetermined list of relevance criteria (Figures 2 and 3).
The Website used for data collection contained detailed instructions on the use of the site and how to record the information required. Subjects were given instructions on the use of the Website and the meaning of the terms used on the site and help pages were provided to explain the search stage choices, relevance judgment choices, and criteria for relevance judgment choices used by the subjects. The Website captured data for each subject using an anonymous user ID for search stage choices, the document selected as a uniform resource locator (URL) and the relevance judgment.
Data collected on the search stage provided an indication of how subjects were progressing through an information search process and enabled the evaluation of research question one. The relevance criteria related to quality of the document (accuracy, source quality, etc.) provided information for evaluation of research question two, and also provided some indication of how and when subjects were evaluating documents, data relevant to question one. Data collected on Websites visited (URLs) provided information for research question three. When these Websites were being visited and the criteria used during each search stage also provided information about how subjects were progressing through the search process, data relevant to research question one.
The search stage model used for this research (see Table 1) was developed from the phases suggested by Wilson (1999) which synthesized the information behaviour of Ellis (1997) with the search process of Kuhlthau (1993). The list presented to the subjects is shown in Description Displayed to Subject column in Table 1. The term initiation was selected because it was considered clearer than beginning or starting. The term browsing was avoided because this term is now commonly used for all Web-related searching and would potentially be confusing to subjects. Instead, the term exploration was used to describe the process of scanning and gathering information. The remaining terms, differentiating, extracting and verifying are from Ellis (1997) and were used in lieu of Kuhlthau's terminology because they were considered more precise. The process of ending the search was considered to be implied by the submission of the final deliverable and would not have generated data relevant to the research questions; it therefore was not presented as a choice to the subjects.
Table 1 contains the criteria identified by subjects as having some bearing on their relevance decision (relevant, not relevant, don't know or partially relevant), the decision whether or not the document (or Website) is relevant to their information need. The relevance criteria chosen by subjects were those factors that contributed to their relevance decision. Prior research by Barry (1994), Barry and Schamber (1998), and Cool et al. (1993) identified the criteria listed in Table 1. A number of studies have identified these criteria and have provided some confirmation as to their consistency across information retrieval tasks (Barry and Schamber 1998; Park 1993; Schamber 1991; Schamber and Bateman 1996; Xu and Chen 2006). A subset of these criteria was presented to subjects not as specific criteria but using the contents of the Description displayed to the subject column in Table 3 (see Appendix C). Subjects were allowed to choose one or more criteria which they felt contributed to their relevance decision.
The criteria selected for this research are a subset of those identified in Table 1. The reasons for these selections are as follows. The Source column in Table 1 identifies the source of the relevance criterion: Barry (1994), or Barry and Schamber (1998) , or Cool et al. (1993). To reduce the potential for confusion and survey exhaustion on the part of the subject, the number of relevance criteria presented was limited to fifteen. In addition to some criteria from the Cool study, criteria identified in the Barry (1994), and Barry and Schamber (1998) studies were used. Excluded were those criteria that were specific only to Schamber's (1991) earlier study and related to document qualities specific to her topic (weather reports) and did not apply to the topics used in this study.
| Criteria | Type | Source* | Used | Description displayed for subject |
|---|---|---|---|---|
| Depth, scope or specificity | Document | Barry, Cool | Yes | Document contains good depth on the topic |
| Accuracy or validity | Document | Barry, Cool | Yes | Document appears to be accurate |
| Currency (recency) | Document | Barry, Cool | Yes | Information is current, recent, up-to-date |
| Tangibility | Document | Barry, Cool | Yes | Information relates to real, tangible issues; not esoteric or theoretical |
| Quality of sources (source quality) | Document | Barry, Cool | Yes | Source is reputable, trusted, considered expert |
| Availability of information | Situation | Barry, Cool | Yes | The extent to which the information is available |
| Verification | Document | Barry, Cool | Yes | The information is consistent with the body of knowledge the field; the information supports the user's point of view |
| Affectiveness | Document | Barry, Cool | Yes | The user's emotional response to the information; pleasure, enjoyment, entertainment |
| Effectiveness of proposed approach | Document | Barry | Yes | How effective is the approach proposed? |
| Consensus within the field | Document | Barry | Yes | How much consensus there is in the field for what is proposed in the document |
| Time constraints | Situation | Barry | Yes | How much time is allowed for the task to be completed? |
| Background, experience or ability to understand | Situation | Barry | Yes | Expression of concern over the ability to understand a document (same as 'understandability') |
| Novelty, content novelty or source novelty | Document | Barry | Yes | The source or content of the document is new to the subject. |
| Geographic proximity | Document | Schamber | No | Refers to weather information in a geographic location |
| Dynamism | Document | Schamber | No | Refers to the ability to dynamically manipulate the information in a document |
| Presentation quality | Document | Schamber | No | Iindication that the source of the information could be manipulated in some way. |
| Structure | Document | Cool | Yes | The structure of the document; how the information is presented and organized |
| Amount of information | Document | Cool | Yes | Document provides sufficient information. |
| Depth | Document | Cool | Yes | Document covers the topic in good depth. |
| Timeliness (age of document) | Document | Cool | Yes | Is the time frame of the document appropriate; (current where recent information is required; written in a certain time period for historical significance) |
| Understandability | Document | Cool | Yes | The document is understandable by the subject (ability to understand) |
| Bias | Document | Cool | Yes | The Document is written with a particular viewpoint. |
| Authority | Document | Cool | Yes | The author or publication has a good reputation in this field. |
Deliverables required of the subject and the month and day they were due are identified in Table 2. Some feedback was provided upon submission of the abstract and detailed outline deliverables to encourage the students to produce a thorough and well-researched presentation. Feedback concerned only the technical aspects of the topic and the scope of their final presentation. Feedback did not concern choice of Websites, search stage choices or relevance criteria choices made by the subjects.
The final deliverable was a ten to thirty slide PowerPoint presentation. Subjects were encouraged to go beyond general information sites (such as Wikipedia), and to use quality sources such as trade journals whenever possible. (Computer trade journal content is widely available on the Web.) Subjects were also encouraged to use between ten and twenty sources to prepare their presentation.
| Deliverable | Description | Due date |
|---|---|---|
| Abstract | Several paragraphs explaining what will be covered in the presentation. | 2 November |
| Detailed outline | One or more pages of outline text which explain the topic, and subtopics which to be presented and the flow of the presentation | 10 November |
| Rough draft of presentation slides | PowerPoint slides containing the rough draft of the presentation | 20 November |
| Final presentation slides | PowerPoint slides containing the final presentation | 7 December |
A total of 80 subjects participated in the study and examined and provided 758 distinct Web page evaluations. Table 5 identifies the number of documents evaluated by subjects for each project deliverable. This table shows that the number of documents evaluated varied significantly for each deliverable (χ2 = 98.9, df = 3, p-value <0.001). Subjects evaluated a total of 81 documents for the project abstract deliverable, but evaluated 225 documents for the detailed project outline. This finding is not unexpected considering the subjects' instructions for the preparation of the report abstract were to create a brief explanation of the project and thus did not require a detailed knowledge of the topic. Table 3 also shows that document selection was evenly distributed over the duration of the research project.
| Project deliverable | % of total | |
|---|---|---|
| Abstract | 81 | 10.69% |
| Detailed outline | 225 | 29.68% |
| Rough draft | 187 | 24.67% |
| Final presentation | 265 | 34.96% |
| Total | 758 | 100.00 |
Table 4 lists the number of search stages reported by subjects. As this table indicates, only five subjects reported being in all the search stages provided as options, and a significant number of subjects (67%) reported being in fewer than four (χ2 = 9.6, df = 1, p-value <0.001).
| Stages reported | Users reporting |
|---|---|
| 5 | 5 |
| 4 | 22 |
| 3 | 20 |
| 2 | 20 |
| 1 | 13 |
| Total | 80 |
Table 5 and 6 identify the percentage of documents evaluated in each search stage. Table 5 lists percentages within search stage, and Table 8 lists percentages within project deliverable. This provides some indication of how subjects proceed through the information search process. As subjects evaluated documents, they indicated which search stage they were in based on a list of search stage explanations (Table 2). Expectations are that research done for preparation of the first deliverable, the project abstract, would be the initiation stage (identified as the 'initial search; start of the search process'). As Table 7 indicates, slightly less than half of the subjects indicated that they were in the initiation stage while preparing the project abstract. This finding indicates that over half the subjects reporting for the project abstract, the first deliverable, reported being in a search stage other than the initiation stage.
Table 6 provides another perspective on the initiation stage and the relationship between search stage and deliverable. As a percentage of documents evaluated within a search stage, during the preparation of the detailed outline, approximately 50% of the subjects reported being in the initiation stage, and they reported this after completing the project abstract, the first deliverable. Additionally, 13% reported being in the initiation stage during the preparation of the rough draft, and 11% reported being in the initiation stage during the preparation of the final presentation. This indicates a non-serial progression through the information search process in relation to project deliverables.
As Table 5 indicates, only 16% of the subjects reported being in the verifying stage while preparing the final presentation deliverable. Subjects were more likely to identify extracting as their current search stage (defined as ‘extracting information to answer the question’) during this time frame. This indicates that these subjects were not concluding the search process in a manner consistent with Kuhlthau's information search process which has an expectation that the subject will perform some verification or evaluation of the research collected in the final stage.
| Search stage | Abstract | Detailed outline | Rough draft | Final presentation |
|---|---|---|---|---|
| Initiation | 47.08% | 25.32% | 16.25% | 8.89% |
| Exploration | 22.81% | 25.67% | 9.96% | 12.72% |
| Differentiating | 7.48% | 10.57% | 2.85% | 9.22% |
| Extracting | 20.62% | 35.69% | 65.84% | 53.18% |
| Verifying | 2.01% | 2.75% | 5.10% | 15.99% |
| Total | 100.00% | 100.00% | 100.00% | 100.00% |
| Search stage | Abstract | Detailed outline | Rough draft | Final presentation | Total |
|---|---|---|---|---|---|
| Initiation | 25.75% | 49.70% | 13.67% | 10.88% | 100.00% |
| Exploration | 14.37% | 58.05% | 9.66% | 17.93% | 100.00% |
| Differentiating | 10.62% | 53.89% | 6.22% | 29.27% | 100.00% |
| Extracting | 5.59% | 34.72% | 27.45% | 32.25% | 100.00% |
| Verifying | 3.62% | 17.76% | 14.14% | 64.47% | 100.00% |
The selection of general information sites (defined in this study as the Wikipedia site) varied significantly in relation to project deliverable due (χ2 = 12, n=80, df = 3, P <0.01) as shown in Table 7. This site selection was approximately 10% of the total documents chosen overall (80 out of 758). Subjects were instructed to search beyond basic informational sites, and there were numerous sites available for their assigned topics. As Table 7 indicates, these sites were used consistently during the preparation of the abstract and rough draft, ranging between approximately 6% and 10% of the total sites chosen. There was a significant increase in the usage of these sites during the preparation of the final presentation. Table 8 provides additional insights into the use of these sites, showing a statistically significant relationship between the search stage and the selection of Wikipedia Web sites (χ2 = 15.7, df = 3, p-value < 0.001). These results indicate that Wikipedia sites represent approximately 40 % of the of the sites evaluated when subjects reported being in the 'extracting' stage (identified as "extracting information to answer the question").
| Deliverable due | Count | Percentages* |
|---|---|---|
| Abstract | 8 | 9.30% |
| Detailed outline | 29 | 9.90% |
| Rough draft | 15 | 5.84% |
| Final presentation | 28 | 14.97% |
| Total | 80 |
| Stage code | Count | Percentages |
|---|---|---|
| Initiation | 20 | 25.97% |
| Exploration | 17 | 22.08% |
| Differentiating | 6 | 7.79% |
| Extracting | 31 | 40.26% |
| Verifying | 3 | 3.90% |
| Total* | 77 | 100.00% |
An examination of the criteria used by subjects to determine relevance provides some indication of their reasons for selecting or rejecting documents as they progressed through an information search process. Since there were a large number of criteria examined by the subjects, Table 11 is simplified to identify those criteria which changed most in terms of their selection as the subject moved through the search process. As this table indicates, subjects were more likely to select amount of information and depth as criteria in the earlier search stages (initiation and differentiating) than in later stages (initiation and verifying). The criteria of novelty of sources, the structure of the document, and time constraints of the subject was selected more in later stages. This may indicate that in later search stages the subjects are looking for new sources of information and possibly documents of different structure, providing a different approach and new information for their subject area. The criterion of time constraints was also more likely to be selected later in the process as deadlines loom. The criteria of and depth were selected less later in the search process, when other criteria such as authority and structure were more likely to be selected. Though authority was selected more in later stages, the criterion of source quality was selected less, suggesting that based on the subjects' selections, subjects may not apply equal weight to authority and source quality as criteria.
| Relevance criteria | Initiation | Differentiating | Extracting | Verifying |
|---|---|---|---|---|
| Affectiveness | 0.57% | 4.37% | 2.77% | 3.52% |
| Amount of information | 8.92% | 12.23% | 10.77% | 7.42% |
| Authority | 1.90% | 3.49% | 2.65% | 3.91% |
| Bias | 1.90% | 1.75% | 1.94% | 3.13% |
| Depth | 8.92% | 12.23% | 9.03% | 7.42% |
| Novelty | 1.14% | 3.93% | 2.06% | 2.73% |
| Recency | 8.35% | 7.86% | 6.97% | 6.25% |
| Source quality | 4.93% | 3.49% | 4.06% | 3.13% |
| Structure | 6.26% | 5.68% | 7.55% | 8.20% |
| Time constraints | 0.57% | 1.31% | 1.29% | 2.34% |
An examination of results grouped by deliverable required provides a slightly different perspective on the progress of the subjects, as shown in Table 10. This table examines criteria selected based on the date the subject made the evaluation. The date was then correlated with the deliverable due in that time frame. Based on this evaluation, subjects working on the final presentation, subjects who were approaching the end of their research effort, were more likely to select amount of information and understandability as criteria in their document choice. The accuracy, source quality, depth and breadth, however, were less likely to be selected by these subjects.
| Criteria code | Rough draft | Final Presentation |
|---|---|---|
| Accuracy | 14.66% | 11.71% |
| Affectiveness | 3.02% | 5.43% |
| Amount of information | 12.07% | 14.00% |
| Authority | 3.23% | 4.29% |
| Bias | 3.23% | 2.14% |
| Breadth | 7.76% | 6.71% |
| Depth | 13.36% | 10.29% |
| Novelty | 1.94% | 3.14% |
| Recency | 8.84% | 9.14% |
| Source quality | 5.17% | 4.00% |
| Structure | 8.41% | 9.14% |
| Time constraints | 2.80% | 2.43% |
| Understandability | 15.52% | 17.57% |
In evaluating results in reference to research question one, these empirical results suggest that millennial generation subjects do not proceed in an orderly fashion through an information search process. Though models such as Kuhlthau's (1993) did not predict a strictly linear progression through the information search process, it was implied that subjects would move through each of the various steps in the process. Subjects in this sample gathered documents consistently over the course of the assignment (see Table 3). Subjects should have acquired good knowledge of the topic by the time they completed their rough draft, but with this sample approximately 35% of the documents used were retrieved after the rough draft was completed, suggesting procrastination or backfilling (adding sources late in the research process as they finalize their report) on the part of subjects. A significant portion of subjects (67 %) reported being in fewer than four of the search stages studied (Table 4) also providing some indication that subjects may not be moving through a search process in an manner consistent with the search stages used in this study.
If subjects had made an orderly, pragmatic progression through an information search process, it would be expected that they report they were ‘verifying information gathered previously during the preparation of the final project deliverable. This search progression would be based on the expectation that subjects had gathered research previously, and were now examining and verifying that research as part of completing the final deliverable. The empirical evidence gathered with this subject pool suggests otherwise. The majority of subjects (53%) evaluating documents for the final presentation identified themselves as being in the extracting stage, described as ‘extracting information to answer the question’. Only 16% of subjects preparing the final deliverable identified themselves as being in the verifying stage. A closer examination of the verifying stage indicates that the selection of this stage as a percentage of the stages reported for a particular deliverable never exceeded 15% of the total (see Table 5), an indication that subjects were not ‘verifying information that has been gathered previously’. Most of the documents evaluated by the subjects (approximately 70%) were evaluated in the final two stages, the latter portion of the project's duration. This amount demonstrates a statistically significant correlation with search stage (χ2 = 28.1214, df = 1, p-value <0.001) suggesting that subjects may be procrastinating or backfilling.
Over half the subjects reporting for the first deliverable reported being in a stage other than the initiation stage. This finding could be an indication that in this time frame, subjects found several documents and reported being in the initiation stage, and then after absorbing those initial set of documents, considered themselves out of the initiation stage and in some other stage. Subjects may have performed some significant portion of their research in this early stage, quickly moving beyond the initiation stage.
In reference to research question two, examination of the results concerning subjects' evaluation of the quality of Web resources provides some indication that subjects were not concerned with the quality, validity, or authority of the documents selected. Subjects selected the categories of source quality and authority (of sources) at a rate of between 55% or less than that of amount of information and depth (see Table 9) suggesting that these criteria were not as important to subjects as simply retrieving some quantity of information on their topic. Categories such as structure, amount of information, recency (currency) and depth were consistently selected more often by subjects, suggesting these criteria were more important to the subjects than source quality or the .
Subjects did not select verifying (presented as 'verifying information that has been gathered previously') for any particular deliverable at a rate higher than 16% (Table 5), suggesting that subjects generally found some other search stage category provided a better description of their behaviour. Based on these results, it appears subjects did not attempt to verify their sources, and it can be assumed they considered the information valid or were not concerned about the validity of the source.
In reference to research question three, subjects did appear to make use of general information sites such as Wikipedia, but they did not select these sites consistently throughout the search process. The instructions presented to subjects on the use of general information sites is obviously an intervening factor to be considered in the evaluation of this statistic. However, the timing of the selection and use of these sites by the subjects suggests that they were likely to select these sites in the extracting stage. Also, the selection of Wikipedia for approximately 15% of sites for the final deliverable suggests that for some subjects, these sites were useful in the later stages of the research project. The difference between the selection of these sites in early stages versus late stages, however, is not statistically significant. For this sample, subjects appeared just as likely to use these sites in early search stages (initiation, exploration) as in later search stages (differentiating, extracting, verifying).
The subjects in this study were from a university in the USA. The articles referenced in the literature review also cited studies performed on subjects in schools in the USA. It is not clear that search behaviour, or more broadly, the behaviour of millennial generation subjects, will be the same in other cultures. Subjects were also undergraduate students with a mixed scholastic background (below average, average, above average). Graduate students or professionals may perform differently.
The task assigned students provided specific instructions to encourage students to perform research. Students were told that they were being graded partially on the quality of their research, and this meant that more quality sources cited would result in a higher grade. Specific instructions given to the subjects also addressed limiting the use of general information sites such as Wikipedia. These intervening factors may have encouraged students to evaluate more documents from quality sites and perform additional research during the later stages of the project. In the absence of these instructions, it is possible Wikipedia sites may have represented a larger portion of the sites selected and referenced.
This study was designed to allow the subjects to work in an environment outside of a lab at their own pace and at their own convenience. The intention was to create an environment where the subject would feel comfortable and behave in a typical manner and provide meaningful results on the common information behaviour of the millennial generation. This meant that the subjects themselves reported their search stage and criteria and relevance as opposed to the method in many information behaviour studies where researchers provide that evaluation. Though subjects were tutored in the use of the system and provided with explanations for what was being reported, and though help prompts and help pages were available to subjects, there is the possibility that some subjects may have misinterpreted these instructions and that these prompts and requirements on reporting may have led to some non-typical behaviour on the part of subjects.
The task assigned to the students involved a subject area that was suitable for Web-based research. This assignment may have encouraged fragmented searches and retrieval since information for many of the assigned topics could be retrieved from multiple sources available on the Web. Other assignments, for example writing a paper on Western philosophy for a humanities course, might involve more focused searches with fewer sources.
The research methods employed allowed subjects to select topicality as a criterion, but did not specifically analyse topicality in relation to the selection of other criteria. This approach was based on the assumption that the document first needed to be on topic before other criteria would be considered (Wang and Sorgel 1998; Crystal and Greenberg 2006). This research focused on the use of more specific relevance criteria.
Additional influences on relevance decisions are known to be user's background or knowledge of the subject domain, and search task. The convenience sample for this research was drawn from a pool of undergraduate students who are business majors in a business school at an American university. All students were taking the same course and were given the same assignment, thus all had the same work task. These influences are mitigated in this study by drawing from a subject pool whose members have similar backgrounds, experiences, and domain knowledge. Though this aspect of the design of these studies attempted to control for variations in domain knowledge, variations in knowledge may exist within the subject pool. The choice of this convenience sample also limits the generalisability of the results.
Though subjects could use any search engine to gather sources, the Yahoo! search engine was used by default since it was easier to integrate search results into the Web-based data collection instruments used in this study. Almost all subjects chose to use the default search engine. The search engine itself and the search results were not being examined in this study, but use of a different search engine as the default (for example, Google) may have produced different results.
These findings suggest that the execution of searches by millennial generation searchers may be problematic. Existing search models are appropriate; it is the execution of the model by the searcher within the context of the search environment that is at issue. millennial generation searchers have come of age with different information gathering resources than their predecessors. The filter of the librarian-mediator relationship is gone and replaced by a profusion of fragmented and sometimes dubious information sources. It is within this environment that they search and gather the information they need. The results presented here provide some indication that members of the millennial generation do not consider verification of Internet sources important, indicating a non-critical view of information found on the Internet. This is consistent with results by Gross and Latham (2011) who report that subjects in their study indicated that information quality was not a concern and express surprise and incredulity that there may be an objective measure of information evaluation.
These findings point towards an information literacy problem that may be based more on a perception rather than lack of information searching skills. The millennial generation has come of age with technology that provides access to a vast variety of information sources. They constantly make choices concerning focus. They clearly know how to filter, the problem is what they choose to filter when evaluating information.
As Maybee (2006) indicated, learning involves a relationship between the learner and the subject matter. Changing the relationship involves changing the learner's perceptions. Subjects identified in Gross and Latham (2011) were not concerned with the quality of information gathered, consistent with the behaviour of the subjects in the study reported here who rarely reported validating information sources (identified as source quality). Gross and Latham further indicate that subjects did not perceive the existence of an objective standard for evaluating information (see Gross and Latham 2011: 173). Finding and evaluating information is subjective, reflecting a distinctly postmodernist view of information and knowledge.
It is this core perception of information quality as a subjective concept, and perhaps even a broader uncritical view of information in general, that must be addressed. Further research should explicate this view and perhaps the broader worldview of subjectivity versus objectivity amongst the millennial generation. A better understanding of why members of this generation resist using objective information evaluation standards may provide strategies for addressing this perception and thus improving the search skills of the members of the millennial generation.
Arthur Taylor is a Professor of Computer Information Systems at Rider University. Before joining Rider University, he worked for over 17 years in the field of information systems. His research interests include relevance, the information search process, and computer supported education. He can be contacted at ataylor@rider.edu
| Find other papers on this subject | ||
|
|
© the author, 2012.
Last updated: 9 February, 2012 |
|
|
|
