Mixed methods data collection using simulated Google results: reflections on the methods of a point-of-selection behaviour study

Introduction

The Researching Students’ Information Choices: Determining Identity and Judging Credibility in Digital Spaces (RSIC) project designed an innovative task-based simulation study to investigate students’ point-of-selection behaviour for school-related projects. The research team defined point-of-selection behaviour as the point at which an information seeker determines whether a resource meets their information need (Buhler et al., 2019). Such a determination is far from simple. The Association of College & Research Libraries (ACRL) framework states that novices must take responsibility to ‘critically examine all evidence—be it a short blog post or a peer-reviewed conference proceeding—and to ask relevant questions about origins, context, and suitability for the current information need’ (ACRL, 2015).

Assessing sources for quality and relevance is particularly problematic for students who, by necessity, must navigate a sea of information as a critical part of their education. Head (2013) described this phenomenon in a study of freshmen undergraduates as an, information tsunami that engulfed them’ (p.28). RSIC project members have seen first-hand how science, technology, engineering and mathematics students struggle with this digital deluge and wanted to develop evidence-based approaches to address it.

To this end, the RSIC project investigated when and how such students decide to select resources and what this means for educators and librarians developing information literacy programmes. In particular, the project examined whether students truly are format agnostic, a term that has been applied to students born in the digital age who it is assumed either cannot or do not consider format when making judgments relating to the use of digital resources (Abram and Luther, 2004; Rowlands et al., 2008). The project also focused on how students determine the credibility, or the trustworthiness and believability (Flanagin et al., 2018), of resources.

This study is significant because the innovative method successfully captured data on several aspects of point-of-selection behaviour over a wide range of educational stages. Given project objectives, the research team chose a mixed-methods data collection approach in conjunction with the task-based simulation study to answer the following research questions:

• How do science and technology students, ranging from 4th grade to graduate school, determine the credibility of online information resources?
  • What cues from a web search results screen do students use to judge the credibility of online information resources?
  • How do students’ characteristics influence their credibility judgments of online information resources?
• Do such students from 4th grade to graduate school differentiate among different types of online information resources during point-of-selection?
  • What cues from web search results screens do the students use to identify online information resources?
  • How do students’ characteristics and experiences influence their identification behaviour?

One hundred seventy-five students from 4th grade through graduate school were selected to participate in the study. The simulation provided a realistic and controlled setting that offered the research team insight into the participants’ actual behaviour as well as a means to compare findings within and across cohorts. In order to identify how and why students choose online resources, the research team designed a series of tasks within the simulation and gathered quantitative and qualitative data about students’ behaviour, thoughts and characteristics. Despite the challenges associated with the mixed methods used in this study, the data provide a more complete picture of resource evaluation and selection behaviour across students of a wide range of educational stages and demographic characteristics.

Literature review

Research methods include data collection and data analysis techniques, which have been used in library and information science research over the years (e.g., Chu, 2015; Greifeneder, 2014; Matusiak, 2017). The data collection techniques used to study information behaviour impact what data analyses can be performed when answering research questions. Several studies have shown that questionnaires and interviews are the predominant data collection techniques used in information behaviour studies (Greifeneder, 2014; Julien and Duggan, 2000; Julien, et al., 2011; Matusiak, 2017; McKechnie, et al., 2002). Focus group interviews, observations, diaries and think-aloud protocols also have been used, but to a lesser degree (Greifeneder, 2014; Matusiak 2017; McKechnie, et al., 2002). These techniques, like all others, have benefits and limitations and should be chosen depending on research objectives and available resources.

Questionnaires

Questionnaires are a relatively inexpensive way to collect structured quantitative data. By collecting a high volume of responses from a representative sample of participants, researchers can use statistical analysis to draw conclusions about the larger population (Connaway and Radford, 2017; Tracy, 2013). Administering questionnaires in absence of the researcher encourages candid responses as participants can be kept anonymous and the introduction of researcher bias is reduced (Connaway and Radford, 2017). However, the researcher also must take care when wording questions to limit bias and avoid increasing the likelihood of obtaining desired results. Care also must be taken to write clear questions that limit misinterpretations that lead to inaccurate responses. Both of these sources of error can be attributed to weaknesses in questionnaire design, such as question sequence, inadequate instructions and failure to accurately explain the scope and purpose of the study, which is why pre-testing is important (Connaway and Radford, 2017).

Individual interviews

Individual interviews are often used to gather richly detailed descriptions of participants’ behaviour. Similar to questionnaires, the researcher must ensure questions are understandable, neutral and not leading. Unlike questionnaires, the researcher’s presence as the interviewer enables a degree of flexibility and tailoring (Case and Given, 2016). There also are drawbacks to this approach. The researcher must be careful not to introduce bias by providing neutral reactions to the interviewee’s responses (Connaway and Radford, 2017). In addition, interviews do not capture behaviour in real time but instead rely on participants to recall what they did (van Someren, et al., 1994). Unfortunately, participants may not remember or be cognizant of what they did or they may misrepresent what they did with or without realizing it (van Someren, et al., 1994). To address these issues researchers often employ the critical incident technique (Flanagan, 1954). The critical incident technique asks participants to reflect on a specific or memorable event, rather than a broad topic or experience, to improve recall and elicit specific details about an area of interest.

Focus group interviews

Like individual interviews, focus group interviews are useful in providing detail on what people think and why they think that way, but do so at the level of the group rather than the individual (Connaway, 1996). Focus group interviews tend to be more efficient in terms of the researcher’s time and effort. They enable the researcher to observe and document verbal as well as non-verbal communication (Case and Given, 2016; Connaway and Radford, 2017). In addition, they can be used to reveal interaction patterns, behaviour and consensus within the group (Connaway, 1996). Yet they are employed less frequently than individual interviews (Greifeneder, 2014; Matusiak, 2017; McKechnie, et al., 2002). This could be based on the cost per interview and the skills required to moderate focus group interviews (Connaway, 1996). The moderator, in particular, factors heavily in the success of the focus group interview, not only in directing the discussion to answer the research questions, but also ensuring even participation among group members and open communication of differences of opinion (Connaway, 1996).

Observations

Observation, another data collection technique, is not widely used in information behaviour research, despite its benefits (Cooper, et al., 2004; Wildemuth, 2017). Like interviews, it can yield rich detail about the phenomenon under study, but the detail is based on watching participant behaviour as it unfolds rather than asking participants about it (Connaway and Radford, 2017). The researcher can participate and observe or only observe. In either case, care must be taken to limit bias during observation, such as the researcher observing behaviour that supports beliefs, but ignoring others or missing behaviour that seems typical or unimportant (Case and Given, 2016; Connaway and Radford, 2017). More generally, researchers have to consider whether and how their presence may be influencing the setting and whether and how the setting may be influencing their ability to systematically collect data (Wildemuth, 2017).

Diaries

Diaries have been used for more than two decades to collect data for studying information behaviour, but recent studies show they are a less used data collection technique (Greifeneder, 2014; Matusiak 2017; McKechnie, et al., 2002). Designed to rely on participants’ self-reports, they are an attempt to overcome some of the drawbacks of interviews and questionnaires (Case and Given, 2016). The researcher tends to request that participants complete diaries at specific times, such as after the occurrence of a certain event or on a time-based schedule (Case and Given, 2016). This enables researchers to collect data about what people do and think without the researchers’ constant presence. However, as is common with self-reported data, investigators only receive information that the participants are willing or have time to share, which increases the chances of collecting incomplete data (Connaway and Radford, 2017). Diaries can be structured, with specific instructions and content that put less of a burden on the participants. However, less structured diaries mitigate any biases or influences from the researcher (Wildemuth, 2017).

Think-aloud protocols

Think-aloud protocols are described as ‘concurrent verbal reports…where cognitive processes, described as successive states of heeded information, are verbalised directly’ (Ericsson and Simon, 1984, p. 16). In other words, participants verbalise their thoughts as they engage in an activity to reveal how they process information during the activity. The researcher asks participants to think aloud while solving a problem and repeats this request as a means to encourage an uninterrupted, continuous account (van Someren et al., 1994). This technique allows researchers to observe the steps of the cognitive process and makes it possible to ascertain where problems are encountered (Wildemuth, 2017). Like interviews and observations, this is time and resource intensive for the researcher. However, the technique allows participants to convey their thoughts in their own language without being led or influenced, thereby limiting bias. Moreover, researchers are not relying on participant recall, since the data are being collected as the action is taking place (van Someren, et al., 1994).

Mixed-methods approaches

Studies using mixed-methods approaches use multiple data collection techniques, either different qualitative or quantitative approaches or a combination of both. As far back as 1979, there have been calls for more mixed-method studies, arguing that they be seen as ‘complementary rather than as rival camps’ (Jick, 1979, p. 602). There also have been more recent discussions of mixed methods research (e.g., Riazi, 2016; Small, 2011; Timans et al., 2019). By using mixed methods, especially qualitative and quantitative together, researchers can design studies that offset the biases and weaknesses of each method (Connaway and Radford, 2017; Cooper, 2014; Jick, 1979). Yet, library and information science researchers tend to limit the number and type of data collection techniques they use in their studies. Julien et al., (2011) found less than one-third of published research studies used a mixed-methods approach. Although a more recent study showed an increase in the use of mixed methods to 45%, the majority of those only used two data collection techniques (Greifeneder, 2014). Matusiak (2017) also found that published studies using multiple data collection techniques most frequently use two.

Part of the challenge is that mixed-method studies take more time, money and effort. Moreover, if the data are collected at different points in time, participant attrition is likely. Mixed-method studies also can be difficult to replicate (Jick, 1979). Given these disadvantages, the question becomes the degree to these drawbacks are outweighed by the researcher’s ability to increase understanding, validate findings, or gain insight into how findings relate or conflict (Brannen, 2005; Connaway and Radford, 2017; Jick, 1979).

Seeing a trend toward qualitative approaches, Vakkari (2008) argued that more explanatory studies are needed in order to situate information behaviour within its origins and context. Others also have found that context and situation are critical to understanding information behaviour (Connaway, et al., 2013). A thoughtful approach to combining several data collection techniques can incorporate these aspects of context and situation that help researchers not only describe what is happening, but how and why it is happening. The key is designing a research study that integrates different components of the study in a way that enables the research questions to be addressed given the resources provided.

Research method

Investigating how students identify and judge the credibility of online resources at the point of selection required controlling for certain aspects of participant recruitment and the environment used to collect data about their information behaviour. To capture participants’ real-time point-of-selection behaviour while maintaining that control, the research team designed a simulation. Simulations are recreations of situations experienced in a natural environment (Gist, et al., 1998). However, they also allow researchers to control certain aspects of the setting given their research objectives (Case and Given, 2016; Gist, et al., 1998).

The simulation provided a realistic and controlled setting for participants and captured quantitative data of participants’ behaviour. It was important for the research team to not only see, but also hear first-hand accounts of how students engage with online information resources. The research team used think-aloud protocols to collect qualitative data on participants’ thoughts as they engaged in the simulation. In addition, questionnaires and interviews were used to recruit participants and gather data on students’ characteristics. The study design and data collection methods were carefully chosen to address the nuances of the research questions while minimizing the limitations of each data collection technique. By collecting quantitative and qualitative data on the participants’ behaviour, thoughts and characteristics, the research team can analyse how and why the participants behave the way they do and whether their characteristics influence these things.

Participant sample

This study examined students from 4th grade to graduate school. Participants were members of two generations, Millennials and Generation Z, meaning students born after 1980 (Dimock, 2019), ranging in age from 9–35 years at the time they responded to the call for participation. Given this age span, the research team wanted to control for two key aspects of participant recruitment. The first was the area of study for post-secondary students. The study recruited community college, undergraduate, and graduate students majoring in science, technology, engineering and mathematics. The second key consideration was recruiting K–12 (kindegarden to grade 12) students operating under common curricula requirements; therefore, the study limited recruiting to a single county in the south-eastern United States. This was the same location as the data collection site for the study.

To develop a sample that met these objectives, two questionnaires were developed, one for the post-secondary students and one for the K–12 students. The questionnaires were used to identify students willing to participate in the study and to collect demographic data that could be used in framing the sample. The questionnaires were administered in 2016–2017. Post-secondary students were surveyed first, followed by K–12 students. For both groups, completion of the questionnaire was incentivised and every 40th student who responded to the questionnaire received a $25 Amazon gift card.

The research team partnered with one university and one community college to recruit post-secondary students. Post-secondary students were surveyed during a three-and-a-half month period. The questionnaire was distributed through several channels, including a link placed on the university and community college library home pages, email invitations sent to science and technology majors by subject librarians, fliers placed in locations where such majors congregate and research assistants who walked the campuses with iPads to collect responses. The university also displayed the questionnaire upon login for 393 of the libraries’ public computers. The students agreed to participate in the questionnaire by clicking the ‘I Accept’ button after reading the informed consent and provided their contact information if they were interested in participating in the study.

To recruit K–12 students, the research team reached out to all media specialists and librarians in the county school system to share information about the study and serve as a conduit to school principals. Schools whose principals agreed to participate were given recruitment packets for parents and guardians. The packets included a letter and a link to the online questionnaire. The packet also included a printed questionnaire and postage-paid, pre-addressed envelope for those who wanted to respond on paper. Research assistants also visited county public libraries to post fliers with a link to an online information page. They also collected responses, using iPads, from children who were with a parent or guardian. Children at the library without a parent or guardian were given a recruitment packet. Lastly, a local home-school group was contacted to e-mail information about the questionnaire to their members. Students were surveyed during a seven and a half month period given the need to get consent from a parent or guardian.

Of the 1,672 questionnaires received from post-secondary students, 1,044 indicated interest in participating in the study. Of those, 883 were valid and used for sample selection. Of the 414 questionnaires received from the parents of K–12 students, 331 indicated interest in participating in the study. Of those, 318 were valid and used for sample selection. Valid questionnaires were those that met study requirements and included contact information. The research team aimed for thirty participants in each student cohort to have a large enough sample for comparisons using demographic variables. A total of 175 students participated in the follow-up simulation study. The total study group was divided into six cohorts as follows: 30 graduate, 30 undergraduate, 30 community college, 26 high school (grades 9–12), 30 middle school (grades 6–8), and 29 elementary school (grades 4–5).

Expecting attrition, the research team randomly sampled sixty students from each cohort using a pre-determined distribution of students who: 1) were or would be first generation college students (i.e., students who did not have a parent or guardian with a four–year degree) and 2) had asked a librarian for help in the last two years. Not only did the research team believe these variables would be important to control, but also that these demographic variables were the only ones that had a level of variation that could be retained through the sampling process. This sampling method was used for all cohorts, except for high school students, where the sample was too small. When attrition rates ran high, the team returned to the larger sample. Of the final sample, 31% were first generation college students and 42% of them had asked a librarian for help in the last two years.

Why a simulation?

In order to study the behaviour of such a diverse group of students, the study employed a simulation. A well-designed simulation offers a variety of methodological benefits. It gives the ability to exercise a high degree of control over the various elements of the research setup. It also allows researchers to define and isolate variables of interest in advance and to more precisely control the environment in which participants encounter those variables. Reducing the variations in environment reduces the number of possible explanations for why and how participants are making decisions. Simulations also increase the comparability of data. By ensuring that each participant encounters the same variables in the same environment, a simulation enables researchers to draw more direct comparisons across participants and groups, strengthening findings.

An additional strength of simulations is that researchers can gather data in real time as participants engage in the behaviour of interest. Depending on how data collection is designed, researchers can gather attitudinal, cognitive and behavioural data while participants engage with the simulation. This enables the collection of more accurate data and reduces recall, social desirability and other types of response biases. The flipside of this is that the behaviour is taking place in an artificial environment. Therefore, how well the data represent real-life behaviour relies directly on the design of the simulation. Researchers must thoroughly consider the phenomenon or situation of interest and design the simulation to realistically replicate it.

Building the simulation

This project sought to isolate students’ point-of-selection behaviour during the initial search for a school-related research assignment. Thus, the focus of this study is not to analyse the search strategies employed by students (i.e., query strings or the iterative search process), but instead to determine how students are evaluating resources that they encounter during their initial search. This includes examining the various judgements made at the point at which a student determines that a particular information object or resource meets an informational need.

To help ensure that participants engaged in naturalistic behaviour, the simulation needed to include not only the activity under study, but also the context for that activity. For point-of-selection behaviour, the context included discovering or being introduced to the topic, receiving an assignment prompt and conducting an initial search. Because previous literature indicates that most students start their searches, even those for school assignments, on Google, the simulation was developed to replicate the appearance of Google’s search engine results pages (Perruso, 2016). Due to its ubiquity, Google also had the advantage of being familiar to participants across cohorts and easy for participants to operate.

A brief video demonstration of the simulation is available.

Selecting a topic and developing prompts

The research team worked with an advisory panel of librarians and instructors to develop a topic that could be used for all cohorts. The advisory panel was selected carefully, to ensure all cohorts were represented and that instructors were relevant subject matter experts. This also ensured that the research team could be advised on the habits, skills and expectations of the participants to develop more realistic simulation activities.

In order to use the same topic across all six cohorts, it had to be complex enough to be engaging for graduate participants, but non-technical enough to be understood by elementary school participants. To ensure that all of the participants could connect with and invest in the activity, the topic needed to have non-academic interest and have multiple sides that could be realistically argued for, but not be so controversial that it would limit participants’ ability to approach it intellectually. The final topic chosen was a life science topic of regional interest: the effect of Burmese pythons on the Florida Everglades.

This topic was then used to develop assignment prompts for each of the six cohorts. Each assignment prompt was developed as a research or inquiry project that represented an authentic assignment that participants might receive at their educational stage. This prompt guided participants’ subsequent information resource selection and served as a touchpoint for their decisions throughout the simulation. See Table 1 for the assignment prompts given for each cohort.

Choosing resources and creating results pages

Participants were all shown the same short news clip to introduce them to the topic, given their assignment prompt and asked to type their initial search into the simulated Google interface. Because the project focused on evaluation, not search, participants in each cohort retrieved the same set of resources in their results pages regardless of what query they typed in. Having a consistent set of resources ensured that researchers could make direct comparisons between participants in a cohort. In addition, the focus on a single topic allowed for a small subset of seven common resources to appear in every participant’s results pages, regardless of cohort. Participants in the same cohort could be compared to each other, but researchers also could directly compare participants’ judgments of these seven resources across cohorts.

To determine which resources should be included in the results pages, the research team and advisory panel formed six small teams, one for each cohort. Each small team came up with a preliminary list of resources, which they then rated based on five criteria, categorised with a container, and scored with their level of preference that the resource be included in the final simulation.

The research team then selected the final resources for the simulation by including those with a variety of ratings and container types in order to compare how various features affected participants’ behaviour and judgments. This meant that the results pages included results, such as scholarly journal articles and books, which normally would not rise to the top pages. Although this is not realistic, the research team felt that the study would be incomplete if it omitted these types of resources. Including both open web and paywalled resources in the results pages allowed the researchers to directly compare participants’ judgments of and behaviour relating to these different types of resources.

In order to construct realistic results pages for the simulation, the simulation roughly displayed resources in the order of Google results and used screenshots of the Google snippet for each resource where available. The resources themselves were captured in a variety of ways depending on the file type (HTML, MP4, PDF, etc.). Webpages were captured as screenshot images. Ads that might potentially date a screenshot, such as specific political advertisements, were removed. Videos were located on YouTube, downloaded, and embedded over the screenshot of the resource. Each resource was reviewed to determine which features would be made interactive or clickable in the simulation, such as an embedded video that could be played or a PDF file that could be opened.

Based on the research team's experience, the input of the advisory panel and the structure of the simulation, only four simulations were constructed for the six cohorts. The three post-secondary cohorts (community college, undergraduate, and graduate) were given the same simulation because the differences in their information processing abilities and reading comprehension level were small enough that separate builds were not necessary.

Technical build

The simulation was built in Articulate’s Storyline software. Storyline has several affordances that made it a good choice for constructing a simulated environment. The software is designed to build online learning environments, so it enabled the creation of structured and sequenced activities. It uses triggers to dynamically change the content based on participants’ actions, which enabled navigation between pages and the use of conditional logic. Triggers also can be used to record the value of variables, which streamlined the collection of behavioural data during simulation sessions. An instructional designer skilled in the use of Storyline was hired to build the simulations and a programmer was hired to write the script necessary to export the data from Storyline into workable data files.

One constraint of Storyline was the creation of links within resources. The fundamental nature of the Internet is to be infinitely linked. This is not possible or feasible in a simulated system. For each link that was made live, additional logic had to be programmed and a new page had to be built for the resource that was being linked to. Storyline is a more or less linear program; participants complete one task and then move to another. Therefore, ensuring that participants could navigate back to the search results became increasingly complicated as more links were added.

Links within resources also created complications for data collection and analysis. In order to compare within and across cohorts, it was necessary to have relative control over which resources each participant encountered. The links within resources usually went to other resources that had not been chosen for inclusion in the results pages, and would consequently give participants the option to view resources that were not accessible from the main task pages. The more links that were added reduced the chances that enough participants would encounter a resource to make meaningful comparisons. Because of these considerations, the only links that were made active were links that the research team thought participants would absolutely want to click on, such as View PDF buttons.

Sequencing of simulation tasks

To ensure that participants completed the tasks as designed, a facilitator script was created. The facilitator script was composed of a screenshot of the first page of each task alongside a brief prompt with the instructions for how to perform the task and a reminder to think aloud while completing it. This was particularly important because of the large number of facilitators conducting sessions and the highly complex nature of the simulation. Having a facilitator script with standardised prompts and visual cues ensured that each simulation session was facilitated consistently, which in turn ensured that the data from each session would be comparable.

Isolating the variables of interest into different tasks allowed the researchers to control data collection. The simulated results pages and resources were duplicated and used to create discrete tasks, each based on the research questions. Data were collected on how participants assessed each of the variables independently, making it easier to conduct analysis and draw conclusions.

The tasks isolated the variables of interest, but in doing so they were more likely to diverge from participants’ natural search behaviour. To address this, the simulation initially was constructed with an open-ended task at the beginning that asked participants to explore the resources as they normally would. Initial pilots with the graduate cohort indicated that having the initial task introduced too much duplication, unnecessarily increasing participants’ cognitive load and the simulation duration. The task was removed and pilots with subsequent cohorts resulted only in small changes to the facilitator script.

For the final version of the simulation, participants were asked to complete five total tasks. Participants started with the Helpful Task, which asked them to explore the resources found on their results pages and identify those they found most helpful. The post-secondary cohorts were instructed to select twenty resources out of forty, the high school cohort ten out of forty, the middle school cohort ten out of thirty, and the elementary school cohort five out of twenty. Although this is more resources than some of these students would be asked to pick for a school project, it was important to have participants select enough resources that the data could be compared and analysed.

Once participants had selected their helpful resources, the Cite Task displayed those resources and asked participants to indicate whether or not they would cite each one in the hypothetical research project. In the third task, the Not Helpful Task, participants were shown the list of resources they did not select during the helpful task and asked to explain why they found the resource not helpful.

Previous research indicates that students often use resources that they do not tell their instructors about (White, 2011). The definition of helpful in the Helpful Task and Not Helpful Task was kept vague to discover what helpful meant to participants. The Cite Task captured the more traditional academic sense of what makes a resource helpful for a school project: citability. Dividing out these three tasks facilitated a deeper exploration of the reasons that students choose, or do not choose, resources at the point-of-selection.

The final two tasks specifically examined elements of the project’s research questions. The Credible Task displayed the list of resources the participants selected as helpful and asked them to rank the credibility of each resource on a scale of 1 (not credible) to 5 (highly credible). The facilitator provided a definition of a credible resource as one that could be trusted and believed. The Container Task displayed a controlled subset of resources from the original search results, and participants were asked to identify which of eight possible containers (blog, book, conference proceeding, journal, magazine, news, preprint, or website) best described each resource. For this task, participants within each cohort evaluated the same set of resources, ensuring that data could be directly compared within a cohort since this addressed one of the study’s main research questions. The post-secondary cohorts had twenty-one resources in the Container Task, the high school cohort had twenty, the middle school cohort had fifteen, and the elementary school cohort had ten. Creating separate tasks that focused on each of these variables ensured that participants’ strategies for determining them could be meaningfully studied whether participants mentioned them in the point-of-selection tasks or not.

Data

The data collection strategy was designed to gather data in three primary participant areas: behaviour during the simulation, thoughts and characteristics. These three types of data allowed the research team to examine relationships among objective indicators of participants’ behaviour, thoughts as they evaluated resources and their previous experience searching for information online. A mixed-method approach was necessary to capture data with this level of diversity and richness.

Because of the way that the data were gathered, they can be analysed using several different units and types of analysis. The data can be analysed by participant, resource, or task, expanding the conclusions that can be drawn about how students choose resources and determine credibility and container at the point of selection. Additionally, the data can be analysed using both qualitative and quantitative methods, which enables descriptive, explanatory and predictive findings that can be combined to increase the richness and depth of those conclusions.

Participant behaviour

One area of interest was understanding how participants judged the resources as they completed the simulation. For the Helpful, Citable, Credible and Container Tasks, the simulation software recorded the participants’ task decision for each resource. This allowed us to create quantitative data to better understand which resources are seen as most helpful, most citable and most credible and how accurately students can judge the containers of each resource.

The simulation software also captured which resources participants clicked on during each task. The click data indicate whether participants relied on the information in the results pages to make decisions or whether they opened a resource for further exploration. These data can be used to explore how clicking into a resource impacted each task judgement and patterns in participants’ search behaviour.

Using the recording of each session, the team calculated the amount of time that each participant spent completing each task in the simulation, as well as the entire simulation. This offered a rough proxy of how much time and effort each participant put into exploring the resources as they completed the tasks.

Participant thoughts

Secondly, there was interest in understanding why participants made their decisions in each task. To do this, the research team collected qualitative data using a think-aloud protocol to complement the data collected by Storyline. All participants were asked to say whatever they were thinking as they completed the tasks, and facilitators reminded the participants to think out loud throughout the simulation. The sessions were recorded and transcribed, and each transcript was cleaned and anonymised.

Using NVivo software, the transcript of each think-aloud protocol was broken down by task and blocked according to which resource in the simulation the participant was talking about. This allowed us to capture data on the thoughts each participant had about each resource. A codebook was developed to capture themes of interest. Codebook development began with members of the research team reviewing ten per cent of the transcripts from each cohort, with a variety of session lengths and types, to identify the most representative set of recurring themes. The team also consulted the existing literature and the terms used in the project’s data collection instruments to ensure that coding would address the research questions.

The codebook captured: 1) the cues, or features of a resource, that participants attended to when making decisions; 2) common judgements that participants made about the resources; and 3) participants’ descriptions of their usual search behaviour when not in the simulated environment. This coding makes it possible to perform content analysis to better understand point-of-selection behaviour from students’ perspective. For example, reviewing every instance when a participant referred to a source as peer-reviewed enables the research team to understand how students perceive peer review, how they determine whether resources are peer-reviewed, and what impact that has on their decision of whether resources meet their needs. Such content analysis explores how and why participants made particular decisions, based on personal descriptions of their experience.

Once acceptable inter-coder reliability was achieved, the codebook was finalised, and team members coded transcripts individually and entered that coding into NVivo. After all individual transcripts were coded, they were merged into combined cohort files for analysis. NVivo software had the added advantage of being able to export data to a spreadsheet, where it could be combined with quantitative data collected in the questionnaires and simulation software. This allowed us to explore relationships between variables that were measured qualitatively and those that were measured quantitatively.

Participant characteristics

Background information on participants was collected from two sources: the questionnaires used to recruit simulation participants and interviews conducted before and after the simulation sessions. These provided a source of quantitative data that focused on the demographic characteristics of the participants as well as their previous experiences with and attitudes about searching for information online.

As noted above, questionnaires were used for recruitment. In the questionnaires, participants were asked about demographic characteristics such as age, gender, national origin, race/ethnicity, whether they had a parent who went to college, and college major (for post-secondary students). These allowed for the exploration of how demographic characteristics impacted simulation behaviour, such as whether first generation college students have difficulty determining information containers (Cyr, et al., 2020).

An interview was conducted before and after each simulation session. The pre- and post-simulation interviews allowed us to create rapport with participants while confirming information collected during the initial screening questionnaires and collecting additional data on participants’ online behaviour and attitudes. The information from both pre- and post-simulation interviews also was used to expand upon the digital visitors and residents framework (White and Connaway, 2011–2014) by adding students in grades 4–11 and collecting additional demographic information such as English as a second language.

Some of the interview questions could prime participants for the simulation so potentially biasing questions were asked after the simulation concluded in order to mitigate that concern. Pre-simulation interviews focused on information about educational background and typical online behaviour. For example, participants were asked if they had ever taken a research course from a librarian, how many research projects they had recently completed and which search engines and social media sites they normally use. Post-simulation interviews focused on attitudes towards finding information online. For example, participants were asked if they believe that knowing information containers is important. Interview data allowed the research team to situate participants’ simulation and think-aloud data in the context of their usual online habits and attitudes, exploring how they impacted their behaviour and thought patterns during the simulation tasks. Lastly, students were debriefed that they had participated in a simulation study where the resources were pre-selected regardless of the search string they entered.

Discussion

This study examined how students identify and judge the credibility of online resources. It is intended to increase the effectiveness of classroom instruction by rooting it in students’ experience, given their educational stage and mode of online engagement. This required a complex study design that controlled participants’ search environment in order to directly compare their behaviour and thoughts within and across educational stages and backgrounds. A wide variety of skills and expertise were needed to create and execute this design, and it took much more time and effort to complete than initially projected.

Skill and expertise

For this project, a large team was assembled because of the wide array of expertise needed to accomplish the project objectives and successfully execute the methods. The core team was composed of science and education librarians, a library and information science educator with K–12 expertise, an educational technologist and information science researchers. The librarians and educator led the advisory panel to develop the content for the simulation and scope the tasks to ensure that they were age-appropriate. The educational technologist led a team composed of an instructional designer and a programmer to build the simulation. The information science researchers led data collection and analysis.

Such a large team required much time and effort to coordinate and manage. However, the project benefited from the variety of skills, expertise, and perspectives. While team members led efforts that aligned with their skills and areas of expertise, the whole team met regularly to discuss and seek feedback from one another on the various aspects of the project. The feedback and contributions from a wide variety of perspectives added value throughout the project, from developing the proposal to disseminating research findings and creating ways to incorporate them into practice.

Time and effort

With a project of this size and scope, it took a lot of time and effort to accomplish the project goals. Rather than relying on observation in a natural setting or a controlled study that is unrealistic, the study employed a simulated environment to collect both qualitative and quantitative data on participants’ behaviour and thoughts. The use of simulations to study information seeking and assessing is rare, and constructing a functional and realistic simulation environment to do so is not something that the research team could find in the literature.

The sample size of 175 participants is much larger than most qualitative studies, and a much wider range of educational stages was recruited compared to previous studies. With this innovative methodology, it was difficult to accurately estimate time and effort involved in each stage of the project.

Despite a relatively good fit to purpose, Storyline still required customization to be able to create an effective simulated environment. The research team had to strike a balance between collecting data that met study goals and creating a realistic experience for participants by allowing them to navigate between pages and click into resources. Additionally, a database was created to safely extract the data when the simulation was run offline. Both customizations required trial and error and coordination between the research team and the developers working on the simulation build.

Despite an initial projection of eight months, data were collected from the 175 participants during a nineteen-month period. For each participant, questionnaire, pre-simulation interview, think-aloud, simulation behaviour and post-simulation interview data were collected. Simulation sessions lasted an average of an hour, not including the time needed to schedule and prepare for each session or the time it took after the session to manage and store the data. Difficulties in scheduling students during particular parts of the year, such as holidays and exams, developing and customising the simulation for each cohort, and the need for the research team to complete some tasks before others led to periods in which data collection was delayed.

In addition, reaching K–12 students and getting informed consent was a multi-step process that included partnerships with the school media specialists and buy-in from the parents. The challenges of recruiting and scheduling these participants resulted in an extended timeframe and an inability to recruit all thirty participants from each cohort. In retrospect, the research team would have developed more strategic, direct relationships with school principals to ensure the questionnaire was distributed to more students.

Once the data were collected, they had to be cleaned, managed and documented. One goal for this project was to make its outputs, including data and project publications, openly available. Both the qualitative and quantitative datasets will be deposited into the Institutional Repository at the University of Florida for reuse, along with the questionnaire instruments, interview protocols and the codebook. Because sharing these materials was a goal from the outset, the research team was careful to clean and manage the data and documentation as they were created. Quantitative data were integrated from multiple formats, checked, cleaned and some variables transformed when necessary. Qualitative transcripts were cleaned and anonymised. Proactively pursuing this at the point of data creation took more time on the front end but made it easier for the research team to be able to use its own data and deposit it for others to reuse.

The qualitative data collected during the simulation sessions then had to be coded. While the team was experienced in qualitative coding, the unique nature of these data and the size of the team and data sample created challenges. One major challenge was determining the unit of analysis when developing the coding practices. In order to analyse the data by participant, task and resource, the team established the resource as the unit of analysis. This meant that a new coding block began each time the participant started talking about a new resource within a task. Ten team members participated in coding, which required the team to develop new strategies for achieving inter-coder reliability and communicating changes in coding definitions and practices. In total, it took approximately twenty-one months to develop the codebook, establish ICR and code the transcripts, which far exceeded the projected time of nine months. In part, the expanded timeframe was also due to the fact that coding each individual transcript took much longer than expected, about eight hours on average.

Conclusion

The methods developed for the RSIC project provided the means to identify the hows and whys of students’ point-of-selection behaviour. The study is significant because, for the first time, simulation software was employed to capture data on how participants actually evaluate search engine results for science and technology projects, not how they think they do. The think-aloud protocol captured why participants made the choices that they did from their own point of view. The questionnaire and interviews captured relevant information about students’ education and online experience. These data on participant behaviour, thoughts and characteristics can be combined to answer questions that could not be answered with one single method. For example, one article on students’ ability to correctly label the containers of information resources looked at the cues that they attended to (captured through the think-aloud protocol), their behaviour during the Container Task (captured by the simulation software), and their educational background (captured by the questionnaire and interviews) (Cyr, et al., 2020).

This would not be possible without the rich data captured using mixed methods. The specific approach this study took in mixed-method research design, combining a task-based simulation and think-aloud protocol, offered several advantages over other mixed-method options. The study design:

• offered a balance between control and realism,
• created data that were comparable across and within a wide range of educational stages, and
• collected data that can be analysed to draw conclusions about:
  • participants and demographic groups,
  • the resources and types of resources included in the simulation, and
  • the behaviour demonstrated when completing specific tasks.

The simulation offered a balance of creating a realistic search environment while also giving the researchers a great degree of control. The results pages looked like a normal Google search output, but enabled control in terms of which resources participants retrieved and the order in which they retrieved them. In addition, the separation of the simulation into different tasks allowed for the capture of data on specific points articulated in the research questions.

In addition to controlling the simulation, the think-aloud protocol allowed the research team to ensure that consistent data were captured when participants talked about their decision-making. Researchers were able to prompt participants during the simulation session so that all participants’ thought processes were captured continuously. For example, they could ask them why they explored the content of a specific resource, what they were looking for as they were reading, and why they found some resources to be unhelpful.

This high level of control increased the comparability of the data, as it ensured that differences in the data were not an artefact of differences in participants’ specific search session. It also ensured that every participant saw a variety of source types, container types and different reading levels. This especially was important in this research project, as the participants came from a wide range of educational stages and demographic characteristics.

The mixed-methods approach used in the study successfully captured data on several aspects of point-of-selection behaviour over a wide range of educational stages. This method could be replicated and expanded in future studies on information behaviour. For example, researchers could investigate how point-of-selection behaviour differs in a non-academic context, how users behave in discovery systems other than Google, or how evaluation processes other than credibility and container are related to selection. Despite the challenges associated with the mixed methods used in this study, the data provide a more complete picture of resource evaluation and selection behaviour across students of a wide range of educational stages and demographic characteristics.

Acknowledgements

This project was made possible in part by the Institute of Museum and Library Services (IMLS) Grant Project LG-81-15-0155.

About the authors

Tara Tobin Cataldo is the Biological/Life Science Librarian and Science Collections Coordinator at the University of Florida’s George A. Smathers Libraries. She has been an academic librarian for 20 years. She can be contacted at ttobin@ufl.edu.
Amy G. Buhler is an Engineering Librarian at University of Florida’s Marston Science Library. She provides research expertise and instructional support to the areas of Agricultural & Biological Engineering, Biomedical Engineering, and Engineering Education. She can be contacted at abuhler@ufl.edu.
Ixchel M. Faniel is a Senior Research Scientist at OCLC in Dublin, Ohio, USA. Ixchel conducts user and library studies related to research data management, reuse, and curation practices and online information behavior. Her research has been funded by the National Science Foundation (NSF), Institute of Museum and Library Services (IMLS), and National Endowment for the Humanities (NEH). She can be contacted at fanieli@oclc.org.
Brittany Brannon is a Research Support Specialist at OCLC Research in Dublin, OH, USA. Her research interests are in information seeking behavior, academic research skills, and scholarly communication. She can be contacted at brannonb@oclc.org.
Lynn Silipigni Connaway leads the Library Trends and User Research team at OCLC Research. In addition to research in these areas, she specializes in research design and methods and is co-author of editions of Basic research methods for librarians and of the 6th edition of Research methods in library and information science. She can be contacted at connawal@oclc.org.
Christopher Cyr is an Associate Research Scientist at OCLC Research in Dublin, OH, USA. His research interests are in the ways that people find information online and the impact that libraries have on their local communities. He can be contacted at cyrc@oclc.org.
Kailey Langer is a graduate student at the University of Florida. She received her Bachelor of Science degree from University of Florida and her research interests involve cognitive and brain changes observed in aging. She can be contacted at kaileylanger@ufl.edu.
Erin M. Hood was a Research Support Specialist at OCLC Research where she worked since 2008 providing project management support. She graduated from Otterbein College in 2001 with a BA in Religion and in 2009 from Kent State University with a MLIS.
Joyce Kasman Valenza is an Associate Professor of Teaching at Rutgers University, School of Communication and Information and the coordinator of the School Library Concentration. Her research interests include search, youth information-seeking, educational technology, social media curation, OER, emerging literacies, and online communities of practice. She can be contacted at joyce.valenza@rutgers.edu.
Rachael Elrod is the Associate Chair of Departmental Libraries and Director of the Education Library at the University of Florida’s George A. Smathers Libraries. Her research interests include the information seeking behavior of students, assessment of library services, and diversity in children’s and young adult literature. She can be contacted at relrod@ufl.edu.
Randy A. Graff is the Director of Educational Technologies at the University of Florida’s Health Science Center. He has been Educational Technology Administrator for 20 years. He can be reached at rgraff@ufl.edu.
Samuel R. Putnam is the Engineering Education Librarian and director of Made@UF, a virtual reality development space, at the University of Florida, Marston Science Library. His work focuses on innovative and multimodal instruction practices as a means to promote information literacy. He can be reached at srputnam@ufl.edu.
Summer Howland was an instructional designer at the University of Florida.

How to cite this paper