How do children reformulate their search queries?

Introduction

Users of information retrieval systems formulate queries to search for information, and reformulate queries to improve search results or gain new information. The process is iterative, with queries reformulated from the user's own prior knowledge or with the help of the information retrieval system (Huang and Efthimiadis, 2009). From an analysis of transaction logs from Web search engines, over one third of all queries submitted are reformulations (Jansen, Booth and Spink, 2009).

Although there is an increasing interest in children's search, very little research examines the query reformulation behaviour of children (see Related Work). This paper is based on an initial study that investigates how children interact with retrieval systems to reformulate queries. This involved an in-depth study of twelve children's searches during an information and communication technology lesson. In particular this paper examines the following: (1) how children use more than one information retrieval system within a single search task, and (2) how children reformulate queries using functionality, such as 'did you mean' and auto-complete, as well as in response to reading search results pages. Specifically we address the following research questions:

• RQ1: why do children reformulate their search queries?
• RQ2: how do information retrieval systems support children with query reformulation?
• RQ3: how do children reformulate their queries?

Related work

Much of the research into children's information-seeking behavior focuses on the differences between child and adult search behaviour (Marchionini, 1989; Bilal and Kirby, 2002; Duarte Torres, Hiemstra and Serdyukov, 2010; Gossen, Low and Nurnberger, 2011) and the impact of search task on search behaviour (Bilal, 2002). Query reformulation is usually discussed in relation to these areas; however, children's query formulation and reformulation has also been considered in relation to the differences between success and lack of success (Marchionini, 1989; Duarte Torres, et al., 2010; Kammerer and Bohnacker, 2012).

The most comprehensive research study into children's query reformulation patterns is the examination of children's search behavior from an analysis of search engine transaction logs (Duarte Torres, et al., 2010). The analysis provides a good overview of the types of query reformulation children enter, and the frequency of reformulation. However, as the analysis is only based on the data collected in search logs, the reasons for the reformulations cannot be adequately explained. Furthermore, search logs by their very nature only record the data for one search engine, and the analysis cannot capture the way queries are reformulated across different information retrieval systems. In observational studies, research has also tended to concentrate on how children search for information in one search system because the search system itself is also being evaluated (Marchionini, 1989; Bilal, 2002). However, in real life situations, similar to adults, children use more than one search system to find information.

Research into how children use information retrieval system assistance for query reformulation, and under what circumstances, is very limited. In part, this is because many of the features offered to support query formulation and reformulation are relatively new and either unavailable in earlier research (Marchionini, 1989; Bilal, 2002) or not yet adopted by children (Druin et al., 2009). It is only recently that children have been observed using assistance (Jochmann-Mannak, Huibers Lentz and Sanders, 2010; Kammerer and Bohnacker, 2012). In this study we use a more naturalistic approach to studying query reformulation by children that is able to overcome the limitations of using only transaction logs and of focusing on search sessions based on interaction with single information retrieval systems.

Method

In this section, the rationale for the study approach, the sampling procedure, the search task, the data collection and analysis methods, and the limitations of the study are discussed in detail.

Rationale

In observational studies, children are usually studied singly, with search tasks recorded outside of the classroom/lesson (for example, Marchionini, 1989; Bilal 2000a, Bilal, 2000b; Bilal, 2002; Madden, Ford, Miller and Levy, 2007; Foss ,, 2012). However, although observation is a key technique for understanding user behavior, observation can only shed light on users' behaviour if it is real-life behaviour that is observed (Martzoukou, 2005). For this reason, this study takes a naturalistic approach to data collection.

The information and communication technology teacher of a Sheffield (UK) primary school agreed to the recording of children's searches in his lesson. The researcher had no input into the planning of the search task or lesson, and as much as possible tried to remain anonymous during the class. However, the children (including those not in the research group) were interested in the study so it cannot be claimed that this research did not have any impact on the children's behaviour.

The children, in common with most other primary school children, worked in pairs when using technology in schools. Although it is more usual to research children's search tasks individually, it was considered more important to collect the data naturalistically, and include observations of children's collaborative behaviour in the data collection.

Sampling

Purposive sampling was used to determine the year group of the children to study, and the children selected from the year group. Following a discussion with the ICT teacher, Year 4 children (aged eight to nine) were chosen on the basis that they were the youngest age where they could mostly search for information independently and, therefore, the recorded searches would represent the searching behaviour of the children, not the supervising adults.

The school has two Year 4 classes and it was decided that six children would be selected from each class as two smaller groups would be easier for the researcher to observe. Furthermore, when testing the screen recording software it became apparent that the school network connection slowed as more computers ran the screen recording software.

The participants were selected by the class teachers: the teachers explained the research to the children, and pupils interested in taking part placed their names in a hat. Each teacher drew six children's names. Letters detailing the research were sent home with the children, and the parents were asked for their consent. Consent was received for all twelve children selected (seven girls and five boys). Immediately before the recording of the children's searches the researcher also explained the purpose of the research to the children, and the children were shown the recording equipment. The children's verbal consent was sought and received before all data collection exercises.

Search task

The research was conducted in a Sheffield (UK) primary school with search task data collected during two ICT lessons that ran consecutively. For each lesson, six pupils aged eight to nine worked in pairs to search for the answers to three questions. The questions were chosen by the class and were based on the topic the children were learning about that term, the human skeleton. At the start of the class the teacher explained to the children the importance of safe searching and the need for reliable information. The children then spent between ten and twenty minutes searching for the answers, with the time dictated by the length of the lesson and the need to set up computers within the lesson time.

Data collection

Data were collected from observations, recording of search tasks, group interviews with the children and an interview with the teacher.

Observations

Observations of the children's collaborative and physical behaviour were recorded by the researcher in a notebook. In practice, there was little time to record observations as the research took place; instead, observations were recorded immediately after the searches were finished.

Screen recordings

The search task was recorded using Camtasia screen recording software. Prior to data collection, Camtasia was temporarily installed on the school equipment.

Interviews with children

To allow for initial analysis of the search recordings, the children were interviewed a few days after the search task. The children were shown screenshots of their searches and were asked (1) how they thought search engines worked (2) the purpose of search tools (3) the reason for the phrasing of their queries (4) what they found easy and what they found hard and (5) if they had a magic wand what it would do. Because children may find it difficult to express their experiences to an adult (Livingstone, Olafsson and Haddon, 2013) the children were interviewed in two groups to boost their confidence. The first group consisted of four children from the first class, and the second group consisted of the six children from the second class plus the remaining two children from the first class who had previously been unable to attend. As it happened, the children were confident communicators and more than able to explain their experiences. Most of the children were very keen to provide explanations, and the hardest part of the interview was to give each child time to answer before another child jumped in.

Interview with information and communication technology teacher

A week after interviewing the children, the teacher was also interviewed. The time lapse allowed for some preliminary analysis of the children's search behaviour and the children's explanations of their search behaviour. The interview questions were semi-structured and the teacher was invited to comment on the preliminary analysis. The teacher confirmed the analysis but suggested that in general children from the school have above average language skills so find query reformulation easier age for age than in some other schools the teacher has taught in.

Verification

Verification of the analysis was sought from the school, and both the teacher and headmaster were given a written report of the findings. No formal feedback was obtained, but the findings were informally discussed with one of the class teachers and the teacher.

Data analysis

The data were analysed both qualitatively and quantitatively with the emphasis on the qualitative analysis to produce a holistic and in-depth explanation of the children's search behaviour (Mason, 2002).

Screen recordings

Initially, the screen recordings were analysed according to the time taken to complete the tasks, the impact of queries on the search results, and the use of assistance. Then, all of the queries were categorised using two classification schemes. Firstly, queries were categorised into informational and navigational queries using Broder's (2002) taxonomy. Informational queries are those used to find information relating to the search task question; navigational queries are those used to find known sites. Secondly, all queries were categorised by query reformulation type based on prior schemes (Jansen, Booth and Spink, 2009; Jesper, Clough and Hall, 2013): New (first query for search question); Assistance ('did you mean' or auto-complete used); Specialisation (the query contains more terms or uses more specific terms); Generalisation (the query contains fewer terms or uses broader terms); Revision (a spelling mistake or grammatical error has been corrected); Previous (the query has been submitted before).

Children's interviews

The children's interviews were coded inductively using thematic analysis (Braun and Clarke, 2006). The initial codes were identified from the data, rather than the research literature, and three broad themes identified: (1) information retrieval system differences, (2) the phrasing of search questions, and (3) choosing between answers.

Information and communication technology teacher interview and children observations

The teacher interview and search observations were analysed inductively for initial codes. The data set was small so the initial codes were not developed into themes; instead, the initial codes were incorporated into the themes found in the children's interviews.

Limitations

Recording detailed observations of children's collaborative and physical behaviours by hand by a single researcher was impractical. An overview of their behaviour was obtained, and confirmed with the teacher, but for a more detailed analysis other data collection methods need to be considered. However, the low key approach meant that the children were relaxed about the research. The addition of more recording equipment or more researchers would have improved the data collection but would have resulted in a less naturalistic study.

The chosen method allows for a deep understanding of how children may use information retrieval systems in real-life, but while the findings apply to twelve children at a Sheffield (UK) primary school, as with other studies based on observation the results are not generalisable (Kelly, Dumais and Pederson, 2009) as the sample is size is small and only one task was analysed. Furthermore, children at a Sheffield (UK) primary school perform above average (Ofsted, 2011) and so the results are not necessarily comparable to children in the same year group at other schools.

Results and discussion

Task performance

During the search session, the children could choose up to three search questions to answer. However, four pairs only searched for one search question, and the remaining two pairs for two questions. In research involving transaction logs and observations, children were found to enter fewer new queries than adults (Duarte Torres, Hiemstra and Serdyukov, 2010) and take longer to complete tasks (Bilal and Kirby, 2002). In this study, the lengthy time taken to complete search tasks can in part be attributed to the time taken to enter queries (for example, the query 'how long is the spine in the average human body' took over two minutes to enter) and the children's reading skills. However, another reason for the lengthy task completion time was that the answers to some of the questions were not easy. Unlike with researcher set tasks, the teacher was unable to check the availability of answers before the lesson, because the questions were decided during the class.

Four pairs used auto-complete and three pairs used 'did you mean' assistance. In the group interviews, all the children demonstrated knowledge of these tools and they discussed the different ways in which these tools could be used. All the children started their searches in Google with five of the six pairs additionally entering searches in Bing, Wikipedia or Dictionary by Farlex.

None of the children could touch type so to enter queries they needed to look for letters on the keyboard. In some studies, the need to look at the keyboard resulted in children only looking at the keyboard and therefore being unable to interact with auto-complete (Druin et al., 2009), but in this study all of the children did periodically look at the screen for this functionality while typing (Jochmann-Mannak, Huibers, Lentz and Sanders, 2010).

All the children worked in pairs and took it in turns at the keyboard, swapping over if (and when) a search question was answered. It was observed in each pair that the child who typed the query decided the query. This contrasts with their behaviour in other areas of the search where the children did confer, notably, which information retrieval system to use, how to spell words, the meaning of words, and location of results in the search results pages, although even then ultimately the child at the keyboard controlled the search. This is similar to studies of adults where finding is a solitary process, but needing and using can be a collaborative activity (Toze, Peet and Toms, 2011).

Query formulation

Altogether, nine navigational queries and twenty-six informational queries were submitted. In the group interviews, the children distinguished between navigational and informational queries, with navigational queries described as easier than informational queries: 'if you were trying to find a specific thing you just type in like if you were trying to find out about a person you could just check, type their name in and then it might find them' (P12).

Although the task was informational in nature, three pairs of children entered navigational queries to find known sites within which they could search for information. During the search task, only one pair directly typed a website address into the address bar; in all other cases children navigated to known sites by entering the name of the site in the search box. Two of the pairs of children entered navigational queries at the start of the task to go to Wikipedia, and during the task, two pairs of children used navigational queries to change information retrieval system before reformulating informational queries, with those children iterating between navigational and informational queries.

From the group interviews it was clear that all of the children put considerable thought into the phrasing of queries and how the phrasing affected the answer. It was not something they always found easy. When asked 'if you had a magic wand, what would it do', two of the children wanted help with the wording of a query: 'If I were finding it hard to look something up then I'd just tell the computer to find the best words to know to look something up and then it would find me the best words that would be easier to make my search' (P7). The children were also concerned with how much detail to put in a query. One child explained 'because if you just put what is the longest bone it could be what if it were an elephant or a cheetah' (P1). The ICT teacher also encouraged the children to be specific with their queries as they will be less likely to come across inappropriate material with more defined queries.

Examining search results

All children had some difficulty finding information in the results pages (Large, Beheshti and Rahmen, 2002; Bilal, 2000a; Large and Beheshti, 2000). They frequently missed relevant information, and found it hard to choose between search results. In the interviews, one child complained of the search results page 'there's like a really long list' (P3), and for five pairs of children relevant or related information was available within the search results that they did not see. These children were able to enter effective searches for questions; they just could not see the answers. Children's failure to extract information has previously been linked to information overload (Bilal, 2000a; Kammerer and Bohnacker, 2012) and loss of focus on search task (Marchionini, 1989). However, from observing the children in this study another explanation is apparent. Using mouse movements as an indicator it is likely that all of the children linearly, rather than selectively, scanned the search results page. The mouse always moved from left to right, from top to bottom of screen. Because the screen was read systematically relevant information further down the screen was missed, and this resulted in unnecessary query reformulation. For example, one pair on seeing the answer for dinosaur bones in the first search result reacted with the following: 'doh, we need to write human skeleton' (P2). But the required answer was actually in the search result immediately underneath.

In the group interviews, the children discussed finding it hard to choose between search results and they were concerned about the reliability of answers: 'if you were looking something up you might want to check that because say you looked on Wikipedia, they might be wrong' (P7). These children were also aware that performance is different for each retrieval system, and one child explained: 'if you use different search engines then you'll find different things' (P9).

Research Question 1: why do children reformulate their search queries?

Altogether there were three reasons for the children's query reformulation: they could not find the required information, they wanted to validate information found, or they wanted to correct queries.

Only three of the informational queries were reformulated because the information was not available. Five of the six pairs of children reformulated queries, despite relevant or related information appearing in the search results page. The reason, in this study, was connected to the linear scanning of results as the children reacted to information in the order in which it appeared. Only one pair checked or verified their answer. They did so by using another retrieval system: swapping from Google to Bing and entering a near-identical query. Three pairs also revised their queries by accepting 'did you mean' suggestions and two pairs corrected their queries for spelling mistakes when the correct spelling was identified in the search results.

Research Question 2: how do information retrieval systems support children with query reformulation?

From an examination of the informational queries, it is apparent that two very different types of informational queries were entered. Four pairs slightly adapted the original task questions and entered queries as specific questions. Two pairs, instead of basing the queries on the search task, entered queries based on the broader class topic. For example, for the search question 'what is the longest bone' one pair entered the query as a specific question, 'what is the longest bone in the skeleton', and another pair entered the query as the broader class topic 'human body'.

In a study that compared children's keyword and natural language queries, children were marginally more successful when using natural language queries because the search results were more applicable to the search task questions (Kammerer and Bohnacker, 2012), and in this study answers to the questions were often available within the search results for the question queries but the results needed to be examined for the broad topic queries. However, there were also other differences between the two query types, as the different styles of queries resulted in different query reformulation behaviour with information retrieval assistance used in different ways.

Question queries

When employing question queries, the children used the retrieval systems for support in query reformulation by applying the 'did you mean' suggestions and extracting information from the search results. Auto-complete was not used for this purpose.

Three pairs of children reformulated queries based on their reading of the search results pages: queries were corrected for spelling mistakes, and also refined to make use of information found. Three pairs of children accepted 'did you mean' suggestions in the search results page. For example, for the query 'what is the longest bone' the 'did you mean' synonym suggestion of 'what is the largest bone' was accepted. For the question queries, auto-complete was only used to correct spellings and provide typing shortcuts. From the recordings, it is apparent that the wording of the queries had been pre-planned and auto-complete was only selected when it matched the exact wording of the planned query, with other appropriate suggestions ignored. For example, for the planned query 'how many bones in a human babys foot', auto-complete made alternative suggestions, but because none of the suggestions exactly matched the planned query, auto-complete was not used.

Broad topic queries

The two pairs of children entering broad topic queries did not reformulate queries based on their reading of the search results pages, and the retrieval systems did not offer 'did you mean' suggestions. The two pairs used auto-complete to assist both the initial query formulation and subsequent reformulations. In the group interviews, it was apparent that some of the children seemed to associate retrieval systems with either having or not having information (for example, 'some of the search engines might not have the question, might not have the answer to your question' (P4)) rather than linking results to differences in query formulation. This may account for why auto-complete was used to suggest queries.

Research Question 3: how do children reformulate their queries?

Of the informational queries, 8 (31%) were classified as New; 7 (27%) Assistance; 4 (15%) Specialisation; 3 (12%) Previous; 2 (8%) Revision; and 1 (4%) New/Assistance. No Generalisation type was used and one query reformulation was submitted by mistake. There were no reformulations of the navigational queries. In the same way that children used different types of support depending on the type of query entered, how the children reformulated the query also varied depending on the query type.

Navigational queries

In this study, none of the children reformulated navigational queries, although one might expect to see this if the navigational queries were difficult to spell or the name of the site the children intended to visit was not fully known. However, two pairs of children did iterate between navigational and informational queries, and after submitting a navigational query the next informational query was a Previous query. In fact, all three of the Previous queries were submitted in a new information retrieval system. That children resubmit previously used queries is well known (for example, Gossen, Low and Nurnberger, 2011), but this is usually linked to children's cognition and recall difficulties (Bilal and Kirby, 2002) rather than an intentional technique. This was clearly a technique other children used too, and when questioned over whether it was necessary to use another retrieval system to check information, a child wisely pointed out 'well you could check on the same but on a different search engine you'll get a wider range' (P9).

Question queries

The children made Assistance, Specialisation and Revision type reformulations when their queries were questions. Five of the Assistance reformulations were made following 'did you mean' suggestions. The suggestions included spelling error corrections, synonym change and broadening queries.

In most instances the children were concerned to make their question queries specific, and notably queries were not generalized except for when following 'did you mean' suggestions. As well as reformulating queries to make them more specific, the children also increased the specificity of the search task questions in the initial New queries. However, the implementation of Specialisation was often misguided. For example, one pair tried to use their topic knowledge to increase specificity and added 'mans average foot' to the task question 'how many bones in your foot'. While adding mans does increase the specificity because a baby's foot has more bones (and the pair knew this), adding 'average' does not increase the specificity because adult feet typically have the same number of bones. The specificity of queries was also increased by including information found in the search results. This was also problematic. For example, one pair changed 'spine' to 'spinal cord' in 'how long is the spine in an average adult' but the length of the spine and spinal cord are not equivalent. Analysis from the study of a transaction log found that it was more common for children to add rather than remove words from a query, but reducing the number of words was more effective. However, all query reformulation types (except for Previous Query) led to children selecting higher ranked results (Torres, Hiemstra and Serdyukov, 2010).

For Revision type reformulations, two pairs of children revised their queries, based on their readings of the search results, to correct grammar and spelling mistakes. For example, despite seeing relevant results, one pair realised there was an error in their query and changed 'how many bones in a human babys foot' to 'how many bones in a human babies foot', ironically introducing another error. The revisions were not necessary and did not improve the search results. However, the children were in a school lesson where they are assessed on spelling and grammar so it is understandable why the corrections were made. During the interview with the ICT teacher the importance of spelling was stressed in terms of speed ('I say when you are doing a search make sure your spellings are correct so it will make your search quicker') and Internet safety in case a misspelling led to inappropriate results.

Broad topic queries

The children only used Assistance type reformulations when their queries were based on the broad topic, and they only used auto-complete to do this. In all of these cases the children used auto-complete to specialise their queries. For example, one pair started their search by typing 'human skeleton' into the search box and then selected 'human skeleton for kids' from auto-complete. After examining the search results page with no success, they again typed 'human skeleton' in the search box and then this time selected 'human skeleton facts' from auto-complete.

Conclusions

This study investigates the query reformulation patterns of children based on an in-depth study of twelve children carrying out a search task in the classroom. The findings suggest that there is a clear role for information retrieval systems to provide support for children's query reformulation: the children used prior knowledge to formulate queries but mostly reformulated queries based on their interactions with the information retrieval systems. To do this the children used a number of techniques, including retrieval system switching, examination of search results, and use of "did you mean" and auto-complete functionality. However, more tailored support could be offered. Firstly, the children had the most difficulty with, and need the most support for, narrowing their queries. Secondly, there were two distinct types of information queries (broad topic and questioning queries) which resulted in different usages of search tools, and these tools could be developed in accordance with how the children are using them. Thirdly, more could be done to accommodate children's linear (rather than selective) scanning of search results as this resulted in unnecessary reformulations.

A further collection of data is planned to see whether these findings are also seen across a larger dataset, how query reformulations vary by search task type, and whether query reformulation patterns change when a search task spans multiple search sessions.

Acknowledgements

Thank you to all who participated from a Sheffield (UK) Primary. Firstly, to the children who generously and eloquently shared their information-seeking experiences. Secondly, to the information and communication technology teacher who very kindly allowed his lesson to be recorded and his time taken up with numerous questions on children's use of search engines. Thirdly, to the headmaster, assistant head and class teachers who consented to the research taking place, and then arranged the collecting of consents from the children.