Theory usage in empirical research in ISIC conference papers (1996-2020)

Introduction

Use of theory as a lens for analysing empirical data is understood to be a fundamental marker of research rigour. Thus, examining a body of research to explore the extent to which it incorporates theory provides a measure of its quality. In addition, exploring which theories are used and the disciplines from which they are drawn helps scholars to situate the field and to understand its intellectual influences.

Informed by previous studies exploring theory use in information behaviour research (e.g., Julien and O'Brien, 2014; Lund, 2019; Pettigrew and McKechnie, 2001) this study explores the use of theory in empirical papers presented at ISIC: The Information Behaviour Conference, from the first conference in 1996 to the most recent conference in 2020. We define theory as a coherent set of ideas or propositions applied systematically, which together have some explanatory power. Our use of the word theory in this paper should be understood to mean theories or models used as conceptual or theoretical frameworks. Common examples from information science would include Kuhlthau's information search process (e.g., 2004) and Dervin's sense-making (1992).

Previous analyses show that a minority of authors use a theoretical frame. Our study was designed to explore this problem more deeply, focusing specifically on the use of theory. Our research question is: how is theory being used in empirical studies presented at the biennial ISIC conferences from 1996 to the present? For each empirical study, we examined the inclusion or lack of theory, the level of theory use, the specific theories used, and the disciplines from which these theories originated.

Literature review

Theory use in information behaviour research has been examined by many authors using multiple approaches. Several authors have examined the proportion of information behaviour studies that use theory. In a series of papers studying the information needs and uses literature from 1984 to 2014, Julien and colleagues (Julien, 1996; Julien and Duggan, 2000; Julien and O'Brien, 2014; Julien, et al., 2011) found that the proportion of empirical studies that mention theory ranged from 18% to 28%. This finding is consistent with other studies of information behaviour theory use (see Table 1) and with studies of the proportion of theory in the overall information science research which ranged from 10% to 34.1% (Feehan, et al., 1987; Järvelin and Vakkari, 1990; Kim and Jeong, 2006; Nour, 1985; Peritz, 1980; Pettigrew and McKechnie, 2001). Notable exceptions are amount of theory in ISIC papers (McKechnie, et al., 2008) and in Ibero-American information behaviour research (González-Teruel, et al., 2021) which found much higher proportions, and amount of theory in information behaviour papers published in Korean information science journals which found a much lower proportion (see Table 1).

Several authors have gone beyond measuring the amount of the theory in information behaviour research to identifying the particular theories used. Theories most often used in information behaviour research are information behaviour theories; in particular, Kuhlthau's information search process model (e.g., 2004) tends to be the most used theory. Table 2 summarizes the findings of the studies that have explored this issue, showing the information behaviour theories that were most commonly used (among the top five most used theories in each study).

Another body of research relevant to the current study is the research into the disciplines of origin of theories in information science papers. For information science research overall, results are mixed. Pettigrew and McKechnie (2001) found that theories tend to come from the social sciences rather than information science, while Jeong and Kim (2005) found that more than half of theories used come from information science, with social sciences the next most influential discipline. Similarly, Julien and O'Brien (2014) found that information behaviour research tended to draw on social sciences theory, but increasingly draws on information science theories.

Recognising that use of theory in some studies is substantial, influencing study design and interpretation of results, while in other studies it is merely an obligatory citation, scholars have developed frameworks to analyse how theory is used. Jeong and Kim (2005) developed five degrees of theory using model and applied it to research in information science (Jeong and Kim, 2005; Kim and Jeong, 2006). Their degrees from least to most substantial use is spot citing, background review, theory discussion, theory application, and analytic evaluation. Most papers used theory for background review. When Kim and Jeong analysed information behaviour papers separately, they found that theory was used for background review as well. Wu et al. (2017) used the five degrees of theory using model to analyse theory use in Taiwan. They found that information behaviour research in Taiwan tended to be at the theory discussion level.

Taking a different approach, Vakkari (2008) analysed theory use in ISIC conference papers from the years 1996 and 2008 with the categories loose, medium, and strong connection. These categories referred to both explanation of the theoretical framework, and relationship between the results and the theoretical framework. He argued that the quality of theory use declined between 1996 and 2008. In 1996, there were 10 medium and 15 strong connection papers, while in 2008 there were 10 loose, 14 medium, and 10 strong connection papers.

Another model for analysing theory use was developed by Kumasi, et al. (2013). Their model was intended to be used by practitioners as well as scholars, and therefore uses language that is easy to understand and was inductively developed using more library-focused journals. Their analytic categories of theory talk, from least to most substantial, are theory dropping, positioning, diversification, conversation, application, testing, and generation. The authors simplify these seven types to a continuum of minimal, moderate and major use of theory. Kumasi et al. (2013) did not quantify their data; their aim was to discuss the types of theory talk rather than measure how many studies used each category. However, González-Teruel et al. (2021) used the analytic categories of theory talk to quantify use of theory in information behaviour. They studied Ibero-American information behaviour research and found that 44% of the papers had minimal or moderate use of theory, while 56% had major use of theory.

The ISIC conference, with Its focus on information behaviour, provides an influential and representative sub-set of research in this sub-field of information science. Analysis of the set of papers presented at the ISIC conferences, held biennially from 1996 to the present, has provided an overview of the structure and aspects of this sub-set of information behaviour work (McKechnie, et al., 2002; McKechnie, et al., 2005; McKechnie, et al., 2006; McKechnie, et al., 2008; McKechnie, et al., 2016; Julien, et al., 2018). This paper continues this longitudinal analysis, focusing on theory use in empirical papers presented at the ISIC conferences between 1996 and 2020. It thus continues the discussion of theory use begun by McKechnie et al. (2008) and Vakkari (2008).

Methods

All ISIC conference proceedings papers, published from the inception of the conference in 1996 to the most recent conference in 2020, were analysed to identify papers that reported empirical research. Papers that discussed theory or argued a point, without including empirical data, were not included in the data set. The total number of papers that fit these parameters was 243. The sampling frame of ISIC conference proceedings constitutes a good representation of authors conducting empirical research in information behaviour during this time period because the ISIC conference is the premier international information science conference focused on information behaviour.

We used content analysis to discern whether theory was included in the paper, the level at which each theory was used (i.e., substantial or unsubstantial), and the discipline from which the theory originated (including information science). Discipline outside of information science was determined by identifying the citation that describes the theory noted in the paper, searching for that citation in Web of Science, and identifying the Journal Citation Reports category for that citation. Where that process was not effective, we consulted Wikipedia for theory origins.

We chose Kumasi, et al.'s (2013) analytic categories of theory talk as a framework for analysing the level of theory use. A first round of coding was conducted by all four authors on a subset of papers published across two proceedings' years (2004 and 2020). Authors were consistent in coding for theories and discipline; however, coding for level of theory use with the seven theory talk categories was difficult. After discussion, we could not reach consensus about coding theory use into the seven theory talk categories. However, we agreed about substantial theory use as corresponding with three categories—application, testing, and generation, and unsubstantial theory use as also corresponding with three categories—dropping, positioning, and diversification. This solution is similar to the major, moderate, and minor theory use categories used by González-Teruel et al. (2021). We did not see instances of theory conversation, likely because our dataset consisted of only empirical studies. After recoding the subset of papers for 2004 and 2020 for substantial and unsubstantial theory use and coming to consensus, the four authors individually coded the remaining papers in the full data set. During that coding, questions or uncertainties were noted and were resolved by another author doing recoding.

Some papers used more than one theory. Each instance of a different theory was coded, but a theory was coded only once per paper. In a few cases, authors referred generically to information behaviour theories or some similar large basket of theories, without citation or specificity. No specific theory could therefore be identified in those instances. Some authors cited one or more theories without actually using them (therefore coded as unsubstantial uses). In Kumasi, et al.'s (2013) terms, that constitutes theory dropping. It appeared as though some authors felt compelled to briefly cite well-known information behaviour theories to establish their credibility, even if the theories were not actually used in the empirical work. A few authors, such as Suorsa and Huotari (2014), proposed new theories or frameworks arising from their empirical work; these were coded as substantial instances of theory.

Several issues were identified during this coding process. First, paper authors do not always clearly specify a theory that is being used, and often do not even provide a citation for a theory or framework used. In addition, authors may use different labels for the same theory. Also, it was occasionally challenging to disarticulate mere concepts from actual theories. We employed McKechnie, et al.'s (2001) definition of theory, so that if an author used any of the words theory, framework, grounded, or underpinnings when using a theory from within or outside of information science, then that was coded as a theory. However, mere concepts, such as habitus or principle of least effort were not coded as examples of theory use. Many empirical papers include multiple concepts which do not rise to the level of theory. Huvila (2020) is an excellent example of a highly conceptual paper, which focuses on the concept of credibility in a very substantial way. Todd (2006) is an example of theory use which is well-integrated throughout the paper. Heinström and Sormunen (2020) also provide an excellent example of a paper in which theory use is substantial. Kuhlthau et al. (2008) provide an example of theory testing.

Unsubstantial use of theory was sometimes evident, even when an author made claims that a number of specified theories informed this study or framed the research, without actually showing any links between the theory or theories and the study design or discussion of results. Other authors cited an exhaustive list of models and theories, as if to mention every information behaviour scholar who has preceded them, or as if to assert their knowledge of the field. It is possible that since these papers are prepared as conference papers, authors are taking shortcuts. Authors may also be assuming that their audience (readers) can in fact make necessary connections between theories used and the empirical work presented.

Results

The results identify the range of theories and their use in this set of empirical papers presented at the ISIC conferences, the disciplines from which these theories originate, and longitudinal trends in these findings. Table 3 shows that 243 full-length empirical papers were published in ISIC conference proceedings from 1996 to 2020, as well as the number of those empirical papers that use theory. The number of empirical papers range from 12 in 2016 to 29 in 2014, with an average of 19 papers per year. Most papers, about 78%, mention theory to some extent.

The papers in each of the 13 years of the ISIC conference use theory to varying degrees, although papers tend toward substantial use of theory. Of the 190 papers that use theory to some extent, 132 papers (69%) use theory substantially. Figure 1 shows the proportion of papers with at least some substantial theory use, papers with no more than unsubstantial theory use, and papers that lack theory use. The conference year when papers had the most substantial theory use was 2006 (12 papers or 75% of that year’s papers), whereas the conference year when papers had the least substantial theory use was 2018 (4 papers or 27%). In 1996 and 2012, only one paper at each of these conferences had no theory use, whereas 2008 and 2018 had the most papers with no theory use (10 papers or 38% and 6 papers or 40% respectively). A chi-squared test shows that there is a significant difference in the proportion of papers that use theory (substantially and insubstantially) and papers that use no theory, between papers written from 1996-2000 and papers written from 2016-2020 (χ2 = 4.102, df = 1, p = .043). Papers presented in 1996-2000 were more likely to use theory in some way than those presented in 2016-2020.

ISIC conference proceedings empirical papers use 229 unique theories. They range from a low of 10 unique theories in 2018 to a high of 53 unique theories in 2012 (see Figure 2). Whether these theories are used substantially or insubstantially varies.

Whereas Figure 1 shows the proportion of papers with substantial, unsubstantial, or no theory use, Figure 3 shows how substantially individual theories were used. Any given paper may use multiple theories in different ways, so this analysis explores instances of theory use in each conference year. Overall, there were 545 instances of theory use. Figure 3 shows the proportion of substantial to unsubstantial use of theories for each conference year. Overall, fewer theories are used substantially (203 or 37%) than insubstantially. The conference year 2004 shows the highest proportion of substantial use of individual theories (13 theories or 57% of the theories used in that year), whereas 2000 and 2014 have the lowest proportion of substantial theory use (12 theories or 23% and 12 theories or 22% respectively).

Research at ISIC draws from both information science theories and those outside the discipline. Of the 545 instances of theory use in the ISIC papers, more theories are drawn from information science in 2012 (59 IS theories or 82 %), whereas and more theories are drawn from outside other disciplines in 2018 (8 non-IS theories or 57%) (see Figure 4).

For any given conference, audiences are exposed to theories from a variety of disciplines, as many as eight (2006 and 2012) and no fewer than four (2004 and 2018) (see Figure 5).

Table 4 shows that of the 543 instances of theory use in ISIC empirical papers, most are from information science. This table also shows how clearly positioned information behaviour is in the social sciences. A chi-squared test shows a significant relationship between the discipline of the theory (information science or non-information science) and level of theory use (substantial or unsubstantial) (χ2 = 3.952, df = 1, p = .0470. Theories from information science were more likely to be used substantially than theories from other disciplines.

A wide variety of information science theories appear in the ISIC research (see Table 5). The most frequently used theories are Kuhlthau's information search process (e.g., 2004), Wilson's models (1981, 1999), and Dervin's sense-making (1992). These theories were used at each of the thirteen conferences. Using a chi-squared test, a significant difference was found between the three most frequently used theories and the rest of the field and the level of use of those theories. Papers that used the three most popular theories were more likely to substantially use those theories than papers that used other theories (χ2 = 6.20, df = 1, p = .013). This trend is even more apparent when we separated the five most popular theories and compared their level of usage to rest of the field. Authors who used the five most popular theories were more likely to substantially use them than authors who used other theories (χ2 = 14.25, df = 1, p < .001).

Use of the top five theories over the two decades of the conferences varies (see Figure 6). Kuhlthau's information search process (e.g., 2004), Wilson's models (1981, 1999), Dervin's sense-making (1992), and Ellis' model (1989) show a similar peak use in early conference years. Savolainen's everyday life information seeking (1995) is unique among the most frequently used theories in that it is trending toward more use during the last two conferences than in earlier conferences.

Discussion

Overall, empirical papers presented at ISIC tend to include at least some theoretical content, and many use theory substantially. This result is consistent with previous studies of theory use in ISIC papers and compares favourably with studies of theory use in the published information behaviour research. When focusing on how theory is used, ISIC papers tend to use theory substantially. This finding is similar to González-Teruel, et al.'s (2021) findings about Ibero-American information behaviour research. Our findings also support Vakkari's (2008) finding that overall theory use declined in 2008 as compared to 1996. Further, we found a significant decrease in theory use between the early and recent years of the conference. Overall, our findings support the claim that the ISIC conference proceedings feature rigorous high-quality research. However, the declining theory use suggests that authors may want to consider deepening the use of theory in their empirical research. Conference reviewers and organizers may want to encourage more use of theory in papers.

The finding that Kuhlthau's information search process (e.g., 2004) is the most used theory in ISIC proceedings is consistent with findings of previous studies. All previous studies have found it to be among the most used theories; in three studies it was the most used theory (McKechnie, et al., 2008; Lund, 2019, 2020), in one study, the second most used (McKechnie, et al., 2005), and in another study, the seventh most used (Pettigrew and McKechnie, 2001). The findings concerning the other top theories are consistent with these previous studies. The findings are also consistent with those of a study of the most common theories taught in reference courses in North America (VanScoy, et al., 2021).

Some of the most used theories, including Kuhlthau's information search process (e.g., 2004), Wilson's models (1981, 1999), Dervin's sense-making (1992), and Ellis' model (1989), all seem to be used less in recent years than in earlier years. By contrast, Savolainen's everyday life information seeking (1995) has garnered greater attention recently. Future longitudinal research could confirm this trend and comparative research on the broader information behaviour literature could confirm whether this trend extends beyond the ISIC proceedings.

Non-information science theories that are used in empirical papers presented at the ISIC conferences primarily originate in the social sciences. The fields contributing the most theory are sociology, psychology, communication,business/management, and education. This list has significant face validity and would be of no surprise to those working in the information behaviour area.

An important consideration in evaluating use of theory is that substantial theory use is not always warranted. In some cases, unsubstantial theory use or lack of theory is appropriate, depending on the method or approach of the study. In one of the ISIC papers analysed, Sakai, et al., et al. (2012) make a case for actively choosing not to incorporate theory into their study. Rather than forcing participants' experience into an inauthentic framework, they argue that ‘we need to put our theoretical concern aside for the purpose of understanding the phenomenon under study and explicate the concepts [in] language that the participants… use'.

The challenge of using Kumasi, et al.'s (2013) theory talk framework suggests that more work could be done, either to better define the theory talk categories, or to develop a tool to analyse use of theory in empirical papers. Scholars who have developed frameworks for analysing theory use acknowledge the challenge of assigning use of theory to specific categories. Jeong and Kim (2005) state that ‘in some cases, it will be somewhat subjective or cognitive judgment' (p. 56), and Kumasi, et al. (2013) explain that their work ‘illustrates the complexities of trying to place studies into distinct categories where there might be overlap in interpretation' (p. 179). Although González-Teruel, et al. (2021) and Wu, et al. (2017) describe their consensus and inter-rater reliability process, they do not mention any challenges they faced in achieving it. However, both González-Teruel et al. and the authors of the current study opted to use broader categories.

While Kumasi et al.'s (2013) theory talk framework was not clear enough to use as an analysis tool, it was immensely useful as a framework for discussion, with its memorable and descriptive category names. It could be useful in helping both students and experienced scholars deepen the use of theory in their research. Coding for how theory was used stimulated enlightening conversations about the purpose of theory and exemplars of substantial theory use. The theory talk categories provided terminology that facilitated this discussion.

Conclusion

These analyses lead us to conclude that the ISIC conference continues to merit its reputation as the premier global conference in information behaviour. Papers presented at the conference continue to be informed by the top theoretical frameworks from information behaviour, as well as lesser-known theories from information science, and theories from other disciplines. Along with González-Teruel, et al. (2021) and Wu, et al. (2017), this study responded to a need for intensively examining theory use in information behaviour research, focusing not just on counting theory use, but examinine how extensively it is used. Although most ISIC empirical papers use theory in substantial ways, there is room for improvement. Many empirical papers mention or cite theory, but do not apply it in research design or in data interpretation. In addition, when theory is used to inform the work, authors should be more explicit about how that theory connects with the study design and data interpretation. Furthermore, use of theory has declined in recent years. Focused attention to substantial theory use by students and more experienced scholars alike, can only increase the rigour and influence of information behaviour research within information science and beyond its boundaries.

Acknowledgements

The authors acknowledge Colin Camp's contribution of statistical analysis and creation of figures.

About the author

Amy VanScoy is Associate Professor in the Department of Information Science at the University at Buffalo, 525 Baldy Hall, Buffalo, NY, USA. She received her Ph.D. from the University of North Carolina at Chapel Hill. Her research interests are in professional ideologies and use of practitioner and formal theories in information work. She can be contacted at vanscoy@buffalo.edu.
Heidi Julien is Professor in the Department of Information Science at the University at Buffalo, 530 Baldy Hall, Buffalo, NY, USA. She received her Ph.D. from the University of Western Ontario in Canada. Her research interests include information behaviour, information and digital literacy, and research methods. She can be contacted at heidijul@buffalo.edu.
Annette Buckley is a doctoral student in the Department of Information Science at the University at Buffalo, 534 Baldy Hall, Buffalo, NY, USA. She is also the Research Librarian for Business and Economics at the University of California, Irvine. Her research interests include information availability and its impact on reference interactions. She can be contacted at annetteb@buffalo.edu.
Jon Goodell is a doctoral student in the Department of Information Science at the University at Buffalo, 534 Baldy Hall, Buffalo, NY, USA. His research interests include the impact of technology in medical and academic libraries. He can be contacted at jongoode@buffalo.edu.

How to cite this paper