Exploring and understanding the causes of competitive intelligence failures: an information behaviour lens

Introduction

Competitive intelligence is very valuable in today’s marketplace (Jaworski, Macinns and Kohli, 2002; Tsitoura and Stephens, 2012; Goria, 2017). Competitive intelligence involves the legal and ethical acquisition of information sources that are publicly available, in order to develop data on competitors, competition, and the market place (Smith, Wright and Pickton, 2010; Nasri and Zarai, 2013; Muñoz-Cañavate and Hípola, 2017). It includes the monitoring of the external environment for opportunities, threats and developments (Strauss and Du Toit, 2010; Tsitoura and Stephens, 2012; Jin and Ju, 2014; Sandal, Gupta, Sharma, Sepat and Kumar, 2017). According to Bergeron and Hiller (2002, p. 355) competitive intelligence refers to ‘the collection, transmission, analysis and dissemination of publicly available, ethically and legally obtained relevant information as a means of producing actionable knowledge’. In the same vein Kahaner (1997, p. 16) defines competitive intelligence as ‘a systematic program for gathering and analysing information about your competitors’ activities and general business trends to further your own company’s goals’. There exists variations of the competitive intelligence life cycle, also referred to as competitive intelligence process, with some authors naming only a few phases, while others present many phases with various naming conventions (Muller, 2002; Bose, 2008; Botha and Boon, 2008; Pellissier and Nenzhelele, 2013; Du Plessis and Gulwa, 2016).

Competitive intelligence professionals are the people responsible for the development of a systematic programme that guides the articulation of intelligence needs, collection, analysis and processing of both external and internal information in order to produce actionable intelligence which supports the strategic and decision making process of the organisation (Du Toit, 2015; Jeong and Yoon, 2017). They often act on behalf of end-users. People who execute tasks related to competitive intelligence often do not claim the title competitive intelligence professional (Baars and Kemper, 2008).

It is the objective of every competitive intelligence professional to produce a product that is successful overtime and avoid failures, however regardless of all the efforts that are directed to the competitive intelligence process, there are instances were concerns about failure are reported (Fleisher and Bensoussan, 2003; Fleisher and Wright, 2009; Nasri, 2011; Garcia-Alsina, Ortoll and Cobarsí-Morales, 2013; Du Toit, 2015; de Almeida, Lesca and Canton, 2016; Jeong and Yoon, 2017). Competitive intelligence failures can have devastating organisational effects which includes a loss of image, reputation, opportunities and profits (Tsitoura and Stephens, 2012; Garcia-Alsina et al., 2013). Reading these reports of concern and the failures noted, it seems as if the wealth of insight gained through information behaviour studies over several decades (Case and Given, 2016; Cole, 2012; Ford, 2015) can extend our understanding. Although information activities such as identification of information needs and information searching are sometimes noted in competitive intelligence articles, it is not with information behaviour insight.

There is limited literature combing competitive intelligence and information behaviour; exceptions being Erdelez and Ware (2001), Garcia-Alsina et al. (2013) and Cerny (2016). Even less has been published on competitive intelligence failures attributed to information behaviour (Sandal et al., 2017). Maungwa (2017) thus initiated a study to explore the understanding and causes of competitive intelligence failures from an information behaviour lens. [He/she] asked: What are the causes of competitive intelligence failures from an information behaviour lens?

This paper reports on selected findings. It will cover clarification of concepts, literature review, choosing the theoretical frameworks, research design and methods, findings with a focus on information activities causing competitive intelligence failures, recommendations and a conclusion.

Clarification of concepts

Four terms are key to understanding an information behaviour perspective to competitive intelligence failure.

Information behaviour: According to Case and Given (2016, p. 5) information behaviour includes the totality of other unintentional or passive behaviours (such as encountering information) and purposive behaviour that do not involve seeking, such as actively avoiding information. This is in addition to active information seeking where Wilson (2000, p. 49) defines information behaviour as ‘the totality of human behaviour in relation to sources and channels of information, including both active and passive information seeking, and information use’. For this paper we accept that information behaviour focuses on people’s information needs, and particularly how they seek, use and manage information in their different roles in various facets of their everyday lives, and that it includes all information activities note by Case and Given (2016) and Wilson (2000).

Competitive intelligence failures: Competitive intelligence failures are a result of inaccurate analytical judgement (Jensen, 2012, p. 261). In addition Johnson (2004, p. 60) views competitive intelligence failures as ‘flaws in analysis resulting from poor or missing data’, and the incorrect articulation of key intelligence needs.

Key intelligence needs: Herring (1999, p. 6) and Muller (2002, p. 2) define key intelligence needs as strategic and tactical requirements that are needed to achieve organisational objectives. He further states that key intelligence needs are considered as issues of critical importance in competitive intelligence. In addition Clayton, Lin and Pitt (2011, p. 1529) affirm that identifying key intelligence needs require identifying relevant indicators, asking the correct questions and recognising early warning signals. Du Toit (2015, p. 112), coming from a library and information science background, and with expertise in information and knowledge management, affirms key intelligence needs to be the organisation’s information needs.

Information needs: An information need arises when an individual senses a problematic situation or information gap, in which his or her internal knowledge and beliefs and model of the environment fail to suggest a path towards the satisfaction of his or her goals (Case, 2007, p. 333). This interpretation can be supplemented with the view of Wilson (1999, p. 252), which states that

information need is not a primary need, but a secondary need that arises out of needs of a more basic kind; and second, that in an effort to discover information to satisfy a need, the enquirer is likely to meet barriers of different kinds.

Literature review: information behaviour, competitive intelligence and competitive intelligence failures

There exists a strong body of literature on competitive intelligence (Fleisher and Bensoussan, 2015; Sepahvand, Nahzarpoori and Veisi, 2016) and less on competitive intelligence failures per se (Du Toit, 2015; Jeong and Yoon, 2017; Sandal et al., 2017). Recently the work of Cerny (2016), Keiser (2016), Kirkwood (2016), Muñoz-Cañavate and Hípola (2017) was published, which covers related issues such as the fundamental motivation to share knowledge among employees and on the information literacy of competitive intelligence students.

As background to the discussion and the scope of reported failures a literature search was done on: ABI/INFORM; Library, Information Science & Technology Abstracts (LISTA); Library and Information Science Abstracts (LISA); ScienceDirect; Web of Science and Emerald Insight.

The consulted literature on information behaviour and competitive intelligence that could guide the study was categorised under (i) competitive intelligence failures attributed to information activities, (ii) individual and organisational factors causing competitive intelligence failures. The literature below explains the scope of competitive intelligence failures, and the value that an information behaviour lens can have in investigating the causes of competitive intelligence failures. Key causes of competitive intelligence failure are summarised in Table 1.

Issues leading to failures in the collection of completive intelligence thus range from the limited scope of sources consulted, and a preference for the convenience and ease of access of internal sources, negligence in validation of information, and the amount of time needed to seek for information leading to overload and inability to deal with information richness which affects the time available for information analysis. Skills of individuals involved in competitive intelligence, such as poor information retrieval skills, inability to apply appropriate methodologies and poor communication skills also contribute to competitive intelligence failures.

Choosing a theoretical framework

Studying competitive intelligence failures through an information behaviour lens required two issues to be addressed: (i) the process of competitive intelligence, and the (ii) information behaviour that might cause competitive intelligence failure. The study therefore considered both information behaviour models and competitive intelligence life cycles for guiding theoretical frameworks. Considering relevance to the study, several information behaviour models were consulted including their underlying components (information needs, information seeking, information exchange, and various influential factors) as noted by Wilson (1981; 1999), Choo (2016), Ingwersen and Järvelin (2005), Bawden (2006) and Meyer (2016). These components are implicit in studies associated with competitive intelligence failures, and can be used to develop a competitive intelligence theoretical framework (Botha and Boon, 2008; Dishman and Calof, 2008; Du Plessis and Gulwa, 2016). Maungwa (2017) chose two frameworks; an adapted version of the competitive intelligence process life cycle based on cycles proposed by Botha and Boon (2008) and Bose (2008); and an eclectic combination of the Wilson (1981, p. 4) model and the information seeking of professionals model developed by Leckie, Pettigrew and Sylvain (1996, p. 180). The chosen competitive intelligence cycle, and information behaviour model, were deemed relevant for this study since they have features that are related to the research question and sub-questions of the original study by Maungwa (2017). The adapted Botha and Boon (2008) and Bose competitive intelligence cycle informed questions about data collection, articulation of intelligence needs, reporting and disseminating. The adapted Wilson (1981) and Leckie et al. (1996) informed questions about data collection, information seeking and feedback from the end-user. The theoretical framework also focuses on various activities in the competitive intelligence process, and how the professionals’ work roles and tasks influence their information seeking behaviour. The two frameworks guiding the study on which this paper report are shown in Figures 1 and 2. When starting with the study, obvious theories that could guide the study were not noted (Maungwa, 2017). Analysis of findings did, however, reveal theories that can be considered for further research, which are mentioned in the discussion.

Research design and methods

Table 3 presents a summary of the research design. This includes the research paradigm, research approach, the time frame of data collection, research method, methods of data collection, sample method and participants.

Findings form empirical component

Findings from the participant profile (limited descriptive data) and the interviews (qualitative data) are presented.

Participant profile

Participants widely varied in their profile and demographic details in terms of highest education qualifications, formal information training, years of experience in competitive intelligence, job descriptions and professional position. Table 3 shows the demographic details of the participants; competitive intelligence professional is abbreviated to CIP, and competitive intelligence educator/trainer is abbreviated as CIE.

Findings from interviews

Thematic analysis was used to analyse the responses from the qualitative data obtained from the individual interviews, and to further classify and present themes and categories that relate to the data (Braun and Clark, 2006, p. 6; Alhojafilan, 2012, p. 40). The study maintained anonymity by assigning a pseudonym to each participant (Elger and Caplan, 2006; Crow, Wiles, Heath and Charles, 2006, p. 9). Table 4 presents the causes of competitive intelligence failures identified from the interviews, categorised as information activities, organisational factors, and individual factors. Some of the participants inferred causes of competitive intelligence failures by mentioning methods, techniques and information activities that reduce the risks of competitive intelligence failures. It is imperative to note that participants made use of the concept information needs and key intelligence needs interchangeably. For our interpretation, key intelligence needs refer to the information needs of the organisation (Du Toit, 2015, p. 112).

Table 4 shows all the factors causing competitive intelligence failures as identified from the discussions by the participants; the proceeding section will only present evidence and verbatim quotes for the most prominent information activities causing competitive intelligence failures.

Identification, determination and articulation of information needs

(a) Inadequate articulation of information needs: The correct articulation and identification of information needs is critical for the success of the competitive intelligence process. A lack of deeper understanding of the organisation’s objectives, goals and mission deprives the competitive intelligence practitioner of identifying the true information needs of the end-user. Proper and accurate procedures for question-negotiation in determination of key intelligence needs should be followed. Jane mentioned that the key step to the correct articulation of the end-user’s information needs, is to be well informed about the organisational strategy:

The competitive intelligence professional must have a very good understanding of the organisation, be well informed about the strategy of the business and have good relations with the senior management who can then be able to translate the intelligence needs to the competitive intelligence professional. (Jane)

For Cynthia the lack of understanding of the organisation prevents the competitive intelligence practitioner to ask the end-user the correct questions:

A lack of information and knowledge about the organisation causes a poor expression of information needs, since the competitive intelligence professional does not know exactly what to ask senior management.(Cynthia)

(b) Forming own notion of information needs during intelligence interviews with the end-user: During intelligence interviews, most participants admitted to omitting information that does not support their views, knowledge base and beliefs therefore forming their own notion. Both Anna and Elena confirmed that there is a practice among competitive intelligence professionals of omitting important information that has the potential to provide them with a deeper understanding of the end-user’s information need.

When they receive the answers, they form their own notion or take the information that support their own view. They actually do not take the entire picture. (Elena)

Sometimes people ask an intelligence question, and then from what they are told they only select what they already know, they do not consider the whole picture, they just want to confirm with their own knowledge base and understanding. (Anna)

(c) Applying inadequate questioning techniques during intelligence interviews: The competitive intelligence professionals should have a good understanding of the tools and standards for collecting and assessing intelligence, the tools and standards must be tailored in the context which the intelligence is to be collected. During intelligence interviews with the end-user, the competitive intelligence professional should ask the correct questions in order to determine the true key intelligence and information needs of the end-user. Brandon mentioned that competitive intelligence professionals should conduct a detailed needs analysis.

I usually plan meetings with senior management and ask questions such as: ‘What is your biggest problem at the moment?’ ‘What is your biggest threat?’ ‘Why do you consider that a problem or a threat?’ (Brandon)

You have to go there and ask very specific questions such as ‘What is it that you need?’, ‘What do you need this information for?’ ‘How do you want the deliverables to be packaged?’ ‘Who in the organisation will be using this intelligence, and what will they be using it for?’ (Bob)

(d) Fragmentations of information needs as opposed to holistic views of information needs: There seems to exist practices of purposeful fragmenting of information needs when some competitive intelligence professionals are told about needs. This is usually done when more than one competitive intelligence practitioner are asked to seek for information. Clint affirmed that the fragmenting of information needs (providing tasks in specifics) is done for valid reasons. This practice however, often, then leads to the partial understanding of an information need(s).

When I task my team, I am very specific in what I tell each individual. I do not get my entire team together and give them the full picture, because they will already know what the answer is. I have been working in the field for a very long time, I already know what we are looking for, but the younger competitive intelligence professionals do not have the expertise that I have, and they cannot see the full picture. (Clint)

(e) Negligence to use source evaluation methods and techniques to verify data and information collected: To avoid collecting the wrong information, it is important that any misunderstanding is clarified with the end-user. Two participants specifically mentioned that competitive intelligence professionals seldom consult with the end-user to clarify any ambiguity and misunderstanding: 

A lot of individuals collect data without clarifying with management if the data they are collecting is actually what is required. (Steven)

The problem with humans is that we never go back and clarify what we initially heard. (Abel)

(f) Fluctuation of intelligence needs: Organisations operate in unstable environments and both internal and external influential factors can impact on the organisations’ information needs and needs changing during the period in which competitive intelligence is collected. It is therefore imperative that competitive intelligence professionals seeks confirmation to ensure that initial information need(s) have been correctly understood and do not change or divert during the information seeking process. John sustained that although the initial information need can be correctly articulated, however due to instability of the organisations’ environment, the information need can expand or take a different direction. The confirmation of the intelligence needs should be maintained at the start of the process as well as during data collection. Changes in information needs and the iterative nature of information seeking and retrieval is well captured in the literature on search heuristics (Savolainen 2017; Wang, 2011; Harter, 1986), browsing (Ellis, 1989), and the progressive nature of some situations, and in fact a theory of progression (McKenzie, 2003).

A lot of individuals collect data without clarifying with management if the data they are collecting is actually what is required, and if they are still on track with the current project. (John)

Application of effective information seeking and retrieval skills

(a) Lack of sophisticated information seeking and retrieval skills: competitive intelligence professionals should have a practical background in data collection, information seeking and retrieval. Poor retrieval skills deprives the competitive intelligence professional of exploring vast avenues of information, thereby limiting the amount of information that can be accessed. Both Brandon and Cynthia highlighted the importance of good information seeking practices.

It is very essential that the competitive intelligence professionals have the correct training, they might not have been trained as librarians or data analysts, but it is very essential that they undertook some courses on data collection. (Brandon)

The competitive intelligence professional should always pick people who they trust to collect data; these should be people with some degree of good information seeking practice. (Cynthia)

(b) Inadequate translation of key intelligence needs into search strategies: the translation of information needs into effective search strategies is critical to obtain the correct information. Collecting intelligence requires specialised efforts in search tactics and search strategies (Harter, 1986; Savolainen, 2016). Steven expressed concern on how competitive intelligence professionals ignore the complexities and importance of effective search strategies.

A lot of people think that competitive intelligence simply refers to information communication technology (ICT). They always compare competitive intelligence to computer systems, and say words like ‘well I have a computer, if I want information, I can simply just go pull it out of the computer’. (Steven)

Management of information richness

(a) The inability to handle information richness leading to information overload: due to limited time and the amount of publicly available information, and various channels and avenues to obtain information, competitive intelligence professionals direct most of their efforts to data collection and less efforts are exerted to data analysis, which leads to experiences of information overload.

Information overload should be considered. With the growing amount of data and information on the web, as well as the advancement in big data, there is so much information and data available. (Stephen)

Evaluation and triangulation of collected data and information sources

(a) Necessity to evaluate and triangulate data and information sources. For competitive intelligence information to have an impact in the decision making process, the information should be factual and accurate. To eliminate errors and redundancies, the collected data should be evaluated. Information source evaluation can be used as a method of triangulation. Five participants mentioned the use of source evaluation methods and techniques to reduce errors. According to Tyrone it is important for competitive intelligence professionals to have the ability to distinguish what is false and what is factual. Steven and Anna also shared their opinions.

You have to make sure that data can be validated, we use triangulation, by collecting data from three different sources, and if they all say the same thing then we have successfully validated our data. (Steven)

To avoid any competitive intelligence failures, we normally use credible sources, we know and have a list of sources and organisations that are fairly validated and reliable. We also cross check if there are any contradictions in the data we collect. (Anna)

Discussion of findings and adapted framework

From the findings and the literature review it is clear that much can be learned from the information behaviour literature on context (Savolainen, 2017) and the importance of the organisation (Choo, 2016). These influence the reasons why competitive intelligence is collected, the urgency for collection, the purpose and the need to consider a bigger goal and vision. The findings also bring to play the need for deeper understanding at the time of question negotiation (Taylor, 1968; 1991) and the necessity to enhance the exploration of dormant information needs such as applying the Johari window approach (Shenton, 2007). Appropriate question negotiation skills, sensitivity to dormant information needs, and overall communication skills can support competitive intelligence professionals in their interpretation of the key intelligence needs expressed by end-users. Such communication problems are well explored in textbooks on reference work, early day work on online searching and the role of the intermediary (Harter, 1986; fid04Fidel, Pejtersen, Cleal and Bruce, 2004). The realities of complication in the communication, processing, interpretation and verbalisation of information needs and translation into search terms, and search strategies have been core issues (Harter, 1986; Dervin, 2015). Apart from the lack of appropriate question negotiation and information seeking and retrieval skills noted by participants, it is clear that competitive intelligence professionals need a solid background in theories of library and information science and information behaviour, which includes knowledge of theories of serendipity and social capital (Fisher, Erdelez and McKechnie, 2005). Although game theory and portfolio theory have been mentioned with regard to competitive intelligence (Stirling, 2012; Perez-Oviedo, 2015) these did not seem appropriate to guide the study. Based on analysis of findings reported here, the work of Case and Given (2016), Ford (2015) and Fisher et al. (2005) can be used to trace theories that can guide future research. In addition, a clear understanding of organisational goals and visions is important. This is in line with work by Treviño and Weaver (1997) who stress the need for competitive intelligence professionals to adhere to their own companies’ practices, objectives and guidelines; to identify themselves and their organisation prior to interviews; and to respect requests for confidentiality.

Recommendations

The following recommendations are provided for further exploration:

(a) Exploration of the concept and the manifestation of intermediation (also referred to as proxy information seeking) in competitive intelligence. Due to the sensitive nature of competitive intelligence (i.e. after all about competition and risk), intermediary information seeking in competitive intelligence gives rise to two issues (i) deliberate partial sharing by the end-user of information on key intelligence needs, which are interpreted as information needs with the competitive intelligence professional who needs to seek information, i.e. collect intelligence; (ii) a competitive intelligence professional searching on behalf of the end-user, which leaves openings for misunderstanding, misconception, forming an own-notion of information needs according to own views and experiences. The latter is subject to all the problems that have been reported by librarians and information professionals searching on behalf of users, and additional problems inherent in the need for secrecy and confidentiality in competitive intelligence.

(b) Training in data collection, and information seeking and searching on an advanced level, but with specific acknowledgement of the scope of information resources and depth required by competitive intelligence. Competitive intelligence professionals should have both a strong theoretical background in information seeking and retrieval, as well as advanced practical skills in information retrieval (Carr, 2003).

(c) Development of skills in the assessment procedures required for the competitive intelligence process and in the screening of individuals who are competent in this process. The success of competitive intelligence relies on the individuals who conduct the process. Application of stringent individual assessment methods and procedures can help to strengthen the process. To improve on the quality of the competitive intelligence process, there should be assessment methods that determine the level of understanding and capabilities of individuals in competitive intelligence related aspects (e.g. information seeking, data collection, reference interviews, and information sources). Such a framework might also help to equip competitive professionals to cope better with information richness. The work of Ford and Mansourian (2006) on information seeking can improve our understanding on how people fail to find information.

(d) Informing competitive intelligence educators and trainers on the need to draw on complimentary disciplines such as library science, information science and the sub-discipline of information behaviour. Insights from these, their models, theories and findings, can explain reasons for competitive intelligence failures, and pave the way for further research and developing procedures, a theoretical base and skill sets that might be incorporated into the training of competitive intelligence professionals. Insights from information behaviour and library reference work, especially need to feature in competitive intelligence training, starting with the work of Peterson (1997), Straw (2000) and Coonine and Levine (2013).

Conclusion

Competitive intelligence are prone to many failures. As shown in this paper such failures often relate to information activities that can be studied from an information behaviour lens. To ensure that the competitive intelligence process is conducted efficiently, competitive intelligence professionals should consider methods, procedures and techniques tailored for the information activities embedded in competitive intelligence and relevant to the context in which competitive intelligence manifests in a manner that can minimise failure. Furthermore competitive intelligence educators and trainers should raise awareness of such failures and shape the training of competitive intelligence professionals to benefit from interdisciplinary input specifically information behaviour, and library and information science.

About the author

Tumelo Maungwa is a lecturer in the Department of Information Science, University of Pretoria. He holds a master’s degree in Information Science from the University of Pretoria and is currently a doctoral candidate. His research interests are in competitive intelligence and information behaviour. Postal address: Private bag X20, Hatfield, 0028, Pretoria, South Africa. He can be contacted at tumelo.sebata@up.ac.za
Dr. Ina Fourie is a full professor in the Department of Information Science, University of Pretoria. She holds a doctorate in Information Science, and a post-graduate diploma in tertiary education. Her research focus includes information behaviour, information literacy, information services, current awareness services and distance education. Currently she mostly focuses on affect and emotion and palliative care including work on cancer, pain and autoethnography. Postal address: Private bag X20, Hatfield, 0028, Pretoria, South Africa. She can be contacted at ina.fourie@up.ac.za

How to cite this paper