Elicitation of requirements for innovative visual patent retrieval based on interviews with experts

Introduction

Patents represent the largest technical collection (Alberts, et al., 2011) used to advance innovative processes. Accordingly, patent search is considered an important task for protecting one's own inventions, but also for identifying competitors' research interests. Yet, there are challenges for patent retrieval that arise from the specifics of the patent domain. On the one hand, the convergence of different disciplines causes a heterogeneity in terms of terminology, which is further complicated by the multilingual and legal background of the patent domain. On the other hand, patents are characterized by their multimodal structure with both structured and unstructured data. It is therefore obvious that conventional text-based retrieval methods do not meet the needs of this domain completely. Thus, new approaches must be found and implemented to expand retrieval facilities. One solution that has been focused on is integrating patent drawings into the patent retrieval process. The purpose of patent drawings is to illustrate technical details of the invention to provide a better understanding of the invention by showing the individual components in their relation to each other (Walter and Schnittker, 2016).

The presented work addresses this issue and is part of the ExpResViP (“Exploitation of Research Results through Visual Patentretrieval“) project, that is funded by the German Ministry of Science and Technology). In this project, the development of an innovative visual search for patent retrieval is intended on the basis of the automatic recognition of text-image-relations and similarities between patent drawings.

The goal of this study is to define requirements for innovative visual patent retrieval. This is achieved by conducting and analysing interviews with experts from different professional fields within the patent domain as the primary data source. The interviews are used to find out how the experts integrate patent drawings into the search process, what meaning the relation between text and drawings has and how they estimate a visual search which does not exist yet. The interview with experts as a special form of a survey method, requiring a special approach and methodological consideration, is a significant aspect of this work. The method used in this study is called Q-rEx and was developed and applied first in this project because of the need for an appropriate way to derive requirements for an innovative system that does not exist yet from the heterogeneous statements of experts from a highly specific domain. Since there is no method available that takes all these aspects in account, Q-rEx is therefore adapted to the research context and combines qualitative data analysis with the requirements engineering process.

Related Work

The research objective covers two areas. One is patent image retrieval that is being addressed because of the integration of patent drawings. The other one is methodological and includes qualitative data analysis methods that are used to derive requirements for an innovative system.

Patent Image Retrieval

By speaking of visual patent retrieval, the usage of patent drawings during the search process is meant. Patent drawings can be technical drawings, flowcharts, and circuits (Piroi, et al., 2011). While they may not contain any text, the individual components must be provided with reference signs referring to the description and the claims. Also, they are kept very abstract and without colour (Walter and Schnittker, 2016). The following figure shows an example of how a patent drawing can look like:

The usage of patent drawings is based on the concept of visual information seeking, which refers to the use and integration of visual elements in the search process (Albertson, 2015). With respect to the patent domain, however, most empirical research focuses on the technical implementation possibilities of patent image retrieval. These relate, among others, to approaches for indexing patent drawings (Bhatti et al., 2018; Gialampoukidis et al., 2019) and to the application of machine learning algorithms to retrieval (Kravets, et al., 2017). For this work, user-centred approaches with regard to the investigation of patent search behaviour are more relevant. Looking at the existing research, it becomes apparent that the focus is rather on the general search behaviour, but not on the concrete integration of patent drawings. Nevertheless, even in these user studies, the relevance of patent drawings becomes evident. Hansen (2011) found out that drawings are seen as technical details of the invention in a clear manner and that they are considered an essential part of evaluating the patent in some areas. Further, experts noted that the required information is available both in text and in image and that many search tasks require the consideration of both modalities. Continuing, Joho et al. (2010) discovered that an image search is perceived as a useful feature for the users and that it should be possible to search for technical drawings in an ideal patent retrieval system.

Qualitative data analysis within requirements engineering

The requirements engineering process focuses an approach that captures, classifies, and organizes the users’ needs of a system so that these can then be translated into requirements or functions that are essential for the development (Ebert, 2014). The whole process is usually not linear, and activities can be interdependent. As a result, the process can be at risk of not being traceable enough. In particular, requirement specifications often cannot be traced back to their origins (Kaufmann and Riehle, 2019). For this reason, it has been recommended that methods from other scientific disciplines can be used to make the analysis of the requirements more comprehensible (Cheng and Atlee, 2007). Most studies on this subject make use of the grounded theory methodology (Kaufmann and Riehle, 2019; Würfel, et al., 2016). However, this procedure is considered very time consuming, and the typical categorization process often needs to be adapted.

Interviews with experts for the requirements elicitation

Interviews with experts can be regarded as a useful method for the present research project, as they allow access to specific knowledge in a particular field. The aim is to collect knowledge in a structured way, which is why semi-structured interviews are considered useful for the project. These are based on a guideline that serves as orientation and contains all questions on relevant aspects regarding the research topic, whereas the order of the questions is variable (Misoch, 20019). The guideline is useful because it makes the interviews comparable, but does not restrict them too much, so that spontaneous questions can be included.

In this study, the role of experts and their highly specialized knowledge in a certain field of action is extensively reflected on to identify the distinctive characteristics of expert interviews for requirements elicitation. Regarding interviews with experts, there is a discrepancy between the frequency of their use and the lack of methodological consideration and reflection. Existing efforts attempt to find a unifying methodological approach for expert interviews, but this would imply that the interviews are not dependent on the present context (Kassner and Wassermann, 2002). Moreover, they are often conducted on the assumption that the experts would provide objective knowledge that can be elicited in its pure form (Bogner and Menz, 2002a) and not be influenced by, for example, the interview situation. But especially the interaction during the interview caused by the perceived role of the interviewers is seen as fundamental for all processes of data production (Bogner and Menz, 2002b). At this point, it is necessary to find out which forms of knowledge are to be elicited and which interview technique is suitable. Since the aim is to generate innovative ideas, it can be useful to show professional competence to the experts as interviewers, so that they assume the interviewers have enough prior knowledge. This is often achieved by perceiving the interviewers as experts from a different domain with shared interests, for example, through their employment at a university (Bogner and Menz, 2002b). For some aspects, however, it can also make sense to appear as a non-expert in the patent domain, so that the experts can formulate their statements in great detail. It becomes clear that it is necessary to clarify who is considered an expert in the given research context, so that it can be discussed which forms of knowledge are to be collected and which interview guidance would be appropriate.

Research Goals and Methods

The elicitation of the requirements for an innovative system is achieved through an appropriate methodological approach. This results from several sub-objectives of this work.

Goals

In this study, a user-centred approach is taken by interviewing patent domain experts regarding their integration of patent drawings. Also investigated is the extent to which the relationship between patent drawings and text has meaning for the search and in which situations the visual search would be preferred to the textual one. The primary goal is to draw requirements for a visual patent retrieval system from this.

Since there is no methodological approach available as a template that enables the derivation of requirements from heterogeneous statements by patent experts, a specific method adapted to the research context must be developed. This method is referred to as “Q-rEx” and integrates qualitative data analysis, requirements engineering and the specificity of the group being targeted, the experts from a specific context, namely the patent domain.

Introducing Q-rEx

Q-rEx (Figure 2) is applicable in a context where qualitative interviews with experts are the main sources of information for deriving system requirements for an innovative system and thus qualitative methods of social sciences and techniques of requirements engineering are combined. The experts have in common that they are working within the same domain (here: patent domain). The differences lie in their professional field within this domain (here: industry, research, patent attorneys, patent office). For the application of the method, a certain degree of prior knowledge about the domain is of benefit.

Q-rEx starts with the discussion of the status 'expert'. Here, it must be determined who is considered an expert in the respective context, what knowledge is to be collected and which interview techniques are to be applied most effectively. As Q-rEx addresses the lack of methodological discussion of expert interviews, this aspect is particularly important in order to better assess the expert interview situation and to take appropriate measures such as defining the role of the interviewers, which are eventually best suited to elicit innovative ideas.

Furthermore, Q-rEx is characterised by iterative and constant coordination and validation because of the participation and cooperation of several members throughout the process. These can be working in a similar as well as in a different field. Several researchers working together becomes extremely relevant when analysing the interviews using techniques of the qualitative content analysis. Here, the categorisation process for analysing unstructured text forms the focus of the data analysis (Mayring, 2015). There are various possibilities for qualitative data analysis, of which a mixed form of deductive and inductive category formation is considered useful in this study (Kuckartz, 2018; Mayring, 2015). An essential aspect of the process is consensual coding (Kuckartz, 2018). It is intended that the categorisation is first carried out separately by several researchers and then discussed together until agreement is reached. This is regarded as reasonable to ensure the reliability of the data. To also guarantee the credibility of the data, Q-rEx relies on peer debriefing (Misoch, 2019). This involves researchers who have a certain expertise in data collection and analysis, but who are not directly involved, taking a critical look at the data obtained and their interpretation.

Within Q-rEx it is also recommended to include additional data sources. These can be workshops (here: e.g., presentation of work packages/intermediate results) and other project-specific documents (here: best practices/idea collection). These can also be analysed qualitatively and be used as a supplement or as reinforcement to the interviews (here: reinforcement).

The derivation of the requirements mainly takes place along the category system. These are formed in several iterations until a natural language formulated requirement can be obtained. The final requirement should be based on a sentence template to give the requirement catalogue a structure that is easier to read and verify (Ebert, 2014). Also, in this phase of Q-rEx, the involvement of several project members is recommended. Especially the developers of the system should be consulted to be able to prioritise the requirements.

Analysis

For the data generation, the interviews were conducted and recorded in a virtual room. Several researchers were constantly involved, so that at least one person was able to take notes. The interviews were conducted using specific strategies meaning the role of the interviewers. As described above, the role of the interviewer as an expert of a different knowledge culture with existing prior knowledge in combination with the role as a non-expert is taken for the elicitation of innovative ideas. The video recordings were transcribed shortly after conducting the interview and compared with the notes to facilitate the process.

The qualitative analysis was carried out based on transcripts of eleven interviews with experts from the patent domain as the primary data source. Accordingly, they were first focused on and analysed using the qualitative content analysis. The goal was to create a category system that structures expert knowledge to later evaluate and transform it into requirements. All this has to happen in a way that is systematic and research-economical, as it is a large and complex set of data where every statement can be of value. This is why a combination of deductive and inductive categorisation was chosen for this process to keep the process systematically but also open. Another aspect that needed to be addressed was the fact that the present data consisted of specialised statements from a complex domain. The assumed difficulty to adequately handle these statements was intercepted through consensual coding, which means several analysts working and coding together until agreement. The following figure visualizes the coding process:

The process started with the help of a deductively developed category system based on the interview guide (1). This was then applied to a selection of interviews (2). Before adjusting the category system, it had to be decided, when the material could be judged sufficient to do so. The researchers decided that this was the case when each professional field has been analysed through at least one interview with an expert from one of these fields. One round with each professional field was further referred to as an “interview set”. Since the knowledge of experts can be considered as specialised knowledge that they can relate to other forms of knowledge, it can be assumed that they can articulate enough structures and procedures within their statements that the researchers can draw on for orientation. Each interview of this set was first coded separately (2a) and then discussed in teams until the coding was consensual to eventually reflect on the results (2b). The category system was then adjusted by adding and adapting categories (3a). For intersubjective comprehensibility, the system was viewed by other researchers who had some expertise in qualitative data analyse (3b). After this first adjustment, the system was applied to the second interview set and tested (4). It was ascertained that the new system worked well and that no changes had to be made to the categories themselves, only to for example the definition of the categories (5). Therefore, the remaining interview set (6), and the first set were also categorised with this category system (7). In the end, all the statements to the respective categories were transferred to another document (8).

The final category system is structured in such a way that system requirements can be drawn directly from some categories. In addition, it is intended to provide a holistic view of the experts' statements regarding their integration of patent drawings, also in combination with patent text, in the search process and their assessment towards a potential visual search. In total, the system includes seven main categories and 20 subcategories. Exemplary categories can be examined in the following table:

To identify the requirements, the text passages assigned to the categories were summarised and structured along the categories in a separate document. After this, the first requirements were formulated and discussed with the analysis team. The results were then confirmed by another data source which was a collection of ideas. This collection contained current standards of patent retrieval systems and corresponding wishes for the system to be developed and was set up by the project team. With the help of this collection of ideas, the requirements obtained from the interviews could be reinforced. This also guaranteed that the limits of the project were not exceeded. After the requirements had been formulated more concretely with the help of the collection of ideas, they were presented to the developers of the system. Together, the requirements were discussed for intersubjective understanding and prioritised in terms of benefits and effort.

The entire analysis process was carried out by several information science researchers and with project members from other disciplines, e.g., the developers and experienced persons in respect to the patent domain. The aim was to achieve an overall agreement on the requirements and at the same time to ensure intersubjective transparency. In addition to supporting data sources for the interviews, workshops were held to provide introductions to the patent domain or to present preliminary findings.

Results

By applying the techniques recommended within the Q-rEx method, system requirements could be successfully derived. These resulted from the importance of drawings for the search process in combination with the patent text. From this, it was also possible to derive user perspectives in which a visual search is particularly useful, possibly also in combination with a textual search. The requirements are plausible and can be traced back to the text passages in the respective transcripts or to the other data sources used.

The meaning of patent drawings and their relation to patent text

In total, ten out of eleven interviewed patent experts said they would use patent drawings during the search process and regard them as very relevant. The speed as a positive feature of patent drawings stands out as particularly important for the patent search. This is because patent drawings show all the essential details and the connections between them on a constructive level and enable a correspondingly fast understanding of the invention without having to read the entire patent document. That’s why ‘one drawing is worth a thousand words’ as one expert stated. Since thorough work in the shortest possible time is an essential aspect of patent searches, which is made difficult by the large number of patent information, corresponding means such as fast drawings are considered significant.

The quick overview with the help of drawings is again beneficial when many technical fields, some of them unknown, must be covered. At the same time, however, they are also helpful when a specific technical field is being supervised. Because of the high level of familiarity with the objects, the experts can capture the invention much faster based on the drawings. What is also considered extremely important is the fact that the drawings are not dependent on language. This makes them very useful for conducting searches in foreign-language areas.

Whether drawings have any meaning, however, depends on their quality and on the respective technical field. Accordingly, the use of drawings only serves a purpose if the standards are adhered to, and the symbols are used correctly. For example, these are predefined by the WIPO (2007) or by the patent law. Furthermore, the use of drawings is not recommended in all technical areas, e.g., no meaning is attributed to them in process descriptions. They are rather of great importance for mechanical components or for electrical circuits, which are very standardized, and generally for objects that are geometrically arranged.

However, if the drawings are used, they are usually combined with the patent text. This is because the drawings only provide reference signs for the individual components of the invention, and these are first explained in the text. Therefore, the patent searchers ‘have to get from the reference signs to the corresponding description very quickly’, as one expert pointed out clearly. Accordingly, if the drawings are not only used for a quick overview but also for a detailed analysis of the patent, they are always used in relation to the patent text.

User perspectives on the usage of a visual search (in combination with a textual search)

The user perspectives in which a visual search is useful result from the perceived positive features of drawings that have been shown above. Further, the perspectives can be described using the process steps of the patent search. In a very abstract model from Koch und Bosch (2011), these are the search query, the analysis of the result set and the detailed analysis of individual patent documents. During these process steps, drawings or rather a visual search is in total seen as useful for the search query as well as for evaluating the results.

Beginning with the process step of the search query, the use of a visual search results from the issue of subjects that are difficult to describe. To assess whether an adequate search query has been made in particularly challenging cases, the use of a visual search query can be seen as a possible solution. ‘And that is actually rather the main driver for us, where we say we would like to have a visual search to get to the documents that are very difficult to grasp with words’ as one expert emphasised. This leads to a search situation in which an image search would be preferred or used as a supplement to the text search to overcome the challenge of the verbal description. Accordingly, the search query should be possible with images and similar images as a result set. At this point, the experts see a combination of image and text-based search queries as useful to reduce the number of matches.

In the next step, the analysis of the result set, the drawings, respectively a visual search, are useful, as many patents must be viewed here. A visual search is particularly efficient for overview searches, which should be as exhaustive as possible. As was already explained within the meaning of drawings, visual search is used in this phase of the patent search process, when an overview of a large or unfamiliar area needs to be obtained, but also to quickly assess familiar object arrangements in a specific area. For novelty searches, a visual search is also considered positive, as one expert explained: ‘In the case of novelty searches, one hit can already be detrimental to novelty. […] And sometimes I get a novelty-damaging hit in a relatively short time due to the drawings, and then I no longer need to go deep into the text. It is simply a time-saver then.’ The same applies to foreign-language patents. The translation of the text takes time and, moreover, may not contain adequate technical wording, which is why it is more effective to use the quick drawings in these cases. According to the experts’ statements, the drawings here must be scrollable quickly in this phase of the patent search.

In the process step of detailed analysis, the patent text is particularly relevant. However, in most cases, the text must be quickly linked to the drawings. For this purpose, the experts stated that the drawings must first be displayable in different views so that details can be identified quickly. This could include the color highlighting of the reference signs for the individual components in the drawings. Another function that was mentioned, is about highlighting the reference signs in the text to better find the explanation of the components. For example, one expert said: ‘And if I were able to colour-code the reference signs in the text so that I could find them super quickly, that would be even more brilliant.’ To subsequently be able to link the component and the explanation in the text, the users need different possibilities. One would be that the reference signs in the drawings are directly explained by a quick info. Other possibilities foresee the highlighting of the reference signs in the text to enable a better navigation within the patent.

Requirements

The requirements for innovative visual patent retrieval were formulated in many iterations consulting different groups of people to be able to have a consensus on these. To ensure that a final common understanding was obtained, the requirements were presented to the system and database developers within a workshop and consensually agreed upon by all involved project members.

The category system provided a solid framework from which the requirements could be derived. Further, it was possible to reinforce and structure them based on the idea collection of the project. To provide a standardisation for better comprehension, sentence templates have been used (Balzert, 2009). These include the users, the system activity and the object referred to. How mandatory the requirements are, is indicated by auxiliary verbs such as must, should and will. The requirements were grouped into different topics. The topic 'filtering and searching information' included requirements such as the following:

• The system must offer users the possibility to perform a search query with drawings only as well as in combination with keywords.
• The system must allow users to colour highlight the reference signs and keywords in the text.

The topic displaying and viewing information contained for instance:

• The system must offer users the possibility to select different views for the drawings.
• The system must offer users the possibility to filter the drawings in the result list.

Another topic, intuitive user navigation and participation, included some of the requirements listed below:

• The system must show users the degree of similarity between the patents displayed.
• The system must offer users the possibility to save selected patents in lists.

The last topic quick info provides requirements such as:

• The system must offer users the possibility of hovering over a reference sign in the drawings and displaying the corresponding description (including translation).
• The system must offer users the possibility of hovering over a reference sign in the text and displaying the corresponding designation (including translation).

Eventually, each requirement was extended by a detailed description to ensure a better understanding of what is demanded. The requirements were then assigned to the system developers and to those responsible for the content and the Graphical User Interface (GUI).

Discussion

For the purpose of this research project, a method had to be developed that would make it possible to derive requirements for a system that does not exist yet, while at the same time taking all relevant factors into account. This method is a combination of qualitative data analysis and requirements engineering techniques, using interviews with experts in the patent domain to elicit requirements. In total, the use of methods and techniques presented within Q-rEx can be considered reasonable for achieving the research goals, namely deriving requirements for an innovative patent retrieval system from unstructured and complex expert knowledge. However, some aspects need to be discussed:

Q-rEx initiated with the discussion of experts as a special target group. It was not only necessary to clarify who is considered an expert in the research context, but also which interview techniques are most appropriate and which specific role the interviewer should fulfil. The method suggested to take the role as a non-expert as well as an expert from a different knowledge culture. Evaluating the experts´ statements, it became apparent, that some experts reacted more to the interviewers taking the role as a non-expert by explaining basic knowledge very detailed. This made the interviews take long and also might have been taking time away from supposedly more interesting topics. Reflecting on this, in this context, it was more useful not to be perceived as too unfamiliar with the patent domain. The aim was for the experts to productively produce knowledge regarding an innovation and not to have to explain too many basic aspects. Therefore, prior knowledge is considered fundamental to the production of innovative concepts and should be shown more to the interviewees. Consequently, the role of an expert from a different domain should be more focused on than the role of a non-expert.

In the analysis of the heterogeneous statements, the focus was on the category system, which was developed systematically and research-economically, but nevertheless openly and exhaustively, so that requirements could be derived in a traceable manner. The testing of the content analytical approach with a mixed form of inductive and deductive categorisation proved to be useful, but it is certainly possible to further develop other techniques such as the identification of causal mechanisms according to Gläser and Laudel (2010). During the analysis, some dependencies could already be identified and were successfully used to answer the research questions. However, it is possible that further dependencies can be identified in terms of user perspectives which in turn can be used to model the requirements in a more traceable manner.

Further, the category system can be adjusted until the end of the categorisation process, but all interviews must be categorised using the finalised system. In this study, the main adjustments were made after the first interview set, which does not mean this is necessarily the case in every context. Even after categorising the last interview, new categories can evolve. It is therefore evident that the application of the method also needs to be tested in other contexts. For the formulation of the requirements another data source, the collection of ideas with current standards of patent retrieval systems, was consulted as a supporting instrument. The collection helped to provide an orientation for the formulation of the requirements and to ensure that the constraints of the project were maintained. Overall, this can be seen as very useful for the process as the prior knowledge of the researchers was restricted. However, as the collection of ideas was only used as a support here, not as an addition, it will be interesting to see how well the categorisation approach is applied to this type of material or even to other data types like workshop notes. Another aspect that needs to be reflected on is the traceability of the requirements. Q-rEx performed well here, because it was possible to link every requirement to its source, which were the exact text passage in the transcripts and the collection of project ideas. Nevertheless, it could be observed that it takes some time and many documents to look up the exact source. For that matter, means need to be found that make the traceability of the requirements faster to view. We suggest some models that show cause and impact patterns visually.

Having support within Q-rEx, whether in terms of other data sources or the interaction of several researchers, is recommended for several reasons. Firstly, it can ensure a proper understanding of the expert knowledge of a particular field. Secondly, while the knowledge is considered extremely valuable, it is not considered objective, but rather reflected by multiple sources and people. Lastly, the interpretations of multiple people create a better opportunity to formulate requirements for a system that does not exist yet.

Conclusion

The aim of this study was to identify the requirements that experts in the patent domain have for innovative visual patent retrieval. To determine these requirements, it was necessary to develop and apply a methodological approach that was appropriate for the research context. This resulted in the development of a method called Q-rEx, which was thus applied for the first time in this work. With the techniques of Q-rEx a rule-guided procedure was obtained which is still open for new data sources. The method allowed requirements to be derived successfully from heterogeneous statements by experts. In further research, the method needs to be used again to test its applicability in other contexts, and to examine whether similar results can be achieved in comparable contexts.

The results of the research showed that patent drawings can be quickly captured because of their simple representation of all significant details of the invention. Drawings are essential for the search because of their speed and, in particular, because they are language independent. They can be more important for certain search occasions and the scope of the areas that need to be covered. Overall, it also became apparent how important the connection between drawings and text is and how this is achieved to reinforce and evaluate the understanding and relevance of both the text and, conversely, the drawings.

In addition, it became clear that a visual search can be regarded as goal-oriented in all search phases and should be combined with a textual search in most cases. The combination of these two modalities serves to increase the accuracy of hits during the search query or to help with search queries that are difficult to formulate verbally. When analysing the result set, a combination can also be useful, although a visual search is often sufficient here. In the more detailed analysis of certain patent documents, the two modalities are often used together, as they can reinforce each other. Not only the visual search but also the general relevance and use of patent drawings should be investigated more in further search to develop a model in which these are integrated under certain circumstances like research occasions and other factors.

Next steps in the project will be the development of a Graphical User Interface (GUI) which will be tested by the patent experts. The individual functions of the system will be evaluated through user tests so that changes can be made appropriately.

Acknowledgements

We would like to thank the participants who generously shared their time and experience for the purposes of this project.

About the authors

Johanna Zellmer is a master’s student at the Institute for Information Science and Language Technology at the University of Hildesheim. She can be contacted at zellmer@uni-hildesheim.de
Dr. Stefanie Elbeshausen works as a postdoctoral researcher in information science at the University of Hildesheim. Her research interests are in the area of (collaborative) information seeking, human-computer interaction and information retrieval. She can be contacted at elbesh@uni-hildesheim.de
Prof. Dr. Christa Womser-Hacker is full professor of Information Science at the University of Hildesheim, Germany. Her research interests are in the area of cross-language information retrieval, information behaviour, and human-computer interaction. She can be contacted at womser@uni-hildesheim.de

How to cite this paper