The effect of visualisation on user experience in recommender systems

Introduction

One of the most valuable and distinguishing assets that an organization possesses is its knowledge (Jasimuddin, 2012). Organizational growth and survival, increased and more effective performance, sustained competitive advantage and improved quality of service are some of the benefits that successful handling of knowledge may bring. The utilisation of knowledge is key to production, power and advancement (Jasimuddin, 2012). With the advent of knowledge economy, the core of enterprises has been shifting from being labour or capital intensive to becoming technology intensive, and currently is now becoming knowledge intensive (Lu and Tseng, 2009). Knowledge management is a critical issue for enterprise management and has been the subject of both academic and industrial research (Rouse, 2002). Parent, et al. (2014) also demonstrates the importance of knowledge management and knowledge sharing. However, the huge amount of created knowledge can also lead to the problem of information overload (Li, et al., 2014). In recent years, recommender systems have expressively reduced the information overload (Portugal, et al., 2017; del Carmen Rodríguez-Hernández and Ilarri, 2021).

Gedikli, et al. (2014) reveals that the content-based tag cloud explanations are particularly helpful to increase the user-perceived level of transparency and to increase user satisfaction. Such tag clouds can be found on a number of popular Social Web sites such as Delicious and Flickr10 where they are used as a visualisation and interaction technique. Incorporating visualisation into recommender systems is beginning to be studied and makes the systems far easier to understand, and provides an effective means of permitting users to interact with systems. Wongsuphasawat, et al. (2016) seek to complement manual chart construction with interactive navigation of a gallery of automatically-generated visualisations. Visualisation techniques have been used by many researchers, and investigators combine a number of different graph layouts for more convincing systems (Saket, et al., 2019; Dimara, et al., 2017; Alsalemi, et al., 2020). Besides, most recommender systems focus on features of the methods and appropriateness, but lack explanations (Wang, et al., 2019). There are significant challenges to conducting human-centred research, including the difficulty of finding and maintaining user communities for research.

This study focuses on the assessment of user experience, thereby gaining information on how to improve user satisfaction and continuance intention through various visual designs. Post-acceptance model of information system continuance has proved to be effective to evaluate user satisfaction and continuance intention (Hadji and Degoulet, 2016; You, et al., 2020). This study proposes a research model to explain the successful adoption of recommender systems by extending post-acceptance model of information system continuance and empirically validating the relationships among the constructs in the model with experimental study. The impact of visual attractiveness is explored by proposed model. The remainder of this paper is organized as follows. First, literatures related to recommender system, information visualisation, tag cloud and post-acceptance model of information system continuance are surveyed. Then, the research methodology and the proposed questionnaire are described. Third, the experiments and the summarised survey results are illustrated. Finally, conclusions are drawn.

Literature review

Recommender systems

Recommender systems are filters which suggest items or information that might be interesting to users. These systems analyse the past behaviour of a user, build a user profile that stores information about user interests, and then analyse that profile to find potentially useful items (de Gemmis, et al., 2015). Some of these systems provide the end user with a personalised item list and an explanation that describes why a specific item is recommended and why the system supposes that the user will like it (Gedikli, et al., 2014). A number of studies have already demonstrated that recommender systems can subtly steer consumer decision (Nanou, et al., 2010; Xiao and Benbasat, 2018; Köcher, et al., 2019).

Survey papers on recommender systems have been published in recent years. There are a few studies of system effectiveness and evaluation criteria from users’ perspectives. Product recommendation agents and other web-based decision aids are deployed extensively to provide online shoppers with virtual advice services. Though the design of product recommendation agents’ functional features has received a substantial amount of attention in academic studies, the social aspects of human–product recommendation agents’ interactions have been studied less (Qiu and Benbasat, 2010). That recommendation agents can substantially improve consumers’ decision making is well understood; but the influence of specific design attributes of the interface on decision making and other outcome measures much less so (Xu, 2014). Konstan and Riedl (2012) reviewed the key advances in collaborative filtering recommender systems, focusing on the evolution from research that concentrates purely on algorithms to research that concentrates on the rich set of questions around the user experience with the recommender. As recommender technology is now becoming widely accepted, the ability to characterise user experience and users’ affective attitudes toward this technology has become extremely important (Pu, et al., 2011). However, several studies argue that there are other factors influencing the user experience. Situational or personal aspects (e.g., product expertise and privacy concerns) can also influence how people interact with and evaluate a system (Knijnenburg and Willemsen, 2009, 2010; Knijnenburg, et al., 2011).

User experience issues are increasingly being studied, in an effort to understand and identify effective methods to elicit preferences and to design guidelines to enhance a recommender’s interface layout (Ozok, et al., 2010). An integrated view of the user experience of recommender systems can be obtained by means of user-centric development and evaluation (Pu, et al., 2011). To understand and improve user experience with recommender systems, it is necessary to conduct empirical evaluations that consider the entire process that defines the user experience (Knijnenburg, 2012). This can provide a better understanding of why and how certain aspects of the system result in better user experience and others do not, thereby helping to further user-centric research and the development of recommender systems (Knijnenburg, 2012). Although, Pu, et al., (2011) had suggested a wide range of criteria regarding users’ experiences (e.g. usability, diversity, user satisfaction), in this research, the user experience is defined as a substitute for usability or user satisfaction without following the theoretical concepts.

Information visualisation

Display designers and content producers are currently soliciting users’ attention in a time when people have tended to develop a degree of immunity against the steady information overload. The main goal for designers is to find the essential features that can implement visualisation that will attract users’ attention (Meyer, 2010). With the rapid growth of knowledge resources and the increasing individual demands of users, the problem of how to spend less effort in finding desired knowledge is becoming increasingly important. By using knowledge mapping techniques, a large and complex set of knowledge resources can be assimilated and navigated more easily (Li and Song, 2011). Abstracting and implementing the visual presentation of views as products of very complex system models is nearly as important to the effectiveness of these efforts to inform decision-making, as the technical competency and completeness of those models. However, the information visualisation of data from complex system models is often considered secondary to technical considerations (Sindiy, et al., 2013).

Explanations in recommender systems have gained increasing importance in the last few years. It has been found that explanations can help increase user acceptance of collaborative filtering recommender systems, helping users to make decisions more quickly, convincing them to buy and even developing trust as a whole (Al-Taie and Kadry, 2014). While most research on website design has focused on functional tasks, the Internet offers many opportunities for leisure as well as experiential activities. Following the evolution of more developed society toward an experience economy, analysing the role of technologies in the presence of prior user experiences makes sense (Pallud and Straub, 2014).

Visualisation is very important in any simulation activity and it is important to use the most appropriate representation techniques to match users’ real needs (Boton, et al., 2013). Existing network analysis tools, however, often lack intuitive interfaces to support the exploration of large scale data. Verbert, et al. (2013) presented a visual recommender system to help guide users during the navigation of network data. Collaborative filtering, similarity metrics, and relative importance are used to generate recommendations of potentially significant nodes for users to explore. In addition, graphic layout and node visibility are adjusted in real-time to accommodate recommendation display and to reduce visual clutter.

Parra (2012) aimed to understand how different visualisations and some personal traits might influence users’ assessment of recommended items, particularly in domains where multidimensional data or contextual constraints are involved. Eppler and Pfister (2013) conducted on-site observations, focus groups, and interviews, together with analysis of the site, documents, tools and software within police headquarters. One main result of the empirical investigation was that the police force’s practice of hybrid (i.e., both digital and physical) knowledge visualisation offers a useful strategy for corporate knowledge management as well.

Verbert, et al. (2013) presented the use of information visualisation techniques to interact with recommender systems. More specifically, both the effectiveness and probability of item selection increase when users are able to explore and interrelate multiple entities, that is, items bookmarked by users, recommendations and tags. One study presented a human cognition framework for information visualisation. This framework emphasises how top-down cognitive processing enables the induction of insight, reasoning, and understanding, which are key goals of the visual analytics community (Patterson, et al., 2014). Visualisation techniques have been used by many researchers who combine a number of different graph layouts for more convincing systems in different domains. Saket, et al. (2019) found the effectiveness of these visualisation types significantly varies across task, suggesting that visualisation design would benefit from considering context-dependent effectiveness. Alsalemi, et al. (2020) presented an overview of the micro-moment based energy efficiency framework that aims to integrate behaviour change theories, effective data visualisation via the end-user application, and personalised recommenders.

Tag cloud

A tag cloud is one type of visual presentations of a set of words, typically a set of tags selected according to some rationale, in which attributes of the text such as size, weight, or color are used to represent features, such as frequency, of the associated terms. Rivadeneira, et al. (2007) describes two studies to evaluate the effectiveness of differently constructed tag cloud for the various tasks they can be used to support, including searching, browsing, impression formation and recognition. A word cloud is a special visualisation of text in which the more frequently used words are effectively highlighted by more prominence in the representation. Word clouds can be a useful tool for preliminary analysis and for validation of previous findings (McNaught and Lam, 2010). Al-Taie and Kadry (2014) focused on the visualisation of explanations in recommender systems and learned what modalities (e.g. text, graphs, tables, and images) can better present explanations to users, through the review of a selection of papers in the literature over the last few years. The results show that explanations with simple graphs and descriptions can better present explanations. Gedikli, et al. (2014) found that the content-based tag cloud explanations are particularly helpful to increase the user-perceived level of transparency and to increase user satisfaction, even though they demand higher cognitive effort from the user. Yoon and Chung (2016) reviewed Boston Bombing related tags from a tag-cloud result and aimed to understand how images are used in communication practices in the Twitter environment.

Two studies evaluated the effectiveness of differently constructed tag cloud for the various tasks they can be used to support, including searching, browsing, impression formation and recognition. Based on these studies, a paradigm was proposed for evaluating tag cloud and ultimately guidelines for tag cloud construction (Rivadeneira, et al., 2007). Romero, et al. (2013) proposed a tag cloud generation module designed to help users to more fully comprehend the retrieved information. As hypertext learning environments are widely used in education, it is important to study and understand the effects and consequences of its use. Hypertext learning environments are non-linear, which means that users must develop ways to navigate through them. Thus, developing interfaces that facilitate and guide navigation is important for learning (Walhout, et al., 2015).

Post-acceptance model of information system continuance

Expectation-confirmation theory is widely used in the consumer behaviour literature to study consumer satisfaction, post-purchase behaviour (e.g., repurchase, complaining), and service marketing in general (Oliver, 1980). Expectation confirmation theory is adapted from the consumer behaviour literature and integrated with theoretical and empirical findings from previous information system use research to theorise a post-acceptance model of information system continuance (Bhattacherjee, 2001).

The model suggests that users' continuance intention is determined by their satisfaction with information system use and perceived usefulness of continued use (Figure 1). User satisfaction, in turn, is influenced by their confirmation of expectation from previous use and perceived usefulness. Post acceptance perceived usefulness is influenced by users' confirmation level.

The perspective of exit-voice theory (Hirschman, 1970) posits that satisfaction will increase customers’ loyalty to a product or service, while dissatisfaction may lead customers to leave or to voice their complaints and seek compensation. Likewise, Bhattacherjee (2001) and Oliver (1980) also suggest that satisfied customers will develop their repurchase intention, whereas dissatisfied customers may discontinue subsequent use. Past studies have provided empirical evidence to support the influence of satisfaction on behavioural intention (Zheng, et al., 2013; Hadji and Degoulet, 2016; You, et al., 2020).

Chiu, et al. (2007) found that students with higher satisfactions to use an online learning system had a greater intention to continue using it. Zhao, et al. (2008) reported that customer satisfaction is positively linked to the behavioural intention to continue using a self-service machine. Oh, et al. (2013) focused on tourists’ adoption of self-service technologies at resort hotels, and found that perceived usefulness is positively related to intention to use. In addition, many other studies have provided evidence of the significant positive effect that perceived usefulness has on the behavioral intention to continue using something (Ku and Chen, 2013), and that satisfaction can increase the technology-related or Internet-related behavioural intentions of customers (Fang, et al., 2011; Zheng, et al., 2013; Hsu, et al., 2014; Qu and Kim, 2019; You, et al., 2020).

Research method

Research model and hypotheses

In this study, post-acceptance model of information system continuance is extended to elucidate successful adoption of the recommender system. The relationship between visualisation effect and continuance intention is also identified and explored. Post-acceptance model of information system continuance is adapted as the basis for modifications. The construct of confirmation is defined in the post-acceptance model as: 'Users expect the system to provide its actual performance with perceived consistency' (Bhattacherjee, 2001, p. 353) However, since this study is for a recommender system, not a general information system, it is not necessary for each subject to have the same familiarity with the system. Therefore, this study removes this construct from the model to avoid the subject's misunderstanding. In addition, Gedikli, et al. (2014) stated that the effectiveness of a recommender system may affect the level of satisfaction, which needs to be studied in greater depth. This study also aims to explore the impact of visual attractiveness. Consequently, the two constructs of effectiveness and visual attractiveness have been added to the proposed model. The model consists of five dimensions: perceived usefulness, effectiveness, visual attractiveness, user satisfaction, and continuance intention. The research model is shown in Figure 2.

Tag clouds were introduced by Gedikli et al. (2011) as another way of visualising explanations in the recommender systems. The assumption is that each recommendable item can be characterised by a set of tags or keywords. Several researchers have found that tag clouds are effective and useful. Romero, et al. (2013) tested the design of a tag cloud generation module algorithmically with no users involved which indicated that it can help users to retrieve information. Through the presentation type of tag cloud, users can easily search the database (Torres-Parejo, et al., 2013). Tag cloud is expressed by its font size (relative to the other tags) and is therefore easily recognised (Batman, et al., 2008). A tag cloud can improve the learning process (Walhout, et al., 2015) and increase user satisfaction (Gedikli, et al., 2014). Saket, et al. (2019) found the effectiveness of visualisation types significantly varies across tasks, suggesting that visualisation design would benefit from considering context-dependent effectiveness. Hence, the following hypotheses:

H1: The use of a tag cloud in a recommender system has higher perceived usefulness than no use.
H2: The use of a tag cloud in a recommender system has higher effectiveness than no use.
H3: The use of a tag cloud in a recommender system has higher visual attractiveness than no use.
H4: The use of a tag cloud in a recommender system has higher user satisfaction than no use.

Perceived usefulness

Usefulness is defined as the user's subjective probability that using a particular information system will enhance his/her performance (Davis, et al., 1989). This in turn influences the user's intention to continue using the system. Kim and Ong (2005) noted that perceived usability and satisfaction are highly correlated. Flavian, et al. (2006) analysed the role of satisfaction and website perceived usability in developing customer loyalty in the e-banking business. The data showed that website perceived usability has a positive effect on customer satisfaction.

Wang and Liao (2008) provided the first empirical test for an adaptation of DeLone and McLean's information system success model in the context of government to consumer eGovernment. The hypothesised relationships between use and user satisfaction is significantly supported by the data. Liaw and Huang (2013) investigated learner self-regulation in e-learning environments. Their statistical results showed that satisfaction of e-learning will be influenced by perceived usefulness.

Numerous research studies provide empirical support for the positive association between usefulness and information technology use intention (Premkumar and Bhattacherjee, 2008; Lee and Lee, 2009; Boakye, et al., 2014). Multiple studies also attempt to identify the antecedents and drivers of post-adoption human behavioural intention and usage behaviour. A few significant antecedents are consumer satisfaction, perceived usefulness, perceived ease of use, subjective norms and perceived enjoyment. Nearly half of the studies (43%) used satisfaction and perceived usefulness as key intrinsic factors to empirically establish the influence of these antecedents on continuous behavioural intention and usage (Shaikh and Karjaluoto, 2015).

Stefanovic, et al. (2016) further evaluated the effectiveness of the Government Information System from the perspective of government employees, and found that perceived usefulness and user satisfaction have a significant, direct relationship. Ramírez-Correa, et al. (2017) used the information system success model to explore the success of a learning management system. The study confirmed that perceived usefulness and satisfaction are highly correlated. Numerous research studies have provided empirical support for the positive association between usefulness and continuance use intention (Hadji and Degoulet, 2016; Hadji, et al., 2016; Demoulin and Coussement, 2020; Al-Fraihat, et al., 2020). Hence, we present the following hypotheses:

H5: Perceived usefulness has a significant positive impact on user satisfaction.
H6: Perceived usefulness has a significant positive impact on continuance intention.

Effectiveness

The definition of effectiveness is, the ability to help users make better decisions; that is, it can help users accurately assess the quality of the recommendation and thus make better decisions (Gedikli, et al., 2014). In a study of which factors will affect e-learning user satisfaction, effectiveness was found to have a positive impact for user satisfaction and perceived usefulness (Calli, et al., 2013). Gedikli, et al. also found that the effectiveness of the system is influential for user satisfaction. This leads to the following hypotheses:

H7: Effectiveness has a significant positive impact on user satisfaction.
H8: Effectiveness has a significant positive impact on perceived usefulness.

Visual attractiveness

Heijden (2003) explored the impact factor of using websites and proposed a new concept (visual attractiveness), defining it as, 'a person thinks the web site looks aesthetic' (p. 544), which is a subjective feeling. Liu, et al. (2013) investigated the relationship between different websites and consumer impulse buying behaviour. The results show that visual attractiveness influences the purchasing behaviour. Cyr, et al. (2010) also found that the configuration of a website is an important factor for user satisfaction. Therefore, the current study considers presentation type as a variable factor, in order to better understand how visual attractiveness will impact user satisfaction. This leads to the hypothesis:

H9: Visual attractiveness has a significant positive impact on user satisfaction.

User satisfaction

The topic of customer satisfaction has held a significant position in the marketing literature since satisfied customers can generate long-term benefits for companies, including customer loyalty and sustained profitability (Homburg, et al., 2006). Customer satisfaction is highly related to behavioural intentions and is regarded as one of the key antecedents of post-purchase behavioural intentions because customer satisfaction has a positive effect on the customer's attitude towards the product or service and it can reinforce the customer's conscious effort to purchase the product or service again in the future (Oliver, 1980, 1999; Liu and Jang, 2009).

In the mobile banking context, continuous intention is found to be solely dependent on the satisfaction of customers (Reji and Ravindran, 2012). In another empirical study, students’ intentions to continue are directly and meaningfully influenced by their satisfaction and perceived usefulness of electronic textbooks (Stone and Baker-Eveleth, 2013); user satisfaction with Web 2.0 applications (Facebook, iGoogle, Plurk, Twitter, and YouTube) and online knowledge groups significantly affects electronic word-of-mouth, which in turn significantly influences their intention to continue (Chen , 2012).

Similarly, satisfaction and perceived usefulness were found to play a significant role in the continuance intention and usage of web analytical services (Park, et al., 2010), Internet-based learning technologies (Limayem and Cheung, 2011) and online communities (Apostolou et al., 2017). Numerous research studies have provided empirical support for the positive association between user satisfaction and continuance intention (Bae, 2018; Li and Fang, 2019; Alalwan, 2020). This leads to the hypothesis:

H10: User satisfaction has a significant positive impact on continuance intention.

Experimental design

This study uses laboratory experimentation and adopts C# as the programming language to construct the experiment platform. Microsoft Visual Studio Express 2013 for the Web is the development environment. The database is built by referring to the information on the job bank's website. The database contains 300 users in 120 types of knowledge rated between 1–10 for a total of 36,000 records. The knowledge includes the relation between a job offering and a skill. The algorithm that is used in the recommender system is adapted from a centring-bunching based clustering algorithm that is used for hybrid personalised recommender systems (Shinde and Kulkarni, 2012). The experimental process is shown in Figure 3.

This study asked participants to use recommender systems to search for a software engineer job. The job bank has a job openings repository, knowledge base, and historical knowledge score data in its database. The software engineers can be divided into four different categories by programming languages: C #, C ++, JAVA and python. What a participant needs to do is to navigate the system and input necessary information, this information will then be used by the system for subsequent recommendations to acquire the most interconnected jobs. Roscoe (2004) noted that as a rule of thumb, between 30 and 500 is an appropriate sample size.

The participants in this research were all graduate students from an information management department, and all had a basic understanding of a recommender system. The total of 120 participants was randomly divided equally into two groups, A and B. Randomly assigning the participants means that the perception of the recommender system is not affected by extraneous factors. All participants (both Group A and Group B) used the same recommender system interface and algorithm to search and browse. The only difference between Group A and Group B was the presentation type of search results. Group A used the tag cloud type of presentation (Figure 4) and Group B did not (traditional type) (Figure 5).

A tag cloud generator was built to analyse a set of jobs, and immediately discover jobs that users mention most often, as well as what they value most. Group A’s results will be presented in tag cloud in which words are shown in varying sizes depending on how often they appear in the data with predefined layouts and colour. On the other hand, group B’s results will be presented in traditional type in which words are shown in same sizes. All participants could spend ten minutes to view the job details, and then fill in the questionnaire.

Measures of the constructs

To ensure the content validity of the scales used in a study, the items selected for the constructs should represent the concepts about which generalisations are to be made. Hence the items selected for the constructs in this study were mainly adapted from prior studies to ensure content validity. Four items, selected from Calli, et al. (2013) and Kwabena, et al. (2014) were adapted and used in this study to measure perceived usefulness. Three items, selected from Tintarev and Masthoff (2012) and Lin, et al. (2010) were adapted and used in this study to measure effectiveness. Five items selected from Loiacono, et al. (2007), Cyr, et al. (2010) and Liu, et al. (2013) were adapted and used in this study to measure visual attractiveness. Four items, selected from Belanche, et al. (2012) and Aversano and Tortorella (2013) were adapted and used in this study measure to user satisfaction. Three items, selected from Kwabena, et al. (2014), Bhattacherjee (2001) and Premkumar and Bhattacherjee (2008) were adapted and used in this study to measure continuance intention.

Each item was adapted to specifically reference recommender systems. Likert scales (1–5), with anchors ranging from “strongly disagree” to “strongly agree,” were used for all questions. After the retesting of the measures, these items were modified to fit the recommender system context studied. The scale was tested for various reliability and validity properties. Cronbach’s alpha was used to assess the reliability of the scale, which provides an indication of the internal consistency of the items measuring the same construct. Construct validity was assessed by both convergent and discriminant validity using factor analysis.

Data analysis

Data were analysed using SPSS version 12.0 and SmartPLS (partial least squares path modelling method) 2.0. Statistical methods used for the questionnaire on user satisfaction were a reliability analysis, a validity analysis and T-tests. A total of 120 responses were obtained. 57% of the respondents were male. Subjects were also asked to consider their own familiarity with the recommender system. 57% subjects considered themselves to be very familiar, and 35% subjects were familiar. Thus most subjects in this study have a reasonable degree of understanding of, and experience with recommender systems.

Results

Measurement model

In SmartPLS, the consistent PLS algorithm, usually called just the PLS algorithm, is the default standard partial least squares modelling procedure. After the path model is created, output reports (Table 1-4, Figure 7) are available after running the PLS algorithm. These questionnaires are considered to be reliable because their Cronbach α values are within a range of 0.811-0.927 (Table 1) and thus are greater than the 0.70 threshold, which indicates that the scales are reliable (Nunnally, 1975).

The questionnaire is based on the literature and then amended by experts, so it has content validity. Finally, this paper uses factorial analysis to assess construct validity. Factor analysis is considered one of the strongest approaches to establishing construct validity and is the most commonly used method for establishing construct validity measured by an instrument. Convergent validity is evaluated by factor loading and average variance extracted (AVE). Convergent validity is established if individual factor loadings are greater than 0.7 and average variance extracted exceeds 0.5 (Hair, et al, 1992).

Table 2 presents the factor loading values of individual measurement items and the average variance extracted of each construct. All the factor loadings values are greater than 0.8 and the average variance extracted values exceed 0.7. The measurement scales exhibit convergent validity. Discriminant validity is determined through an examination of a cross loading table. A cross loading table must indicate that the measurement items load highly on their theoretically assigned factors and not highly on other factors (Gefen and Ridings, 2003).

Table 3 shows that the loadings of individual items on its assigned construct are higher than with other construct.

The square root of average variance extracted can also be used to determine discriminate validity, which is confirmed if the square root of average variance extracted is greater than the off-diagonal elements of a correlation matrix (Fornell and Larcker, 1981). Table 4 shows that all constructs satisfy this criterion and demonstrate adequate discriminant validity.

Structural model

The t-test compares two sample means by calculating Student’s t-value and displays the two-tailed probability of the difference between the means. This study compared the results of the SPSS analysis from two groups, presentation with tag cloud and presentation without tag cloud, to test the hypotheses examining the effect of visualisation (H1-H4). The SPSS analysis is shown in Table 5 and Figure 6. These results clearly indicate that Group A (using tag cloud presentation) was superior to Group B (not using tag cloud presentation), since the five constructs all reached significance levels of 0.01. Thus, hypotheses: H1, H2, H3, H4, H5 are supported.

After running the PLS algorithm, the report of R-squared was found (Figure 7). R-squared is a goodness-of-fit measure for linear regression models. This statistic indicates the percentage of the variance in the dependent variable that the independent variables explain collectively. The results account for 57.3% of the variance in perceived usefulness, 77.9% of the variance in user satisfaction, and 75.3% of the variance in continuance intention. Thus the results indicate that the research model significantly explains the successful adoption of recommender systems, and all six hypotheses (H5–H10) are significant and supported. The strongest relationship exists between user satisfaction and continuance intention. Perceived usefulness is an important factor that explains user satisfaction. The effect size is relatively small for the relationship between effectiveness and user satisfaction, though the relationship is significant.

This study further compared the results of the PLS analysis from two groups, presentation with tag cloud (Group A) and presentation without tag cloud (Group B), to examine the moderating effect of presentation type (H5 - H10). Table 6 presents a comparison of the results between the path coefficients. However, different presentation types would not lead to significantly different impacts between perceived usefulness, effectiveness, visual attractiveness, user satisfaction and continuance intention. We infer that the reason why the representation type is not significant is that the main effect is very substantial when evaluating a recommender system. Whether different representation types are used to determine the main effect seems to be relatively insignificant and the result of moderating effect is not apparent.

Implications

This study proposes a research model to explain the successful adoption of recommender systems by extending post-acceptance model of information system continuance and empirically validating the relationships among the constructs in the model with experimental study. All hypotheses are supported and the visualisation effort (tag cloud) positively affects perceived usefulness, effectiveness, visual attractiveness, and user satisfaction. Implications of this study for researchers and practitioners are discussed below, together with its limitations, which point to opportunities for future research.

This study confirms the influence that perceived usefulness has on user satisfaction and continuance intention with recommender systems. Perceived usefulness is a key aspect of the user's subjective probability to use recommender systems and enhance his/her performance. This result implies that the field study on features of recommender systems from industry /users is essential to find factors in promoting recommender system success. This study also finds that effectiveness has significant influence in the perceived usefulness and user satisfaction with recommender systems. Site developers should focus on enhancing effectiveness to raise user satisfaction in recommender systems since user satisfaction is a highly influential variable for business success (Chen, 2012). This study also finds that visual attractiveness is a critical factor for user satisfaction with recommender systems. Appearance, configuration and presentation type are key aspects of recommender systems which have a significant, positive relationship with visual attractiveness. In our study, the use of tag cloud indeed has higher effectiveness. For practical suggestions, several points are addressed:

1. At present, the software engineer positions in the job bank are not categorised according to different programming languages, so job-seekers can only search by manually inputting keywords. If the job bank could categorise jobs, as done in this study and provide recommendations to users, that might improve job-seekers' satisfaction with this website.
2. The visual presentation of the recommender system can improve user satisfaction. Therefore, website designers should not only consider the algorithm behind the recommender system, but also visual presentation of the recommender system.
3. Industries developing recommender systems should also promote these benefits to prospective adopting organizations. Researchers are encouraged to continue using continuance intention in future recommender system research since this is an important construct.

This study is limited by the laboratory setting. The external validity could be enhanced by designing an experimental setting that reflects relevant aspects of real decision tasks and by using a task that has real-world data. For example, because this laboratory experiment involves one decision aid that is designed to support an individual performing a single function, generalising the results to recommender system usage in a broader sense requires more experiments to determine the effect on different types of recommender system.

Conclusions

The research model was developed and validated through utilising the extended post-acceptance model of information system continuance adapted to the context of recommender systems. This study examined the impact that factors relevant to recommender systems have on continuance intention. Perceived usefulness, effectiveness, visual attractiveness, and user satisfaction were used to measure continuance intention. The results demonstrate that the proposed framework for the evaluation of recommender systems is established and successfully implemented. From the results of this study, we observe that in order to improve the satisfaction or continuance intention of users, it is necessary to improve the perceived usefulness, effectiveness and visual attractiveness of a site. Moreover, regardless of which indicators are considered, the visual presentation (tag cloud) is significantly better than the non-visual presentation (traditional), which indicates that the visualisation presentation of the recommender system can indeed significantly improve the user’s satisfaction and continuance intention. Therefore, good visual design is essential for end users. If we only focus on slight changes of the algorithm, users may not be able to perceive a substantial difference. However, direct improvements to the user interface may greatly enhance user willingness to continue using this recommender system.

Acknowledgements

This research has been partially supported by funds from the Taiwan Ministry of Science and Technology (MOST MOST 107-2410-H-260 -014 -MY3 ; MOST 109-2410-H-018-015-).

About the authors

Wen-Yau Liang is Professor of Information Management at National Changhua University of Education, Taiwan. His research interests are object-oriented design, artificial intelligence, intelligent agent and electronic commerce. His contact address is wyliang@cc.ncue.edu.tw
Chun-Che Huang, corresponding author, is a Professor in the Department of Information Management at National Chi Nan University, Taiwan and directs the Laboratory of Intelligent Systems and Knowledge Management. His contact address is cchuang@ncnu.edu.tw.
Tzu-Liang (Bill) Tseng is Professor of Industrial, Manufacturing and Systems Engineering at The University of Texas at El Paso. His research focuses on the computational intelligence, data mining, bio- informatics and advanced manufacturing. He has been serving as a principle investigator of many research projects, funded by the National Science Foundation, NASA, the Department of Education, and Kentucky Engineering and Science Foundation. He can be contact at btseng@utep.edu
Zih-Yan Wang received his Master degree in Information Management from the National Changhua University of Education, Taiwan. His e-mail address is az8546@hotmail.com

How to cite this paper