Learning analytic devices – co-forming, re-forming, in-forming

Introduction

This work-in-progress paper explores the intersection of theorising in human-data-interaction, information studies and learning analytics as part of a discussion about the role informative artefacts play as agents of learning. In building on previous work about the sociomateriality of information practices, creative ecologies of learning and learning analytics, we offer a conceptual explanation for classroom activities where we sense that the artefacts being produced are players in the learning. Using this explication of the role played by these informative artefacts in the individual and collective sensemaking of our students illustrates how the act of representing has value for learning. Making visible the qualities of representation and inscription in analytic artefacts contributes to the understanding of how the knowledge embodied in these artefacts comes to be represented and communicated in any given context (or within any given community). In the rest of this paper we introduce learning analytics, highlighting the synergies between concerns for ‘algorithmic accountability’, and work in the information science domain. We articulate our work with respect to two short classroom-based examples, before concluding with points for further discussion.

Learning analytics as an information agent

The last five years, from the first learning analytics and knowledge conference in 2011, have seen a growing interest in the use of data from learning contexts, to support the learning occurring in and through those contexts. The field of ‘learning analytics’ has developed as one response to this challenge, making use of the increasing amount of data from learning interactions in (often online) software platforms to gain insight into learning. The field is thus emerging as a discipline occupying a ‘middle space’ (Suthers and Verbert, 2013) between learning, computational, and information sciences. This bridge between learning analytics and information studies offers opportunity for a transdisciplinary conversation about the dynamic, hyper-coordinated spaces within which people create, curate and use information.

Thus, in learning analytics, transdisciplinary work is engaged to bring to bear the tools and theorised approaches of each domain on the boundary object of the learning target. For example, a growing interest in learning analytics is in its potential for the analysis of processes of learning (providing student feedback on dialogue, for instance) and the social and discursive constituents thereof (Buckingham Shum and Ferguson, 2012). In such analysis, each domain brings to bear its own theoretical and empirical perspectives on the problem, for example, how learning dialogue is to be conceptualised, structured, and investigated. Elements of each analysis will be domain-specific – providing, for example, particular methods or constructs emergent from one domain or another. However across domains a shared understanding of the core constructs and methods must be built, to develop a ‘transdisciplinary’ model of an analytic device that is grounded in a shared perspective. Thus, it has been argued (Knight and Buckingham Shum, Forthcoming), that these devices in their operationalisation of constructs into technologies and contexts commit to particular assumptions about learning. Indeed, it has been suggested (Knight, Buckingham Shum, and Littleton, 2014) that these commitments can be thought of in terms of a triad of epistemology, pedagogy, and assessment (which learning analytics are a form of), such that devices implicitly or explicitly commit to particular stances on what it is to know, how one learns (or comes to know), and how that knowledge is assessed.

Learning analytic devices as inscribed informative artefacts

In this paper we thus argue that learning analytic devices become information artefacts inscribed with particular commitments that both shape, and become shaped by, the learning contexts in which they are deployed. That is, analytic devices both shape the ways that learning is understood and enacted as objects of assessment, and are interpreted, reinterpreted, and acted with as a dynamic part of that very context. This broad discussion has noted that code ‘acts’ in education (Williamson, 2015), such that:

as algorithms are increasingly being designed to anticipate users and make predictions about their future behaviours, users are now reshaping their practices to suit the algorithms they depend on. This constructs ‘calculated publics,’ the algorithmic presentation of a public that shapes its sense of itself. (Williamson, 2015, p. 30)

Thus, the ways in which analytic devices become active agents in learning – both inscribed with policy and practice commitments, and enacted or enactive informative artefacts – has led to calls for greater ‘algorithmic accountability’ (Diakopoulos, 2014), to ensure that the pedagogic aims of analytic devices are transparent across a range of stakeholders.

One means through which algorithmic accountability can be approached is to make the invisible more visible, by acknowledging the representational work of coding and classifying embedded in an analytic device, for instance. This articulation work, which Suchman and Trigg (1993) also refer to as craftwork, draws attention to representational devices as central actors in the structuring of practice. Building on this theoretic tradition, Anderson (2007) argues there are two conditions of human experience essential to acknowledge when studying information practices in context:

1. The situated, embodied character of human experience; and
2. Active examination and analysis of the socio-material (and increasingly socio-technical) context.

The legibility of analytic devices

Conveying learning analytic information across stakeholder audiences with their respective skills and needs (from individual students up to institutional leaders) is a challenge, requiring consideration of collaborative sensemaking (Knight, Buckingham Shum, and Littleton, 2013). As approaches such as learning analytics become increasingly available, the need to explore human interactions with this data/information grows, with fields such as ‘Human Data Interaction’ (building on work in human computer interaction – HCI) emerging to explore how to "support end-users in the day-to-day management of their personal digital data..." aligning with our own view of data as of an "inherently social and relational character" (Crabtree and Mortier, 2015, p. 1). In such approaches, interactions with analytic devices would be seen as a "distinctively socio-technical problematic, driven as much by a range of social concerns with the emerging personal data ‘ecosystem’ as it is by technological concerns, to develop digital technologies that support future practices of personal data interaction within it" (Crabtree and Mortier, 2015, p. 3). Of particular interest to our concerns is the notion of ‘data legibility’ (Crabtree and Mortier, 2015, p. 18), a concern to understand the ways in which personal data is made legible through visualisation and processing, to support people in understanding and investigating their own data.

Analytic devices, as objects that both shape and are shaped by learning contexts require complex analyses to make them legible to learners and educators. To do so, analysis of the theory and operationalisation behind any given learning-target (for example, a particular form of dialogue), alongside the methods for collation and feedback, should be given. Moreover, agents should understand how their data-feedback is both an ends of the analytic, and a fundamental shaping component in the analytic device. Thus, from Suchman’s theorising (2007; Suchman and Trigg, 1993), we recognise that a ‘situated action’ unfolds in the doing, in interaction with the circumstances that make that situation. The close relationship between product and process in document work is thus increasingly recognized by researchers in the information and learning sciences. Lund’s broad defining of a document refers to:

… any results of human efforts to tell, instruct, demonstrate, teach or produce a play, in short to document, by using some means in some ways, is very focused on activities around making documents, in other words on practices (Lundh, 2010, p. 744).

In information studies, researchers like A. Lundh (2010; A. H. Lundh and Dolatkhah, 2016), Trace (2007) and Sköld (2013) note that this activity must be understood as constituting both physical and mental activities at both individual and collective levels of practice. Building on this notion, we have begun to consider how analytic devices are brought into meaningful interaction in learning contexts, as both a shaping part of those contexts and a shaped output from them through minded acts of documentation. Thus, in the earlier example, the analytic devices both shape the feedback on dialogue and are shaped by the dialogue. We thus argue that analytic devices become inscribed in their unfolding cultural context.

Informative artefacts as representation of and representation for learning

With this paper we wish to open discussion about the collaborative meaning making that lends value to these informative artefacts as agents of learning. To do so we draw on insights gleaned from two examples from our classrooms, briefly outlined here.

Communal conversations centring on the production and interpretation of deliberately analogue artefacts are fundamental to the learning design of each of these exercises. The significance of the artefacts formed and reformed in the activities described above is illustrative of the valuable social roles that Brown and Duguid (1996) attribute to documents and their assertion that, "…we need to see the way that documents have served not simply to write, but also to underwrite social interactions; not simply to communicate, but also to coordinate social practices." It is through the deliberate and visible act of producing and representing that the students learn about the subjects under study. We speak of the social life of these artefacts because their formation is in partnership with the learners’ and yet separate from their control within each of these classroom contexts. The significance of activity and the movement of ideas, as represented in the material forms students craft, are critical to each learning community’s conversations, giving these documents a social as well as an informational role. Through the constant refreshment and reinvention of the material forms that the students exchange with one another (and ultimately with their instructors) information is produced. The artefacts produced in each classroom are instrumental in getting students to make their thinking more visible to themselves and to others (peers and teachers), confirming Brown and Duguid’s contention that looking at the content alone cannot explain their value.

Framing the informative artefacts crafted through collaborative work in these classroom examples as boundary objects allows us to appreciate their socio-materiality and value as carriers and constructers of ideas within and beyond the classroom. As has been discussed elsewhere (see, for example, Anderson, 2007; Huvila, 2012) applying the concept of the boundary object (Star and Griesemer, 1989) to the study of information practices helps to make the sociotechnical contexts of the making, using, sharing and curating of information visible. Critical is the appreciation of the role boundary objects perform as translational devices. Because the making and nurturing of boundary objects initiates and advances shared understanding, they are important agents of learning in the classroom. Here we illustrate two contexts where those boundary objects tool very material, analogue forms. Through conversation about and circulation of the analogue representations of ideas in both of these examples, students form their views about the topics under discussion in each respective classroom/subject.

The emergent, evolving boundary objects -- shaped in the slowly unfolding, collaborative activities within the two examples presented here -- help the individual learner to make their own kind of sense of the activity in question in concert with a growing communal understanding. In these examples we can recognise and value both information and learning perspectives. Information studies helps us value the collective forming and reforming of these material artefacts as information practices and the minded acts of documentation. Learning analytics helps us explore these artefacts as agents of learning, encouraging us to identify ways that the individual and collective practices of information curation, creation and use they enable shape the ways that learning is understood and enacted as objects of assessment.

Conclusion: the sociomateriality of learning

The recursive link between data and information discussed by Crabtree and Mortier (2015) compels us to pursue further discussion of analytics (and the objects containing them) as information. Both of our examples embed an unfolding co-creation in the learning activities where data is both an object and a player. Consequently, the analytic device reshapes the context in which the interaction takes place and reshapes the artefactual agents. Therefore, the ‘same’ data may be presented at various levels of a stakeholder diagram (from individual students through to teachers), at various times, while representing varied information through a range of possible manifestations. Through ‘playing’ with the artefacts being developed in each classroom example presented here, students come to engage practically with the space of possibilities and meanings in representation; commensurately, each artefact itself may be seen as a player in learning context.

We are deliberate in our choice of the term ‘player’ when describing the evolving artefacts crafted through learner interaction in both of the examples as players of learning. In earlier work presenting classrooms as creative ecologies of learning, Anderson (2013) discusses the value of nurturing the creative capacities of students through classroom strategies that make a deliberate effort to build in reflective practice (pause) and opportunities to tinker with ideas and take risks (play) to become better at engaging with information in unexpected and exponentially changing ways. It is a very activity-oriented position that connects to discussions of the haptic value of drawing and doing in the classroom (see, for example, A. H. Lundh and Dolatkhah, 2016; Madsen, 2013). We see meanings unfold over engagement with these artefacts, or as Knorr-Cetina refers to them "epistemic objects" with their "unfolding, dispersed and signifying (meaningproducing) character" (2001, p191-3). Playfully allowing for multimodal interaction, the artefactual agents in our two examples create a range of possibilities: as material objects (representing a transition from analogue-activities to the encoded/inscribed-digital) and as informative objects (via their documentary status).

The desire to support playful engagement with the ideas at the heart of both exercises informed the fluid, transient nature of their design and, we believe, contributed to the learning value of the informative artefacts produced via such individual and collective sensemaking. Further discussion and exploration of additional learning contexts and analytic players shaping learning contexts will contribute a fuller understanding of the ways that creative unfolding shapes learning with, and from, these playful and informative agents of learning.

About the authors

Theresa Dirndorfer Anderson is an Associate Professor in the Connected Intelligence Centre at the University of Technology Sydney, where she is Course Director of the Master of Data Science and Innovation. She received her Bachelor's degree (honours) in Diplomatic History and Soviet/Russian Studies from the University of Pennsylvania (US), a Master’s Degree in International Relations and Security Studies from the University of Lancaster (UK), a Master’s Degree in Information Studies and PhD from the University of Technology Sydney (Australia). She can be contacted at: theresa.anderson@uts.edu.au
Simon Knight is a Postdoctoral Research Fellow in the Connected Intelligence Centre at the University of Technology Sydney, where he also teaches into the Master of Data Science and Innovation. He received his B.Sc (Intl Hons) in Philosophy and Psychology from Leeds University (UK), a Master’s Degree in Philosophy of Education from University of London (UK), MPhil in Educational Research from University of Cambridge and Doctor of Philosophy, Knowledge Media Institute (KMI), Open University (UK). He can be contacted at: Simon.Knight@uts.edu.au