North reasons why evaluating visualizations by insights may be preferable to evaluating them by tasks. “The purpose of evaluation is insight,” and insight requires different evaluative methods and goals.
“A default and implicit definition is to equate insight with user tasks, such as finding extreme values. That is, the answers to questions about the data constitute insight.” The author claims that benchmark tasks often are “too simplistic and constrained to provide a useful indication of… insight.” He lists the following constraints:
Tasks are predefined, and do not allow for natural exploration,
Tasks “need definitive completion times,”
They “must have definitive answers&8230; leaving little room for qualitative insight,”
“They require simple answers.” No complexity or uniqueness.
Insights are complex, deep, but also inexact, qualitative, and unexpected. In order to capture this, North recommends that evaluations “test more synergistic, complex tasks that involve some uncertainty.” These include characterizing distributions of values, estimation tasks, or “forcing users to interpret the visualization into a textual answer” which “ensures that they have developed their mental model of the data.”
The author suggests that evaluators “change the benchmark tasks from an independent to a dependent variable.” This may involve
an open-ended protocol (e.g. think aloud),
qualitative insight analysis (going beyond “dry data analysis” to “inferences and hypotheses”)
an emphasis on domain relevance
North mentions that such evaluations may require long trial times, more effort to capture and code results, motivated and domain-knowledgeable users, as well as experts to interpret results.
The paper does not really move “toward measuring” insight (North has been more specific in an earlier paper), but does motivate more knowledge-centric evaluations.
North, C. (2006). Toward Measuring Visualization Insight. IEEE Computer Graphics and Applications. 26, 3, May-Jun. 2006 (pp. 6-9) DOI
Moggridge talks about his book, Designing Interactions, which is a collection of interviews. Most of the clips are interesting bits of history and anecdotes from big names (e.g. Will Wright, Larry Page and Sergei Brin, Terry Winograd, Doug Engelbart).
From all of these, he claims to see two themes that lead to successful design: people and prototypes. That is, ethnography and discussion combined with rough versions and testing.
You thought Bill Verplank’s (19:00) bit was interesting. He lists three questions that designers should answer when approaching things from the users’ perspective:
How do I do?: “How do you affect the world?”
Handles: “I can grab the world and manipulate it.” Continuous control.
Jigsaw is a suite of linked visualizations made for intelligence investigators: a tool to aid in understanding, navigating, hypothesizing, and making sense of entities (names, places, things) mentioned in a large corpus of documents. Jigsaw aims to represent these entities in useful ways, as well as act as an “external cognition aid.” The authors describe Jigsaw’s aims, its various visualizations, a scenario of its use, and potential future work.
The Jigsaw application is centered on a text-based query. This query is the starting point for the exploration of the connected document-entity space. This exploration is carried out in a number of different views:
a tabular view of sortable columns of entities, connections between them shown by colouring and drawing lines between them;
a semantic graph view displaying connections between entities and reports;
a scatter plot view giving an overview of the relationships between any two entity categories; and
a text view displaying the original reports with entities highlighted.
At the VA workshop, Stasko introduced several new Jigsaw displays:
a calendar view;
a cluster view, where documents are organized into clusters based on queries;
the shoebox: a visual notebook for analyst-defined groupings, connections, and miscellaneous notes.
The sheer number of views and complexity of the data requires many windows and reasonably complicated interaction. The authors note that Jigsaw is best when it is used on several monitors. The complication and size also brings up a great number of design and interaction issues. Though there has been little to no formal evaluation of Jigsaw, it did win the university division of the 2007 VAST contest.
Of particular interest to you is the authors’ assertion “that analysts must carefully read reports to best understand them. What we seek to provide is a type of interactive visual index onto the reports.” This is both an admission that the text processing is quite superficial and that much of the task requires large amounts of human brain power and insight.
Stasko, J., Gorg, C., Liu, Z., Singhal, K. (2007). Jigsaw: Supporting Investigative Analysis through Interactive Visualization. IEEE Symposium on Visual Analytics Science and Technology 2007 (VAST 2007). Oct. 30 2007-Nov. 1 2007 (pp. 131-138). DOI, Site
Polaris is a tool for visualizing and analyzing queries of large databases. Polaris builds on the idea of Pivot Tables, organizing representations in tables which can be layered (that is, ability to display mixed plots). The paper also describes the method of creating visual specifications, which are the analysis task, database query, and way of defining the graphical representation of the results. In describing Polaris, the authors incidentally describe their data and visual models (in which nominal data is interpreted as ordinal, etc.)
Visual specifications are “the state of the [user] interface.” By drag-and-dropping database fields into certain areas, the user describes the relevant data to be represented, as well as the way in which it will be represented. Table axes and layers are also “implicitly specified;” the authors describe the hidden table algebra used to organize ordinal and quantitative fields into axes and table headings.
The authors also describe the types and purpose of graphics available in Polaris (ordinal-ordinal, ordinal-quantitative, quantitative-quantitative) and their mapping to retinal properties (shape, size, colour, orientation, but not texture). They also mention the exploratory, methods of interaction (brushing, tooltips; sorting and filtering). Examples of Polaris in use are also given; useful for seeing its practical applications.
The paper is well written and sourced. Polaris itself appears to be a very powerful tool. (It has been commercialized; it’s now Tableau.) It should be noted that the most difficult aspect of the visual representation is still under the users’ control: what the authors call the “mark.” Polaris automates queries and representation, but does not determine the best low-level representation of data.
Stolte, C., Tang, D., & Hanrahan, P. (2002). Polaris: A System for Query, Analysis, and Visualization of Multidimensional Relational Databases. IEEE Transactions on Visualization and Computer Graphics 08, 1, Jan.-Mar. 2002 (pp. 52-65). DOI, Site
The authors describe issues related to the interpretation, redundancy, and conflict in published heuristics for evaluation. The authors apply Schneiderman’s mantra (overview, zoom and filter, etc.), Amar & Stasko’s task-based framework (expose uncertainty, concretize relationships, etc.), and a set proposed by Zuk & Carpendale as one would apply them in a heuristic evaluation (in HCI).
Considering a heuristic evaluation of a tool (that is, a meta-analysis) found the three groups of heuristics “did at times find the same problem from different perspectives.” Zuk & Carpendale’s heuristics were not well organized and of high specificity, “most useful of evaluating perception, Schneiderman’s “were most useful for evaluating usability,” Amar & Stasko’s “for evaluating the discovery process.” Further classification is difficult: “we are not yet at the stage of producing a taxonomy [of heuristics].”
The authors acknowledge that no one (at this point) really knows how best to evaluate a visualization, and say that it may be useful to include a domain expert, usability expert, and visualization expert in an evaluation. Domain knowledge is particularly important when applying Amar & Stasko’s heuristics. Domain experts “may also have the right to override heuristics based on domain knowledge or other constraints.”
Zuk, T., Schlesier, L., Neumann, P., Hancock, M. S., & Carpendale, S. (2006). Heuristics for information visualization evaluation. Proceedings of the 2006 AVI Workshop on Beyond Time and Errors: Novel Evaluation Methods For Information Visualization (BELIV 2006). DOI
The authors present “a part of a scheme for mapping the morphology of the [infovis] design space,” a way of describing how variables in data are represented according to a variety properties. These properties include those the type of data, how it is filtered by the logic of the tool, how it is processed automatically (thanks to human perception/cognition), and controlled properties (specifically text, which requires a greater cognitive investment from users).
Data types
Ordinal
Quantitative
QX: intrinsically spatial (e.g. geographical)
Nominal
N×N: nominal set mapped to itself (e.g. graphs)
Mediation
Unmediated: the Data itself
Filter or function (e.g. sorting, multidimensional scaling, sliders, etc.)
D′: recoded data (e.g. nominal data mapped to screen coordinates)
Automatic processing
Position in space-time (X, Y, Z, T)
*: non-semantic use of space-time
Retinal properties
Colour
Size
—: connection
[]: enclosure
Marks
Point
Line
Surface
Area
Volume
Controlled processing
Tx: text
The authors collect these properties in a table and go through classifying certain visualization methods by how they use (or don’t) them.
The proposed classification system aims to act as “a framework for designing” visualizations. It seems it would serve well when trying to justify a novel representation: a careful mapping of data to the graphical and perceptual principles which will describe it.
Card, S. K., & Mackinlay, J. (1997). The structure of the information visualization design space. Proceedings of the 1997 IEEE Symposium on Information Visualization (INFOVIS 1997). DOI
The authors present a measure of visual clutter called feature congestion, how difficult it is to add a new salient item to a display. They describe how it is found and how it relates to variance in contrast and colour. They compare the measure with subjective human ratings of clutter and find a high level of agreement.
“Clutter is the state in which excess items, or their representation or organization, lead to a degradation of performance at some task.” The task discussed in the paper is visual search. In order to improve performance of this task, search targets must be perceptually salient; to be salient, they must differ along some dimension (colour, luminance, motion, orientation, etc.) from distractors.
The authors’ method calculates the variance of certain features—colour and luminance contrast—in a display, at different levels of detail. They then combine across scales, taking the maximum for a given pixel at any scale, and then average for the entire image. This produces a measure of feature congestion: just how much of the perceptual feature space has been covered by items in the display, something inversely proportional to the ease of adding a salient (”outlier”) element. This measure appears to rank weather and city-scale maps by clutter about as well as an untrained human.
It’s interesting to think of clutter as difficulty of having certain information stand out. That a statistical algorithm can be used to evaluate images is even better. The authors of this paper have recently released MATLAB code to measure clutter in images (though it appears to calculate a different measure than the one described in this paper).
Rosenholtz, R., Li, Y., Mansfield, J., & Jin, Z. (2005). Feature congestion: a measure of display clutter. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2005). DOI
Few has written a practical guide to designing and arranging business-oriented dashboards. The book covers basic perceptual principles as they relate to design, a variety of methods of displaying data, and a bunch of examples (cautionary, mostly) and rules-of-thumb. It’s written in clear, casual language, aimed, it seems, at non-design professionals (software designers and business people). Those better versed in design could get away with reading the chapter abstracts and headings.
The author clearly defines a dashboard as a “visual display of the most important information needed to achieve one or more objectives which fits entirely on a single computer screen so it can be monitored at a glance.” This seems much more static (single screen, little-to-no interaction) than what I would imagine most infovis tools would be.
Few gives thirteen common mistakes in dashboard design:
Exceeding boundaries of a single screen
Inadequate context for the data
Displaying excessive detail or precision
Choosing a deficient measure
Inappropriate representation of data (aka. “display media”)
Meaningless variety
Poorly designed display media
Encoding quantitative data inaccurately
Arranging the data poorly
Ineffectively highlighting important data (if at all)
Useless decoration
Misusing colour
Unattractive display
Few discusses a number of tried-and-true methods for representing data (bar and line graphs, scatter and box plots); making it clear that area and angle-based representations such as pie charts should not be used for anything, ever. He likes treemaps, is no fan of star plots.
Few also introduces bullet graphs: a display of a single key measure and a target, sometimes in the context of certain ranges. These he presents as an alternative to thermometers and radial gauges. They’re space-efficient and easier to stack, though unfamiliar.
For the remainder of the book, Few goes on to elaborate—more often repeat—his advice on layout. Most everything he speaks of agrees with or is taken from the work of other prominent interface and information design specialists: reducing the amount of non-data ink (pixels), avoiding cute but poor displays, being consistent, etc. Given the examples he’s found of actual dashboards, it’s all sorely needed.
The authors claim that supporting analysis of data beyond just representing it will make a visualization more useful. They identify two analytic gaps: the rationale gap and the worldview gap. For each gap, they propose three tasks to be used as goals in design or to be met in an evaluation.
The rationale gap is “between perceiving a relationship and expressing confidence in the correctness and utility of [it].”
Expose uncertainty “in data measures and aggregations, and [show] the possible effect of this uncertainty on outcomes.”
Concretize relationships
Formulate cause and effect “by clarifying possible sources of causation.”
The worldview gap is “between what is shown to a user and what actually needs to be shown… for making a decision.”
Determine domain parameters: combining knowledge and metadata, showing what’s relevant and important.
Multivariate explanation by “providing support for discovery… of useful correlative models and constraints.”
Create and confirm hypotheses
These tasks promote analytic primacy, the aim of which is to create systems that generate high-level, actionable knowledge about a domain. This aim stands in contrast to what the authors call “representational primacy” which is the “pursuit of faithful data replication and comprehension”, a notion that “can be limiting” and “probably represents uncertainty as to how to best support [users'] needs.”
The identification of these tasks certainly seems important and much needed. The authors’ tasks clarify the core purpose of a visualization, to place tools back in terms of their usefulness in supporting understanding and action. That said, the tasks are quite difficult and high-level themselves. Zuk et al. suggest that domain knowledge is important when applying these heuristics.
Amar, R., & Stasko, J. (2004). A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations. Proceedings of the IEEE Symposium on Information Visualization (INFOVIS 2004). DOI
The authors devise a method for evaluating users’ insights when interacting with a visualization. This is significantly different from evaluation methods currently in practice because it’s less about the usability and perceptual ease, more about the practical benefits (and goals) of a visualization. A group of visualizations of microarray data were evaluated using this new method.
The authors define an insight as “an individual observation about the data by the participant, a unit of discovery,” which can be said to have certain aspects:
Fact: the actual finding
Time required to reach the insight
Significance (aka. Domain value)
Hypothesis: whether this leads to the formation of a new direction of inquiry
Breadth vs. Depth: the level of detail and generalization
Expected: whether the insight came as a result of a directed search or as a surprise
Correctness
Category (which may be domain specific)
Overview
Patterns
Groups/clusters
Details/specific
The authors found that “multiple visual representations [in a single visualization application] gave Spotfire users more confidence that they did not miss information” and appeared to result in higher numbers of insights. (Also “despite having a large feature set, [Spotfire] has a learning time almost equivalent to the simple tools. This is likely due to… brushing and dynamic query concepts,” linked views and the like.)
Unfortunately, their study is somewhat weak: small numbers of subjects, many conditions, no details on what analysis was carried out or measures of effect size. (Ugly, straight-from-Excel graphs too.)
Saraiya, P., North, C. & Duca, K. (2004). An Evaluation of Microarray Visualization Tools for Biological Insight. Proceedings of the 10th IEEE Symposium on Information Visualization (INFOVIS 2004). DOI
The paper describes a study comparing pan & zoom and rubber sheet navigation, finding that pan & zoom was significantly faster and required less mental effort than rubber sheet navigation. Further, they found that, contrary to previous research, overviews did not improve performance, “but were still perceived as beneficial by users.”
Of course, it is possible that these findings may hold only for the tree analysis tasks studied in this paper. Subjects were asked to find nodes, topological distances between nodes, and compare subtrees. Perhaps in other settings (cartographic data, microscopic data, non-hierarchical networks; ones familiar to users) rubber sheet or overview navigation will be more useful in that they can establish a meaningful context. The authors acknowledge this and say they plan to address it in future studies.
The authors also mention that the strategies for accomplishing tasks had an impact on performance, and so may also cause certain methods of navigation to work better than others. (Task-based evaluation is better than concluding about the absolute value of abstract methods, yeah.)
You read this paper for a course on ANOVA experiment design, and so were interested primarily in how well the experiment was described, carried out, and the data analyzed. It was all done quite well—the authors even reported effect sizes.
Nekrasovski, D., Bodnar, A., McGrenere, J., Guimbretière, F., & Munzner, T. (2006). An evaluation of pan & zoom and rubber sheet navigation with and without an overview. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2006). DOI
The authors discuss the mapping of communicative goals to visual representations through cognitive tasks, particularly how this his done through their AutoBrief system.
“Visualizations are used… to succinctly convey complex relations or to organize a large amount of information… [and] are planned to achieve communicative goals.” The authors propose that users specify these goals (ex. “insufficient airport capacity is the cause for some late cargo”) which are then translated to cognitive tasks (selecting a subset of data, accessing values, contrasting them, etc.) which are, in turn, mapped to certain visual representations.
The authors discuss their AutoBrief system in great detail: the syntax for expressing goals, the working of the tasks, the resultant graphics. You believe that much of what is accomplished by this automation is the responsibility of a designer.
The idea that visual representations must serve communicative goals is a good one. You should be careful to justify your task model and representations with examples of how they can serve communicative goals. (Perhaps by giving short, conversational examples of each task, as if a user were explaining her conclusions and insights with your visualization.)
Kerpedjiev, S., & Roth, S. F. (2000). Mapping communicative goals into conceptual tasks to generate graphics in discourse. Proceedings of the 5th International Conference on Intelligent User Interfaces (IUI 2000). DOI
The authors summarize a variety of methods of evaluating the quality of tools’ interfaces, user experiences, and perceptual effectiveness. These can and should, according to the authors, be applied in visualization research.
The authors say that visualization tools should:
Visually represent data to enhance analysis
Display users’ mental models, interpretations, ideas, hypotheses, and insights
Help users improve mental models by finding supporting/contradictory evidence
Help users organize and share ideas
They claim “human factors methodology and stringent evaluation techniques by the visualization community is in its infancy.” Many of the methods they describe are well known and taken for granted in HCI: user-centred design, cognitive walkthroughs, rapid prototyping, etc. The authors also mention less task-based methods such as perception- and cognitive-based design. They illustrate the usefulness of these with many examples.
In the introduction, the authors try to make a distinction between “continuous model visualization” and “discrete model visualization;” you weren’t clear on what that was, and why they felt it was necessary. It didn’t seem very important. The authors referred to one of their earlier papers for “a complete description and justification” of the terminology.
Tory, M., & Möller, T. (2004). Human factors in visualization research. IEEE Transactions on Visualization and Computer Graphics 10, 1, Jan. 2004 (pp 72-84). DOI
Plaisant argues that current evaluative techniques must be changed and enhanced; that long-term studies, meaningful measures, and evidence of practical benefit will help increase adoption of the products of infovis research. The also calls for more case studies (success stories) and repositories of test data to create benchmarks.
The author identifies how current evaluations focus on controlled experiments (comparing design elements, tools), usability evaluations, and a few case studies. She says that testing should focus on demonstrating utility in the real-world, “matching tools with users, tasks, and real problems.” Adoption requires that potential users see the relevance and benefit of the research right away.
Plaisant also suggests that more long-term studies be carried out in order to capture the experience of well-trained users. She also would like some way of discovering and measuring the “the chances of discovery”, of when a tool can “answer questions you didn’t know you had.”
I believe that this paper will be useful both when designing the evaluation of a visualization as when writing up the report, to ensure that the evidence is stated in a way that will get people interested in actually using it.
Plaisant, C. (2004). The challenge of information visualization evaluation. Proceedings of the Working Conference on Advanced Visual Interfaces (AVI 2004). DOI
The authors present the parallel coordinate tree, which combines multidimensional analysis with a tree structure representation for the visualization of survey data. They also include a lens distortion method for navigating the representation.
The parallel coordinate tree shows the top- (3 indices), mid- (23 quality dimensions), and low-level (80 questions) values collected in the annual survey, each path representing a certain segment of respondents. The intensity of grey is used to denote parent-child relationship. Filtering and highlighting options appear at the bottom of the window.
Parallel coordinate trees may be more limited than what is demonstrated by the data used in this paper: it’s of balanced, similar values in a uniform range, with a shallow hierarchy without sparse sub-trees. “Preliminary experiments have shown, that this method also works with somewhat unbalanced trees with up to a few hundred leaves and about five hierarchical levels deep.”
Parallel coordinates are best for showing relationships between dimensions (and the order of these dimensions along the horizontal are very important). The tool appears to want to show values, not relations between responses to related questions. Borders between different subtrees are not clearly marked, and the continuous line at each level implies a relationship between the data at the leaves that may not exist and can be misleading. (In other words, you’d have preferred values to be represented by some discontinuous way: dots on each axis, with changes in the saturation of their colour, in addition to the shade of the background, to emphasize location in the hierarchy.)
Brodbeck, D., & Girardin, L. (2003). Visualization of large-scale customer satisfaction surveys using a parallel coordinate tree. Proceedings of the 2003 IEEE Symposium on Information Visualization (INFOVIS 2003). DOI, PDF
Opinion Observer provides “a visual side-by-side and feature-by-feature comparison” of products, the basis of which are data mined from natural language reviews taken from sources on the web.
Colours represent different products. The height of the bars represent the number of positive (above the horizontal axis) and negative (below) mentions of that feature; a numeric summary of this appears at the bottom of the graph area.
While the visualization does facilitate a feature-by-feature comparison of products, it does not provide any other level of representation (for example, products vs. products). Also, the method of mining does not distinguish between strong or weak opinions or even neutral mentions of a feature. This coarse analysis results in a clean visualization, but perhaps a misleading one.
The authors spend most of the paper discussing their data mining method and the structure of the mining system and almost none on the justification for, and effectiveness of, the visual representation of the data. It appears to be considered just one feature among many of the Opinion Observer application.
Liu, B., Hu, M., & Cheng, J. (2005). Opinion Observer: Analyzing and Comparing Opinions on the Web. Proceedings of the 14th International Conference on World Wide Web. DOI
The professor’s advice on writing clear research papers, giving good presentations, as well as dealing with reviews of the work.
Many things to keep in mind while writing a paper, likely the most important is to explain what and why before how at every level, be it paragraph, section, or the entire paper. Also, to explicitly state the contribution of your research.
She also identifies certain kinds of papers (some may be mixed):
technique: here’s how to do…
design study: justifying visual encoding choices
system (important to tell us why we should care)
evaluation
model: taxonomy as aid to thinking, finding gaps
Munzner, T. (2006). Writing infovis papers. Information Visualization (CPSC 533C, Fall 2006). PDF
The authors argue that there need to be guidelines on how to use video in HCI that protects subjects and does not misrepresent research/product. These guidelines are difficult to adapt from other professions and must be created for use in HCI. The authors develop pretty thorough ethical guidelines, despite thinking them only a first-step:
Before recording
Establish what constitutes informed consent
Inform subjects of the presence of cameras
Explain the purpose of the video
Explain who will have access to the video
Explain possible settings for showing the videotape
Explain possible consequences of showing the video
Describe potential ways video might be disguised
After recording
Treat videotapes of users as confidential
Allow users to view videotapes
If use of the videotape changes, obtain permission again
In Editing
Avoid misrepresenting data
Distinguish between working prototypes and finished products
Label any changes made to enhance technology
Presenting
Protect users’ privacy
Do not highlight clips that make users look foolish
Educate the audience
Do not rely on the power of video to make a weak point
Summarize data fairly
Distributing
Do not use videos for purposes for which they were not intended
The reasons for applying these (or similar) constraints on video use are anecdotal or hypothetical: the current system may not be “broken.”
It makes me think of how I (mis)used video in my undergrad HCI project. We were given no guidelines (other than “edit it down to less than ten minutes”) on how to present things, protect subjects, and keep ourselves from using our user tests inappropriately.
Mackay, W. (1995). Ethics, lies and videotape…. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 1995). DOI
The author describes a framework for understanding research in general and to outline and compare various research strategies, comparative, and inferential methods.
It’s rather long and covers a lot of material that (should be) familiar to anyone involved in any kind of research. There’s also quite a bit of careful terminology to keep track of.
It summarizes a huge range of research methods and strategies very well, pointing out weaknesses and strengths in general. This puts many methods that may come up in research/reading in a perspective, among a range of others, which can help in analysis of the research itself.
Reminds me of the psychology and HCI courses I took in undergrad: experiment design, research methods, etc.
McGrath, J. (1994). Methodology matters: Doing research in the behavioural and social sciences. In Baecker, R. M., Grudin J., Buxton, W. A. S., & Greenberg, S. (Eds.) Readings in Human-Computer Interaction: Toward the Year 2000 (pp. 152–169).
