From Interaction to Intent

Inferring User Objectives from Provenance Logs

From mouse hovers alone, a model can tell which of seven analysis tasks a person is doing in a scatterplot of high-dimensional data, at about four times the rate of guessing.

What the model guesses right now

It only sees hovers up to this moment, the same way it would in a live tool.

  • Name cluster34%
  • Find similar pointsactual task12%
  • Identify outliers4%
  • Generate clusters8%
  • Add data to projection12%
  • Compare clusters22%
  • Map synthesized to original dimension8%

Top strip, the task the person was actually doing. Bottom strip, the model's top guess at each moment. Drag to scrub.

Hover 1 of 119. Actual Find similar points, guess Name cluster.

Synthetic sample for layout testing, not real participant data. Replace it by running web/scripts/export_sample.py on the Zenodo dataset.

The question

Visual analytics tools record every hover, click and selection a person makes. These provenance logs are usually kept for undo or for replaying a session later. We asked whether they also say something about intent, that is, whether a tool could work out what a person is trying to find while they are still looking.

What people did

Crowdworkers explored scatterplots of high-dimensional data, made with PCA, t-SNE and UMAP, and each was given one of seven analysis tasks such as naming a cluster, finding points similar to a given one, or comparing two clusters. We recorded every hover. The seven tasks fall into three broader patterns of movement1The grouping is defined from theory, not learned. Local exploration stays inside one neighbourhood, global scanning covers the whole plot, and comparative probing moves back and forth between regions., which the figure above shows as three color families.

The data covers four source datasets. Models are trained on basketball, Pokémon and weather data and tested on recipes, which they never see during training.

Two models, two jobs

The first stage classifies one complete hover sequence. An XGBoost model works from 39 summary features of the whole sequence, and a bidirectional GRU works from 24 features computed at every step.

The second stage is the harder, live version of the problem. Several tasks are stitched into one session, and a forward-only GRU with a CRF layer labels the current task at every step using only the hovers seen so far. This is the model the replay figure imitates.

How well it works

With seven tasks, chance is about 14 percent. Both first-stage models land around four times that.

Model Labels Cross-validation accuracy Held-out recipes accuracy
XGBoost 7 tasks 0.549 0.621
BiGRU 7 tasks 0.568 0.598
XGBoost 3 groups 0.678 0.690
BiGRU 3 groups 0.692 0.730

Labelling tasks live, in stitched sessions, is harder. The online model reaches a segment F1 of 0.40 with seven tasks and between 0.54 and 0.56 with three groups in cross-validation.

Order matters. Shuffling the steps of each sequence lowers the BiGRU’s macro F1 by 0.043, and removing the features that carry state across steps, such as revisits and coverage so far, lowers it by 0.096, the largest drop in any ablation.

Reproduce it

The code turns the published dataset into every table and figure in the paper.

make data   # download the dataset from Zenodo
make all    # reproduce every table and figure

Expected numbers are listed in the repository’s results/REFERENCE.md.

Citation

@misc{holter2026interaction,
  title         = {From Interaction to Intent: Inferring User Objectives from Provenance Logs},
  author        = {Holter, Steffen and St{\"a}hle, Tobias and Narechania, Arpit and El-Assady, Mennatallah},
  year          = {2026},
  eprint        = {2607.04501},
  archivePrefix = {arXiv},
  note          = {Under review}
}