From Interaction to Intent
Inferring User Objectives from Provenance Logs
From mouse hovers alone, a model can tell which of seven analysis tasks a person is doing in a scatterplot of high-dimensional data, at about four times the rate of guessing.
@misc{holter2026interaction,
title = {From Interaction to Intent: Inferring User Objectives from Provenance Logs},
author = {Holter, Steffen and St{\"a}hle, Tobias and Narechania, Arpit and El-Assady, Mennatallah},
year = {2026},
eprint = {2607.04501},
archivePrefix = {arXiv},
note = {Under review}
}What the model guesses right now
It only sees hovers up to this moment, the same way it would in a live tool.
Top strip, the task the person was actually doing. Bottom strip, the model's top guess at each moment. Drag to scrub.
Synthetic sample for layout testing, not real participant data. Replace it by running web/scripts/export_sample.py on the Zenodo dataset.
The question
Visual analytics tools record every hover, click and selection a person makes. These provenance logs are usually kept for undo or for replaying a session later. We asked whether they also say something about intent, that is, whether a tool could work out what a person is trying to find while they are still looking.
What people did
Crowdworkers explored scatterplots of high-dimensional data, made with PCA, t-SNE and UMAP, and each was given one of seven analysis tasks such as naming a cluster, finding points similar to a given one, or comparing two clusters. We recorded every hover. The seven tasks fall into three broader patterns of movement1The grouping is defined from theory, not learned. Local exploration stays inside one neighbourhood, global scanning covers the whole plot, and comparative probing moves back and forth between regions., which the figure above shows as three color families.
The data covers four source datasets. Models are trained on basketball, Pokémon and weather data and tested on recipes, which they never see during training.
Two models, two jobs
The first stage classifies one complete hover sequence. An XGBoost model works from 39 summary features of the whole sequence, and a bidirectional GRU works from 24 features computed at every step.
The second stage is the harder, live version of the problem. Several tasks are stitched into one session, and a forward-only GRU with a CRF layer labels the current task at every step using only the hovers seen so far. This is the model the replay figure imitates.
How well it works
With seven tasks, chance is about 14 percent. Both first-stage models land around four times that.
| Model | Labels | Cross-validation accuracy | Held-out recipes accuracy |
|---|---|---|---|
| XGBoost | 7 tasks | 0.549 | 0.621 |
| BiGRU | 7 tasks | 0.568 | 0.598 |
| XGBoost | 3 groups | 0.678 | 0.690 |
| BiGRU | 3 groups | 0.692 | 0.730 |
Labelling tasks live, in stitched sessions, is harder. The online model reaches a segment F1 of 0.40 with seven tasks and between 0.54 and 0.56 with three groups in cross-validation.
Order matters. Shuffling the steps of each sequence lowers the BiGRU’s macro F1 by 0.043, and removing the features that carry state across steps, such as revisits and coverage so far, lowers it by 0.096, the largest drop in any ablation.
Reproduce it
The code turns the published dataset into every table and figure in the paper.
make data # download the dataset from Zenodo
make all # reproduce every table and figure
Expected numbers are listed in the repository’s results/REFERENCE.md.
Citation
@misc{holter2026interaction,
title = {From Interaction to Intent: Inferring User Objectives from Provenance Logs},
author = {Holter, Steffen and St{\"a}hle, Tobias and Narechania, Arpit and El-Assady, Mennatallah},
year = {2026},
eprint = {2607.04501},
archivePrefix = {arXiv},
note = {Under review}
}