Reliable real-time failure detection is critical for deploying learned robotic policies, yet defining failures explicitly is infeasible in open-world settings. This motivates failure detection from nominal demonstrations alone. Existing methods, however, struggle to balance robustness to benign variations with the ability to capture semantic and interaction-level errors. We introduce Robot-WATCHDOG, a self-supervised framework that models manipulation scenes as object-centric spatio-temporal graphs. Building on this representation, we investigate two complementary failure scores: Trajectory Projection and Correlation (TPC), a non-parametric measure of geometric deviation and relational consistency, and GnnT, a Graph Attention Network with a Transformer that predicts future object states. We evaluate both on a public benchmark and on a new dataset of semantically complex manipulation tasks, across six tasks and ten random seeds. Both scores outperform image-based failure detectors, particularly on semantic and interaction-level failures. TPC provides the strongest performance in AUPR and MCC on both dataset averages, with low variance across seeds.
A substantial class of failures in robotic manipulation does not arise from abnormal motion or corrupted observations, but from incorrect interactions between otherwise familiar objects: grasping the wrong item, violating task-order constraints, or placing an object in an incorrect spatial relation to others. These errors often produce perfectly nominal kinematic profiles and only subtle pixel-level changes, which makes them nearly invisible to action-consistency metrics and to standard visual anomaly detectors alike.
The central idea of this work is to recast failure detection as monitoring the evolution of object interactions over time. Rather than treating visual and kinematic signals as separate modalities, we represent the scene as a spatio-temporal graph of interacting objects, where nodes encode object states and edges encode their pairwise spatial relationships throughout execution. This formulation generalizes prior monitoring approaches — visual detectors can be read as operating on node appearance, kinematic monitors as tracking motion along trajectories — while additionally exposing the relational structure required to detect task-relevant interaction failures.
Each episode is converted from RGB observations into a sequence of 2D trajectories over tracked labeled objects, using a vision-language initialization model together with a tracking pipeline. This yields a temporal graph sequence \( \{ \mathcal{G}_t \}_{t=1}^{T} \), where at each time step the scene is abstracted as \( \mathcal{G}_t = (V, E_t, X_t) \). Vertex features encode absolute positions and instantaneous velocities, \( x_{i,t} = [\, p_{i,t},\; p_{i,t} - p_{i,t-1} \,] \), while each edge encodes the explicit pairwise spatial relation between two entities from their relative displacement \( o_{ij,t} = p_{j,t} - p_{i,t} \): \( e_{ij,t} = \left[\, o^{(x)}_{ij,t},\, o^{(y)}_{ij,t},\, d_{ij,t},\, \cos(\theta_{ij,t}),\, \sin(\theta_{ij,t}) \,\right] \in \mathbb{R}^5 \).
This representation has three useful properties for failure detection: (i) translation invariance through relative edge encoding, reducing sensitivity to workspace shifts; (ii) explicit modeling of pairwise object interactions, so that failures involving object identity or spatial arrangement are represented directly; and (iii) temporal consistency through velocity features and graph sequences, enabling detection of both instantaneous and sequential deviations.
Within this representation we define two scoring functions, both referenced or trained on nominal demonstrations only, and both turned into binary decisions by the same conformal calibration. At every timestep each scorer produces one value per tracked object; the scalar failure score \( S_{\text{total}} \) is their sum over objects.
TPC — geometric and relational consistency. The first scorer is non-parametric: it compares the observed episode directly to the set of expert trajectories, without fitting any model. For each tracked object, Trajectory Projection and Correlation computes the orthogonal projection distance from the observed trajectory to the closest expert trajectory of the same label, aggregated over time with exponentially increasing weights so that recent deviations dominate. Since the per-point distance does not depend on the prefix, the score of a streaming episode is maintained incrementally at constant cost per timestep. To capture deviations in interaction structure rather than in individual kinematics, TPC adds a correlation penalty: the \( \ell_1 \) divergence between the pairwise Pearson correlation matrices of the observed objects and of the nominal dataset.
GnnT — predictive spatio-temporal dynamics. The second scorer replaces the direct comparison to expert trajectories by a learned model of the nominal dynamics, and flags deviations from predicted object states. Spatial structure is encoded with GATv2 dynamic attention, so that the network can weight task-relevant interactions (e.g. an end-effector approaching a target object) more strongly than distant or weakly coupled entities. To model action–reaction delays, a sliding window of the past \( m \) spatial embeddings is flattened into \( m \times C \) tokens and processed by a Transformer encoder, with a composite positional encoding accounting for relative time within the window, object identity, and global episode progress. The current-step tokens are projected by an MLP to predict object positions over a horizon \( L \). The network is trained solely on nominal demonstrations, minimizing trajectory MSE together with an edge-consistency term: \( \mathcal{L} = \lambda_p \mathcal{L}_{\text{pos}} + \lambda_e \mathcal{L}_{\text{edge}} \). This formulation can represent action–reaction delays that a purely geometric comparison cannot, at the cost of requiring enough demonstrations to fit the model.
Calibration by conformal prediction. Both scores are calibrated the same way, without task-specific threshold tuning and without any failure examples. Non-conformity scores are computed on a held-out calibration set of nominal episodes, and for a user-specified tolerance \( \alpha \in (0,1) \) a time-varying threshold \( \eta_t \) is built with a functional split-conformal construction: its width is the \( (1-\alpha) \)-quantile of the maximum deviation of calibration episodes from the mean nominal profile. Since we use the uncorrected empirical quantile, the resulting level is treated as approximate.
We evaluate Robot-WATCHDOG on two complementary datasets. The first is BotFails, a public benchmark for robotic failure detection collected with LeRobot. Of its 10 manipulation tasks we retain the 4 involving rigid, visually distinguishable objects — Table-setting, Dish storing, Vegetable sorting and Groceries sorting; tasks involving deformable or fluid interactions are excluded, as they violate the discrete object abstraction underlying our method.
The second is a new dataset released with this work, made of two manipulation tasks with increased relational and dynamic complexity. Table-setting is a structured assembly task on an ALOHA platform, where the robot must place plates and utensils in predefined spatial configurations and in the correct temporal order — emphasizing relational correctness. Waste sorting is a dynamic task on an SO-100 arm, where objects arrive on a moving conveyor belt, so that positions evolve continuously and both initial conditions and interaction timing vary from episode to episode. In total the dataset provides 272 episodes and 170,752 annotated frames of RGB video, proprioception and language, split into expert demonstrations for training and test sets containing both nominal and failure trajectories.
| Task | Split | Robot | # Episodes | # Frames | FPS | Cameras | Action dim |
|---|---|---|---|---|---|---|---|
| Table-setting | Expert | ALOHA | 100 | 67,341 | 15 | 4 views | 9 |
| Table-setting | Test | ALOHA | 21 | 13,471 | 15 | 4 views | 9 |
| Waste sorting | Expert | SO-100 | 100 | 59,562 | 30 | Top view | 6 |
| Waste sorting | Test | SO-100 | 51 | 30,378 | 30 | Top view | 6 |
We compare against trajectory-matching baselines (AP-RMS, AP-Fréchet, Soft-DTW), learned representations (logpZO, lopO, AE-Recon) and the visual-semantic detector FIDeL. Every number is averaged over ten seeds, each redrawing the expert train/calibration partition, the network initialisation and the conformal split.
Both object-centric scores outperform image-based detectors, and TPC is consistently the stronger of the two: averaged per dataset it reaches AUPR 0.811 / maximum MCC 0.751 on BotFails and 0.810 / 0.719 on Watchdog, against 0.623 / 0.488 and 0.648 / 0.631 for GnnT, while the strongest baselines reach only 0.442 / 0.320 (AP-RMS, BotFails) and 0.346 / 0.344 (AP-Fréchet, Watchdog). Trajectory metrics evaluate objects independently, cannot express relational deviations, and are the least stable of all, with seed-to-seed standard deviations up to 0.19 AUPR on the Watchdog average — an order of magnitude above TPC (0.013). Density, reconstruction and visual baselines are steadier but plateau well below both of our scores. The learned score only approaches TPC in the continuously dynamic conveyor scene, which indicates where learned dynamics may justify their additional complexity.
Robot-WATCHDOG relies on a discrete object-centric abstraction, which limits applicability to tasks involving highly deformable materials, fluids, or interaction domains without a clear object decomposition (e.g. painting or soldering). Performance also depends on the upstream vision-language tracking pipeline: severe occlusions, missed detections or identity switches can degrade graph quality and propagate to the anomaly detector. Finally, the current formulation models object centroids only, and therefore cannot capture failures involving orientation or fine-grained pose. Extending the graph representation with 6D pose estimates is a natural direction for future work.
Illustration by Charlotte Szwarc
@inproceedings{rolland2026robotwatchdog,
title={Robot-WATCHDOG: Failure Detection through Object-Centric Graph Representation},
author={Rolland, Quentin and Mayran de Chamisso, Fabrice and Mouret, Jean-Baptiste},
booktitle={Conference on Robot Learning (CoRL)},
year={2026},
}