Trust, Don't Trust, or Flip: Robust PBRL with Multi-Expert Feedback

Figure 1: Overview of TriTrust-PBRL (TTP). A shared reward $R(\tau)$ and expert trusts $\{\alpha_k\}$ are learned jointly from heterogeneous, disjoint preferences. Under a correct reward branch, reliable experts develop $\alpha_k > 0$, noisy experts $\alpha_k \approx 0$, and systematic adversaries $\alpha_k < 0$ (inversion). The two-path decoupled objective aggregates feedback with detached confidence weights on the reward path.

Signed Trust ($\alpha_k$)

Jointly learns a scalar trust parameter per expert without expert features or side metadata. Automatically distinguishes reliable ($\alpha_k > 0$), noisy ($\alpha_k \approx 0$), and adversarial ($\alpha_k < 0$) feedback.

Inversion Instead of Discard

Sample-level filters throw away corrupted labels. TTP's negative trust flips systematically inverted preferences, turning adversarial annotators into high-value training signal even over disjoint comparisons.

Two-Path Decoupling

Separates reward updates $\mathcal{L}_R$ (with detached confidence weights $w_k(t)$) from trust updates $\mathcal{L}_A$. Combined with Stabilized zero-init, it locks onto the true branch and prevents reward gradient leaks.

Overview

Abstract

Preference-based reinforcement learning (PBRL) learns reward functions from pairwise trajectory comparisons, offering an alternative to explicit reward engineering. In practice, preference data often comes from heterogeneous annotators: some accurate, some noisy, and some systematically adversarial. Existing PBRL methods either weight all feedback equally or filter unreliable samples; both struggle when annotators are anti-aligned rather than merely noisy, because filtering discards signal that could be recovered by inversion.

We introduce TriTrust-PBRL (TTP), which jointly learns a shared reward model and expert-specific trust parameters. During training, trusts move toward positive (trust), near-zero (down-weight), or negative (invert) values, so systematically adversarial experts contribute usable signal instead of being thrown away. We accompany the method with a gradient analysis of trust separation and an identifiability result for reward and trust up to affine reparameterization. We also make the remaining global sign ambiguity explicit, together with the reliable-majority condition that resolves it.

Empirically, synthetic diagnostics map when branch selection and trust separation succeed, and end-to-end PEBBLE experiments on four continuous-control domains (MetaWorld and DM Control) show that, under a reliable majority with adversarial or noisy experts, TTP recovers strong policies where uniform-weight PEBBLE collapses and improves over sample-level robust baselines in the adversarial setting. TTP needs no expert features beyond an identity index and plugs directly into existing PBRL pipelines.

Interactive Rollouts

Benchmark Environments & Policy Videos

Evaluation rollouts on continuous-control tasks spanning robotic manipulation (MetaWorld) and locomotion (DMControl) under 25% adversarial corruption, formatted natively at 256×256 resolution (15 fps).

DMControl 256×256 • 15fps

Cheetah-Run

DMControl • Dense Locomotion

Long-horizon forward running. Under 25% adversarial feedback ($\beta=[1,1,1,-1]$), TTP learns stable forward running while standard PEBBLE stalls. Notably, we achieve state-of-the-art Oracle SAC performance solely by leveraging feedback from heterogeneous scripted teachers.

256×256 • 15fps Matches Oracle SAC Return ~800+ 1,000 queries

MetaWorld 256×256 • 15fps

Sweep-Into-v2

MetaWorld • Precision Contact

Sweeps an object into a designated hole. With sparse success signals, TTP inverts adversarial feedback to reach >90% success rate.

256×256 • 15fps Success >90% Sparse reward

MetaWorld 256×256 • 15fps

Door-Open-v2

MetaWorld • Articulated Object

Grasps handle, rotates, and pulls door open. TTP rapidly identifies the adversarial annotator and achieves near 100% door opening.

256×256 • 15fps Success ~100% Multi-stage

DMControl 256×256 • 15fps

Walker-Walk

DMControl • Bipedal Locomotion

Tests feedback efficiency across 500 to 5k query budgets. TTP achieves upright bipedal walking even in low query regimes (shown: β=[1, 1, 1, 1, -1] at 1,000 comparisons, return ~941).

256×256 • 15fps β=[1,1,1,1,-1] 1,000 queries Return ~941

Formulation & Theory

Joint Reward & Signed Trust Learning

Bridging Bradley-Terry preference modeling with per-expert signed trust dynamics and two-path decoupled optimization.

The Signed Bradley-Terry Model

Consider $K$ experts $\{e_1, \dots, e_K\}$ providing pairwise preferences over private, disjoint datasets $D^{(k)} = \{(\tau_i, \tau_j, y_{ij}^{(k)})\}$, where $y_{ij}^{(k)} \in \{0, 1\}$. We model expert $k$'s preference probability using the trajectory reward gap $\Delta R_{ij} = R(\tau_i) - R(\tau_j)$ and a learnable scalar trust $\alpha_k$:

$$P\big(y_{ij}^{(k)} = 1 \mid R, \alpha_k\big) = \sigma\big(\alpha_k (R(\tau_i) - R(\tau_j))\big) = \frac{1}{1 + \exp\big(-\alpha_k \Delta R_{ij}\big)}$$

Gradient Dynamics of Trust Separation

Differentiating the negative log-likelihood with respect to $\alpha_k$ reveals three distinct dynamical behaviors:

$$\frac{\partial \mathcal{L}}{\partial \alpha_k} = \sum_{(\tau_i, \tau_j) \in D^{(k)}} \big[\sigma(\alpha_k \Delta R_{ij}) - y_{ij}^{(k)}\big] \cdot \Delta R_{ij}$$

Reliable Experts ($\alpha_k > 0$): Labels align with positive $\Delta R_{ij}$, yielding a negative gradient that monotonically pushes $\alpha_k$ upwards.
Noisy Experts ($\alpha_k \approx 0$): Random feedback oscillates across comparisons, causing statistical cancellation: $\left.\frac{\partial \mathcal{L}}{\partial \alpha_k}\right|_{\alpha_k \approx 0} \approx 0$.
Adversarial Experts ($\alpha_k < 0$): Systematic anti-alignment drives $\alpha_k$ into the negative regime, converting negative feedback into inverted positive signal.

Two-Path Decoupled Objective

To prevent gradient interference between reward learning and trust estimation, TTP decouples training into two distinct computation paths using stop-gradients $\mathrm{sg}(\cdot)$:

$$\mathcal{L}_R = \mathbb{E}_{k, (\tau_i, \tau_j)}\Big[ w_k(t) \, \ell_{ij}^{(k)}\big(\mathrm{sg}(\bar{\alpha}_k) \Delta R_{ij}\big) \Big], \qquad w_k(t) = \frac{K |\tilde{\alpha}_k|}{\sum_{k'} |\tilde{\alpha}_{k'}|}$$ $$\mathcal{L}_A = \mathbb{E}_{k, (\tau_i, \tau_j)}\Big[ \ell_{ij}^{(k)}\big(\bar{\alpha}_k \, \mathrm{sg}(\Delta R_{ij})\big) \Big]$$ $$\mathcal{L}_{\mathrm{TTP}} = \mathcal{L}_R + \mathcal{L}_A$$

Algorithm 1: TriTrust-PBRL PyTorch SGD

1:Input: Expert data $\{D^{(k)}\}_{k=1}^K$, rates $\eta_R, \eta_\alpha$

2:Initialize: $R \leftarrow$ zero-last-layer; $\alpha_k \leftarrow 0.01$

3:for step $t = 1, 2, \dots, T$ do

4: Sample minibatch $(\tau_i, \tau_j, y_{ij}^{(k)})$ from all experts

5: $\Delta R_{ij} \leftarrow R(\tau_i) - R(\tau_j)$

6: $\tilde{\alpha}_k \leftarrow \tanh(\alpha_k)$

7: $\bar{\alpha}_k \leftarrow \tilde{\alpha}_k / \mathrm{detach}(\max_{k'} |\tilde{\alpha}_{k'}|)$

8: $w_k \leftarrow \mathrm{detach}(K |\tilde{\alpha}_k| / \sum_{k'} |\tilde{\alpha}_{k'}|)$

9: $\mathcal{L}_R \leftarrow \mathrm{mean}\big(w_k \ell_{ij}^{(k)}(\mathrm{sg}(\bar{\alpha}_k)\Delta R_{ij})\big)$

10: $\mathcal{L}_A \leftarrow \mathrm{mean}\big(\ell_{ij}^{(k)}(\bar{\alpha}_k \mathrm{sg}(\Delta R_{ij}))\big)$

11: $\mathcal{L}_{\mathrm{TTP}} \leftarrow \mathcal{L}_R + \mathcal{L}_A$

12: Update $R \leftarrow R - \eta_R \nabla_R \mathcal{L}_{\mathrm{TTP}}$

13: Update $\alpha_k \leftarrow \alpha_k - \eta_\alpha \nabla_{\alpha_k} \mathcal{L}_{\mathrm{TTP}}$

14: Project $\bar{\alpha}_k$ step cap $|\Delta \bar{\alpha}_k| \le \delta_{\bar{\alpha}}$

15:return Reward model $R(\cdot)$ and trusts $\{\alpha_k\}$

Empirical Findings

Interactive Results Explorer

Explore end-to-end benchmark learning curves, learned trust separation trajectories, feedback budget heatmaps, and controlled synthetic diagnostics.

Continuous-Control Benchmark Performance

Evaluation curves comparing TTP (Ours) against Oracle SAC, Uniform PEBBLE, RIME, and MCP under 25% adversarial feedback ($\beta=[1,1,1,-1]$) and noisy feedback ($\beta=[1,1,1,0]$). Click any plot to zoom.

Sweep-Into-v2 (Adversarial) Success Rate (%) vs Environment Steps

Door-Open-v2 (Adversarial) Success Rate (%) vs Environment Steps

Cheetah-Run (Adversarial) True Return vs Environment Steps

Sweep-Into-v2 (Noisy) Success Rate (%) vs Environment Steps

Door-Open-v2 (Noisy) Success Rate (%) vs Environment Steps

Cheetah-Run (Noisy) True Return vs Environment Steps

Learned Trust Trajectories ($\bar{\alpha}_k$)

Per-expert trust evolution on MetaWorld Sweep-Into-v2 under adversarial conditions ($\beta=[1,1,1,-1]$). Experts 0, 1, and 2 (reliable) surge into positive trust ($\approx +0.85$), while Expert 3 (adversary) shifts negative ($\approx -0.85$).

Expert 0 ($\beta_0 = +1$) Reliable: Trust moves positive ($\bar{\alpha}_0 \to +0.85$)

Expert 1 ($\beta_1 = +1$) Reliable: Trust moves positive ($\bar{\alpha}_1 \to +0.85$)

Expert 2 ($\beta_2 = +1$) Reliable: Trust moves positive ($\bar{\alpha}_2 \to +0.85$)

Expert 3 ($\beta_3 = -1$) Adversarial: Trust inverts negative ($\bar{\alpha}_3 \to -0.85$)

Feedback Efficiency Across Query Budgets

DMControl Walker-Walk performance under varying pairwise comparison budgets ($500$, $1{,}000$, $2{,}000$, $3{,}000$, and $5{,}000$ queries) and heterogeneous mixtures with $K=5$ experts.

Walker-Walk True Episode Reward Heatmap across Feedback Budgets & Annotator Mixtures Demonstrates robust policy recovery across high and low query regimes.

Controlled Synthetic Mechanism Diagnostics

Isolating branch selection, trust separation, trajectory horizon $T$, and initialization scale under known ground truth $R^*$ without MDP confounding.

Branch Symmetry: 3R1N 3 Reliable / 1 Noisy: Noisy expert converges near 0

Branch Symmetry: 3R1A 3 Reliable / 1 Adversary: Full sign inversion

Branch Symmetry: 1R3A 1 Reliable / 3 Adversaries: Locks onto inverted branch

Horizon Sensitivity: 3R1N Stable across $T \in [10, 200]$ with scaled learning rate

Horizon Sensitivity: 3R1A 100% correct branch recovery across trajectory lengths

Partial & Stochastic Adversaries Trust adapts proportionally to corruption strength

Ablation Studies

Component Analysis & Practical Enhancements

Isolating the role of $\tanh$-bounding, max-normalization, and detached confidence weights $w_k$ on a shared reward head (mixture 2R1N1A, 200 seeds).

Variant	Tanh Bounding	Max-Norm ($\bar{\alpha}$)	Confidence $w_k$	$\bar{\alpha}_R$	$\bar{\alpha}_N$	$\bar{\alpha}_A$	$\|\mathrm{corr}\|$
Raw	—	—	—	$+0.18$	$+0.01$	$-0.16$	$0.97$
+Tanh		—	—	$+0.18$	$+0.01$	$-0.17$	$0.97$
+Tanh, +Max-norm			—	$+0.88$	$-0.02$	$-0.87$	$0.94$
Full TTP (Detached $w_k$)				$+0.86$	$-0.01$	$-0.85$	$0.94$
w/o $w_k$			—	$+0.86$	$0.00$	$-0.87$	$0.94$
Attached $w_k$			Attached*	$+0.43$	$0.00$	$-0.27$	$0.89$

*Note: Max-normalization is the critical catalyst for signed separation, expanding trust to $[-1, 1]$. Attaching $w_k$ to the trust path compresses magnitudes and lowers reward correlation.

Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback