q Jiaming Wei · Research Portfolio
UCL MSc Dissertation · Web Agents · Representation Routing

When does a web agent actually need expensive multimodal context?

The oracle says sometimes. The learned router says the harder part is predicting when. This project turns that gap into the result: a real routing opportunity can exist while the supervision needed to learn it is structurally scarce.

See the result Repository Watch demo
Submitted · REALM @ EMNLP 2026 Second Workshop for REsearch on Agent Language Models
Submitted · VLM4RWD @ NeurIPS 2026 Grounded and Faithful Vision-Language Models for Real-World Deployment
01 · Result

A negative router result with a positive scientific explanation.

The project first proves that routing is worth attempting, then shows the learned policy fails under honest evaluation, and finally diagnoses where the learning problem breaks.

Opportunity
+3.45–16.07pp

There is something worth routing.

An oracle representation portfolio improves success while costing 13.7–35.3% less than the strongest fixed reference in the corresponding comparison.

Learnability
0 / 6

The learned router does not recover the ceiling.

Under true nested cross-validation, none of the six evaluated VWA cells Pareto-dominates the trivial always-cheapest policy.

Diagnosis
2–27%

The bottleneck is label supply.

A useful “which representation mattered?” label appears only after the task is solved. Low base success means useful supervision is scarce before classifier choice becomes the main issue.

02 · Question

Not “vision or no vision?” — which part of the representation matters?

The six-mode study factorizes text content, prompt style, and rendered image context so that multimodality is not treated as one monolithic switch.

Research question
When is expensive multimodal context truly necessary — and can that “when” be predicted cheaply enough not to erase the benefit?
Headline answer: necessity is state-dependent, but in the measured regime the decision is not learnable cheaply enough. The failure mechanism is part of the contribution.
Text content × prompt style
DOM prompt
SoM prompt
DOM text
domtext-only baseline
phantom_promptprompt axis isolated
SoM-style text
phantom_textcontent axis isolated
phantom_somno rendered annotation image
visionraw screenshot
somannotated screenshot + matching text

Important: the phantom modes are experimental factorization tools, not claimed as universal cheap replacements for full SoM.

03 · Evidence chain

The result only matters because the argument closes end to end.

A failed router alone is weak evidence. The scientific story needs an oracle ceiling, honest controls, a failed learned policy, and a mechanism that explains the failure.

01

Prove the opportunity exists first.

The oracle can choose representations task by task and beats fixed policies on success–cost trade-offs. Without this step, “the router failed” could simply mean there was nothing useful to route.

Primary evidencephase1_full_prereg_decision.json
router_objective_ordering.md
02

Show the representations are complementary — then keep the noise caveat.

Phantom arms uniquely solve subsets of tasks, but the project does not inflate that into a replacement claim. Same-condition rerun instability is large enough that small drop-one effects must be interpreted cautiously.

Diagnostic boundarydrop-one structure + same-mode rerun noise
caveat kept explicit
03

Evaluate the router without leaking model selection into the answer.

True nested cross-validation, a label-shuffle null, and the fixed always-cheapest baseline make the negative result meaningful. The learned policies can hit trade-off points, but none dominates the trivial policy.

Primary evidencerouter_triage_learnability.md
6 VWA evaluation cells
04

Locate the failure before proposing “a better classifier.”

Routing labels are endogenous to task success: if a task is not solved, it cannot reveal which representation was necessary. At low base success, resplitting or tuning a classifier cannot manufacture missing solve events.

Primary evidencerouter_label_supply_diagnosis.md
4/6 cells lack a trainable label regime
04 · Mechanism

The router is trained on a signal the agent must first earn.

This is why the failure is not summarized as “the classifier was weak.” The scarcity sits upstream of classifier selection.

How a useful routing label is created
Run the task1agent interacts with a live web environment
→
Solve event?2–27%observed base success regime
→
Routing labelrarewhich representation actually mattered becomes observable

The training set is therefore filtered by the agent’s own competence. When solve events are rare, the label process becomes the bottleneck. More flexible classifiers do not fix missing events.

05 · Engineering

The research result sits on top of a reliability system.

Live web benchmarks are stateful, authenticated, slow, and easy to contaminate. The infrastructure treats evaluation integrity as part of the research method, not backstage plumbing.

Experiment contract
preregistered configs→ race-safe queues→ web-agent runner→ versioned episodes→ validation→ analysis

The goal is not simply to run many episodes. It is to make every scored claim traceable through resets, logs, scored-set accounting, contamination checks, and deterministic analysis entry points.

Race-safe orchestrationPer-site serialization, reset discipline, restart-safe queues.
Schema-versioned recordsEpisode logs that survive migrations and partial runs.
Contamination watchdogsAuth, quota, stale state, and invalid-run checks before analysis.
Regression disciplineTests for failure modes that previously created silent false confidence.
06 · Claim boundaries

What the project establishes — and what it deliberately does not.

Negative results are easy to overstate in either direction. The portfolio keeps the claim ladder visible.

Supported

  • Representation needs vary by task; a meaningful oracle routing ceiling exists.
  • The learned router fails to Pareto-beat always-cheapest in the six nested-CV VWA cells evaluated.
  • Low solve rates make informative routing labels structurally scarce in the observed regime.
  • Phantom representations are useful for factorizing signal sources and studying complementarity.

Not claimed

  • phantom_som is not a universal cheaper replacement for full SoM.
  • The result does not say representation routing is impossible on stronger models or other benchmarks.
  • Dollar/token cost is not presented as a direct measurement of energy use or carbon emissions.
  • Small complementary effects are not treated as mechanism proof when they sit inside observed rerun instability.
07 · Research status

Submitted to two 2026 workshops, with the portfolio kept truthful about status.

Both are listed as submissions, not acceptances. The public page will be easy to update when decisions arrive.

Submitted

VLM4RWD @ NeurIPS 2026

Grounded and Faithful Vision-Language Models for Real-World Deployment · NeurIPS 2026.

Official workshop site ↗
08 · Demo

A visual walkthrough of the phantom routing space.

The demo is supporting material for the representation factorization. The scientific headline above is intentionally broader than the old “phantom mode wins” narrative.

Project walkthrough · representation space, experiment design, and evidenceLocal asset · no external player
09 · Go deeper

For reviewers, recruiters, and people who want the receipts.

Code + evidenceGitHub repositoryImplementation, analysis, tests, experiment controls. Research overviewCurrent READMECanonical public summary and research boundaries. ExplainerPhantom spaceAnimated representation-space walkthrough. Authorquarkspace.topResearch, evaluation, agent safety, and selected projects.