FeVOS: Foresight Expression Video Object Segmentation

Lan, Kehan; Ying, Kaining; Ding, Henghui

FeVOS teaser comparing observed-frame referring segmentation with foresight expression segmentation. — Comparison of related datasets with Foresight Expression Video Object Segmentation (FeVOS). Unlike existing datasets (Ref-DAVIS, MeViS) that ground expressions describing observable events (e.g., in the sink, moved), our task requires predicting which object will be involved in future events based on observed visual cues. In this case, given the foresight expression “What tool will be used?”, the model must analyze temporal context (dirty pot) and spatial cues (hand states) to anticipate the correct target.

01 Abstract

Reasoning about what happens next

Existing Referring Video Object Segmentation tasks focus on referring expressions describing events, actions or appearances of relevant objects within the observed frames, lacking evaluation in scenarios that require pre-decisive spatio-temporal reasoning, thereby limiting their applicability. To address this, we propose Foresight Expression Video Object Segmentation, a task that queries future events in upcoming video segments and requires masks of the objects in the observed frames as visual answers. For example, in ego-centric scenes, the question “What tool will be used?” demands reasoning over spatio-temporal cues to predict the masks of the next tool to be used, which helps with the understanding of future actions and decisions. To support this task, we introduce FeVOS, a dataset with 968 video clips, 14,525 foresight expressions, and 2,904 chain-of-thought annotations to provide explicit and interpretable reasoning steps. We further develop FeVOS-R1, an MLLM-based model trained on our dataset via a two-stage pipeline of supervised fine-tuning and reinforcement learning. FeVOS-R1 not only achieves state-of-the-art performance on FeVOS, but also demonstrates strong generalization to existing RVOS benchmarks. We hope this work can inspire more research on predictive reasoning in video perception.

New Task

Foresight expressions query future events; masks answer in the observed frames.
New Dataset

968 clips, 14.5K expressions and 2.9K chain-of-thought reasoning traces.
New Model

FeVOS-R1 pairs SFT with GRPO for end-to-end, reasoning-aware segmentation.

02 Dataset

FeVOS, built for predictive grounding

Given a video clip and a predictive expression describing future events, the model outputs pixel-level segmentation masks for objects in the observed frames. Unlike traditional RVOS, our task requires reasoning about future actions or events based solely on cues from observed frames.

Through our annotation pipeline, we curated FeVOS, which contains 968 video clips with 14,525 foresight expressions and corresponding pixel-level segmentation masks. Each expression is paired with precise annotations identifying target objects across all frames in the observation segment. Additionally, we generated 2,904 synthetic chain-of-thought annotations to provide explicit reasoning supervision.

968Video clips

14,525Foresight expressions

30,125Referenced frames

2,904CoT annotations

A Look Inside the Annotations

Every figure below is computed directly from the released FeVOS metadata — expressions, splits, and chain-of-thought traces.

Anticipatory by Design

Share of expressions carrying an explicit future-tense cue.

0% future-tense

Future-tense 10,917
Implicit 3,608

Question Types

How the 14,525 foresight expressions are phrased.

What57.1%
Which32.0%
Who7.8%
Where / other3.1%

Foresight Cue Words

Most frequent future-pointing tokens across all expressions.

will9,223
next3,380
collide1,655
upcoming1,425
later1,393
going1,177

Frames per Clip

Observation length varies widely — 31 frames on average.

<10

10–20

20–30

30–40

40–60

60+

Reasoning Trace Length

Chain-of-thought traces average 102 words of spatio-temporal reasoning.

<60

60–80

80–100

100–120

120–140

140+

Watch the Model Anticipate

Real validation clips. The violet mask marks the object a future event will involve — tracked across the observed frames before the event happens.

FeVOS predicted target highlighted across video frames

LIVE Real-world

Foresight expression

What will be cut?

Predicted target

the bamboo leaves

Representative FeVOS samples with foresight expressions and reasoning annotations. — Representative FeVOS samples — foresight questions, chain-of-thought reasoning, and the predicted target masks across observed frames.

Word cloud of FeVOS foresight expressions. — Foresight expressions are dominated by future-tense cues — *will*, *next*, *going*, *collide*.

Word cloud of FeVOS chain-of-thought reasoning annotations. — Chain-of-thought traces reveal the spatio-temporal reasoning behind each predicted target.

03 Method

FeVOS-R1: reason, then segment

Sa2VA is a unified framework that integrates MLLM with SAM2 for referring video object segmentation. The architecture consists of three key components: a vision encoder that extracts visual features from video frames, a large language model that processes both visual embeddings and text prompts to generate responses with special segmentation tokens, and SAM2's mask decoder that produces pixel-wise segmentation masks conditioned on the hidden states of segmentation tokens.

We implement our method FeVOS-R1 based on Sa2VA via a two-stage training paradigm. Given an input video, we first sample a sequence of frames and encode them using the vision encoder to obtain visual embeddings. These embeddings, along with a text prompt, are processed by the LLM to generate a response containing a special token [SEG]. The hidden states of [SEG] are then projected and fed into SAM2's mask decoder to predict a segmentation mask sequence for the input frames.

To equip the model with basic reasoning capabilities, we perform supervised fine-tuning using our synthetic CoT dataset. While supervised fine-tuning provides preliminary knowledge of reasoning, the reasoning process remains suboptimal in quality and weakly aligned with segmentation objectives. We employ GRPO to further refine the reasoning process that leads to high segmentation quality with task-specific objectives.

Unlike existing visual grounding methods that require models to output intermediate representations such as bounding box coordinates in JSON format, we leverage the [SEG] token to support end-to-end optimization that directly maximizes segmentation accuracy.

Stage 1

Supervised Fine-tuning

Learns preliminary reasoning from synthetic chain-of-thought supervision with text and mask losses.

Stage 2

Reinforcement Learning

GRPO refines the reasoning process toward high-quality masks with an IoU reward and KL penalty.

Overview of the FeVOS-R1 two-stage training pipeline.

04 Results

State of the art

We comprehensively benchmark a range of recent video segmentation models on the proposed FeVOS dataset. Zero-shot models exhibit substantial difficulty with our predictive reasoning task, with J&F scores below 31.0. When directly fine-tuning Sa2VA on FeVOS using standard SFT without CoT enhancement, performance improves markedly from 25.4 to 35.8. Our complete training pipeline, incorporating both CoT-guided reasoning and RL-based optimization, further pushes the performance to 42.3, achieving an additional +6.5 gain over the SFT baseline.

This demonstrates that explicit reasoning chains and reward-guided optimization are essential for capturing subtle visual cues required for accurate future event prediction. Notably, the performance on FeVOS is substantially lower than on ReVOS and MeViS, highlighting the increased complexity and challenge posed by predictive segmentation, which requires models to anticipate future events from visual cues rather than grounding expressions about observable events.

On ReVOS, the directly fine-tuned baseline Sa2VA* suffers a performance drop compared to its zero-shot counterpart, suggesting overfitting to FeVOS, while our method achieves 60.3, outperforming both baselines with particularly strong gains on the reasoning subset. On MeViS, our method reaches 49.5, representing a +3.0 improvement over the baseline and surpassing all comparable-sized models. These results demonstrate that our CoT-augmented and RL-enhanced training strategy not only improves in-domain performance but also substantially enhances cross-domain generalization, particularly in reasoning-intensive scenes.

Method	Backbone	J	F	J&F
Zero-shot baselines
ReferFormer	ResNet-50	16.4	20.0	18.2
LMPM	Swin-T	17.1	20.7	18.9
VISA	Chat-UniVi-7B	22.9	28.2	25.6
VideoLISA	LLaVA-Phi-3-V-3.8B	22.9	29.4	26.1
VideoGLaMM	Phi3-Mini-3.8B	21.7	26.7	24.2
VRS-HQ	Chat-UniVi-7B	28.8	33.3	31.0
GLUS	Chat-UniVi-7B	27.4	31.7	29.6
Sa2VA	InternVL2.5-4B	23.7	27.2	25.4
Fine-tuned on FeVOS
GLUS*	Chat-UniVi-7B	31.0	35.9	33.5
Sa2VA*	InternVL2.5-4B	33.1	38.4	35.8
FeVOS-R1 (ours)	InternVL2.5-4B	39.5	45.1	42.3

Main results on the FeVOS dataset. * indicates the model is fine-tuned on FeVOS.

Method	Backbone	ReVOS			MeViS
Method	Backbone	Ref.	Reas.	All	J	F	J&F
ReferFormer	ResNet50	16.9	12.8	14.9	-	-	-
ReferFormer	Video-Swin-B	32.7	23.4	28.1	29.8	32.2	31.0
LMPM	Swin-T	34.1	18.8	26.4	34.2	40.2	37.2
LISA	LLaVA-7B	45.7	36.1	40.9	35.1	39.4	37.2
TrackGPT	LLaVA-7B	-	-	-	37.6	42.6	40.1
VISA	Chat-UniVi-7B	50.9	43.0	46.9	40.7	46.3	43.5
VISA	LLaVA-7B	51.0	43.2	47.1	-	-	-
VideoLISA	LLaVA-Phi-3-V-3.8B	-	-	-	41.3	47.6	44.4
VideoGLaMM	Phi3-Mini-3.8B	-	-	-	42.1	48.2	45.2
VRS-HQ	Chat-UniVi-7B	62.1	56.1	59.1	47.6	53.7	50.6
GLUS	Chat-UniVi-7B	58.3	51.4	54.9	48.5	54.2	51.3
Sa2VA	InternVL2.5-4B	62.5	55.6	59.1	-	-	46.4
Sa2VA*	InternVL2.5-4B	61.0	55.2	58.1	43.4	49.7	46.5
FeVOS-R1 (ours)	InternVL2.5-4B	62.8	57.8	60.3	46.2	52.7	49.5

Quantitative results on ReVOS and MeViS.

05 BibTeX

Cite this work

@inproceedings{FeVOS,
  title={{FeVOS}: Foresight Expression Video Object Segmentation},
  author={Lan, Kehan and Ying, Kaining and Ding, Henghui},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}
}