PoseBridge: Bridging the Skeletonization Gap for Zero-Shot Skeleton-Based Action Recognition

Sanghyeon Lee; Jinwoo Kim; Jong Taek Lee

NeurIPS 2026

PoseBridge

Bridging the Skeletonization Gap for
Zero-Shot Skeleton-Based Action Recognition

Sanghyeon Lee Jinwoo Kim Jong Taek Lee^†

School of Computer Science and Engineering, Kyungpook National University

Daegu, South Korea † Corresponding author

Paper + Appendix arXiv GitHub BibTeX

Paper: arXiv v2, September 2026 Code release forthcoming

The idea Skeleton coordinates keep motion, but can lose the cues that tell actions apart. PoseBridge preserves those cues inside the pose estimator and carries them into zero-shot recognition.

01 / THE PROBLEM

Similar motion. Different actions.

Putting on a hat, headphones, or glasses can produce similar hand-to-head trajectories. Joint coordinates alone do not make the object and its interaction with the body explicit.

Conventional skeleton-text alignment loses object cues during pose estimation. PoseBridge preserves pose-anchored semantics to distinguish a hat, headphones, and glasses. — Figure 1 The **skeletonization gap** begins before skeleton-text alignment. PoseBridge reuses intermediate features from the same pose estimation process to retain action-relevant evidence.

Bridge the information loss at its source. Use the pose estimator as a source of body-grounded semantics, alongside the skeletons it already produces.

Read the abstract

Zero-shot skeleton-based action recognition (ZSSAR) is typically treated as a skeleton-text alignment problem: encode joint-coordinate sequences, align them with language, and classify unseen actions. We argue that this alignment is often too late. Skeletons are not complete action observations, but compressed outputs of human pose estimation (HPE); by the time alignment begins, human-object interactions and pose-relative visual cues may no longer be explicit. We call this upstream semantic loss.

To address it, we propose PoseBridge, an HPE-aware ZSSAR framework that bridges intermediate HPE representations to skeleton-text alignment. Rather than adding an RGB action branch or object detector, PoseBridge extracts pose-anchored semantic cues from the same HPE process that produces skeletons, then transfers them through skeleton-conditioned bridging and semantic prototype adaptation.

Across NTU-RGB+D 60/120, PKU-MMD, and Kinetics-200/400, PoseBridge improves ZSSAR performance under the evaluated protocols. On the Kinetics-200/400 PURLS benchmark, which contains in-the-wild videos with diverse scenes and action contexts, PoseBridge shows the clearest separation, improving the strongest compared baseline by 13.3–17.4 points across all eight splits. Our code will be publicly released.

02 / THE METHOD

Preserve first. Then bridge.

Two coupled stages carry pose-anchored semantics from human pose estimation into skeleton-text alignment.

STAGE 01 · WITHIN HPE

Keep the visual evidence
around the body.

Hierarchical refinement injects fine-grained shallow features into deeper pose-structured features. Body-aware pooling selects cues around predicted joints.

Caption-based semantic alignment encourages these cues to retain action-relevant meaning while preserving the pose estimation objective.

OUTPUT Skeletons + pose-anchored semantic cues

Multi-level pose estimator features undergo hierarchical refinement and body-aware pooling, then align with caption features from a frozen text encoder. — Figure 2 Semantic preservation within HPE.

STAGE 02 · ZERO-SHOT RECOGNITION

Connect motion and meaning.

The frozen pose estimator supplies skeletons and cached semantic cues. A skeleton-conditioned bridge lets motion query the temporal cues; prototype adaptation brings language targets closer to the pose-semantic space.

Skeleton features query pose-anchored cues using cross-attention. Seen-class pose centroids adapt text prototypes; shifts from related seen classes adapt unseen prototypes. — Figure 3 The bridge enriches the recognition query, while prototype adaptation calibrates its semantic targets.

The recognition setting

PoseBridge uses skeletons, intermediate HPE cues, and text. The same pose estimation pass provides both visual outputs; an independent RGB action backbone or object detector is not needed. Unseen-class prototypes use shifts learned from related seen classes, with no unseen visual samples.

03 / MAIN RESULTS

The clearest gains are in the wild.

Evaluated on five benchmarks: NTU-RGB+D 60/120, PKU-MMD, and Kinetics-200/400.

+13.3–17.4pp

over the strongest compared baseline
across all eight Kinetics ZSL splits

55.6% on Kinetics-200 180 seen / 20 unseen

52.3% on Kinetics-400 360 seen / 40 unseen

Kinetics-200/400

ZSL top-1 accuracy (%) ↑ · Paper Table 3

In-the-wild recognition under the PURLS benchmark. Split labels denote seen / unseen classes.

Kinetics-200 and Kinetics-400 zero-shot top-1 accuracy
Method	Kinetics-200				Kinetics-400
Method	180/20	160/40	140/60	120/80	360/40	320/80	300/100	280/120
DeVISE	22.2	12.3	8.0	5.7	18.4	10.2	9.5	8.3
ReViSE	25.0	13.3	8.1	6.2	20.8	11.8	9.5	8.2
PURLS	26.0	15.9	10.2	7.8	22.5	15.1	11.4	11.0
PURLS*	32.2	22.6	12.0	11.8	34.5	24.3	17.0	14.3
TDSM	38.2	24.4	15.3	13.1	38.9	26.2	18.5	16.1
PoseBridge	55.6	38.3	28.6	26.4	52.3	39.6	32.9	32.1
Gain vs. TDSM	+17.4	+13.9	+13.3	+13.3	+13.4	+13.4	+14.4	+16.0

* Multiple text prompts. Other rows, including PoseBridge, use a single prompt. Gains are percentage points.

NTU-RGB+D 60/120

Standard Xsub splits · Paper Table 1

ZSL top-1 accuracy (%) and GZSL harmonic mean H (%). Selected recent comparisons from the paper.

NTU60 and NTU120 standard-split ZSL and GZSL results
Method	NTU60 · 55/5		NTU60 · 48/12		NTU120 · 110/10		NTU120 · 96/24
Method	ZSL	H	ZSL	H	ZSL	H	ZSL	H
SCoPLe	84.1	70.8	53.0	57.9	74.5	62.3	52.2	52.2
Neuron	86.9	71.4	62.7	59.1	71.5	63.3	57.1	53.6
TDSM	86.5	—	56.0	—	74.2	—	65.1	—
FS-VAE	86.9	75.7	57.2	52.1	74.4	63.3	62.5	54.7
BSZSL†	83.0	78.0	53.0	48.8	77.7	75.8	56.1	61.6
SKI-VLM†	82.2	—	52.0	—	77.5	—	59.3	—
PoseBridge	88.8	83.0	73.2	68.0	80.3	68.0	70.9	62.6

† Additional RGB pathway with a pretrained CLIP visual encoder. “—” means unreported. PoseBridge leads ZSL in all four standard splits and GZSL-H among compared methods without an extra RGB pathway.

Beyond a single class split

Random Xsub splits · Paper Table 2

Average of three random seen/unseen splits. ZSL accuracy (%) and GZSL-H (%).

Random-split results on NTU60, NTU120, and PKU-MMD
Method	NTU60 · 55/5		NTU120 · 110/10		PKU-MMD · 46/5
Method	ZSL	H	ZSL	H	ZSL	H
SA-DVAE	84.2	75.3	50.7	47.5	66.5	54.7
SCoPLe	83.7	77.7	53.3	54.1	71.4	54.9
TDSM	88.9	—	69.5	—	70.8	—
FS-VAE	—	—	—	—	71.2	59.0
PoseBridge	90.5	78.2	74.8	66.0	77.5	71.9

GZSL-H improves by 11.9 points on NTU120 and 12.9 points on PKU-MMD over the strongest compared baseline. Selected comparisons; full results are in the paper.

Lightweight recognition

ZSSAR stage only · Paper Table 4

Measured on one NVIDIA RTX A6000, including skeleton and text encoders. These are recognition-stage measurements after skeleton extraction, not end-to-end video throughput.

Recognition-stage computational cost and throughput
Method	Trainable / total params (M)	GFLOPs ↓	FPS ↑
Neuron	2.3 / 126.7	7.5	1165.4
TDSM	260.4 / 384.2	10.6	772.5
FS-VAE	0.2 / 64.4	3.7	1237.0
SKI-VLM	2.8 / 215.9	281.0	22.6
PoseBridge	3.8 / 128.2	7.3	1398.0

Pose estimation and cue extraction incur additional cost. See Appendix D.3 for component-wise pipeline costs.

ZSL: classify unseen actions only. GZSL: classify both seen and unseen actions; H balances their accuracies, H = 2SU / (S + U).

Full experiments & ablations

04 / A CLOSER LOOK

Recover the cues that distinguish actions.

“Folding paper” and “typing on a keyboard” can look similar in skeleton space. Pose-anchored cues provide evidence around the hands and manipulated objects.

Baseline Grad-CAM attends broadly to the body and the two action classes overlap in t-SNE. PoseBridge attends around the hands, paper, and keyboard and shows clearer class separation. — Figure 4 Grad-CAM and t-SNE visualizations from the paper. Red: folding paper. Blue: typing on a keyboard.

ATTENTIONFrom body localization to action-relevant regions. PoseBridge emphasizes hand-object interaction areas in these examples.

REPRESENTATIONClearer separation for motion-ambiguous actions. The illustrated zero-shot matching features are less mixed.

Scope and limitations

PoseBridge assumes access to intermediate pose-estimation features during skeleton extraction, so it uses an HPE-aware protocol. Its cues can become unreliable under heavy occlusion, severe motion blur, pose estimation failures, or when objects are very small or outside the body-attended region. See Appendix G for the full discussion.

05 / CITE POSEBRIDGE

Build on this work.

Download .bib

If you find PoseBridge useful in your research, please cite our paper.

BibTeX · NeurIPS 2026

@inproceedings{lee2026posebridge,
  title     = {PoseBridge: Bridging the Skeletonization Gap for
               Zero-Shot Skeleton-Based Action Recognition},
  author    = {Lee, Sanghyeon and Kim, Jinwoo and Lee, Jong Taek},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  eprint    = {2605.11497},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2605.11497}
}

Read the paper ↗Follow the code release ↗