3D ConvNetのための教師なし行動ローカライズクロッピングによる動画リターゲティング
Unsupervised Action Localization Crop in Video Retargeting for 3D ConvNets
各フレームで均質な動きのパッチから顕著領域を特定し、3D空間のpolyBezierフィッティングで時間的に滑らかな1:1クロップを生成する教師なし動画クロッピング手法を提案し、動画分類精度でランダムクロップを上回ることを示した。
著者: Prithwish Jana, Swarnabja Bhaumik, Partha Pratim Mohanta
分類: cs.CV, cs.LG, cs.RO
原文アブストラクト
Untrimmed videos on social media or those captured by robots and surveillance cameras are of varied aspect ratios. However, 3D CNNs usually require as input a square-shaped video, whose spatial dimension is smaller than the original. Random- or center-cropping may leave out the video's subject altogether. To address this, we propose an unsupervised video cropping approach by shaping this as a retargeting and video-to-video synthesis problem. The synthesized video maintains a 1:1 aspect ratio, is smaller in size and is targeted at video-subject(s) throughout the entire duration. First, action localization is performed on each frame by identifying patches with homogeneous motion patterns. Thus, a single salient patch is pinpointed per frame. But to avoid viewpoint jitters and flickering, any inter-frame scale or position changes among the patches should be performed gradually over time. This issue is addressed with a polyBezier fitting in 3D space that passes through some chosen pivot timestamps and whose shape is influenced by the in-between control timestamps. To corroborate the effectiveness of the proposed method, we evaluate the video classification task by comparing our dynamic cropping technique with random cropping on three benchmark datasets, viz. UCF-101, HMDB-51 and ActivityNet v1.3. The clip and top-1 accuracy for video classification after our cropping, outperform 3D CNN performances for same-sized random-crop inputs, also surpassing some larger random-crop sizes.
関連論文
- 加速度画像を用いた動き表現動画認識
