Transformerを用いた手術シーンにおける視覚的質問応答
Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer
手術シーンを対象とした視覚的質問応答（Surgical-VQA）タスクを設計し、分類回答と文回答の2種類のデータセットを構築するとともに、視覚と言語のトークン間の相互作用を強化する残差MLPベースのVisualBertエンコーダを提案した。
著者: Lalithkumar Seenivasan, Mobarakol Islam, Adithya K Krishna, Hongliang Ren
分類: cs.CV, cs.AI, cs.LG, cs.RO, eess.IV
原文アブストラクト
Visual question answering (VQA) in surgery is largely unexplored. Expert surgeons are scarce and are often overloaded with clinical and academic workloads. This overload often limits their time answering questionnaires from patients, medical students or junior residents related to surgical procedures. At times, students and junior residents also refrain from asking too many questions during classes to reduce disruption. While computer-aided simulators and recording of past surgical procedures have been made available for them to observe and improve their skills, they still hugely rely on medical experts to answer their questions. Having a Surgical-VQA system as a reliable 'second opinion' could act as a backup and ease the load on the medical experts in answering these questions. The lack of annotated medical data and the presence of domain-specific terms has limited the exploration of VQA for surgical procedures. In this work, we design a Surgical-VQA task that answers questionnaires on surgical procedures based on the surgical scene. Extending the MICCAI endoscopic vision challenge 2018 dataset and workflow recognition dataset further, we introduce two Surgical-VQA datasets with classification and sentence-based answers. To perform Surgical-VQA, we employ vision-text transformers models. We further introduce a residual MLP-based VisualBert encoder model that enforces interaction between visual and text tokens, improving performance in classification-based answering. Furthermore, we study the influence of the number of input image patches and temporal visual features on the model performance in both classification and sentence-based answering.
