Exploiting Edge-Oriented Reasoning for 3D Point-based
Scene Graph Analysis

EdgeGCN
  • 1University of Sydney
  • 2University of New South Wales

Overview

overview

Scene understanding is a critical problem in computer vision. In this paper, we propose a 3D point-based scene graph generation (\(\bf{SGG_{point}}\)) framework to effectively bridge perception and reasoning to achieve scene understanding via three sequential stages, namely scene graph construction, reasoning, and inference. Within the reasoning stage, an EDGE-oriented Graph Convolutional Network (\(\texttt{EdgeGCN}\)) is created to exploit multi-dimensional edge features for explicit relationship modeling, together with the exploration of two associated twinning interaction mechanisms between nodes and edges for the independent evolution of scene graph representations. Overall, our integrated \(\bf{SGG_{point}}\) framework is established to seek and infer scene structures of interest from both real-world and synthetic 3D point-based scenes. Our experimental results show promising edge-oriented reasoning effects on scene graph generation studies. We also demonstrate our method advantage on several traditional graph representation learning benchmark datasets, including the node-wise classification on citation networks and whole-graph recognition problems for molecular analysis.

Video

(contains audio w/ subtitles)

Datasets & Evaluation Scripts

  • 3D $\mathbf{SGG_{point}}$ on Real-World 3D Scans: 3DSSG-O27R16 (based on 3RScan + 3DSSG).
    • 3D Scans (3RScan Dataset): paper, dataset
    • Scene Graph Annotations (3DSSG): paper, dataset
    • Our own preprocessed 3DSSG-O27R16 dataset is available at HERE. Note: our 3DSSG-O27R16 is derived from 3RScan + 3DSSG, which was initially collected and annoated by Johanna Wald Et Al. To fully honour their data ownership and efforts in creating the datasets, please restrictly follow the 4-step instructions below to obtain it.
      1. Fill in and submit this online form (3RScan Terms of Use - as requested in 3RScan) to register your interests in the datasets.
      2. A welcome webpage will be redirected immediately to display the full access to the raw 3RScan and 3DSSG datasets.
      3. Take a screenshot of this welcome webpage and send it (with your name and school pls) to our SGGpoint@gmail.com to receive the access password.
      4. Go to the Download Page to obtain our 3DSSG-O27R16 dataset, where the detailed dataset info/structure could be found in Dataset Description.
  • 3D $\mathbf{SGG_{point}}$ on Synthetic 3D Scenes: SUNCG + SceneGraphNet.
  • Traditional Graph Representation Learning: Planetoid + MoleculeNet.
    • Node classification for citation analysis on Planetoid dataset (Cora, CiteSeer, and Pubmed): paper, script
    • Graph recognition for molecular analysis on MoleculeNet dataset (Tox21 and BBBP): paper, script

BibTeX

If you find our data or project useful in your research, please cite: 
@InProceedings{SGGpoint,
    author    = {Zhang, Chaoyi and Yu, Jianhui and Song, Yang and Cai, Weidong},
    title     = {Exploiting Edge-Oriented Reasoning for 3D Point-Based Scene Graph Analysis},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {9705-9715}
}

Acknowledgments

The website template was borrowed from Zhenqin Li's project.