NA-MIC Project Weeks

Back to Projects List

The future of rendering in VTK and Slicer

Key Investigators

• Simon Drouin (École de Technologie Supérieure)
• Steve Pieper (Isomics, Inc., USA)
• Murat Maga (University of Washington, USA)
• Andras Lasso (Queen's University, Canada)
• Sara Rolfe (Seattle Children's Research Institute, USA)
• Jean-Christophe Fillion-Robin (Kitware, Inc., USA)
• Stephen Aylward (Kitware, Inc., USA)
• Rafael Palomar (NTNU, Norway)
• Sankhesh Jhaveri (Kitware, Inc., USA)
• Matt McCormick (Kitware, Inc., USA)
• Forrest Li (Kitware, Inc., USA)
• Lucas Gandel (Kitware SAS, USA)
• Jaswant Panchumarti (Kitware, Inc., USA)
• Shreeraj Jadhav (Kitware, Inc., USA)
• Tom Birdsong (Kitware, Inc., USA)

Presenter location:

Description

The goal of this breakout session is to gather all parties interested in the future of rendering in VTK and Slicer, present ongoing development by Kitware and others and discuss potential future directions and clinical and biomedical needs.

During the Breakout Session

Links and notes are organized at https://hackmd.io/Mq81LxbYTfqrwOBRjxrb6Q. It is a markdown based document we can collaboratively & interactively edit.

Once the breakout session is over, we will contribute the information back to this page. See Notes below.

Topics

• VTK Evolution
  • Status of the replacement for OpenGL (WebGPU)
  • Integration of VTK and VTK.js (common shaders?)
• Rendering
  • Global illumination
  • Support for high resolution volumes
  • Support for shared graphics contexts
  • Support for GPU pre-processing of volumes and meshes
  • Creation of an experimental rendering module:
    • Support for multiple volumes/surfaces handled by the same pipeline
    • Modifiable shaders
    • Multipass rendering
    • Arbitrary number of transfer functions
    • Better support for animation and high-resolution rendering
• AR-VR
  • New Slicer Mixed-Reality module (for Hololens remoting)
  • Support for OpenXR in Slicer Virtual Reality module
  • Status of WebXR in vtk.js

Notes

VTK C++: WebGPU

Contact: Sankhesh Jhaveri @ Kitware

WebGPU effort in VTK aims to provide a future-proof rendering backend as an alternative to OpenGL.

• VTK’s data model and visualization pipeline will have no architectural changes.
• For the most part, there will be no frontend user-facing changes to the rendering classes either. Applications would still have to instantiate vtkRenderWindow, vtkRenderer, vtkActor, etc.
• There will be changes with respect to how platform-specific windows are created. These windows no longer need to be tied to the graphics backend i.e. no need for vtkXOpenGLRenderWindow, etc.
• For advanced users of VTK who modify rendering logic, use shader replacements, etc. there will be significant changes. This should be expected, IMO.

Experimental:

• RenderingWebGPU: https://docs.vtk.org/en/latest/modules/vtk-modules/Rendering/WebGPU/README.html
• Draft: WebGPU: Native graphics backend. See https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10239

Questions:

• multi-volume rendering
  • Will it be the default ? In the OpenGL backend, there were two backends.
  • vtkMultiVolume issues
    • SetVisibility issue
    • UserMatrix not taken into account for the first volume (issue) (works in TestGPURayCastMultiVolumeOverlapping)
    • Cropping
    • Shading
    • Only shows up in Composite w/ Shading, but nothing in MIP (Error: Shader failed to compile in raycastervs.glsl)
• context sharing ?
  • Goal is to reduce the memory footprint by avoiding having duplicate of teture memory (e.g multiple 3D views of the same volume data)
  • Availability of result of compute shader for rendering.
• Compute shaders
  • WebGPU ?

Notes:

• moltengl: Applications built for OpenGL ES 2.0 can use MoltenGL to run on top of Metal,
• Unity allows to easily compose complex rendering pipeline.
• RenderingWebGPU:
  • The shader is the center piece
  • Possibly use SPIRV-Reflection to decipher shader inputs, their attributes/dimensionality and expose a VTK interface to populate shader bindings with vtkImageData, vtkDataArray, etc.
• Compute Shader
  • https://github.com/Punzo/SlicerAstro/tree/master/vtkOpenGLFilters
  • Simon will share details about experiment for implementing compute shader
• AI and WebGPU for training ?
  • Rendering into texture ?
  • https://virtualgl.org/
  • Idea would be to support Differential Rendering
    • https://towardsdatascience.com/differentiable-rendering-d00a4b0f14be
• Review meeting ?

WebAssembly

Contact: Jaswant Panchumarti @ Kitware

• Slides: WASM/WebGPU in VTK

• Docs: Getting Started / Using WebAssembly

• Leverage efforts done in the context of https://wasm.itk.org/

VTK C++: OpenXR

Contact: Lucas Gandel @ Kitware

Improvement roadmap for OpenXR and OpenXR Remoting support in VTK:

• OpenXR controller model support:
  • vtkOpenXRManager already supports for querying the controller models in a buffer (see here)
  • The buffer should be forwarded to vtkGLTFReader using vtkResourceStream/vtkResourceParser
  • Need to add vtkResourceStream/vtkResourceParser support to vtkGLTFReader (see PLYReader )
• OpenXR Remoting hand interaction improvements
  • The current interactor style does not allow for accessing different poses from a single Move3D event (see discussions in this MR)
  • Possible options are:
    • Add support for storing additional poses (maybe as additional TrackedDevice)
    • Add support for hands joints tracking and implement basic gestures recognition
• Add support for scene understanding in OpenXR Remoting, to occlude the VTK scene with the real world.

SlicerVirtualReality

• Are we ready to transition to OpenXR ?
  • Before doing so, we would need to generalize the function vtkOpenVRRenderWindow::GetOpenVRPose() currently specific to Rendering/OpenV and only available in Slicer/VTK

SlicerMixedReality

Contact: Jean-Christophe Fillion-Robin @ Kitware

See https://github.com/KitwareMedical/SlicerMixedReality/pull/2

Question: What to do once we start working on adding OpenXR support to SlicerVirtualReality.

vtk.js: Rendering Non-uniform image series

Contact: Shreeraj Jadhav @ Kitware

See https://docs.google.com/presentation/d/1mrMe8w2G5hgRan0KzdwqrxgKGLfyR-h3mM7Kj5KYz4c/edit#slide=id.p

vtk.js: Interactive, in-browser cinematic volume rendering of medical images

Contact: Stephen Aylward & Forrest Li @ Kitware

The diversity and utility of cinematic volume rendering (CVR) for medical image visualisation have grown rapidly in recent years. At the same time, volume rendering on augmented and virtual reality systems is attracting greater interest with the advance of the WebXR standard.

See https://doi.org/10.1080/21681163.2022.2145239

See https://volview.kitware.com/

vtk.js: WebXR

Contact: Tom Birdsong @ Kitware

• Blog: VTK.js Transforms Web-based Visualization with Immersive Virtual and Augmented Reality
• Blog: Holograms Over the Web: Kitware Extends vtk.js to Support Looking Glass Factory’s Displays
• Example and FAQs: https://kitware.github.io/vtk-js/docs/develop_webxr.html
  • Supported devices
  • Feature Support Roadmap

VTK & ITK interoperability

Web:

• ITK IO are compiled to WASM and re-used in vtk.js based web application
• ITK-WASM is a building block for VTK-WASM

OME-Zarr:

• https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1
• https://github.com/InsightSoftwareConsortium/itkwidgets/blob/main/examples/integrations/zarr/OME-NGFF-Brainstem-MRI.ipynb

Slicer + Large Image Rendering:

• We currently do not have any readily available implementation in Slicer, from a high level, the idea would be to:
  • support streaming dataset at different resolutions and updating the rendering accordingly.
  • leverage ITK+Zarr integration and ensure information flow all the way through the VTK pipeline
    • Consolidate ITKVTKGlue modules
• Improve ITK module: ITKIOOMEZarrNGFF. See https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF

Illustrations

Background and References

• WebGPU in VTK
• WebGPU in Slicer from Project Week 37.