Skip to main content

WebGL2

This project requires WebGL2. This specification was finalized in January 2017. It is supported by all popular modern browsers. NiiVue exploits WebGL2 features that are not available in WebGL1. Specifically, the images are represented using non-Power of two 3D textures. The shaders used by WebGL2 are written using the OpenGL ES 3.0 version of the OpenGL Shading Language (GLSL). WebGL2 added additional texture formats that provide native support for most NIfTI format data types.

WebGL2 is a shader-based language. NiiVue includes many in-built shaders - the pull-down menu below allows you to explore a few of these. In addition, you can write your own custom shaders, as showcased in this live demo.

Loading...

NiiVue uses WebGL not only to display images but also to accelerate many of the functions. For example, the grow cut drawings and the image reslicing described below.

Strengths and Weaknesses​

WebGL is a modern shader-based language. It lacks the fixed function pipeline, quad primitives and helper functions of legacy OpenGL. The benefit of this approach is that it forces the developer to use efficient methods that map closely to modern hardware design. However, it does have a steep learning curve. The minimal design of WebGL allows ANGLE to map it directly to the most efficient graphics library available (METAL, Vulkan, DirectX or Modern OpenGL). However, as it was designed as the least common denominator between these competing libraries, it does lack some useful features like geometry shaders. Therefore, one must adopt alternative approaches to implement features such as wireframes and illuminated streamlines. It also lacks clip planes and the depth buffer is read only.

While the emerging WebGPU has not yet been finalized, it can provide better performance than WebGL2, in particular when there are a huge number of draw calls. However, NiiVue is designed to only require a handful of draw calls, and therefore the visualization is already well optimized for modern devices. On the other hand, WebGPU can provide a huge boost for AI models, and we illustrate this potential with our tinygrad and ONNX implementations of the brain chop models.

Textures​

The term Textures refers to bitmap images that are stored on the graphics card. WebGL2 can support 1D, 2D, and 3D textures. The WebGL context can only have a limited number of textures active at one time (with the command activeTexture deterimining which textures are available). A WebGL 2 context provides a minimum of 32 total loaded textures, with a fragment program able to access at least 16 concurrently. You can think of these active textures as slots that are available for the shaders to access. NiiVue consistently uses the same slots for specific textures. While this is not required by WebGL, it simplifies the code (e.g. otherwise many calls to bind textures to specific slots of the style gl.activeTexture(gl.TEXTURE2); gl.bindTexture(gl.TEXTURE_3D, this.overlayTexture)). This means that each draw call does not need to explicitly set the active textures. Therefore, slots 0..8 should be considered reserved and if required these slots should be returned to the intended use when not needed. The code currently uses slots 9..15 transiently, so these could be easily used by other functions.

  • TEXTURE0_BACK_VOL: Background volume. This 3D scalar bitmap stores the voxel intensities of the background image.
  • TEXTURE1_COLORMAPS: Active colormaps. This 2D RGBA bitmap converts the scalar volume voxel intensities to RGBA values (e.g. Grayscale, Viridis). Note that each voxelwise layer can have two unique colormaps (for positive and negative values) and meshes can also have colormaps.
  • TEXTURE2_OVERLAY_VOL: Overlay volumes. This 3D RGBA bitmap stores the blended values of all loaded overlays.
  • TEXTURE3_FONT: Font. This is a 2D bitmap that stores the multi-channel signed distance field typeface.
  • TEXTURE4_THUMBNAIL: Thumbnail. This is a 2D bitmap that can be rapidly displayed. When a user clicks on the thumbnail the (large and slow) volumes and meshes will be loaded.
  • TEXTURE5_MATCAP: Matcap. This is a 2D bitmap used for exclusively by the matcap mesh shader (transiently enabled with the shader, so available for other functions). It is also used as a temporary 2D texture: this is used to blend multiple overlays into a single texture (TEXTURE2).
  • TEXTURE6_GRADIENT: Gradient texture for shiny volume rendering shaders that use a gradient map for lighting.
  • TEXTURE7_DRAW: Drawing texture: this stores any active drawing, having the same resolution as TEXTURE0.
  • TEXTURE8_PAQD: Probability Atlas Quad Datatype provides a compact encoding for probability atlases.
  • TEXTURE8_GRADIENT_TEMP: Transient texture used when generating TEXTURE6_GRADIENT.
  • TEXTURE9_ORIENT: Transient texture used when re-orienting voxel-based volumes to RAS orientation.
  • TEXTURE10_BLEND: Transient texture for blending different overlay volumes into the single TEXTURE2_OVERLAY_VOL.
  • TEXTURE11_GC_BACK: Transient texture used growcut shader: background voxel intensity.
  • TEXTURE12_GC_STRENGTH0: Transient texture used growcut shader: edge strength that ping-pongs between read and write.
  • TEXTURE13_GC_STRENGTH1: Transient texture used growcut shader: edge strength that ping-pongs between write and read.
  • `TEXTURE14_GC_LABEL0: Transient texture used growcut shader: drawing color that ping-pongs between read and write.
  • TEXTURE15_GC_LABEL1: Transient texture used growcut shader: drawing color that ping-pongs between write and read.
Overlays​

Overlay images are resliced to match the resolution of the background image. Multiple overlays may be loaded, but they are all blended together to generate a single RGBA texture. This overcomes WebGL limits on the number of active textures loaded, and improves the speed of rendering.

JavaScript is slower than natively compiled programs for many computations, using WebGL can dramatically increase performance providing near native performance. Therefore, while using the GPU instead of the CPU can accelerate performance regardless of language, the benefits are greater for JavaScript. Reslicing 3D volumes is compute intensive. Specifically, the GPU-based WebGL reslicing algorithm is also about an order of magnitude faster than the standard JavaScript code.

The reslicing algorithm uses the R8UI, R16I, R16UI and R32F base formats for the NIfTI DT_UINT8, DT_INT16, DT_UINT16, and DT_FLOAT32 datatypes. One limitation of WebGL is that these texture formats are not filterable, meaning that only nearest interpolation is available. Therefore, overlays may appear blocky if there is not a one-to-one correspondence between the background image and an overlay. On the other hand, this is often desired for thresholded statistical maps (where many voxels are artificially zeroed). Another quirk of WebGL is that three shader programs are required to support unsigned integer, signed integer and floating point textures (using usampler3D, isampler3D and sampler3D respectively).

A GLSL compute shader applies a 4×4 affine transformation matrix to map the overlay image onto the background volume. Since WebGL operates on 2D framebuffers, the shader processes one 2D slice of the 3D background volume at a time. This approach leverages standard WebGL2 functionality and does not rely on emerging WebGL or WebGPU compute extensions.

Future Directions​

This section describes methods that could enhance NiiVue visualization. Each will require development resources and the complexity impacts the simplicity and maintainability of NiiVue. Further, each impacts performance which may impact the experience on low-powered devices. Therefore, if implemented they would likely be optional effects.

  • Barycentric Effects could allow wire-frame visualization. As WebGL lacks geometry shaders, implementing these effects is possible but has performance implications.
  • Streamline Effects can enhance the appearance of fibers. As WebGL does not include geometry shaders, imposters could be used. This method has performance implications.
  • Ambient Occlusion models how valleys (e.g. sulci) are more shaded from light than ridges (gyri). VolView provides one JavaScript approach, Hernell provides another approach. One method may be to use a blurred opacity map to simulate shading. All these approaches have performance implications.
  • The desktop-based mrtrix and dsi-studio can display Orientation Distribution Functions that allow users to evaluate the quality of diffusion data. Developing similar functionality for the web would be useful. These methods have performance implications.