Skip to main content

WebGL2

Introduction​

This project requires WebGL2. This specification was finalized in January 2017. It is supported by all popular modern browsers. NiiVue exploits WebGL2 features that are not available in WebGL1. Specifically, the images are represented using non-Power of two 3D textures. The shaders used by WebGL2 are written using the OpenGL ES 3.0 version of the OpenGL Shading Language (GLSL). WebGL2 added additional texture formats that provide native support for most NIfTI format data types.

NiiVue uses WebGL not only to display images but also to accelerate many of the functions. For example, the grow cut drawings and the image reslicing described below.

Strengths and Weaknesses​

WebGL is a modern shader-based language. It lacks the fixed function pipeline, quad primitives and helper functions of legacy OpenGL. The benefit of this approach is that it forces the developer to use efficient methods that map closely to modern hardware design. However, it does have a steep learning curve. The minimal design of WebGL allows ANGLE to map it directly to the most efficient graphics library available (METAL, Vulkan, DirectX or Modern OpenGL). However, as it was designed as the least common denominator between these competing libraries, it does lack some useful features like geometry shaders. Therefore, one must adopt alternative approaches to implement features such as wireframes and illuminated streamlines. It also lacks clip planes and the depth buffer is read only.

Textures​

The term Textures refers to bitmap images that are stored on the graphics card. WebGL2 can support 1D, 2D, and 3D textures. The WebGL context can only have a limited number of textures active at one time (with the command activeTexture deterimining which textures are available). A WebGL 2 context provides a minimum of 32 total loaded textures, with a fragment program able to access at least 16 concurrently. You can think of these active textures as slots that are available for the shaders to access. NiiVue consistently uses the same slots for specific textures. While this is not required by WebGL, it simplifies the code (e.g. otherwise many calls to bind textures to specific slots of the style gl.activeTexture(gl.TEXTURE2); gl.bindTexture(gl.TEXTURE_3D, this.overlayTexture)). This means that each draw call does not need to explicitly set the active textures. Therefore, slots 0..7 should be considered reserved and not used for other functions. The code currently uses slots 8..15 transiently, so these could be easily used by other functions.

  • TEXTURE0_BACK_VOL: Background volume. This 3D scalar bitmap stores the voxel intensities of the background image.
  • TEXTURE1_COLORMAPS: Active colormaps. This 2D RGBA bitmap converts the scalar volume voxel intensities to RGBA values (e.g. Grayscale, Viridis). Note that each voxelwise layer can have two unique colormaps (for positive and negative values) and meshes can also have colormaps.
  • TEXTURE2_OVERLAY_VOL: Overlay volumes. This 3D RGBA bitmap stores the blended values of all loaded overlays.
  • TEXTURE3_FONT: Font. This is a 2D bitmap that stores the multi-channel signed distance field typeface.
  • TEXTURE4_THUMBNAIL: Thumbnail. This is a 2D bitmap that can be rapidly displayed. When a user clicks on the thumbnail the (large and slow) volumes and meshes will be loaded.
  • TEXTURE5_MATCAP: Matcap. This is a 2D bitmap used for exclusively by the matcap mesh shader (transiently enabled with the shader, so available for other functions). It is also used as a temporary 2D texture: this is used to blend multiple overlays into a single texture (TEXTURE2).
  • TEXTURE6_GRADIENT: Gradient texture for shiny volume rendering shaders that use a gradient map for lighting.
  • TEXTURE7_DRAW: Drawing texture: this stores any active drawing, having the same resolution as TEXTURE0.
  • TEXTURE8_GRADIENT_TEMP: Transient texture used when generating TEXTURE6_GRADIENT.
  • TEXTURE9_ORIENT: Transient texture used when re-orienting voxel-based volumes to RAS orientation.
  • TEXTURE10_BLEND: Transient texture for blending different overlay volumes into the single TEXTURE2_OVERLAY_VOL.
  • TEXTURE11_GC_BACK: Transient texture used growcut shader: background voxel intensity.
  • TEXTURE12_GC_STRENGTH0: Transient texture used growcut shader: edge strength that ping-pongs between read and write.
  • TEXTURE13_GC_STRENGTH1: Transient texture used growcut shader: edge strength that ping-pongs between write and read.
  • `TEXTURE14_GC_LABEL0: Transient texture used growcut shader: drawing color that ping-pongs between read and write.
  • TEXTURE15_GC_LABEL1: Transient texture used growcut shader: drawing color that ping-pongs between write and read.
Color Schemes​

The user can choose different colormaps for displaying dark to bright voxels. In addition to grayscale, one can choose from the viridis color palettes (Cividis, Inferno, Plasma and Viridis) which are designed to be both salient and compatible with the most common forms of colorblindness. Since WebGL2 does not support 1D textures, these are codes as 2D bitmap textures (sampler2D, with a width of 256 pixels and a height of one pixel). This explains the GLSL definition sampler2D colormap, with reading using texture(colormap, vec2(f, 0.5)) (where f is a fraction from 0..1, and 0.5 indicates sampling in the vertical middle of the bitmap).

Overlays​

Overlay images are resliced to match the resolution of the background image. Multiple overlays may be loaded, but they are all blended together to generate a single RGBA texture. This overcomes OpenGL limits on the number of active textures loaded, and improves the speed of rendering.

JavaScript is slower than natively compiled programs for many computations, using WebGL can dramatically increase performance providing near native performance. Therefore, while using the GPU instead of the CPU can accelerate performance regardless of language, the benefits are greater for JavaScript. Reslicing 3D volumes is compute intensive. Specifically, the GPU-based WebGL reslicing algorithm is also about an order of magnitude faster than the standard JavaScript code.

The reslicing algorithm uses the R8UI, R16I, R16UI and R32F base formats for the NIfTI DT_UINT8, DT_INT16, DT_UINT16, and DT_FLOAT32 datatypes. One limitation of WebGL is that these texture formats are not filterable, meaning that only nearest interpolation is available. Therefore, overlays may appear blocky if there is not a one-to-one correspondence between the background image and an overlay. On the other hand, this is often desired for thresholded statistical maps (where many voxels are artificially zeroed). Another quirk of WebGL is that three shader programs are required to support unsigned integer, signed integer and floating point textures (using usampler3D, isampler3D and sampler3D respectively).

The algorithm of the reslicing shader is shown in the Figure below. Each texture is a unit normalized cube (with the voxels accessed in the range 0..1 in the X,Y,Z dimensions, regardless of the number of voxels in the column, row and slice dimension). A 4x4 matrix provides the affine transformation mapping the overlay (red in figure below) to the background image (black in image below). Since WebGL works with 2D framebuffers, the shader is run for every 2D slice of the background image's 3D volume. Therefore, this compute shader requires only the standard WebGL2 functions, without requiring the nascent compute extensions.

Overlay image

Future Directions​

This section describes methods that could enhance NiiVue visualization. Each will require development resources and the complexity impacts the simplicity and maintainability of NiiVue. Further, each impacts performance which may impact the experience on low-powered devices. Therefore, if implemented they would likely be optional effects.

  • Barycentric Effects could allow wire-frame visualization. As WebGL lacks geometry shaders, implementing these effects is possible but has performance implications.
  • Streamline Effects can enhance the appearance of fibers. As WebGL does not include geometry shaders, imposters could be used. This method has performance implications.
  • Ambient Occlusion models how valleys (e.g. sulci) are more shaded from light than ridges (gyri). VolView provides one JavaScript approach, Hernell provides another approach. One method may be to use a blurred opacity map to simulate shading. All these approaches have performance implications.
  • The desktop-based mrtrix and dsi-studio can display Orientation Distribution Functions that allow users to evaluate the quality of diffusion data. Developing similar functionality for the web would be useful. These methods have performance implications.