Data processing and analysis software

On this page, we list and describe some data processing and analysis tools and software packages that have proved to be useful for data taken at TOMCAT. The list is by no means exhaustive and many interesting options might be missing. We hope that the information will become more complete over time and are thankfor for any feedback or input. Also note that despite our best efforts to keep things up to date, many tools are rapidly evolving, so the information provided here might already be outdated with respect to the latest developments. Nevertheless, we hope that you will find the following descriptions, hints and tips useful to get started with your data processing and analysis tasks.
Some of the applications or packages discussed on this page are available on our remote access data analysis cluster Ra (see Computing), while others are more adequate for usage and installation at a user's home institution. Please contact us if you think a particular solution should be made available on our Ra compute resources such that we can request an installation from out IT specialists managing the system.
We would like to encourage you to share your data analysis and processing experience with us, to provide feedback on the provided tools and make us aware of other great software packages that are missing from the following description, and to share your data analysis code if it might find a broader user base at TOMCAT.
DMP reader
Fiji) / Imagej

Prefer Fiji over current versions of ImageJ2 since the latter has still not fully transitioned from using Java 6 to Java 8.

Python Matlab

tif files h5
3D slicer

TomViz pro: reads tiff stacks out of the box

ParaView pro: Installed on Ra cons: does not read tiff stacks --> publish script to convert tiff stacks to vtk files.

VisIt

Drishti
Fiji (see above) is a great tool to view image data packaged in hdf5 files. However, while the standard plugin supplied with the normal Fiji distribution provides great features, it does not currently allow one to open large image series as virtual stacks. In most cases, this results in out of memory errors when trying to view larger tomographic data sets with hundreds to thousands of full frames.

The PSI provides a dedicated hdf5 reader plugin with the ability to view the data as virtual stacks, which can be found in the following GitHub repository:

https://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5

However, the installation of this PSI plugin unfortunately seems to require a deactivation of the hdf5 plugin that is packed in the standard Fiji distribution. Follow the below instructions to install the PSI hdf5 plugin on your system.

7.1.1 Prerequisites

The PSI hdf5 plugin requires Java 7 or higher to run. The latest versions of Fiji or ImageJ are bundled with Java 8, so this requirement is easily fulfilled by newer installations. You can check whether Java-8 is included with Fiji as follows:
  1. Open the update dialog from the menu via Fiji > Help > Update...
  2. Wait for the application to finish checking for new updates
  3. In the ImageJ Updater Window, click on "Advanced Mode"
  4. Type java-8 into the "Search" field.
  5. If you see an entry lib/Java-8 in the results box below, then Java 8 is ready to be used on your system (see screenshots below).
Fiji Updater AdvancedMode.png
Fiji Updater search java8.png


If you do not have Java 8 available, consider updating Fiji or grabbing the latest version from the Fiji website

7.1.2 Disabling the standard hdf5 plugin

As mentioned previously, unfortunately the standard hdf5 plugin (HDF5_Vibez) of Fiji interferes with the PSI plugin. You will be able to tell which plugin is currently active when trying to import an hdf5 file (Fiji > File > Import > HDF5...). Once you have selected an hdf5 file, the standard plugin opens an import dialog that looks as follows:

Fiji plugin hdf5 Vibez.png


To disable the standard hdf5 plugin, follow these steps.
  1. Open the update dialog from the menu via Fiji > Help > Update....
  2. Wait for the application to finish checking for new updates.
  3. In the ImageJ Updater Window, click on "Advanced Mode".
  4. Type hdf5 into the "Search" field.
  5. The results box below should now list the plugins/HDF5_Vibez entry (see screenshot below).
  6. Select that plugins/HDF5_Vibez entry and click "Uninstall" below, then hit "Apply Changes".
  7. The "Status/Action" column of the plugins/HDF5_Vibez entry should now say Not installed.
  8. Restart Fiji.
Fiji Updater search hdf.png

7.1.3 Installing the PSI hdf5 plugin

The source code and installation instructions for the PSI hdf5 plugin can be found in GitHub:

https://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5

Follow these steps to install the plugin from the repository:
  1. Grab the latest release of the plugin here:
    https://github.com/paulscherrerinstitute/ch.psi.imagej.hdf5/releases
  2. Save the zip file to a convenient location and unzip the contents there. This will result in a new directory with the same name as the zip file (e.g.: HDF5_Viewer-0.12.0)
  3. Navigate into that new directory (e.g. =cd HDF5_Viewer-0.12.0).
  4. Locate the installation directory (FIJI_DIR) of Fiji. The default locations are the following, but you may also have a local installation in a different location:
    • macOS: FIJI_DIR = /Applications/Fiji.app
    • Linux:
    • Windows:
  5. Copy the contents of that directory (there should be a lib and a plugins folder) to the Fiji application path:
    • On OS X:rsync -avh lib /Applications/Fiji.app/ rsync -avh plugins /Applications/Fiji.app/
    • On Linux:
    • On Windows:
  6. Close any running Fiji applications, and reopen Fiji.
  7. Try to import an hdf5 file: Fiji > File > Import > HDF5....
    • If the installation of the pluging was sucessful, the file import dialogue should look like in the screenshot below once you have selected an hdf5 file.
    • If the HDF5... loader is not available in the Fiji > File > Import menu, the installation did not succeed. Check that all the files downloaded for the new plugin are actually present in the correct locations, i.e. in $FIJI_DIR/plugins and $FIJI_DIR/lib.
Fiji plugins hdf5 PSI.png
In addition to the usual 8-bit or 16-bit unsigned integer tiff file formats used to export reconstructed slices at TOMCAT, another option includes the creation of 32-bit floating point number images. These are saved as simple binary dump files (.DMP) by our systems. To read these .DMP files in Fiji, you will need to install the corresponding plugin. Download the source and follow the installation instructions in the git repository containing the code:

https://gitlab.psi.ch/tomcat_public/TOMCAT-DMP-Files

https://gitlab.psi.ch/tomcat_public/TOMCAT-DMP-Files/blob/master/ImageJ/README.md

Note that the same repository also offers scripts and functions to read .DMP files in MATLAB, IDL, or Python.