Multimodal registration: specification

Code: src/spam_gui/modules/mmr.py (planned: src/spam_gui/modules/mmr/) · Tests: tests/gui/workflow.py (planned: tests/gui/modules/mmr.py), tests/test_multimodalSteps.py · Status: done; the reference implementation of SPEC.md

1. Purpose

• What it does: registers two 3D images of the same sample acquired with different modalities (e.g., x-rays and neutrons): it finds Φ, the transformation that moves image 1 onto image 2 (the reference).

• spam functions called, unchanged:

  • spam.DIC.multimodalRegistration (the registration itself, step 6), on the images quantised to the bins;
  • spam.DIC.binning (step 1), spam.deformation.decomposePhi / computePhi (Φ shown and typed as translation, rotation, zoom, shear).

• Conventions, as spam-mmr-graphical (spam.visual.visualClass): Φ deforms image 1 onto image 2, about the centre of the images; the TSV files are those spam-mmr writes.

• Inputs: two 3D images, each a TIFF file or a folder of TIFF slices, of any grey-level type, up to several GB each (larger than the memory).

• Outputs (step 6, « Save… »; n1, n2 the names of the images, b the binning):

  File	Ticked at first	Content
  n1-n2-PhiMMR-binb.tsv	yes	Φ for these images (spam's registration file)
  n1-n2-PhiMMR-bin1.tsv	yes	the same Φ for the original images (translation × binning); only for binned images
  n1-registered-binb.tif	yes	image 1 moved by Φ, on the voxels of image 2
  n1-n2-mmr-phases-binb.tif	yes	the phase of each voxel (0: none)
  n1-n2-mmr-residual-binb.tif	yes	the residual of each voxel
  n1-n2-mmr-log-binb.txt	no	the iterations, as shown in step 6

  The fields can be saved for the whole images or for the region registered only.

• Analysis spam does not have yet (src/spam_gui/tospam/DIC/multimodalSteps.py, listed in TO_SPAM.md): what the joint histogram and phase diagram steps compute, step by step, where spam.DIC.multimodal.gaussianMixtureParameters and phaseDiagram decide everything at once: jointHistogram, findHistogramPeaks, fitGaussianPeak, gaussianHasMaximum, phaseDiagramSigma, sigmaForFullCoverage, quantiseImage. With full coverage, the phase diagram is spam's (tested).

2. Steps

1. Load images

• One intro paragraph: coarse binning first, then finer ones, ticking the results of a coarser one.
• 1. Original images (full resolution): « Image 1 (moved) », « Image 2 (reference) », each a path with « File… » and « Folder… », and its size and type under it. « Reset: work with other images ».
• 2. Binning to work at: buttons 1, 2, 4, 16, 64 (tooltip: « Each block of b×b×b voxels averaged (spam.DIC.binning) »), the size at that binning and whether it is read from the files; « Use the results kept at binning … » (the results to carry, see section 3); « Load the images at binning b »; « Close the other binnings (frees memory) » (ticked).
• The images given on the command line (spam-gui mmr image1 image2) fill the two paths; they are loaded with « Load the images at binning … », not at once.

2. Eye registration

• One intro line, then a grid: Translation (voxels), Rotation (degrees about Z, Y, X), Zoom (1: none), Shear (ZY, ZX, YX; 0: none). Each change moves image 1 at once.
• In sync or not with Φ: an indicator (green: in sync, amber: out of sync, e.g., once Φ is updated from the MMR registration), « Sync Phi from eye registration », « Sync eye registration from Phi ».
• Pairs of points (spam_gui.gui.points, layer « mmr points »): « Add a pair » (a click on a feature of image 1, then on the same feature of image 2, in any window; a yellow prompt says what to click; « Cancel » or Esc stops), « Remove last », « Move points », the points' radius (they are spheres, seen in the cuts near them), and a table of point 2 minus point 1 moved by Φ. The pairs are a visual aid: Φ is not fitted to them.

3. Crop

• In the coordinates of image 2, drawn in yellow in the windows (layer « mmr crop »).
• Shape: « Rectangle » (from, to along Z, Y, X) or « Cylinder » (axis, radius, length, centre).
• Margins: « Whole image », « 10 % », « 20 % » (the default), « 50 % » of the image left out, or set by hand.
• At least 16 voxels in each direction.

4. Joint histogram

• 1. Grey level mapping: for each image, its min and max (number boxes, and a slider over the histogram of the crop's grey levels), « Min and max of the crop », « Limits of the layer controls ». Intro: below the min: the first bin; above the max: the last.
• 2. Bin and peak selection: the number of bins (64 by default), how peaks are found (min distance, peaks listed), « Compute histogram », and the grey levels in one bin.
• 3. Joint histogram and peak selection: the histogram (2D from above, or a 3D surface, see « Decisions »), and a table of the peaks found: ticked or not, position, fit radius. A click on the 2D histogram ticks a peak. Each ticked peak is fitted by a Gaussian; a fit radius changed is fitted again once it stops changing (350 ms).
• Computed on the crop, with image 1 moved by the current Φ (the eye registration's at first). The note under it says which Φ; a ⚠ note says when the grey levels or bins changed since (stale, SPEC.md 3.2).
• Searching the peaks again (other min distance or number) uses the histogram already computed; a ticked peak found again within 1.5 bins stays ticked.

5. Phase diagram

• Each bin of the joint histogram is given to the closest Gaussian if its distance is below sigma; the others (white) are ignored by the registration.
• « Distance »: « Maximum distance » or « Mahalanobis »; « Sigma » (a slider and a box) and « Full coverage » (the smallest sigma for which every bin is given to a phase: the default, as spam's).
• The windows show the voxels of each phase in its colour (layers « mmr phase 1 », … , spam_gui.phases), updated when sigma or the distance change; hidden when going back to an earlier step.

6. MMR registration

• Options: « Margin (voxels) » (5), « Maximum iterations » (50), « Convergence: min |ΔPhi| » (0.001), « Register the cropped region only » (ticked: the box of the crop; for a cylinder, the box around it).
• It starts from the current Φ (the footer), and says which.
• The memory it needs (about 60 bytes per voxel of the region) and the memory available are shown; a registration that does not fit is asked about first (dialogs.warn) and not run if refused.
• « Run registration »: in a thread, the iterations in a log; it ends converged, at the maximum iterations, diverging, or with a singular matrix (spam exits there: caught, and said).
• « Update current Phi from MMR registration »: Φ becomes the result. The layers « mmr phase field » and « mmr residual » show the result.
• Save: the files of section 1, ticked as listed; « Save… » asks for a folder.

Footer: Current Φ

« Current Phi (applied to image 1) », shown whatever the step: « Initial » (image 1 as loaded), « Eye » (the eye registration's Φ), « MMR » (the MMR registration's Φ); where it comes from (with the binning it was obtained at); the matrix (Φ = ⎡…⎤); translation, rotation, zoom. Collapsed: only its three buttons. Always the height it has with an MMR Φ.

Summary (anytime)

Four boxes: « Initial images », « Binnings registered », « Transformation, for the original images », « Accuracy gained at finer binnings » (how much each registration moved image 1 from the one before, over the region registered).

3. State

MMRState (the module's makeState):

The module also keeps the pairs of points, the layers of the two images, image 1 moved by Φ (a MovedVolume), the phase layers, and the results at each binning (results: (names of the images, binning) → results).

Several binnings: the results of each binning are kept for the session and restored when it is loaded again. Carried to another binning (converted), when ticked in step 1: the eye registration's Φ, the crop, the MMR Φ. Made again at each binning: the joint histogram and the phase diagram, because their peaks change with the binning. Resetting step 6 keeps PhiMMR (still a registration of these images) until other images are used.

4. Large images and memory

• At binning 1, an image is mapped from its file when it can be (a single uncompressed TIFF); otherwise it is read into memory if it fits, else into a scratch file. Binned images are read a few slices at a time.
• Image 1 moved by Φ is computed for what is shown (a cut, a region): volumes.MovedVolume. On the crop, for the joint histogram, it is kept in memory if it fits, else in a scratch file.
• The joint histogram, the quantised images and the fields saved are computed by parts, in threads, with a progress bar and « Cancel ».
• The registration needs its region in memory: about 60 bytes per voxel (parts.registrationMemory). Said in step 6, asked about before it starts if it does not fit.
• 3D views need both images whole in memory (about 8 bytes per voxel): refused otherwise, with « Load the images at a coarser binning (step 1) for a 3D view ».

5. Decisions

• Φ convention: as spam-mmr-graphical (Φ moves image 1 onto image 2). Why: the same files and results as spam's modules.
• No automatic fit of Φ to the pairs of points: the pairs help to see the match. Why: (to confirm with the maintainer).
• The joint histogram and the phase diagram are made again at each binning, while the eye Φ, crop and MMR Φ are carried. Why: their peaks change with the binning.
• Joint histogram in three numbered panes, in the order they are used, referring to each other by number. Why: SPEC.md 4.1; this step is where the rule comes from.
• 3D joint histogram: fixed isometric view (parallel projection, elevation 35.26°, azimuth −45°: the corner of the last bin of image 1 and the first of image 2 in front), turned only by a « Rotate » slider (−180° to +180°, 0° in the middle), mouse rotation and zoom off; a coarser surface (32 points per axis) while dragging, the whole one (up to 96) once released; the angle kept when redrawn. Why: (to confirm with the maintainer).
• Current Φ as a footer, always the height it has with an MMR Φ. Why: Φ is used at every step (to confirm); the fixed height: the panel must not jump (SPEC.md 4.4).
• Eye registration in sync or not with Φ, shown in green or amber, with buttons to sync either way. Why: once Φ is updated from the MMR registration, changing the eye registration no longer moves image 1: the user must see it (to confirm).
• Crop at least 16 voxels in each direction; default margin 20 %. Why: (to confirm with the maintainer).

6. Tests

• Unit tests: tests/test_multimodalSteps.py (joint histogram, peaks, Gaussian fit, the phase diagram equal to spam's at full coverage, quantising); tests/test_parts.py (quantise, fields, the moved crop); tests/test_volumes.py (MovedVolume equal to applyPhi, phase volumes).
• Window test (tests/gui/workflow.py): it makes an « x-ray » and a « neutron » image of the same random structure (96³ at full resolution), image 2 moved by a known Φ (translation and rotation), and goes through the module at binning 2 then binning 1, with napari settings of its own. Its MMR checks:
  • every step: no sideways scroll, and the left panel keeps its width (in « Single »);
  • navigation: step 2 refused without images, and a crop under 16 voxels refused, each with its ready() message;
  • step 2: translation, rotation, zoom and shear move image 1 at once, by computePhi of them; in sync; reset; « Add a pair » (the prompt, the pair in its layer and in the table), « Remove last »; after the MMR registration, out of sync, said why, with the sync buttons;
  • step 3: the default rectangle (20 % margin), drawn; the 50 % margin; « Cylinder », drawn as one;
  • step 4: the joint histogram of the crop (5 peaks), equal to that of the whole crop; three numbered panes; the grey levels in one bin said; more peaks listed: those ticked still ticked; grey levels changed: said out of date, then up to date again; the 3D view isometric, not turned by the mouse; turned by its slider without making the surface again; back to 2D without the slider; a pane collapsed;
  • Current Φ: « Initial », « Eye », « MMR » available as they should be; « Φ = ⎡…⎤ »; the same height whatever Φ; collapsed to its three buttons;
  • step 5: a phase diagram of two phases; a layer per phase in the windows, hidden when going back to an earlier step and shown again; full coverage at first; a smaller sigma covers less and is said; « Mahalanobis »;
  • step 6: « Register the cropped region only » unticked: the whole images, more memory; at each binning: converged, and Φ within half a voxel of that binning of the true one; the log, the result, « Update current Phi from MMR registration » available, then applied (image 1 moved by it, « MMR » checked); the memory needed said; a registration larger than the memory asked about, and not run when refused;
  • binning 1: read from the files; steps 2 and 3 carried from binning 2, steps 4 and 5 made again;
  • files saved; image 1 registered equal to applyPhi; fields of the whole images; the joint histogram with image 1 moved equal to applyPhi on the whole image;
  • images too large for the memory: no 3D view, nothing read for it;
  • Summary: both binnings.
• Screenshots (tests/gui/screenshots.py): mmr-1-load-images, mmr-2-eye-registration (and -triple), mmr-3-crop, mmr-4-joint-histogram (two peaks ticked; and -3d), mmr-5-phase-diagram, mmr-6-mmr-registration (after a registration), mmr-summary; at 1366×768, 1920×1080, 3840×2160, both themes.

7. Open questions

• Why the pairs of points do not give Φ (a fit, as spam's eye registration could): a choice, or not done yet?
• The reasons for the 16-voxel minimum crop and the 20 % default margin.
• Which of steps 1 to 3 (load, eye registration, crop) become shared steps (modules/common/), and what changes in them for that.