R&D for a highly accurate 3D reconstruction, rigging and visualization of an ocelatted lizard.
In collaboration with scientists of Laboratory of Natural and Artificial Evolution, University of Geneva, we researched various automated imaging methods for the full reconstruction at high resolution of the skin of a timon lepidus, the largest endemic lizard in Europe. 3D microscopy data cannot be used by photogrammetry reconstruction algorithms due telecentric lenses.
So I have come up with a semi-automated Python worklow for a full reconstruction by projecting and stitching high frequency details (that are imaged per block) with OpenCV feature-matching and Blender laplacian mesh editing. The resulting reconstruction is rigged and it allows dynamic coloration of skin scales following numerical RD simulations.
Using improved photometric stereo and structure from motion developed by E. Jahanbakhsh as a canvas, high frequency details from structured light microscopy were projected and stitched together. This last part was the most difficult one. A more detailed explanation will follow soon.
Details of the sub-sampled geometry
Simulations of the evolution of colors
L. Manukyan et al. showed in a Nature publication that the rules that drive the patterns of black and green scales can be inferred with a mathematical model of reaction diffusion of which a cellular automaton behaviour emerges.
A canvas for 3D simulation
This model is down-sampled to keep geometry to a usable level. What matters at this scale is the geometry of the skin scales which impacts next generation 3D reaction-diffusion simulations. This reconstruction is useful for simulating more accurately the evolution of the colors of skin scales and it allows to do research on other parts than the back of the lizard.
Rigging, Mo-cap and presentation in VR
Together with scientist Paule Dagenais, we set up and performed a motion capture of reptiles such as the ocellated lizard. Even with cleaned up data, applying motion capture data directly onto the rigged 3D model requires substantial work. Simulations of discrete reaction-diffusion algorithms can be directly mapped on to the rigged model and I have developed VR prototypes in Unity to show interactive visualization of such data.
The Fascinating Brain, German National Parliament
R&D to reconstruct a large-scale neuronal circuit and to visualize it alongside a full brain segmentation and connectivity pathways for Ebrains.eu. The resulting murals represent current and future neuroscience and this video shows some details of the largest mural I created.
December 2020 news: the long-running journal Sciences & Avenir published this artwork in n°886.
This 6.5 meters long data-driven mural features neuroscientific research at multiple scales, from mathematical analysis of whole brain activity down to cortical activity of millions of cells. This was commissionned for the launch of Ebrains.eu, an European infrastructure project that supports neuroscientific research. The mural artworks I produced were initially presented to the public and to government representatives at the German National Parliament, Berlin.
In this illustration, I reveal multimodal data of The Virtual Brain with functional pathways in the human brain alongside a view over a ridge of the temporal lobe with millions of human neurons. This is the future of neuroscience that many neuroscientists and engineers are building. This fresco represents a woman on purpose with the hope that more of them will lead neuroscientific research.
Reconstruction of the first large-scale digital circuit of human neurons in the temporal lobe.
A view on the neural architecture of the temporal lobe within our brain
Here's a detail of the mural artwork at full size in a bright region. Spines are visible at this scale. A tiny fraction of the actual amount of synapses is represented here with 1% of spines per cell.
This mural artwork is the first digital reconstruction of human grey matter of the temporal lobe. White matter is hidden below this mountains of cells. Colors reveal structural properties of the circuit such as layers, cell morphology type and synaptic class. Full mural artwork size: 47000*27000px, 4m by 2.2m at 300dpi.
5% of neurons are visualised from a circuit of 50 million cells. The system looks so complex that we can hardly distinguish spines – at a very low density per cell here due to memory limit. Neither glia nor vasculature are in this model. Cell nuclei are visible but there are no other organelles here.
For this mural, I reviewed scientific literature, I curated data and I researched and developed two new tools. The first one is a curvature-dependent neuronal circuit builder that handles non trivial placement of neurons. The second one is a new 3D reconstruction tool made for Blender 2.8 that leverages my C++ mod of Blender/Cycles for large-scale visualization of splines, which I used for both murals visible here.
Paul-Löbe-Haus, Bundestag, Berlin where my artworks were presented. Picture by Ansgar Koreng.
About these scientific visualizations
Several mural artworks were commissionned by the Human Brain Project and were revealed at Bundestag German parliament, Paul-Löbe-Haus, Berlin on the 27th of November until December 19th, 2019. After that, the exhibition moved to the state parliament of Northrhine-Westfalia in Düsseldorf from January 21st to 31st, 2020.
Scientific data provenance: CNCR Amsterdam, Neuromorpho.org, Institut de Neurosciences de Marseilles, Charité University Berlin and Human Brain Project.
For the production of the murals, I engineered software solutions with Python and C++ on top of Blender with my custom modification for large-scale graph visualization. This was deployed with Docker onto a Cray supercomputer at CSCS Lugano.
The images I produced were used for Ebrains communication such as on ebrains.eu and on a VR booth at FENS Virtual Forum 2020.
Special thanks to
Human Brain Project
CNCR Amsterdam, Netherlands
Christiaan PJ de Kock
Mohan et al
CSCS supercomputing center
Charité University Berlin
Institut de Neurosciences de Marseille
Another detail of the mural artwork at full size, this time in a dark region. Nuclei (within somata) and spines on branches are more visible in this case.
The Neuro-Glia-Vascular ensemble
This visualization is part of the engineering research I did as part of a long term collaboration with scientist Eleftherios Zisis to reveal his research on the reconstruction of a digital circuit of neurons, astrocytes and blood vessels.
In the brain, neurons are supported and regulated by other types of cells known as the glia. Among the glia, astrocytes are territorial cells – shown here in blue – that, among other complex interactions with their surroundings, take nutrients from the blood vessels in red and distribute them to neurons in yellow. But astrocytes do more than that. For instance, they support a part of the synaptic activity within their domain. This visualization shows neocortical grey matter roughly 1.5mm tall and 0.5mm wide.
Custom Python library for meshing and visualization, Blender, Cycles raytracer modification
Neuronal connectivity patterns
Looking at digital simulations of neuronal activity, scientists found connectivity patterns: neurons seem to have a preference to connect with a chosen set of neurons and the resulting connectivity forms patterns like the "rich club" in the middle of this illustration or the feed-forward vertical pattern. The emergence of a communication pattern among cells reveals a functional structure within what appears to us as a chaotic forest of cells.
Nature Neuroscience TOC, July 17
Blender, Sverchok, Cycles.
Cellular variety in a 1mm^3 neocortical brain tissue
This artwork illustrates data obtained with electron microscopes of a small brain tissue.
A variety of cells are visible here, from microglia in blue/turquoise to neurons in purple, astrocytes in orange-yellow with a perivascular process and also myelinated axons in dark blue and blood vessels, here not reconstructed as endothelial cells, which shows the limits of the reconstruction.
The 3D models except red blood cells were automatically reconstructed from EM stacks by scientist Corrado Cali.
Front. Neurosci., 25 September 2018
Digital 3D reconstruction of capillaries in the neocortex
Using field-based meshing – aka "metaballs" in Blender, I developed a way to automatically reconstruct 3D surfaces for large datasets. This method builds the mesh by blocks so it can build any dataset.
The graph skeleton was calculated by E. Zisis and the raw data was provided by Bruno Weber et al. ETHZ.
2nd place award, professional illustrators category
AEIMS 18 medical illustration congress
Eleftherios Zisis et al.
Astrocytes growing perivascular processes
A major step after reconstructing vasculature is to have synthesized astrocytes filling a good part of the volume occupied by blood vessels and neurons. Here neurons are not shown. The astrocytes in blue grow special branches in white that are about to wrap nearby capillaries in red to get nutrients and distribute them to neurons, among many other roles theses cells perform.
- Scientific poster, SfN 2017
- Le grand Atlas du cerveau, 2018
This is the first illustration that could benefit from a mesh-less concept based on Cycles raytracer, enabling large-scale visualisation in Blender.
This inspired BBP visualisation team to explore similar techniques for large-scale visualization.
Meshless objects: 260 million splines
Surface meshes: 12 million polygons
Synaptome of a layer 5 pyramidal neuron
A close-up view of the 4138 local connections – input spines and output boutons – that this neuron forms with other cells (invisible here) in a neocortical circuit. Given points on the surface of the branches and end points, I developed a way to represent spines in a realistic way, based on scientific papers documenting neuron spine types and sizes.
2nd place award, professional illustrators category
AEIMS 18 medical illustration congress
Custom Python library for meshing and visualization, Blender, Cycles.
Eilif Muller et al.
Topology, a mathematical tool for interpreting brain activity
Illustration concept of algebraic topology coupled with neuronal simulation. Using simulation data on the left hand, I revealed the other "multidimensional side" in an orthogonal projection. Black and white here is a visual code I chose for domains that are beyond our sight. When many neurons are connected together at a given time, they form simplices of high dimensional orders and even "cavities", such as the one illustrated here.
This publication generated a small buzz in the scientific community with over 90’000 views in two weeks and a lot of questions asked to BBP.
Max Nolte et al
Volumetric dataset illustrated
From a biological reference
Volumetric data of all sub-regions within the rat hippocampus, with a Nissl staining from an image stack that was the basis for calculating these volumes.
Aesthetic Merit, Seeing Life Science, EPFL 2018.
Image published in Nature Nanotechnology
Cover of The Analytical Scientist
Illustration of the tiophenol reaction to specific wavelength laser within the 7Å gap between gold particles.
Molecular cavity optomechanics
Nano particles of gold are positioned to make a nano gap of about 7Å. When this gap is lit with a precise wavelength, tiophenol molecules react, "store" photons and scatter them soon after [Raman scattering]. Philippe Roelli et al described the inner workings of this phenomenon which resulted in a lot of attention in the scientific community and this will pave the way for new quantum technology.
- Nature Nanotechnology TOC Volume 11 Issue 2
- The Analytical Scientist #37
Tobias Kippenberg, Philippe Roelli et al.
Nicolas Antille © 2015-2020
About these scientific illustrations
Vision & work
I strive to make meaningful and aesthetic visualizations for each and every project, in close collaboration with scientists. As an engineer, I read scientific papers, analyze data and I do research and development to create algorithms and tools – often based on open source ones – that process data produced by the hard work of scientists. As an artist, I use such tools to reveal scientific research with novel and insightful visualizations.