Angelo Franciosini (PhD , 2017 / 2020)
Trajectories in natural images and the sensory processing of contours (PhD position, 2017 / 2021)
Aix-Marseille Université's Ph.D. program in Integrative and Clinical Neuroscience
- Vision, Neural Networks, Bio-Inspired Computer Vision, contours, learning
- Thesis director
- Dr. Laurent PERRINET, Director's research unit: Institut de Neurosciences de la Timone (INT)
Summary of the proposed research project
- State of the art
Binding the different features of objects in images is at the core of visual perception. As such, the visual system needs to detect local edges and to bind them together to form contours at a higher, more global level. A state-of-the art theory is that of the “association field”: the confidence of an edge depends on the configuration of neighboring edges. For instance it is facilitated for co-linear or co-circular edges. This process takes advantage of the statistical regularities of edges that are present in natural images. In particular, we have developed a method to quantify the association field in different classes of natural images (Perrinet & Bednar, 2015). Using an existing library, it is possible to compute histograms of edge co-occurrences from the sparse representation of static natural images. We have already shown that these different statistics were sufficient to categorize images, for instance to know if they contain an animal or not. At the neural level, modeling the representation of the image, such as that formed in the primary visual cortex of primates (V1), this heuristics translates to a set of rules that adapts dynamically the activity of isolated neurons representing edges into the coherent population activity of contours. Yet, we miss an understanding of the link between these statistics and the probabilistic rules that binds features together and how this information is dynamically encoded in V1.
In this computational neuroscience project, we will exploit our current expertise in computer vision for the statistical integration of visual of objects to translate them in the form of probabilistic predictive models for biological vision. Our core hypothesis is that in natural scenes, contours follow coherent trajectories and that this knowledge is integrated (learned) by the visual system to optimally inform the representation of the image.
First, we will learn the different classes of edge co-occurrences that are relevant to natural images. Using an existing unsupervised learning algorithm, we will learn these as an independent components analysis. Such an algorithm extends well to a deep-learning convolutional neural network, but importantly, it will be informed by our expertise of modeling neural networks in low-level visual areas by including horizontal connectivity. We expect that relevant features will be mainly the predictable arrangements, such as co-linear or co-circular pairs of edges, but also highly surprising ones, such as T-junctions or end-stopping features. Importantly, we will be able to compare this representation with that present in higher level areas and to refine our knowledge on the representation of natural-like images. Second, we have previously found that using synthetic textures could further advance our understanding of neural computations and perception. These random synthetic textures, coined “Motion Clouds” were initially targeted to quantify the integration properties of visual motion perception (Leon et al, 2012, Simoncini et al, 2012). Informed by the generative model of edge co-occurrences studied above, an extension to such stimuli would be to include dependencies between different elements. As such, we will be able to manipulate the level of dependency between different elements, whether in space, time or feature space (orientations). A potential outcome will be to use these in neurophysiological and psychophysical experiments within the team. In particular, the ability to select different classes of dependencies learned above will make it possible to evaluate the relative contribution of each component to the association field.
- Expected results
Finally, those two tasks converge to a long-term goal of understanding the impact of the spatio-temporal structure of natural images in the neural computations implementing visual processing in low-level visual areas and perception. Indeed, the regularities observed in static images can be extended to dynamical scenes by observing that a co-occurrence in time can be implemented by simple geometrical operations as they are operated during that period. For instance a co-circularity can be described as a smooth roto-translational transformation of an edge along a smooth trajectory. Importantly, such a distinction should allow us to determine the hierarchy of different features relevant to describe the full statistics of the feature space (that is, of spatio-temporal edge co-occurrences). We expect to see that the different independent features should decompose at various scales both in space and in time. This translates into a probabilistic hierarchical model that would combine dependencies from different cues. In particular, we expect to see the emergence of differential pathways for form and motion.
The project is based on existing expertise and libraries in computer vision and computational neuroscience. The extension of this expertise to the dynamical domain will be made possible thanks to an existing collaboration (JM Morel at ENS-Cachan, G Peyré at ENS-Ulm). The groundbreaking nature of the work takes advantage of the interaction with neurophysiological and psychophysical experiments thanks to the use of synthetic textures (collaboration with F Chavane, INT; Y Fregnac, UNIC) as planned in a the parallel (approved) grant Horizontal-V1.
Simoncini C, Perrinet LU, Montagnini A, Mamassian P, Masson GS (2012) More is not always better: dissociation between perception and action explained by adaptive gain control. Nature Neuroscience, 15:1596–1603
Friston KJ, Adams RA, Perrinet LU, Breakspear M (2012) Perceptions as Hypotheses: Saccades as Experiments. Frontiers in Psychology, 3
Perrinet LU, Adams RA, Friston KJ. (2014) Active inference, eye movements and oculomotor delays. Biological Cybernetics, 108(6):777-801
Perrinet LU, Bednar JA (2015) Edge co-occurrences can account for rapid categorization of natural versus animal images. Scientific Reports, 5:11400
Khoei M, Masson GS, Perrinet LU (2017) The flash-lag effect as a motion-based predictive shift. PLoS Computational Biology, 13(1):e1005068
we keep a bibliography on the project @ https://www.mendeley.com/community/edgetracks/
List of publications
- Angelo Franciosini, Laurent U. Perrinet. On the Origins of Hierarchy in Visual Processing, URL . In Curves and Surfaces 2018, Arcachon, 2018 abstract
- Victor Boutin, Franck Ruffier, Laurent Perrinet. From biological vision to unsupervised hierarchical sparse coding, URL . In submitted to iTwist, 2018, 2018 abstractThe formation of connections between neural cells is essentially emerging from an unsupervised learning process. During the development of primary visual cortex (V1) of mammals, for example, one may observe the emergence of cells selective to localized and oriented features. This leads to the development of a rough contour-based representation of the retinal image in area V1. We modeled the formation of this representation along the thalamo-cortical pathway using a sparse unsupervised learning algorithm in a hierarchical network. This algorithm alternates (i) a coding phase to encode the information and (ii) a learning phase to find the proper encoder (also called dictionary). We replicated and adapted the Multi-Layer Convolutional Sparse Coding (ML-CSC) model from Michael Elad's group \citeSulam2017. As an application, we have trained our implementation on a database containing images from faces. The extracted features show similarities with some of the neuron's receptive field found in V1 and beyond. Furthermore, our results demonstrate the potential application of such a strategy to the fast classification of images, for example in hierarchical and dynamical architectures..
This work was supported by the Ph.D. program in Integrative and Clinical Neuroscience which received funding by the Aix-Marseille Université through the prestigious status of "Excellence Initiative" (A*MIDEX) awarded by the French Government.