Description

An essential task in Arti cial Intelligence (AI) is the prediction of future inputs from a given sequence, such as predicting the next frame of a video, with prominent applications such as self-driving cars. The human brain is very good at this task, without this ability we would be too slow to catch a ball or jump out of the way. A theory called "predictive coding" in neuro- science introduced by Ballard and Rao in 1999, explains the phenomena.

A recently developed predictive network from Massachusetts Institute of Technology (MIT) called 'PredNet' leverages the ideas of predictive coding for next-frame video prediction. Interestingly, PredNet has shown a very brain-like ability to be fooled by illusions of motion (when a static image appears to be moving). This leads us to pose the question of whether PredNet can be used to study perceptual disruptions in the brain associated with mental disorders, particularly schizophrenia.

In this study, we show how several types of modi cations to PredNet designed to simulate di erent models of perception disruption in schizophre- nia from neuroscience literature, a ects its ability to be fooled by illusions of motion, and the overall e ect these disruptions have on the predictive ability.

Read the report to know more: PDF

PredNet

While many strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem with unsupervised learning is that training on unlabeled examples to learn about the structure of a domain remains a dicult problem. This problem arises mainly on computer vision models that are typically trained on static images whereas, in the real world, visual objects are alive with movement, driven both by the self-motion of the viewer and the objects within a scene.

To address this, Bill Lotter, Gabriel Kreiman, and David Cox developed a deep convolutional recurrent neural network they called PredNet [Lotter et al., 2016]. While building on the previous work in next-frame video prediction [Softky, 1996, Palm, 2012, Goroshin et al., 2015, Mathieu et al., 2015b, Wang and Gupta, 2015], the architecture of PredNet is heavily inspired by the concept of predictive coding from neuroscience literature [Rao and Ballard, 1999].

Read the report to know more: PDF

Results

The results have been summarized in the report and also in the presentation. Have a look through the presentation for an abstract idea of things and then dig deeper into the world of predictive networks, illusions and schizophrenia by reading the report.

Presentation

Report

References

Click on respective names to open pdf
Year Name Abstract Author
Mar 2017 Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning While great strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem of unsupervised learning - leveraging unlabeled examples to learn about the structure of a domain - remains a difficult unsolved challenge. Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world. We describe a predictive neural network ("PredNet") architecture that is inspired by the concept of "predictive coding" from the neuroscience literature. These networks learn to predict future frames in a video sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers. We show that these networks are able to robustly learn to predict the movement of synthetic (rendered) objects, and that in doing so, the networks learn internal representations that are useful for decoding latent object parameters (e.g. pose) that support object recognition with fewer training views. We also show that these networks can scale to complex natural image streams (car-mounted camera videos), capturing key aspects of both egocentric movement and the movement of objects in the visual scene, and the representation learned in this setting is useful for estimating the steering angle. Altogether, these results suggest that prediction represents a powerful framework for unsupervised learning, allowing for implicit learning of object and scene structure. David Cox, William Lotter, Gabriel Kreiman
March 2018 Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction The cerebral cortex predicts visual motion to adapt human behavior to surrounding objects moving in real time. Although the underlying mechanisms are still unknown, predictive coding is one of the leading theories. Predictive coding assumes that the brain's internal models (which are acquired through learning) predict the visual world at all times and that errors between the prediction and the actual sensory input further refine the internal models. In the past year, deep neural networks based on predictive coding were reported for a video prediction machine called PredNet. If the theory substantially reproduces the visual information processing of the cerebral cortex, then PredNet can be expected to represent the human visual perception of motion. In this study, PredNet was trained with natural scene videos of the self-motion of the viewer, and the motion prediction ability of the obtained computer model was verified using unlearned videos. We found that the computer model accurately predicted the magnitude and direction of motion of a rotating propeller in unlearned videos. Surprisingly, it also represented the rotational motion for illusion images that were not moving physically, much like human visual perception. While the trained network accurately reproduced the direction of illusory rotation, it did not detect motion components in negative control pictures wherein people do not perceive illusory motion. This research supports the exciting idea that the mechanism assumed by the predictive coding theory is one of basis of motion illusion generation. Using sensory illusions as indicators of human perception, deep neural networks are expected to contribute significantly to the development of brain research. Eiji Watanabe , Akiyoshi Kitaoka, Kiwako Sakamoto, Masaki Yasugi and Kenta Tanaka
1999 Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects We describe a model of visual processing in which feedback connections from a higher- to a lower-order visual cortical area carry predictions of lower-level neural activities, whereas the feedforward connections carry the residual errors between the predictions and the actual lower-level activities. When exposed to natural images, a hierarchical network of model neurons implementing such a model developed simple-cell-like receptive fields. A subset of neurons responsible for carrying the residual errors showed endstopping and other extra-classical receptive-field effects. These results suggest that rather than being exclusively feedforward phenomena, nonclassical surround effects in the visual cortex may also result from cortico-cortical feedback as a consequence of the visual system using an efficient hierarchical strategy for encoding natural images. Rajesh P. N. Rao & Dana H. Ballard
August 2014 What visual illusions teach us about schizophrenia Illusion, namely a mismatch between the objective and perceived properties of an object present in the environment, is a common feature of visual perception, both in normal and pathological conditions. This makes illusion a valuable tool with which to explore normal perception and its impairments. Although still debated, the hypothesis of a modified, and typically diminished, susceptibility to illusions in schizophrenia patients is supported by a growing number of studies. The current paper aimed to review how illusions have been used to explore and reveal the core features of visual perception in schizophrenia from a psychophysical, neurophysiological and functional point of view. We propose an integration of these findings into a common hierarchical Bayesian inference framework. The Bayesian formalism considers perception as the optimal combination between sensory evidence and prior knowledge, thereby highlighting the interweaving of perceptions and beliefs. Notably, it offers a holistic and convincing explanation for the perceptual changes observed in schizophrenia that might be ideally tested using illusory paradigms, as well as potential paths to explore neural mechanisms. Implications for psychopathology (in terms of positive symptoms, subjective experience or behavior disruptions) are critically discussed. Charles-Edouard Notredame, Delphine Pins, Sophie Deneve and Renaud Jardri
November 2013 Circular inferences in schizophrenia A considerable number of recent experimental and computational studies suggest that subtle impairments of excitatory to inhibitory balance or regulation are involved in many neurological and psychiatric conditions. The current paper aims to relate, specifically and quantitatively, excitatory to inhibitory imbalance with psychotic symptoms in schizophrenia. Considering that the brain constructs hierarchical causal models of the external world, we show that the failure to maintain the excitatory to inhibitory balance results in hallucinations as well as in the formation and subsequent consolidation of delusional beliefs. Indeed, the consequence of excitatory to inhibitory imbalance in a hierarchical neural network is equated to a pathological form of causal inference called ‘circular belief propagation’. In circular belief propagation, bottom-up sensory information and top-down predictions are reverberated, i.e. prior beliefs are misinterpreted as sensory observations and vice versa. As a result, these predictions are counted multiple times. Circular inference explains the emergence of erroneous percepts, the patient’s overconfidence when facing probabilistic choices, the learning of ‘unshakable’ causal relationships between unrelated events and a paradoxical immunity to perceptual illusions, which are all known to be associated with schizophrenia. Sophie Deneve, Renaud Jardri
October 2016 Circular inference: Mistaken belief, misplaced trust At the clinical level, psychosis can be formalized as the formation of aberrant beliefs or percepts and has been proposed to result from disruptions in the excitation/inhibition (E/I) balance in cortical microcircuitry. However, these two conceptual approaches toward psychosis have yet to be correlated. Here, we review recent empirical and computational studies that enable an integrated understanding of how the brain may generate beliefs along a spectrum ranging from normal to pathology. We mainly focus on hierarchical predictive coding and circular inference. We will expose how these two frameworks may account for hallucinations, delusions, and reduced susceptibility to illusions, and we will additionally critically discuss their respective strengths and weaknesses as well as potential future research directions. Sophie Deneve, Renaud Jardri
December 2008 Perceiving is believing: a Bayesian approach to explaining the positive symptoms of schizophrenia Advances in cognitive neuroscience offer us new ways to understand the symptoms of mental illness by uniting basic neurochemical and neurophysiological observations with the conscious experiences that characterize these symptoms. Cognitive theories about the positive symptoms of schizophrenia — hallucinations and delusions — have tended to treat perception and belief formation as distinct processes. However, recent advances in computational neuroscience have led us to consider the unusual perceptual experiences of patients and their sometimes bizarre beliefs as part of the same core abnormality — a disturbance in error-dependent updating of inferences and beliefs about the world. We suggest that it is possible to understand these symptoms in terms of a disturbed hierarchical Bayesian framework, without recourse to separate considerations of experience and belief. Paul C. Fletcher, Chris D. Frith
May 2013 The computational anatomy of psychosis This paper considers psychotic symptoms in terms of false inferences or beliefs. It is based on the notion that the brain is an inference machine that actively constructs hypotheses to explain or predict its sensations. This perspective provides a normative (Bayes-optimal) account of action and perception that emphasizes probabilistic representations; in particular, the confidence or precision of beliefs about the world. We will consider hallucinosis, abnormal eye movements, sensory attenuation deficits, catatonia, and delusions as various expressions of the same core pathology: namely, an aberrant encoding of precision. From a cognitive perspective, this represents a pernicious failure of metacognition (beliefs about beliefs) that can confound perceptual inference. In the embodied setting of active (Bayesian) inference, it can lead to behaviors that are paradoxically more accurate than Bayes-optimal behavior. Crucially, this normative account is accompanied by a neuronally plausible process theory based upon hierarchical predictive coding. In predictive coding, precision is thought to be encoded by the post-synaptic gain of neurons reporting prediction error. This suggests that both pervasive trait abnormalities and florid failures of inference in the psychotic state can be linked to factors controlling post-synaptic gain – such as NMDA receptor function and (dopaminergic) neuromodulation. We illustrate these points using biologically plausible simulations of perceptual synthesis, smooth pursuit eye movements and attribution of agency – that all use the same predictive coding scheme and pathology: namely, a reduction in the precision of prior beliefs, relative to sensory evidence. Rick A. Adams, Klaas Enno Stephan, Harriet R. Brown, Christopher D. Frith and Karl J. Friston
January 2013 The function of efference copy signals: Implications for symptoms of schizophrenia Efference copy signals are used to reduce cognitive load by decreasing sensory processing of reafferent information (those incoming sensory signals that are produced by an organism’s own motor output). Attenuated sensory processing of self-generated afferents is seen across species and in multiple sensory systems involving many different neural structures and circuits including both cortical and subcortical structures with thalamic nuclei playing a particularly important role. It has been proposed that the failure to disambiguate self-induced from externally generated sensory input may cause some of the positive symptoms in schizophrenia such as auditory hallucinations and delusions of passivity. Here, we review the current data on the role of efference copy signals within different sensory modalities as well as the behavioral, structural and functional abnormalities in clinical groups that support this hypothesis. Laura K.Pynn, Joseph F.X. DeSouza
July 2015 Active inference, communication and hermeneutics Hermeneutics refers to interpretation and translation of text (typically ancient scriptures) but also applies to verbal and non-verbal communication. In a psychological setting it nicely frames the problem of inferring the intended content of a communication. In this paper, we offer a solution to the problem of neural hermeneutics based upon active inference. In active inference, action fulfils predictions about how we will behave (e.g., predicting we will speak). Crucially, these predictions can be used to predict both self and others – during speaking and listening respectively. Active inference mandates the suppression of prediction errors by updating an internal model that generates predictions – both at fast timescales (through perceptual inference) and slower timescales (through perceptual learning). If two agents adopt the same model, then – in principle – they can predict each other and minimise their mutual prediction errors. Heuristically, this ensures they are singing from the same hymn sheet. This paper builds upon recent work on active inference and communication to illustrate perceptual learning using simulated birdsongs. Our focus here is the neural hermeneutics implicit in learning, where communication facilitates long-term changes in generative models that are trying to predict each other. In other words, communication induces perceptual learning and enables others to (literally) change our minds and vice versa. Karl J. Friston and Christopher D. Frith

Team

Dr. Antonina Kolokolova

Antonina is an Associate Professor at the Department of Computer Science , Memorial University of Newfoundland. Her research interests are in theoretical computer science, in particular complexity theory and logic, including proof complexity.


Umar Shaik

Umar is pursuing MSc in Computer Science from Memorial University of Newfoundland


Daniel Powers

Daniel is pursuing BSc in Computer Science from Memorial University of Newfoundland


Hilary Sinclair

Hilary holds a BEng (Mech) and is currently pursuing a BSc in Computer Science from Memorial University.