Neural networks and brain function

[Rajesh Rao]

> This failure tells us something about the limitations of the cortical
> memory system, and thus, why we might need a hippocampus.
   
Speaking of the cortex, some promising results have been obtained in
recent years with regard to explaining cortical receptive field
properties and interpreting cortical feedback/lateral connections
using statistical principles such as maximum likelihood and Bayesian
estimation. This line of research goes back to the early ideas of
redundancy reduction and predictive coding advocated by Attneave
(1954), MacKay (1956), and Barlow (1961). More recent incarnations of
these ideas have been in the form of networks that attempt to learn
sparse efficient codes (Olshausen and Field, Lewicki and Olshausen),
networks that aim to maximize statistical independence of outputs
(approaches of Bell and Sejnowski, van Hateren and Ruderman using ICA
- http://www.cnl.salk.edu/~tewon/ica_cnl.html), networks that try to
learn translation-invariant codes (Dana Ballard and myself), and
networks that exploit biological constraints such as rectification for
efficient coding (Lee and Seung, Hinton and Ghahramani, Dana Ballard
and myself). Application of these algorithms to natural images produce
spatial and spatiotemporal receptive field properties qualitatively
similar to those observed in the visual cortex (the important and
related line of research on correlation-based models of development by
Ken Miller and others has already been mentioned in this thread).

In the realm of hierarchical models, the early proposal of MacKay and
more recently Mumford, ascribing to feedback connections the role of
predicting or anticipating inputs, has been formalized in terms of
learning generative models of input signals, the idea being that the
feedback pathways might represent a learned statistical model of how
the inputs are being generated ("synthesis" as opposed to "analysis"
in the feedforward pathways). Examples include the work of Dayan,
Hinton, Neal and Zemel (Helmholtz machine), Kawato, Hayakawa and Inui
(forward-inverse optics model), Dana Ballard and myself (extended
Kalman filter model), and related work by people such as Pece, Softky,
Ullman, and others (I apologize if I inadvertently missed someone -
please post a reply to add to this list). The work on hierarchical
models is also closely related to the algorithms in the previous
paragraph in that both rely on the idea of generative models, the
differences being in the type of constraints imposed and the
definition of statistical efficiency used.

Although the results obtained thus far have been encouraging, the
precise details regarding the neurobiological implementation of these
algorithms in the cortex is far from clear. There is also a need for
models that allow efficient learning of non-linear hierarchical
generative models while at the same time respecting cortical
neuroanatomical constraints. This gives me the excuse to advertise
(somewhat shamelessly) a post-NIPS workshop on statistical theories of
cortical function: the web page
http://www.cnl.salk.edu/~rao/workshop.html contains more details and
links to the web pages of some of the people pursuing this line of
research.


References:

@article	(Attneave54,
author	=	"F. Attneave" ,
title	=	"Some informational aspects of visual perception", 
journal	=	"Psychological Review" ,
volume	=	"61" ,
number  =       "3" ,
year	=	"1954" ,
pages	=	"183-193"
)

@incollection{MacKay56,
author =        "D. M. MacKay",
title =         "The epistemological problem for automata", 
editors =       "C. E. Shannon and J. McCarthy",
booktitle =     "Automata Studies", 
pages =         "235-251",
publisher =     "Princeton, NJ: Princeton University Press",
year =          "1956"
}

@incollection{Barlow61,
author =        "H. B. Barlow",
title =         "Possible principles underlying the transformation of sensory messages",
editor =       "W. A. Rosenblith",
booktitle =     "Sensory Communication",
pages =         "217-234",
publisher =     "Cambridge, MA: MIT Press",
year =          "1961"
}

(Other references to work mentioned above can be obtained from the web
pages of the researchers - see the workshop page given above for some
useful links).

---
Rajesh P.N. Rao, Ph.D.               Internet: rao at salk.edu
The Salk Institute, CNL & Sloan Ctr  VOX: 619-453-4100 x1215
10010 N. Torrey Pines Road           FAX: 619-587-0417
La Jolla, CA 92037                   WWW: http://www.cnl.salk.edu/~rao/