The neural coding problem

[eplunix!peter@eddie.mit.edu]

In a very useful note Marius Usher recently (3/31/95) brought up 
the neural coding problem for discussion:

> Perhaps the most crucial question in the study of cortical function is
> whether  the brain uses a mean rate code or a temporal code.
> Recently a number of models have been proposed in order to account
> for the variability of spike trains (discussed by Softky and Koch, 1993).
> As it seems, each of these models can account for variability, despite their
> very different assumptions and implications regarding the "neural code".

> We are writing this note in order to highlight the specific predictions in
> which these models differ, hoping in particular to direct the attention of
> experimentalists to the "missing " data required to disambiguate between
> these theoretical models and their implications about the neural code.

For the last five years I have been investigating population interspike 
interval codes in the auditory system which may subserve perception 
of the low pitches of complex tones (periodicity pitch) and the 
discrimination of phonetic elements. As a consequence, I have given
a great deal of thought to how central auditory structures
might use the wealth of timing information which is available in the
auditory periphery. 

Here are some thoughts that may facilitate the more general
discussion of the neural coding problem:

1. Very many neural codes based on temporal patterns or times-of-arrival
(including interneural synchrony codes) are possible, but only a very
small subset of the possible codes, particularly of the higher-order
pattern codes, have yet been seriously considered, either experimentally
or theoretically. We should not rule out more complex codes on the basis
of not finding evidence for the simplest or most obvious ones.

2. Neural codes are generally not mutually exclusive. A given spike train
can be interpreted in different ways by different neural assemblies
downstream. Thus discharge rates could be used by some cell populations,
temporal patterns by another, and patterns of spike latencies by another.
A possible example of this can be found in the ascending auditory pathway
of many mammals, where there are several brainstem pathways which 
subserve auditory localization. Some pathways appear to convey binaural
level differences encoded in discharge rates while others appear to 
convey interaural time differences encoded in spike latencies and 
interneural synchronies. There are almost undoubtedly real
neural architectures which gracefully fuse both rate 
and time-based information (cf. Licklider), but very few "duplex" models
have been proposed.

3. Often several aspects of neural discharge covary. For example,
in the peripheral auditory system the roles of discharge rates 
and temporal patterns are hard to separate, since both kinds 
of information are present together in nearly all auditory populations.

4. While information can be encoded in the discharges of individual neurons, 
it seems likely from reliability considerations that information is 
encoded in the activity of populations of neurons. We are very familiar
by now with the possibility of distributed rate codes, but it is also
possible to have distributed temporal codes. An example of a distributed
synchrony code is the "volley principle" in the auditory system. An
example of a distributed temporal pattern code would be a population
interspike interval code, where the all-order interval distribution of
a population conveys information concerning stimulus periodicities.
Distributed temporal codes can be either synchronous or asynchronous.
Every hypothetical neural code has a corresponding hypothetical 
processing architecture.

5. Deciding whether a particular pattern of discharge is a "code" (i.e.
that it has a functional role in the representation, transmission, and
processing of information) is a difficult problem, since there are only a
few systems whose function is understood well enough to see immediately
what role a given putative encoding would play.

   Possibly the most direct way to demonstrate that a given 
discharge pattern has functional significance is to impose a 
particular pattern of activity on a neural population, 
e.g. by electrical stimulation, and to observe 
the perceptual and behavior consequences. Specific electrical time patterns 
are known to evoke particular sensations in many diverse
sensory modalities: audition (single-channel cochlear implants,
Eddington), somatoception (Mountcastle), gustation (Covey, DiLorenzo), 
and even color vision (Young). 
   The next best thing is to look for correspondences between neurophysiology
and psychophysics by comparing how closely a putative neural representation 
covaries with the percepts/behaviors which the representation
hypothetically subserves. On the perceptual side this is a 
stimulus-coding problem -- does the code covary with perceptual performances?
If discharge rates saturate and representations based on rates are degraded
at high stimulus intensities when perceptual discriminations are unchanged or
even improve, then this is evidence against a functional role for rate
representations (in lieu of elaborate compensatory mechanisms, which then
must be found and incorporated into the representation's description). 
   
6. Putative codes can be ruled out by showing that the information needed
to perform a particular perceptual or behavioral task is not present in the
discharge activity of a particular population. It is important not to erect
"straw man" codes when trying to rule out possible coding schemes. In 
general, the kinds of temporal codes thus far considered in the literature
have been only the most simple and obvious ones, and much more consideration
needs to be given to population synchrony codes (a la Abeles' synfire codes) 
and asynchronous temporal pattern codes. (a la Abeles' neurophysiological 
results).

7. If one finds a correspondence between discharge rates and some 
perceptually- or behaviorally-relevant distinction, this does not necessarily
rule out a time code. Because rate-codes have been the conventional assumption
of most of neuroscience, often physiological investigations stop when 
scientists find what they are looking for, i.e. rate-based 
correspondences with perception or behavior. However, underlying the
rate-based responses may be complex time patterns of excitation and 
inhibition that may better correspond to the psychophysics than the
rate code itself (arguably this is the case in explaining 
frequency selectivity in the peripheral auditory system -- 
while one can point to rate-based "frequency-channels" in the
auditory nerve, the interspike interval distributions of the auditory nerve
fibers yield much more robust and higher quality information (Goldstein)
which, like the percepts, does not degrade with higher stimulus sound 
pressure levels. A similar situation exists in the fly visual system 
-- Bialek, Reichardt)

8. Long temporal integration times do not preclude temporal coding. In the
auditory system there are a number of reasons to believe that the time
window for fusing sounds is on the order of 5-10 msec (e.g. Chistovitch), 
whereas there is a longer build-up process associated with the apparent 
loudnesses of sounds of short durations (Zwislocki, Chistovitch). 

(We have many examples of rate-based processing models in the literature, 
but a dearth of time-based ones -- I will therefore outline a 
possible temporal integration mechanism as an example). Let us suppose that
we have a complex acoustic stimulus with a low pitch, 
say a vowel with a fundamental frequency F0 (voice pitch) of 100 Hz). 
The most frequent interspike interval in the population of 
auditory nerve fibers and (probably) most cochlear nucleus populations
will be 1/F0 = 10 msec. At the auditory cortex, this voice pitch will be
seen in periodicities of auditory evoked potentials (e.g. Steinschneider et
al), so there are evidently populations of neurons which are discharging 
either singly or in spatially-distributed volleys at intervals of 10 msec.
There are probably other units which have discharge periodicities related
to 10 msec which are not synchronized relative to the rest of the population. 
There are many recurrent pathways within the auditory cortices and the 
thalamus where spike trains containing a disproportionate numbers of 
these intervals can circulate. It is not then hard to imagine a temporal 
cross-correlation process between intervals circulating in these loops 
and incoming temporal patterns, and as 10 msec intervals are differentially 
faciliated based on their prevalence in the reverberating loops, this kind of 
structure would produce an asynchronous build-up of 10 msec intervals 
over longer time windows. It's only a sketch, but such mechanisms do not
seem to be out of the question.

Marius Usher also gave an example in favor of rate-coding:
> Proponents of the coincidence detection principle may need to find an 
> explanation for the wealth of evidence showing integration in the perceptual
> system. For example, the  Bloch law (Loftus and Ruthruff 1993) shows that, for
> stimuli of duration shorter than 100 msec, perceptual discrimination depends
> only on the INTEGRAL of the stimulus (a high contrast 10 msec stimuli, 
> produces exactly the same affect on perception as a 20 msec stimuli 
> of half contrast).

While I am not a visual scientist, I do know that images of very
short duration (tachistoscopically presented) can be recognized, and that,
like in the auditory system, the time windows for perceptual fusion are
much shorter than for integration of intensity-related information. There
are many alternatives to rate-based models of intensity discrimination, but
these are generally underdeveloped. Two general classes of alternative
models would be latency-based models (latency, latency variances) 
and temporal correlation models (population interval models). The latency
codes need gating/reset mechanisms in addition to buildup loops.
Apparently the latency of visual evoked potentials corresponds well to 
subjective brightness (see S.S. Stevens, "Sensory power functions 
and neural events" in Principles of Receptor Physiology, 
Loewenstein, ed. Springer-Verlag, 1971.), so that there are
some examples, even in vision, of possible codes not based on rate.
In addition Reichardt et al and Bialek et al found evidence for 
temporal cross-correlation operations in insect motion detection and 
Chung, Raymond, and Lettvin found interspike intervals corresponding to
various luminance conditions in the frog.

At the risk of heresy, I think that there could be a general 
temporal correlation theory of vision, particularly of form vision. 
I would think that there  would be a great deal of 
spatio-temporal structure which would be imposed on retinal 
responses whenever a structured image moves across the receptor array, 
which, as I understand is the normal state of affairs -- 
the eye is always moving, even in "fixation". I have never 
understood how a rate-based model of visual form accounts for this 
(and what is it that is integrated over 100 msec if the image is constantly 
moving?). For an alternative temporal model of visual form
it would be useful to know exactly how reliable 
are the latency distributions of vertebrate 
retinal ganglion cells to edges crossing their fields 
and how much temporal cross-correlation information 
might exist between ganglion cells in a local region. 
Does anyone know  offhand if (where) such data exists?

If the temporal correlations in the retinal responses 
are what matter, then higher contrast stimuli should
produce more spatio-temporal structure. It may be the case that the 10 msec 
high-contrast stimulus may generate as many temporal cross-correlations 
as the 20 msec low-contrast stimulus, and that these 
cross-correlation patterns are integrated at higher levels through 
recurrent temporal cross-correlation. It would also be worth checking
whether the rate-model corresponds to the psychophysics under a wide
variety of conditions (very low and very high light levels, in the presence
of visual noise, chromatic light, etc.). In the auditory system, auditory
nerve discharge rate models work fairly well for moderate sound pressure 
levels but not very well for either levels near threshold or high levels.

9. (last thought). There also are many perceptual phenomena which are not easily
explained using average rate models, but which are explicable in terms of
temporal codes. Some of these are: the low pitch of unresolved harmonics,
the pitch of repetition noise, the pitch of AM noise, the perception of
different vibratory frequencies in somatoception, achromatic color
(Benham's Top), the Pulfrich effect (latency & perception of depth), the
perception of visual "spatial beats" which is the analogue
of the "missing fundamental" in audition (Hammett & Smith), 
and all of the electrical stimulation examples alluded to above. 
Bekesy reportedly was able to localize stimuli differing 
in arrival time by 1-2 msec using many different modalities, 
(e.g. audition, somatoception, taste, olfaction) 
presumably on the basis of latency differences (Bower, Bekesy).

There is really a bewildering array of physiological and psychophysical
evidence that needs to be examined in some sort of systematic way for
correspondences. I've started to collect and collate the disparate 
evidence for temporal coding, and I have yet to find a sensory modality
for which there is not some evidence available in the literature.

The neural coding problem is fundamental because
until we understand the nature of the neural signals involved, 
we may miss those aspects of neural activity which are essential to 
the functional organization of the system.

Dr. Peter Cariani
Eaton Peabody Laboratory of Auditory Physiology
Massachusetts Eye & Ear Infirmary
243 Charles St., Boston, MA 02114 USA

4/3/95

email:  eplunix!peter at eddie.mit.edu
tel: (617) 573-4243
FAX: (617) 720-4408


References
--------------------------------------------------------------------------

Abeles, M., H. Bergman, E. Margalit, and E. Vaadia. "Spatiotemporal firing 
patterns in the frontal cortex of behaving monkeys." J. Neurophysiol. 70 (4 
1993): 1629-1638. See also Abeles et al in Concepts in Neuroscience, 4(2):
131-158, 1993.

Bekesy, Georg von. "Olfactory analogue to directional hearing." Journal of 
Applied Physiology 19 (3 1964a): 369-373.

Bekesy, Georg von. "Rythmical variations accompanying gustatory stimulation 
observed by means of localization phenomena." Journal of General Physiology 47 
(5 1964b): 809-825.

Bialek, W., F. Rieke, R. R.  van Stevenink, and Warland de  Ruyter D. "Reading a 
neural code." Science 252 (28 June 1991): 1854-1856. Fly vision.

Bower, T. G. R. "The evolution of sensory systems." In Perception: Essays in 
Honor of James J. Gibson, ed. Robert B. MacLeod and Herbert Pick Jr. 141-152. 
Ithaca: Cornell University Press, 1974. (Bekesy anecdotes)

Bullock, T.H. "Signals and neural coding." In The Neurosciences: A Study 
Program, ed. G.C. Quarton, T. Melnechuck, and F.O. Schmitt. 347-352. New York: 
Rockefeller University Press, 1967. General review.

Cariani, P. and B. Delgutte. "Interspike interval distributions of auditory 
nerve fibers in response to variable-pitch complex stimuli." Assoc. Res. 
Otolaryng. (ARO) Abstr. (1992): 

Cariani, P and B. Delgutte. "The pitch of complex sounds is simply coded in 
interspike interval distributions of auditory nerve fibers." Soc. Neurosci. 
Abstr. 18 (1992): 383.

Cariani, P. As if time really mattered: temporal strategies for neural coding 
of sensory information. Communication and Cognition - Aritificial Intelligence, 
1995, 12(1-2):161-229. Preprinted in: Origins: Brain and Self-Organization, K 
Pribram, ed., Lawrence Erlbaum Assoc., 1994; 208-252.

Chistovitch, L. A. Central auditory processing of peripheral vowel spectra.
J. Acoust. Soc. Am. 77(3):789-805. Time window for fusion of spectral shapes.

Chung, S.H., S.A. Raymond, and J.Y. Lettvin. "Multiple meaning in single visual 
units." Brain Behav Evol 3 (1970): 72-101. Interval codes in frog vision.

Covey, Ellen. "Temporal Neural Coding in Gustation." Ph.D., Duke University, 
1980. Time pattern codes in rodent taste.

Delgutte, B. and P. Cariani. "Coding of the pitch of harmonic and inharmonic 
complex tones in the interspike intervals of auditory nerve fibers." In The 
Processing of Speech, ed. M.E.H. Schouten. Berlin: Mouton-DeGruyer, 1992.

Di Lorenzo, Patricia M. and Gerald S. Hecht. "Perceptual consequences of 
electrical stimulation in the gustatory system." Behavioral Neuroscience 107 
(1993): 130-138. Time pattern codes in rodent taste.

Eddington, D. K., W.H. Dobelle, D. E. Brackman, M. G. Mladejovsky, and J. 
Parkin. "Place and periodicity pitch by stimulation of multiple scla tympani 
electrodes in deaf volunteers." Trans. Am. Soc. Artif. Intern. Organs XXIV 
(1978): 

Festinger, Leon , Mark R. Allyn, and Charles W. White. "The perception of color 
with achromatic stimulation." Vision Res. 11 (1971): 591-612.

Hammett, S.T. and Smith, A.T. Temporal beats in the human visual system.
Vision Research 34(21):2833-2840. Missing (spatial) fundamentals.

Goldstein, J. L. and P. Srulovicz. "Auditory-nerve spike intervals as an 
adequate basis for aural frequency measurement." In Psychophysics and Physiology 
of Hearing, ed. E.F. Evans and J.P. Wilson. London: Academic Press, 1977.

Kozak, W.M. and H.J. Reitboeck. "Color-dependent distribution of spikes in 
single optic tract fibers of the cat." Vision Research 14 (1974): 405-419.

Kozak, W.M., H.J. Reitboeck, and F. Meno. "Subjective color sensations elicited 
by moving patterns: effect of luminance." In Seeing Contour and Colour, ed. J.J. 
Kulikowski Dickenson, C.M. 294-310. New York: Pergamon Press, 1989.

Licklider, J.C.R. "A duplex theory of pitch perception." Experientia VII (4 
1951): 128-134. Mixed time-place autocorrelation model.

Licklider, J.C.R. "Three auditory theories." In Psychology: A Study of a 
Science. Study I. Conceptual and Systematic, ed. Sigmund Koch. 41-144. Volume I. 
Sensory, Perceptual, and Physiological Formulations. New York: McGraw-Hill, 
1959.

Macrides, F. "Dynamic aspects of central olfactory processing." In Chemical 
Signals in Vertebrates, ed. D. Muller Schwartze and M. M. Mozell. 207-229. 3. 
New York: Plenum, 1977. Time patterns in smell. See also more recent work
by Gilles Laurent in insect olfaction. Science, 265: 1872-75, Sept 23, 1994.

Macrides, Foteos and Stephan L. Chorover. "Olfactory bulb units: activity 
correlated with inhalation cycles and odor quality." Science 175 (7 January 
1972): 84-86. Temporal code for smell.

Morrell, F. "Electrical signs of sensory coding." In The Neurosciences: A Study 
Program, ed. G.C. Quarton, T. Melnechuck, and F.O. Schmitt. 452-469. New York: 
Rockefeller University Press, 1967. Review.

Mountcastle, Vernon. "The problem of sensing and the neural coding of sensory 
events." In The Neurosciences: A Study Program, ed. G.C. Quarton Melnechuk, T., 
and Schmitt, F.O. New York: Rockefeller University Press, 1967. Review.

Mountcastle, Vernon. "Temporal order determinants in a somatosthetic frequency 
discrimination: sequential order coding." Annals New York Acad. Sci. 682 (1993): 
151-170. Problem of vibration discrimination/neural representation.

Mountcastle, V.B., W.H. Talbot, H. Sakata, and J. Hyvarinen. "Cortical neuronal 
mechanisms in flutter-vibration studied in unanesthetized monkeys. Neuronal 
periodicity and frequency discrimination." J. Neurophysiol. 32 (1969): 452-485.

Reichardt, Werner. "Autocorrelation, a principle for the evaluation of sensory 
information by the central nervous system." In Sensory Communication, ed. Walter 
A. Rosenblith. 303-317. New York: MIT Press/Wiley, 1961. See also Egelhaaf & 
Borst. A look into the cockpit of the fly: visual orientation, algorithms, and
identified neurons. J. Neuroscience, Nov. 1993 13(11):4563-4574.

Uttal, W.R. The Psychobiology of Sensory Coding. New York: Harper and Row, 1973.
Review of the coding problem.

Young, R.A. "Some observations on temporal coding of color vision: 
psychophysical results." Vision Research 17 (1977): 957-965. Electrical 
temporal pattern stimulation produces colored phsophenes.

Zwislocki, J. 1960. Theory of temporal auditory summation. J. Acoust. Soc.
Am. 1960 32(8):1046-60.