Connectionist symbol processing: any progress? LSA & HAL models

[Curt Burgess]

> One of things that has recently renewed my interest in the
> idea of using distributed representations for processing
> complex information was finding out about Latent Semantic
> Analysis/Indexing (LSA/LSI) at NIPS*97.  LSA is a method
> for taking a large corpus of text and constructing vector

I think LSA is an important approach in this symbol processing debate.   
A model similar in many ways to LSA is our Hyperspace Analogue to  
Language (HAL) model of memory (also at NIPS*97 [workshops]). One  
difference is that LSA (typically) is implemented in a matrix of word by  
larger text unit dimensions. HAL is a word by word matrix. There are  
other differences - one being how dimensionality is reduced.  One big  
advantage of HAL and LSA is that they use learning procedures that scale  
up to real world language problems and thus can use large corpora as  
input.  It would be difficult to put 300 million words through a SRN the  
number of times required for any learning to take place (!).  With global  
co-occurrence models like HAL or LSA, this scalability isn't a problem.  
HAL and LSA also use continuous valued vector representations which  
results in very rich encoding of meaning.

We've addressed the scalability issue by comparing HAL's algorithm to a  
SRN in a chapter available on my lab's web page  
(http://locutus.ucr.edu/Reprints.html) - get the Burgess & Lund, 1998,  
under review, document).  We show that the same input into a SRN and HAL  
will get virtually identical output.  The beauty of this is that one can  
use vector representations acquired in a global co-occurrence model in a  
connectionist model knowing that these vectors are what would likely be  
produced if they were learned via a connectionist methodology. In this  
chapter we also address a variety of other related issues (what is  
similarity? the symbol-grounding problem, the relationship between  
associations and categorical knowledge, modularity and syntactic  
constraints, developing asymmetric relationships between words, and, in a  
limited way, using high-dimensional models to mimic higher-level  
cognition). The chapter was written to be a little provocative.

There are 6 or 7 papers detailing the HAL model available as PDFs and  
another 6 or 7 that you can order with the reprint order form. The latest  
issue of Discourse Processes (edited by Peter Foltz) is a special issue  
on quantitative approaches to language and is full of LSA and HAL papers.  
 I will be editing a special journal issue that will have more HAL and  
LSA papers (a followup to the high-dimensional semantic space symposium  
at psychonomics last year).  At the SCiP (Society for Computers in  
Psychology) conference in Nov (the day before Psychonomics), we will have  
a symposium on high-dimensional semantic memory models and Tom Landauer  
is giving the keynote talk (titled "How modern computation can turn  
cognitive psychology into a real science" - I suspect also a little  
provocative!). I gave the keynote at last years SCiP meeting and this is  
available on our website and is in the last issue of BRMIC (Burgess, C.  
(1998). From Simple Associations to the Building Blocks of Language:  
Modeling Meaning in Memory with the HAL Model.  Behavior Research  
Methods, Instruments, and Computers, 30, 188 - 198.). It's a good brief  
introduction to the range of problems we've addressed.

The HAL and LSA work certainly are related to the "context vector"  
research that Steve Gallant was talking about.

I guess that's enough...

Curt
---
   Dr. Curt Burgess, Computational Cognition Lab
   Department of Psychology, University of  California
   Riverside, CA 92521-0426  URL: http://locutus.ucr.edu/
   Internet: curt at cassandra.ucr.edu
   MaBellNet: (909) 787-2392       FAX: (909) 787-3985