Simple pictures, tough problems.

[Neza van der Leeuw]

Hello,
 
I am currently working on the "grounding problem", in the sense that I try to
derive meaning of language from pictures coupled with sentences (my own version
of the Berkeley ICSI L0 project: Miniature Language Acquisition; A touchstone
for cognitive science. TR-90-009). Right now I am working with a self-organising
neural net (kind of Kohonen), having bitmaps and strings as input. These bitmaps
represent circles, squares and triangles in a bitwise manner.
 
The problem is that I want the architecture to generalize. This means that it
should group small and large circles, rather than circles of size X with squa-
res of size nearly X. In the current implementation, coding is done by repre-
senting each line of the picture by a random vector. But when doing this, the
net generalizes not on second-order properties like form, but on first-order
properties (roughly: where the black spots in the picture reside). Thus,
circles are seen similar to squares when their sizes "match".
 
One could avoid this problem by choosing a representation in language-like
propositions, but this seems to me to be the "solving the problem by doing it
yourself" approach. Some represenational mechanism should provide the answer
to my problem, but I want it to be "picture-like" instead of "language-like".
Otherwise I bounce back into my favourite "grounding problem" again.
 
Does anyone know of any work that has been done in this field? Any hints, refe-
rences or other useful messages are very welcome. As I am more a computer lin-
guist than a grafical scientist my knowledge of the whole thing is rather
limited.
 
If some useful stuff comes out, I will write a nice report and send it to the
list, if others are interested in the matter as well.
 
Many thanks, and don't get overworked during these "holiday months" (whatever
that may be).
 
Neza van der Leeuw
Dept. of Computational Linguistics
Faculty of Arts
University of Amsterdam
Spuistraat 134
1012 VB Amsterdam
The Netherlands