<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#ecca99">
<p><font size="+1"><font face="monospace">It would have to be updated with with DL-RNN or LSTMs..</font></font></p>
<p><font size="+1"><font face="monospace">S</font></font><br>
</p>
<div class="moz-cite-prefix">On 6/13/22 9:13 AM, Gary Marcus wrote:<br>
</div>
<blockquote type="cite" cite="mid:9D553D9E-CA9A-43E6-840F-0DD05F3C4D9E@nyu.edu">
<div dir="ltr"> I do remember the work :) Just generally Transformers seem more effective; a careful comparison between Y&P, Transformers, and your RNN approach, looking at generalization to novel words, would indeed be interesting.</div>
<div dir="ltr">Cheers, </div>
<div dir="ltr">Gary </div>
<div dir="ltr"><br>
<blockquote type="cite">On Jun 13, 2022, at 06:09, <a class="moz-txt-link-abbreviated" href="mailto:jose@rubic.rutgers.edu">
jose@rubic.rutgers.edu</a> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<p><font size="+1"><font face="monospace">I was thinking more like an RNN similar to work we had done in the 2000s.. on syntax.</font></font></p>
<p><font size="+1"><font face="monospace">Stephen José Hanson, Michiro Negishi; On the Emergence of Rules in Neural Networks. Neural Comput 2002; 14 (9): 2245–2268. doi:
<a class="moz-txt-link-freetext" href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__https%3A%2F%2Fdoi.org%2F10.1162%2F089976602320264079__%3B!!BhJSzQqDqA!WCsRlT1zpBKD3ai8Ov_I79iH_HCdTlAMymGIe2ZsIdTnfZawzlMQNGZWisMjmcLBgH6SbBUZ6rtr_exEspS4Igo%24&data=05%7C01%7Cstephen.jose.hanson%40rutgers.edu%7Cdf1d37eb5a494e8cca8108da4d4261ba%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637907244767470752%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DiJWII1n%2F24jAzC%2B47UpHF1%2FdhfCHKRia%2BTdl2jZsd0%3D&reserved=0" originalsrc="https://urldefense.com/v3/__https://doi.org/10.1162/089976602320264079__;!!BhJSzQqDqA!WCsRlT1zpBKD3ai8Ov_I79iH_HCdTlAMymGIe2ZsIdTnfZawzlMQNGZWisMjmcLBgH6SbBUZ6rtr_exEspS4Igo$" shash="h8LTrBeN3L9aMtbMhXt+F5dEtqCN/JoOMDNy0/msHUeHkLYL+EyG1P2YT/ka9baIPAqz6VJTsn5Xq1DWS/706wjxDetPDlFMSMDA0gKClRVkeSKIgWhLRyIobAH+I9HkNdGwIyltUHCZ6OC3XTHb7ML948ZkJd6PL1oHBhVpjAs=" moz-do-not-send="true">
https://doi.org/10.1162/089976602320264079</a></font></font></p>
<p><font size="+1"><font face="monospace">Abstract<br>
A simple associationist neural network learns to factor abstract rules (i.e., grammars) from sequences of arbitrary input symbols by inventing abstract representations that accommodate unseen symbol sets as well as unseen but similar grammars. The neural network
is shown to have the ability to transfer grammatical knowledge to both new symbol vocabularies and new grammars. Analysis of the state-space shows that the network learns generalized abstract structures of the input and is not simply memorizing the input strings.
These representations are context sensitive, hierarchical, and based on the state variable of the finite-state machines that the neural network has learned. Generalization to new symbol sets or grammars arises from the spatial nature of the internal representations
used by the network, allowing new symbol sets to be encoded close to symbol sets that have already been learned in the hidden unit space of the network. The results are counter to the arguments that learning algorithms based on weight adaptation after each
exemplar presentation (such as the long term potentiation found in the mammalian nervous system) cannot in principle extract symbolic knowledge from positive examples as prescribed by prevailing human linguistic theory and evolutionary psychology.<br>
</font></font><br>
</p>
<div class="moz-cite-prefix">On 6/13/22 8:55 AM, Gary Marcus wrote:<br>
</div>
<blockquote type="cite" cite="mid:60B7CD76-2DCE-4DB9-9ECD-96D046916BB8@nyu.edu">
<div dir="ltr">– agree with Steve this is an interesting paper, and replicating it with a neural net would be interesting; cc’ing Steve Piantosi. </div>
<div dir="ltr">— why not use a Transformer, though?</div>
<div dir="ltr">- it is however importantly missing semantics. (Steve P. tells me there is some related work that is worth looking into). Y&P speaks to an old tradition of formal language work by Gold and others that is quite popular but IMHO misguided, because
it focuses purely on syntax rather than semantics. Gold’s work definitely motivates learnability but I have never taken it to seriously as a real model of language</div>
<div dir="ltr">- doing what Y&P try to do with a rich artificial language that is focused around syntax-semantic mappings could be very interesting</div>
<div dir="ltr">- on a somewhat but not entirely analogous note, i think that the next step in vision is really scene understanding. We have techniques for doing object labeling reasonably well, but still struggle wit parts and wholes are important, and with
relations more generally, which is to say we need the semantics of scenes. is the chair on the floor, or floating in the air? is it supporting the pillow? etc. is the hand a part of the body? is the glove a part of the body? etc</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Best,</div>
<div dir="ltr">Gary</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jun 13, 2022, at 05:18, <a class="moz-txt-link-abbreviated" href="mailto:jose@rubic.rutgers.edu" moz-do-not-send="true">
jose@rubic.rutgers.edu</a> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<p><font size="+1">Again, I think a relevant project here would be to attempt to replicate with DL-rnn, Yang and Piatiadosi's PNAS language learning system--which is a completely symbolic-- and very general over the Chomsky-Miller grammer classes. Let me
know, happy to collaborate on something like this.<br>
</font></p>
<p><font size="+1">Best</font></p>
<p><font size="+1">Steve<br>
</font></p>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</blockquote>
<div class="moz-signature">-- <br>
<img src="cid:part3.207D6B48.20BF6C56@rutgers.edu" border="0"></div>
</body>
</html>