<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Since a Turing machine cannot "<span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif;font-size:14px;background-color:rgb(255, 255, 255);display:inline !important">not
explain the human problem solving", super-Turing computation should be considered. There are two recent papers exploring computation consistent with super-Turing operation: <a href="https://journals.aps.org/prresearch/abstract/10.1103/PhysRevResearch.3.013120" id="LPlnkOWALinkPreview">https://journals.aps.org/prresearch/abstract/10.1103/PhysRevResearch.3.013120</a> and <a href="https://link.springer.com/chapter/10.1007/978-3-030-86380-7_38" id="LPlnkOWALinkPreview_1">https://link.springer.com/chapter/10.1007/978-3-030-86380-7_38</a> . </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif;font-size:14px;background-color:rgb(255, 255, 255);display:inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif;font-size:14px;background-color:rgb(255, 255, 255);display:inline !important">The APS paper is open source.
The studies include noise which the brain also has. They simulate a quantal nature. Neuron excitation is also quantal.</span></div>
<br>
<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview _EReadonly_1">
<div id="LPBorder_GTaHR0cHM6Ly9qb3VybmFscy5hcHMub3JnL3BycmVzZWFyY2gvYWJzdHJhY3QvMTAuMTEwMy9QaHlzUmV2UmVzZWFyY2guMy4wMTMxMjA." class="LPBorder941076" style="width: 100%; margin-top: 16px; margin-bottom: 16px; position: relative; max-width: 800px; min-width: 424px;">
<table id="LPContainer941076" role="presentation" style="padding: 12px 36px 12px 12px; width: 100%; border-width: 1px; border-style: solid; border-color: rgb(200, 200, 200); border-radius: 2px;">
<tbody>
<tr valign="top" style="border-spacing: 0px;">
<td>
<div id="LPImageContainer941076" style="position: relative; margin-right: 12px; height: 160px; overflow: hidden;">
<a target="_blank" id="LPImageAnchor941076" href="https://journals.aps.org/prresearch/abstract/10.1103/PhysRevResearch.3.013120"><img id="LPThumbnailImageId941076" alt="" height="160" style="display: block;" width="160" src="https://cdn.journals.aps.org/journals/PRRESEARCH/key_images/10.1103/PhysRevResearch.3.013120.png"></a></div>
</td>
<td style="width: 100%;">
<div id="LPTitle941076" style="font-size: 21px; font-weight: 300; margin-right: 8px; font-family: wf_segoe-ui_light, "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px;">
<a target="_blank" id="LPUrlAnchor941076" href="https://journals.aps.org/prresearch/abstract/10.1103/PhysRevResearch.3.013120" style="text-decoration: none; color: var(--themePrimary);">Noise optimizes super-Turing computation in recurrent neural networks</a></div>
<div id="LPDescription941076" style="font-size: 14px; max-height: 100px; color: rgb(102, 102, 102); font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; margin-bottom: 12px; margin-right: 8px; overflow: hidden;">
The authors use noise to restore chaotic behavior and show consistency with super-Turing theory and operation in neural networks.</div>
<div id="LPMetadata941076" style="font-size: 14px; font-weight: 400; color: rgb(166, 166, 166); font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;">
journals.aps.org</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<br class="Apple-interchange-newline">
<table style="font-size:14px;background-color:rgb(255, 255, 255);padding:12px 36px 12px 12px;width:434.062px;border-width:1px;border-style:solid;border-color:rgb(200, 200, 200);border-radius:2px">
<tbody>
<tr valign="top" style="border-spacing:0px">
<td>
<div style="margin:0px 12px 0px 0px;height:160px;overflow:hidden"><a target="_blank" href="https://link.springer.com/chapter/10.1007/978-3-030-86380-7_38" style="margin:0px"><img alt="" height="160" width="105" style="margin:0px;display:block" src="https://static-content.springer.com/cover/book/978-3-030-86380-7.jpg"></a></div>
</td>
<td style="width:123.262px">
<div style="font-weight:300;font-size:21px;font-family:wf_segoe-ui_light, "Segoe UI Light", "Segoe WP Light", "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;margin:0px 8px 12px 0px">
<a target="_blank" href="https://link.springer.com/chapter/10.1007/978-3-030-86380-7_38" style="margin:0px;color:var(--themePrimary)">Noise Quality and Super-Turing Computation in Recurrent Neural Networks - SpringerLink</a></div>
<div style="font-family:wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;margin:0px 8px 12px 0px;color:rgb(102, 102, 102);max-height:100px;overflow:hidden">
Figures 3 and 4 show the number of output sequences which remain consistent with chaos (y-axis) vs. \(log_2\) of the repeat length of the noise (x-axis) for the logistic and Hénon maps digital RNNs for both optimum and near-optimum noise magnitudes and the
two noise types (MATLAB vs. LFSR). As mentioned in Sec. 2, noise magnitudes of four times LSB is regarded as optimum for the Hénon map ...</div>
<div style="font-family:wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif;margin:0px;color:rgb(166, 166, 166)">
link.springer.com</div>
</td>
</tr>
</tbody>
</table>
<div class="BteeS" tabindex="0" title="Remove link preview" style="font-size:14px;margin:0px;cursor:pointer;height:32px;pointer-events:all;top:4px;text-align:center;right:4px;background-color:rgb(255, 255, 255)">
<i data-icon-name="Cancel" class="w6wS4 root-364" style="display:block;font-style:normal;font-weight:300;speak:none;font-family:controlIcons;color:var(--neutralSecondary);font-size:17px;line-height:32px;pointer-events:none;margin:auto"></i></div>
<br class="Apple-interchange-newline">
<br>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
</div>
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<div class="BodyFragment"><font size="2">
<div class="PlainText">Emmett <span style="color: rgb(237, 92, 87);">Redd</span> Ph.D. mailto:EmmettRedd@missouristate.edu<br>
Professor (417)836-5221<br>
Department of Physics, Astronomy, and Materials Science<br>
Missouri State University Fax (417)836-6226<br>
901 SOUTH NATIONAL<br>
SPRINGFIELD, MO 65897 USA Dept (417)836-5131<br>
<br>
<p style="margin-top:0pt; margin-bottom:0pt; text-align:left; direction:ltr; unicode-bidi:embed; vertical-align:baseline">
<span style="font-size:10pt; font-family:"Times New Roman",Times,serif; color:black">Perfect logic and faultless deduction make a pleasant theoretical structure, but it may be right or wrong; the experimenter is the only one to decide, and he is always right.
–</span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt">Lé</span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt">on</span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt"> </span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt">Brillouin</span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt">,
</span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt; font-style:italic">Scientific Uncertainty, and Information</span><span style="font-family:"Times New Roman",Times,serif; font-size:10pt"> (1964).</span></p>
<br>
</div>
</font></div>
</div>
</div>
</div>
</div>
<div id="signature_bookmark"></div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Connectionists <connectionists-bounces@mailman.srv.cs.cmu.edu> on behalf of Andrzej Wichert <andreas.wichert@tecnico.ulisboa.pt><br>
<b>Sent:</b> Thursday, July 21, 2022 4:07 AM<br>
<b>To:</b> Schmidhuber Juergen <juergen@idsia.ch><br>
<b>Cc:</b> connectionists@cs.cmu.edu <connectionists@cs.cmu.edu><br>
<b>Subject:</b> Re: Connectionists: weight guessing quickly solves n-bit parity</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:center">
<span style="color:#5E0009; font-weight:bold">CAUTION: </span>External Sender</div>
<br>
<div>Dear Juergen,
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">Symbols do not, by themselves, represent any utilizable knowledge, they cannot be used for a definition of similarity criteria between themselves. The use of symbols in algorithms which
imitate human intelligent behavior led to the famous physical symbol system hypothesis \index{physical symbol system hypothesis} by Newell and Simon (1976) ``The necessary and sufficient condition for a physical system to exhibit intelligence is that it
be a physical symbol system.'' Symbols are not present in the world; they are the constructs of a human mind and simplify the process of representation used in communication and problem solving. </div>
</div>
<div class=""><br class="">
</div>
<div class="elementToProof">A Turing machine can simulate any algorithm, the same with RNN, but they do not explain the human problem solving.</div>
<div class=""><br class="x_webkit-block-placeholder">
</div>
<div class="">There is a difference, for example human problem solving can be described by production systems. The most successful model is the SOAR architecture. Such systems were very successful, they learn and can give you an explanation for their doing.</div>
<div class=""><br class="">
</div>
<div class="">
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">in August 1998 Dave Touretzky asked on the connectionistic e-mailing list:</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">``Is connectionist symbol processing dead?’'</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal"><br class="">
</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal"><br class="">
</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">From <a href="mailto:ml-connectionists-request$@$mlist-1.sp.cs.cmu.edu" data-auth="NotApplicable" class="">
ml-connectionists-request$@$mlist-1.sp.cs.cmu.edu</a> Tue Aug 11 17:35:10 1998</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">From: Dave$\_$<a href="mailto:Touretzky$@$cs.cmu.edu" data-auth="NotApplicable" class="">Touretzky$@$cs.cmu.edu</a></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">To: <a href="mailto:connectionists$@$cs.cmu.edu" data-auth="NotApplicable" class="">
connectionists$@$cs.cmu.edu</a></div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">Subject: Connectionist symbol processing: any progress?</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">Date: Tue, 11 Aug 1998 $03:34:27$ -0400</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal"><br class="">
</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">I'd like to start a debate on the current state of connectionist symbol</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">processing? Is it dead? Or does progress continue? ... People had gotten</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">some interesting effects with localist networks, by doing spreading activation</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">and a simple form of constraint satisfaction.... This approach does not create</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">new structure on the fly, or deal with structured representations or variable</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">binding. Those localist networks that did attempt to implement variable binding</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">did so in a discrete, symbolic way that did not advance the parallel constraint</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">satisfaction/heuristic reasoning agenda of earlier spreading activation</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">research. ... So I concluded that connectionist symbol processing had reached</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">a plateau, and further progress would have to await some revolutionary new</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">insight about representations. ... The problems of structured representations</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">and variable binding have remained unsolved. No one is trying to build</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">distributed connectionist reasoning systems any more, like the connectionist</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal">production system I built with Geoff Hinton...</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal"><br class="">
</div>
</div>
<div class="" style="margin:0px; font-stretch:normal; line-height:normal"><br class="">
</div>
<div class="">24 years past and not much progress was done. It seems that the progress is only related to pure brute force of computers, but not much insight beside a wishful thinking. The whole DL movements stops the progress in understanding how the brain
works. We need some new fresh ideas beside error minimization.</div>
<div class=""><br class="">
</div>
<div class="">I think one of the main problems is the publish or perish altitude, the famous Google impact factor. One does not care what some one is doing, one just checks his Google impact factor.. This is like the saying, eat more shit, one million flies
cannot be wrong. </div>
<div class="">Like some mathematician said, computer science is not science at all, but it force us to follow its ideas.</div>
<div class=""><br class="">
</div>
<div class="">Andreas</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="x_Apple-interchange-newline">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">--------------------------------------------------------------------------------------------------</span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">Prof.
Auxiliar Andreas Wichert </span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none"><a href="http://web.tecnico.ulisboa.pt/andreas.wichert/" data-auth="NotApplicable" class="">http://web.tecnico.ulisboa.pt/andreas.wichert/</a></span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">-</span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none"><a href="https://www.amazon.com/author/andreaswichert" data-auth="NotApplicable" class="">https://www.amazon.com/author/andreaswichert</a></span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">Instituto
Superior Técnico - Universidade de Lisboa</span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">Campus
IST-Taguspark </span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">Avenida
Professor Cavaco Silva Phone: +351 214233231</span><br class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<span class="" style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline!important; float:none">2744-016
Porto Salvo, Portugal</span> </div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 20 Jul 2022, at 15:55, Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" data-auth="NotApplicable" class="">juergen@idsia.ch</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class=""><br class="">
I have never understood the difference between "symbolic" and "sub-symbolic" reasoning. A recurrent neural net (RNN) is a general computer that can do both. n-bit parity for arbitrary n can be solved by a tiny RNN with few connections, sequentially reading
bits one by one [1]. The best learning algorithm is NOT gradient descent. Instead keep randomly initializing the RNN weights between -100 and +100 until the RNN solves parity for a few training examples of various large sizes n (this will take just 1000 trials
or so). Now the RNN will probably generalize to ANY n. BTW, try that with a Transformer - it will never generalize like that.<br class="">
<br class="">
[1] J. Schmidhuber and S. Hochreiter. Guessing can outperform many long time lag algorithms. Technical Note IDSIA-19-96, IDSIA, 1996<br class="">
<br class="">
Jürgen<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
On 19 Jul 2022, at 23:08, Danko Nikolic <<a href="mailto:danko.nikolic@gmail.com" data-auth="NotApplicable" class="">danko.nikolic@gmail.com</a>> wrote:<br class="">
<br class="">
Dear Barak,<br class="">
<br class="">
Thank you for the pointers. I have to read the papers. I need to understand then why, if parity is so easy to learn, my deep learning models had such a hard time that it led to an exponential growth in the number of needed parameters with each additional bit
added to the input. Strange.<br class="">
<br class="">
I will report back.<br class="">
<br class="">
Best<br class="">
<br class="">
Danko<br class="">
<br class="">
Dr. Danko Nikolić<br class="">
<a href="http://www.danko-nikolic.com" data-auth="NotApplicable" class="">www.danko-nikolic.com</a><br class="">
https://www.linkedin.com/in/danko-nikolic/<br class="">
-- I wonder, how is the brain able to generate insight? --<br class="">
<br class="">
<br class="">
On Mon, Jul 18, 2022 at 1:12 PM Barak A. Pearlmutter <barak@pearlmutter.net> wrote:<br class="">
On Mon, 18 Jul 2022 at 08:28, Danko Nikolic <danko.nikolic@gmail.com> wrote:<br class="">
In short, learning mechanisms cannot discover generalized XOR functions with simple connectivity -- only with complex connectivity. This problem results in exponential growth of needed resources as the number of bits in the generalized XOR increases.<br class="">
<br class="">
Assuming that "generalized XOR" means parity, this must rely on some<br class="">
unusual definitions which you should probably state in order to avoid<br class="">
confusion.<br class="">
<br class="">
Parity is a poster boy for an *easy* function to learn, albeit a<br class="">
nonlinear one. This is because in the (boolean) Fourier domain its<br class="">
spectrum consists of a single nonzero coefficient, and functions that<br class="">
are sparse in that domain are very easy to learn. See N. Linial, Y.<br class="">
Mansour, and N. Nisan, "Constant depth circuits, Fourier Transform and<br class="">
learnability", FOCS 1989, or Mansour, Y. (1994). Learning Boolean<br class="">
Functions via the Fourier Transform. Theoretical Advances in Neural<br class="">
Computation and Learning, 391–424. doi:10.1007/978-1-4615-2696-4_11<br class="">
<br class="">
--Barak Pearlmutter<br class="">
<br class="">
<br class="">
<br class="">
On 18 Jul 2022, at 18:01, Gary Marcus <gary.marcus@nyu.edu> wrote:<br class="">
<br class="">
sure, but a person can learn the idea for n-bits from a few examples with a small number of bits, generalizing it to large values of n. most current systems learn it for a certain number of bits and don’t generalize beyond that number of bits.<br class="">
<br class="">
On Jul 18, 2022, at 7:17 AM, Barak A. Pearlmutter <barak@pearlmutter.net> wrote:<br class="">
<br class="">
<br class="">
<br class="">
On Mon, 18 Jul 2022 at 14:43, Danko Nikolic <danko.nikolic@gmail.com> wrote:<br class="">
<image.png><br class="">
<br class="">
It is a hard problem to learn for a connectionist network.<br class="">
<br class="">
We don't need to invent new terminology, like "inverters problem" or "generalized xor." This is parity. Four (4) bit parity.<br class="">
<br class="">
https://en.wikipedia.org/wiki/Parity_function<br class="">
<br class="">
Parity is *not* a hard function to learn. Even for a connectionist network.<br class="">
<br class="">
It is an interesting function for historic reasons (n-bit parity cannot be loaded by a k-th order perceptron, for k<n, although there are loopholes if a random bits are available or if you are allowed to only almost load it) and because it's an interesting
function for many mathematical constructions. See the above wikipedia page for some details. But it is not super difficult to learn.<br class="">
<br class="">
--Barak Pearlmutter.<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
On 18 Jul 2022, at 00:58, gary@ucsd.edu <gary@eng.ucsd.edu> wrote:<br class="">
<br class="">
Sorry, I can't let this go by:<br class="">
And it is good so because generalize XOR scales worse than power law. It scales exponentially! This a more agressive form of explosion than power law.
<br class="">
I'm not sure exactly what you mean by this, but a single-hidden layer network with N inputs and N hidden units can solve N-bit parity. Each unit has an increasing threshold, so, one turns on if there is one unit on in the input, and then turns on the output
with a weight of +1. If two units are on in the input, then a second unit comes on and cancels the activation of the first unit via a weight of -1. Etc.<br class="">
<br class="">
g.<br class="">
<br class="">
<br class="">
On Sat, Jul 16, 2022 at 12:03 AM Danko Nikolic <danko.nikolic@gmail.com> wrote:<br class="">
Dear Thomas,<br class="">
<br class="">
Thank you for reading the paper and for the comments.<br class="">
<br class="">
I cite: "In my experience, supervised classification scales linearly in the number of classes."<br class="">
This would be good to quantify as a plot. Maybe a research paper would be a good idea. The reason is that it seems that everyone else who tried to quantify that relation found a power law. At this point, it would be surprising to find a linear relationship.
And it would probably make a well read paper.<br class="">
<br class="">
But please do not forget that my argument states that even a linear relationship is not good enough to match bilogical brains. We need something more similar to a power law with exponent zero when it comes to the model size i.e., a constant number of parameters
in the model. And we need linear relationship when it comes to learning time: Each newly learned object should needs about as much of learning effort as was needed for each previous object.
<br class="">
<br class="">
I cite: "The real world is not dominated by generalized XOR problems."<br class="">
Agreed. And it is good so because generalize XOR scales worse than power law. It scales exponentially! This a more agressive form of explosion than power law.
<br class="">
Importantly, a generalized AND operation also scales exponentially (with a smaller exponent, though). I guess we would agree that the real world probably encouners a lot of AND problems. The only logical operaiton that could be learned with a linear increase
in the number of parameters was a generalized OR. Finally, I foiund that a mixure of AND and OR resulted in a power law-like scaling of the number of parameters. So, a mixture of AND and OR seemed to scale as good (or as bad) as the real world. I have put
this information into Supplementary Materials.<br class="">
<br class="">
The conclusion that I derived from those analyses is: connectionism is not sustainable to reach human (or animal) levels of intelligence. Therefore, I hunted for an alternative pradigm.<br class="">
<br class="">
Greetings,<br class="">
<br class="">
Danko<br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<div style="background-color:#FFEB9C; width:100%; border-style:solid; border-color:#9C6500; border-width:1pt; padding:2pt; font-size:10pt; line-height:12pt; font-family:'Calibri'; color:Black; text-align:center">
<span>This message originated outside Missouri State University. Please use caution when opening attachments, clicking links, or replying.</span></div>
<br>
</div>
</div>
</body>
</html>