Connectionists: Rising Stars in AI Symposium at KAUST: Protocol Flaws

[Juyang Weng]

Danko,

I must provide a deeper insight into your comment: "Write down: The
original study reported the performance of A. The true performance is B.
And B < A."

If the truth is that B can report only 10% correct, A can report 99.99%
correct or any higher percentage, if one uses a flowed
protocol---Post-Selections Using Validation Sets (PSUVS) or Post-Selections
Using Test Sets (PSUTS) depending on whether A reports validation
performance or test performance.

I  have proven the following theorem, in a manuscript currently being
evaluated by a journal (an archival version of my IJCNN 2021 paper about
Post-Selections):

Theorem 4 (Post-Selection Illusion): Given any nonzero validation error
rate and nonzero test error rate, the nearest neighbor classifiers with a
confidence threshold in Definition 4 satisfies the validation error rate
and the test error rate, if the computational resources and the time spent
on Post-Selections are not bounded.

With an explosion of the number of false papers, the AI field is in a great
crisis of protocol flaw.

Best regards,
-John

On Wed, Mar 16, 2022 at 5:51 PM Juyang Weng <juyang.weng at gmail.com> wrote:

> Danko,
>
> (A) You wrote "Have a student or a group of students redo the studies (at
> least those for which the code and data are available) and publish the
> correct performance results."
>
> I tried to contact the authors of a few so-called "high profile" papers.
> Many of them did not respond at all.  Some of them even contacted my MSU
> supervisor to press me to be silent.    If some data sets are available at
> Github or elsewhere, something is always missing.  It is always not a click
> of a run to get all the results.  In fact, a click of a run saves their
> time instead of removing something. Did they do a careful job to make sure
> that duplication is too costly?
>
> (B) You wrote: "Write down: The original study reported the performance of
> A. The true
> performance is B. And B < A."
>
> From the theory in my paper in IJCNN, it is much much worse than B < A.
> The Post-Selection technique is not product-credible at all, since it
> requires a customer to cast dice, probably many times.  Saggio et al. of
> Google in Nature is among the few who responded to Nature's request, which
> I appreciate.  The paper mentioned training 10,000 random networks, but the
> authors did not show the performance of these 10,000 networks, only 165
> (luckiest?) of them.
>
> (C) You wrote: "Would you say that this entire issue is a part of the
> general replicability
> crisis in science?"
>
> Not as a scientific historian, I guess that this rampant scale of general
> replicability crisis in AI is probably the greatest in human history if we
> consider the number of AI papers that hide the Post-Selection stages.
> arXiv rejected my reports to Science and Nature about the Protocol flaws
> without stating any reasons.  Is arXiv not peer reviewed as it claimed?
>
> Best regards,
> -John
> ----
> Date: Wed, 16 Mar 2022 06:48:59 +0100
> From: Danko Nikolic <danko.nikolic at gmail.com>
> To: Juyang Weng <juyang.weng at gmail.com>
> Cc: Post Connectionists <connectionists at mailman.srv.cs.cmu.edu>
> Subject: Re: Connectionists: Rising Stars in AI Symposium at KAUST:
>         Protocol        Flaws
> Message-ID:
>         <CANVbDY0sUdYpuVYZHGDijP3o_tmEMc5ddFf7xb=
> qyiZBCHAhnw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Thank you Juyang,
>
> " Please suggest what we should do, as much resource is being wasted."
>
> A suggestion: Have a student or a group of students redo the studies (at
> least those for which the code and data are available) and publish the
> correct performance results.
>
> Write down: The original study reported the performance of A. The true
> performance is B. And B < A.
>
> This is a lot of work. I know.
>
> Would you say that this entire issue is a part of the general replicability
> crisis in science?
>
> Danko
>
> Dr. Danko Nikoli?
> www.danko-nikolic.com
> https://www.linkedin.com/in/danko-nikolic/
>
>
> --
> Juyang (John) Weng
>


-- 
Juyang (John) Weng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220317/1e9cf521/attachment.html>