Connectionists: A Six-Unit Network is All You Need to Discover Happiness

[Brian Mingus]

It's a great title! I believe the word that characterizes it is "sardonic."

Yet speaking of not actually solving the stated problem, how is it possible
that connectionists failed so miserably to get AI working?

Check out Gary's other publications:
https://scholar.google.com/citations?user=5Aut7EEAAAAJ&hl=en&oi=sra

He seriously knows what's up. Gary knows how to compute on the edge of
chaos.

Me personally, having been trained in the Randy O'Reilly lab, I watched ten
years of failure to get models working.

And then suddenly, Eureka! It was basically LSTM all along. But didn't
Mozer have BPTT before that, and, guys, seriously, didn't we have the SVD
in the late 1800s? “Oh, but you have to train it in a very special way! You
need dropout and layer-wise semi-supervised pre-training.” I think the
papers published before and since then have shown this to not be true.

"The Unreasonable Effectiveness of Data" by Norvig:
https://research.google.com/pubs/archive/35179.pdf

Um, OK, let's run with that. We are still messing around with MNIST (tiny
data) and surprisingly, those methods tend to generalize. So while what
Norvig is getting at regarding the effectiveness of simple models on big
data is still a good point, why wasn't AI solved long before 1990?
Overclocking CPUs is not a new thing. You had the speed, you had the data,
and 640K of RAM is actually literally enough to get by on.

In fact, the problem is exactly this simple: I recently had the idea to
take the backprop implementation from scikit learn and implement the
following pseudocode to solve mnist:

Generate a random vector the size of an image with numbers of any scale or
distribution (I did not try bignum, but did try very big ints). Train on
this vector. Test on every image in the training set to see if any new
images were correctly classified. If yes, keep the model, if no, revert to
the previous model and generate a new random vector. Repeat until you
master the training set, or get bored. And then observe that you generalize
very well to the test set.

That method is totally brain-dead. It *literally* worked on my first try.
And I think we can do the same thing with random matrix multiplies. In
fact, I have seen O'Reilly train brain models by approximating a
correlation matrix by fiddling with basically a single floating point
round-off error and checking to see if the correlations increased.

And Sutton & Barto recently published that RL is basically just brute
force. "RL" boils down to: when you get lucky, don't be an idiot and forget
what you saw. This turns out to not be pushing the problem back into the
human brain. It's actually that easy. And it's not just for MDP's - POMDPs
feel the same love.

Personally, and I just feel the need to get this on the record, I think
there's a third option, and it's literally The Third Option:
https://en.wikipedia.org/wiki/The_Third_Option

I believe we had AI in the 70s and probably the 50s, 40s and maybe earlier
to a point. Actually though, an "SVD" on punch cards would not actually be
that onerous. Given that I was funded by the CIA (technically IARPA) and
nearly every other defense org that funds academics, I'm wondering how it's
possible that O'Reilly got all that defense funding in the first place
while we seemingly thrashed our weights trying to solve an easy problem.

In case I die tomorrow, let me just put the question out there, as your
diligent and friendly connectionist moderator for over ten years now: Was
"Connectionism" and "PDP" etc. actually a CIA operation? Is DeepMind a
staged rollout of AI that's been backdoored by NIST-level mathematical
geniuses? Is Schmidhuber then a spy as well?

Actually, how many spies are on this list. Raise your hands :) I'm serious.

It's actually an important question. The problem turned out to be so easy,
and connectionists apparently failed so badly, that if we are really just
now getting a handle on it we might actually be looking at a singularity.
On the other hand, it would be amazingly good news if the CIA got, say, 30
years or more ahead of this problem. Which is an excellent reason for me to
ask in and of itself. Are we getting some help here from Uncle Sam or not?

Sincerely,

Brian Mingus

http://linkedin.com/in/brianmingus
https://scholar.google.com/citations?user=T_sFnwoAAAAJ&hl=en

PS: Is anyone looking to hire a strong generalist in the, say, AI Safety
space, or anything fun that doesn't lead to the end of the world? If so hit
me up!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20200108/bedcbf00/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: resume_mingus_ml.pdf
Type: application/pdf
Size: 467696 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20200108/bedcbf00/attachment-0001.pdf>