Connectionists: Conditions for convergence of RMSProp, even in non-convex settings

[Danilo Mandic]

Dear Connectionists,

You may be interested in the following article which has just appeaared in Cognitive Neurodynamics.  The analysis employs the Holder inequality to establish the hyperparameter range within RMSProp which guarantees its convergence even in non-convex setting.

https://link.springer.com/article/10.1007/s11571-022-09845-8

A similar analysis for the Adam optimizer is underway.

Regards,

Danilo