AdaSwarm: A Novel PSO optimization Method for the Mathematical Equivalence of Error Gradients

by   Rohan Mohapatra, et al.

This paper tackles the age-old question of derivative free optimization in neural networks. This paper introduces AdaSwarm, a novel derivative-free optimizer to have similar or better performance to Adam but without "gradients". To support the AdaSwarm, a novel Particle Swarm Optimization Exponentially weighted Momentum PSO (EM-PSO), a derivative-free optimizer, is also proposed which tackles constrained and unconstrained single objective optimization problems and looks at applying the proposed momentum particle swarm optimization on benchmark test functions, engineering optimization problems and habitability scores for exoplanets which show speed and convergence of the technique. The EM-PSO is extended by approximating the gradient of a function at any point using the parameters of the particle swarm optimization. This is a novel technique to simulate gradient descent, an extremely popular method in the back-propagation algorithm, using the approximated gradients from the particle swarm optimization parameters. Mathematical proofs of gradient approximation by EM-PSO, thereby bypassing the gradient computation, are presented. The AdaSwarm is compared with various optimizers and the theory and algorithmic performance are supported by promising results.


A Swarm Variant for the Schrödinger Solver

This paper introduces application of the Exponentially Averaged Momentum...

Fairly Constricted Particle Swarm Optimization

We have adapted the use of exponentially averaged momentum in PSO to mul...

A hybrid swarm-based algorithm for single-objective optimization problems involving high-cost analyses

In many technical fields, single-objective optimization procedures in co...

Hamiltonian Monte Carlo Particle Swarm Optimizer

We introduce the Hamiltonian Monte Carlo Particle Swarm Optimizer (HMC-P...

A Bi-population Particle Swarm Optimizer for Learning Automata based Slow Intelligent System

Particle Swarm Optimization (PSO) is an Evolutionary Algorithm (EA) that...

Design of multifunctional metamaterials using optimization

This paper explores the use of optimization to design multifunctional me...

Quadratically constrained quadratic programming for classification using particle swarms and applications

Particle swarm optimization is used in several combinatorial optimizatio...

Please sign up or login with your details

Forgot password? Click here to reset