Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

11/12/2017
by   Takuya Akiba, et al.
0

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset