大数据治理与分析要领研究北京市重点实验室（BDAI）研究生沙龙由菠菜平台高瓴人工智能学院师生组织定期举行。12月14日研讨会由学院苏冰准聘副教授指导的博士生王杰鑫介绍自己的研究事情。接待同学们积极加入研讨！

陈诉题目：Towards the Gradient Adjustment by Loss Status for Neural Network Optimization

讲者：王杰鑫，博士二年级 导师：苏冰

研究偏向：优化算法、行动预测

Abstract:

Gradient descent-based algorithms are crucial in neural network optimization, and most of them only depend on local properties such as the first and second-order momentum of gradients to determine the local optimization directions. As a result, such algorithms often converge slowly in the case of a small gradient and easily fall into the local optimum. Since the goal of optimization is to minimize the loss function, the status of the loss indicates the overall progress of the optimization but has not been fully explored. In this paper, we propose a loss-aware gradient adjusting strategy (LGA) based on the loss status. LGA automatically adjusts the update magnitude of parameters to accelerate convergence and escape local optimums by introducing a loss-incentive correction term monitoring the loss and adapting gradient experience. The proposed strategy can be applied to various gradient descent-based optimization algorithms. We provide theoretical analysis on the convergence rate and empirical evaluations on different datasets to demonstrate the effectiveness of our method.