Catastrophic Forgetting

Catastrophic forgetting refers to the phenomenon where a neural network loses previously acquired knowledge upon learning new information. This occurs primarily in traditional neural network architectures that are trained sequentially on different tasks without retaining the knowledge from earlier tasks. Mathematically, this can be expressed as a significant increase in the loss function L for previously learned tasks when the model is fine-tuned on new tasks, leading to a high generalization error on earlier data. Techniques such as Elastic Weight Consolidation (EWC) and progressive neural networks have been proposed to mitigate this issue by preserving important weights or creating separate pathways for new tasks. Catastrophic forgetting is a critical challenge in the field of continual learning and relates to the broader concept of lifelong learning, where models are expected to adapt to new information without losing prior knowledge.