Integrating temporal difference methods and self-organizing neural networks for reinforcement learning with delayed evaluative feedback

Author(s): Lu, N. | Tan, A. | Xiao, D. |

Year: 2008

Citation: IEEE TRANSACTIONS ON NEURAL NETWORKS Volume: 19 Issue: 2 Pages: 230-244

Abstract: This paper presents a neural architecture for learning category nodes encoding mappings across multimodal patterns involving sensory inputs, actions, and rewards. By integrating adaptive resonance theory (ART) and temporal difference (TD) methods, the proposed neural model, called TD fusion architecture for learning, cognition, and navigation (TD-FALCON), enables an autonomous agent to adapt and function in a dynamic environment with immediate as well as delayed evaluative feedback (reinforcement) signals. TD-FALCON learns the value functions of the state-action space estimated through on-policy and off-policy TD learning methods, specifically state-action-reward-state-action (SARSA) and Q-learning. The learned value functions are then used to determine the optimal actions based on an action selection policy. We have developed TD-FALCON systems using various TD learning strategies and compared their performance in terms of task completion, learning speed, as well as time and space efficiency. Experiments based on a minefield navigation task have shown that TD-FALCON systems are able to learn effectively with both immediate and delayed reinforcement and achieve a stable performance in a pace much faster than those of standard gradient-descent-based reinforcement learning systems.

Topics: Machine Learning, Applications: Other, Models: Fuzzy ARTMAP,

PDF download




Cross References


  1. A massively parallel architecture for a self organizing neural pattern recognition machine
    A neural network architecture for the learning of recognition categories is derived. Real-time network dynamics are completely characterized through mathematical analysis and computer simulations. The architecture ... Article Details

  2. Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps
    A new neural network architecture is introduced for incremental supervised learning of recognition categories and multidimensional maps in response to arbitrary sequences of analog or binary input vectors, which may ... Article Details

  3. Adaptive resonance associative map
    This article introduces a neural architecture termed Adaptive Resonance Associative Map (ARAM) that extends unsupervised Adaptive Resonance Theory (ART) systems for rapid, yet stable, heteroassociative learning. ARAM can be ... Article Details

  4. Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system
    A Fuzzy Adaptive Resonance Theory (ART) model capable of rapid stable learning of recognition categories in response to arbitrary sequences of analog or binary input patterns is described. Fuzzy ART incorporates computations ... Article Details

  5. Rule extraction: From neural architecture to symbolic representation
    This paper shows how knowledge, in the form of fuzzy rules, can be derived from a supervised learning neural network called furry ARTMAP. Rule extraction proceeds in two stages: pruning, which simplifies the network ... Article Details