Deep Learning - How Recurrent Neural Networks (RNNs) work: A Gentle Introduction

Introduction

In contrast to a classic feedforward network, a Recurrent Neural Network (RNN) allows backward connections. An RNN is particularly suitable for processing time series data and is able to take into account the time dependency of data. This enables an RNN to have a kind of short-term memory. The core component of an RNN is the cell. In this tutorial we want to take a closer look at an RNN cell and find out how it works.

RNN Cell

An RNN cell represents a hidden layer in an RNN. The cell receives an input and generates an output. The special thing about the cell is that it has a feedback loop, which means that output goes into the cell as input. This is known as hidden state.

The abbreviations have the following meaning:

i^t: Input
h^t: Hidden state