Multi-stage ORGaNICs: A Hierarchical Recurrent Neural Circuit Model

The visual cortex is known for its modular and hierarchical structure, with feedback connections between brain areas that modulate functions like communication and normalization. To account for these reciprocal feedback connections, we introduce the Multi-stage ORGaNICs (Optimized Recurrent Generative Networks with Inhibitory Control) model, a hierarchical recurrent circuit model with feedback from higher cortical areas (e.g., V4) to lower cortical areas (e.g., V1).

Model Architecture

The architecture of the Multi-stage ORGaNICs model is illustrated in the figure below:

Each region in the model consists of three neural populations:

Principal excitatory neurons $\mathbf{y}$
Modulatory inhibitory neurons $\mathbf{a}$
Modulatory excitatory neurons $\mathbf{u}$

The subscripts 1 and 2 denote the layer number or hierarchy in the model.

Dynamics of Principal Excitatory Neurons

The dynamics of the principal excitatory neuron $\mathbf{y}_1$ in the first layer (V1) is governed by the following equation:

$$ \begin{align*} \tau_{y_1}\frac{\mathrm{d}\mathbf{y}_1}{\mathrm{d} t } = -\mathbf{y}_1 + \left ( \frac{\beta_1^{+}}{1+\beta_1^{+}} \right ) \cdot \mathbf{z}_1 + \left( \frac{1}{1+\mathbf{a}_1^{+} } \right ) \left( \mathbf{W}_{11} \sqrt{ \mathbf{y}_1^+} + \left (\frac{\gamma_4^{+}}{1+\gamma_4^{+}}\right ) \cdot \mathbf{W}_{41} \sqrt{ \mathbf{y}_4^+} \right) \end{align*} $$

where:

$\mathbf{y}_1$ and $\mathbf{y}_4$ are the membrane potentials of neurons in V1 and V4, respectively.
$\mathbf{z}_1$ is the input drive from the preceding cortical area (LGN).
$\beta$ and $\gamma$ are tuning parameters that modulate the input and feedback drives, respectively.
$\mathbf{W}_{11}$ and $\mathbf{W}_{41}$ are the recurrent and feedback weight matrices, respectively.
$\mathbf{a}_1$ is the modulatory inhibitory neuron population that dynamically normalizes the recurrent and feedback drives.

Normalization and Modulatory Neurons

When $\beta$ and $\gamma$ are set to unity and $\mathbf{W}_{11}$ is an identity matrix, the $\mathbf{y}_1$ neurons in V1 follow the normalization equation exactly:

$$ \lfloor \mathbf{y}_1 \rfloor^2 = \frac{\lfloor \mathbf{z}_1 \rfloor^2}{\sigma^2 + \mathbf{N}_{1} \lfloor \mathbf{z}_1 \rfloor^2} $$

This normalization is achieved dynamically by the modulatory neurons $\mathbf{a}_1$ and $\mathbf{u}_1$, whose dynamics are governed by:

$$ \begin{align*} \tau_{a_1}\frac{\mathrm{d}\mathbf{a}_1}{\mathrm{d}t} &= -\mathbf{a}_1 + + \left (\frac{g_1^{+}}{1+g_1^{+}}\right ) \frac{\mathbf{W}_{41} \sqrt{ \mathbf{y}_4^+}}{\sqrt{ \mathbf{y}_1^+}} + (1 + \mathbf{a}_1)\circ \mathbf{u}_1^{+} + \alpha_1 \frac{\mathrm{d}\mathbf{u}_1}{\mathrm{d}t} \\ \tau_{u_1}\frac{\mathrm{d}\mathbf{u}_1}{\mathrm{d}t} &= -\mathbf{u}_1 + \left(\frac{b_1^{+}\sigma}{1+b_1^{+}}\right)^2 + \mathbf{N}_{1} \left ( \mathbf{y}_1 ^{+} \circ \mathbf{u}_1^{+2} \right ) \end{align*} $$

Both modulatory neurons have an input term and a recurrent term. An increase in $\mathbf{y}_1$ leads to an increase in $\mathbf{u}_1$, which in turn increases $\mathbf{a}_1$. The increased $\mathbf{a}_1$ then decreases $\mathbf{y}_1$, achieving normalization.

Conclusion

The Multi-stage ORGaNICs model provides a powerful framework for studying the modular and hierarchical structure of the visual cortex, incorporating feedback connections and dynamic normalization. By capturing the complex interactions between cortical areas, this model offers insights into the mechanisms underlying visual processing and communication in the brain.