cs224n RNN和语言模型 （The Vanishing Gradient Issue）

cs224n RNN和语言模型 （The Vanishing Gradient Issue） 

cs224n RNN和语言模型中提及梯度消失，这里展示在两个隐藏层的简单神经网络中分别使用sigmoid、relu激活函数的区别，这是Andrej Karpathy为cs231n 一个小型网络演示构建的，代码如下：

# -*- coding: utf-8 -*-
# Setup
import numpy as np
import matplotlib.pyplot as plt

#%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading extenrnal modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
#%load_ext autoreload
#%autoreload 2


#generate random data -- not linearly separable 
np.random.seed(0)
N = 100 # number of points per class
D = 2 # dimensionality
K = 3 # number of classes
X = np.zeros((N*K,D))
num_train_examples = X.shape[0]
y = np.zeros(N*K, dtype='uint8')
for j in  ran