I have designed a 2 layered deep neural network for a classifier with 2 units in the hidden layer. I use linear activation functions with a sigmoid at the final layer. I use a data visualization tool and see that the decision boundary is in the shape of a sine curve. I have tried to train with 200 data points with known class labels and see that the training error is too high. What do I do ?

1. Increase number of units in the hidden layer
2. Increase number of hidden layers
3.  Increase data set size

4. Change activation function to tanh

5. Try all of the above

The answer is d. When I use a linear activation function, the deep neural network is realizing a linear combination of linear  functions which leads to modeling only a linear function. Increasing  the depth or number of hidden units does not help. The network is not complex enough to learn a sine function and is underfitting the data. Change the linear function to tanh to realize a non-linear decision boundary helps.