Intro to Deep Learning

This notebook is a basic introduction to deep learning. The goal is not to use a high-level library immediately, but to see the core pieces clearly:

1. What deep learning does: learn an unknown function.
2. What a neural network contains: parameters, linear transformations, activation functions, and layers.
3. What training means: reduce prediction error.
4. What backpropagation computes: gradients of the loss with respect to parameters.
5. A small example: write Julia code from scratch to learn a nonlinear function.

1. Why should economists care about deep learning?

In many economic problems, the object we want is an unknown function:

$$ y = f(x). $$

Examples include:

• value function: $V(k, z)$
• policy function: $c(k, z)$ or $k'(k, z)$
• pricing kernel
• conditional expectation: $E[y \mid x]$
• classifier: map features into a class

Traditional methods choose a function class by hand, such as polynomials, Chebyshev polynomials, or splines. Deep learning uses a parameterized function $g_\theta(x)$ to approximate $f(x)$:

$$ f(x) \approx g_\theta(x). $$

Training a neural network means choosing $\theta$ so that $g_\theta(x)$ is close to the observed data.

2. The key idea: change representation, then approximate

Deep learning is not only curve fitting. It also searches for a useful representation of the data.

Traditional function approximation is often written as:

$$ f(x) \approx h(x). $$

A neural network is closer to:

$$ f(x) \approx g(\phi(x)). $$

Here:

• $x$ is the raw input.
• $\phi(x)$ is a representation learned by earlier layers.
• $g(\cdot)$ uses that representation to predict the output.

In simple terms, the network bends, stretches, and combines the original variables into useful intermediate features, then predicts $y$ from those features.

3. Julia setup

We use basic Julia plus Random, Statistics, LinearAlgebra, and Plots. This keeps the internal steps visible.

4. Basic neural-network architecture

A feedforward fully-connected neural network is built from layers:

• input layer: the raw features $x = (x_1, x_2, \ldots, x_N)$.
• hidden layers: intermediate transformations of the input.
• output layer: the final prediction $\hat y$.

Two common words:

• depth: the number of hidden layers.
• width: the number of neurons in a hidden layer.

Fully connected means every neuron in one layer is connected to every neuron in the next layer.

Key design choices are called hyperparameters. They are not learned directly by gradient descent; the researcher chooses them before training, then checks train/test performance.

Choice	Symbol	Meaning
activation function	$\phi(\cdot)$	nonlinear transformation inside hidden layers
number of neurons	$M$	width of a hidden layer
number of hidden layers	$J$ or $L$	depth of the network
epochs	--	how many passes/updates we run during training
batch size	--	how many observations are used in each gradient update
learning rate	$\eta$	step size in gradient descent

For a general network, define $h^{(0)}=x$. Each hidden layer does two things:

1. affine transformation:

$$ a^{(\ell)} = W^{(\ell)}h^{(\ell-1)} + b^{(\ell)}, $$

2. nonlinear activation:

$$ h^{(\ell)} = \phi(a^{(\ell)}). $$

After $L$ hidden layers, the output layer is:

$$ \hat y = \psi(W^{(L+1)}h^{(L)} + b^{(L+1)}). $$

Here $\phi$ is the hidden-layer activation. The output activation $\psi$ depends on the task.

Common hidden-layer activations:

Name	Formula	Main idea
identity	$\phi(z)=z$	no nonlinearity
sigmoid	$\phi(z)=1/(1+e^{-z})$	maps to $(0,1)$
tanh	$\phi(z)=\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$	maps to $(-1,1)$
ReLU	$\phi(z)=\max(0,z)$	simple and common
leaky ReLU	$\phi(z)=\max(z,\alpha z)$	keeps a small negative slope
softplus	$\phi(z)=\log(1+e^z)$	smooth version of ReLU

Common output-layer choices:

Task	Output activation	Formula	Example
regression, any real value	identity	$\psi(z)=z$	price, value function, policy value
positive regression	softplus or exp	$\psi(z)=\log(1+e^z)$ or $\psi(z)=e^z$	variance, consumption, capital
number in $(0,1)$	sigmoid	$\psi(z)=1/(1+e^{-z})$	probability, share, rate
multi-class classification	softmax	$\psi_j(z)=\frac{e^{z_j}}{\sum_k e^{z_k}}$	class probabilities

5. A small task: learn an unknown function

Suppose the true function is:

$$ f(x) = \sin(3x) + 0.3x^2. $$

For now, pretend we do not know this formula. We only observe data points $(x_i, y_i)$. We will train a neural network to learn the map $x \mapsto y$.

This is the basic supervised-learning regression problem.

6. Start with the simplest model: linear regression

The linear model is:

$$ \hat y = a + bx. $$

Here $a$ and $b$ are parameters. Training means choosing $a,b$ to minimize mean squared error:

$$ L(a,b)=\frac{1}{N}\sum_{i=1}^N (a + bx_i - y_i)^2. $$

We first write gradient descent by hand.

The linear model loss falls, but the model can only draw a straight line. The true function bends and oscillates, so a well-trained linear model still cannot represent it.

This is the basic idea of model capacity: if the model class is too simple, it cannot express the true function.

7. A minimal neural network

Now we use a neural network with one hidden layer:

$$ z_1 = W_1x + b_1, $$

$$ h = \tanh(z_1), $$

$$ \hat y = W_2h + b_2. $$

Intuition:

• $W_1x+b_1$: create many different linear transformations of the input.
• $\tanh(\cdot)$: add nonlinear bending. Without nonlinearity, stacked linear layers are still linear.
• $W_2h+b_2$: combine intermediate features into the final prediction.

Here $h$ is a simple representation.

The diagram below shows the same architecture. The notebook later uses more hidden neurons for training, but the structure is the same.

8. Backpropagation: chain rule for gradients

The training objective is still mean squared error:

$$ L(\theta)=\frac{1}{N}\sum_{i=1}^N(\hat y_i-y_i)^2. $$

We need the effect of each parameter on the loss:

$$ \frac{\partial L}{\partial W_1},\quad \frac{\partial L}{\partial b_1},\quad \frac{\partial L}{\partial W_2},\quad \frac{\partial L}{\partial b_2}. $$

Backpropagation is just the chain rule. Since:

$$ x \rightarrow z_1 \rightarrow h \rightarrow \hat y \rightarrow L, $$

we start from $L$ and move backward layer by layer.

9. Train the network

Gradient descent updates parameters by:

$$ \theta \leftarrow \theta - \eta \nabla_\theta L(\theta), $$

where $\eta$ is the learning rate. If it is too small, training is slow. If it is too large, training can become unstable.

This figure shows the key ability of a neural network: it is not limited to straight lines. It can approximate complex functions by combining many nonlinear hidden units.

The intuition behind universal approximation is simple: with enough hidden units and a suitable activation function, even a basic feedforward network can approximate a wide class of functions.

10. What did the hidden layer learn?

Each hidden neuron creates an intermediate feature:

$$ h_j(x)=\tanh(w_jx+b_j). $$

The final prediction is a weighted sum of these features:

$$ \hat y = b_2 + \sum_j W_{2j}h_j(x). $$

This is the simplest version of representation learning.

11. Model capacity: what if we add more hidden neurons?

More hidden neurons make the model more flexible. Flexibility is useful, but it is not automatically better:

• Too few: underfitting. The model cannot learn the true shape.
• Moderate: the model can fit while staying reasonably smooth.
• Too many: the model may chase noise. In practice, deep learning also uses regularization, dropout, early stopping, and more data.

Below we compare several hidden-layer sizes.

12. Why do activation functions matter?

Without activation functions, stacked linear transformations are still just one linear transformation. For example:

$$ W_2(W_1x+b_1)+b_2 = (W_2W_1)x + (W_2b_1+b_2). $$

Nonlinear activations are what let neural networks express complex functions.

Common activation functions:

• sigmoid: $\sigma(z)=1/(1+e^{-z})$
• tanh: $\tanh(z)$
• ReLU: $\max(0,z)$
• softplus: $\log(1+e^z)$

Activations determine how the network bends and transforms the input space.

13. Training error is not enough: test on new points

During training, the network only sees the training points. A lower training loss means the network fits those points better, but it does not prove that the network has learned the underlying function.

The usual check is to create a test set:

• training set: points used to update the parameters.
• test set: new points not used during training.

There are two related ideas:

• In-domain test set: new points drawn from the same region as the training data. Here this means new points in $[-2,2]$.
• Out-of-domain extrapolation test: new points outside the training region. Here this means points in $[3,5]$.

The first check asks whether the model generalizes to new observations from the same environment. The second check is harder: it asks whether the model can extrapolate beyond what it saw during training.

In this toy example, we know the true data-generating function, so we can create both kinds of test points ourselves. In real data work, we usually hold out part of the observed data before training. If we also care about extrapolation, we need a separate out-of-domain check.

Now compare MSE on three samples:

• training points: points used to estimate the parameters.
• in-domain test points: new points from the same range as the training data.
• out-of-domain test points: new points from $[3,5]$, outside the training range.

A useful rule of thumb:

• low train MSE and low in-domain test MSE: good interpolation/generalization inside the training region.
• low train MSE but high in-domain test MSE: likely overfitting.
• low in-domain test MSE but high out-of-domain test MSE: the model learned the local shape but does not extrapolate well.

The out-of-domain result should be interpreted carefully. It is not just a normal test set; it is an extrapolation test. The model was trained only on $[-2,2]$, so it has little information about the shape of the true function on $[3,5]$.

This is why train/test evaluation must match the question we care about. If we only need interpolation, an in-domain test set is the right first check. If we need reliable predictions outside the training region, we must test that directly, and many models that look good in-domain can fail there.

14. Manifold hypothesis: Learning High Dimensional Functions

The manifold hypothesis says that high-dimensional data often have low-dimensional structure. The object may be represented in a large space, but the economically relevant observations may move near a much smaller set.

Here the input is a distribution. After discretization on 60 asset-grid points, a distribution is a vector:

$$ \mu = (\mu_1, \mu_2, \ldots, \mu_{60}) \in \mathbb{R}^{60}. $$

But not every vector in $\mathbb{R}^{60}$ is a meaningful distribution. A valid distribution must be nonnegative and sum to one. In many economic models, the distribution also moves because of a few underlying forces, such as average assets, dispersion, tail mass, or mass near a borrowing constraint.

This is exactly where deep learning can be useful. A neural network can use the full histogram and learn its own representation:

$$ \mu \rightarrow \phi(\mu) \rightarrow \hat y. $$

In a heterogeneous-agent model, a policy object may depend on an individual state, an aggregate shock, and the full cross-sectional distribution:

$$ y = g(k,z,\Phi). $$

The distribution state $\Phi$ is infinite-dimensional. A practical approximation is to replace $\Phi$ with a finite histogram $\mu$ and learn:

$$ g(k,z,\Phi) \approx g_\theta(k,z,\mu). $$

The toy example below removes $k$ and $z$ so that we can focus only on how to learn a function of a distribution:

$$ y = g(\mu). $$

Toy data-generating process

The original HA problem has an infinite-dimensional distribution state. In real applications, training data would come from solving or simulating the economic model. Here we do not want to solve a full HA model, so we create synthetic supervised-learning data:

$$ (\mu_i,y_i), \qquad i=1,\ldots,n. $$

The purpose of the toy DGP is not realism. It gives us a controlled example where we know the true target function and can compare train/test MSE.

The asset grid is:

$$ a_j \in [0.001,0.999], \qquad j=1,\ldots,60. $$

Each distribution is generated from only three latent variables:

$$ z=(z_1,z_2,z_3) \sim N(0,I_3). $$

Define three smooth shape functions on the asset grid:

$$ q_{1j}=\sin(2\pi a_j), $$

$$ q_{2j}=\exp[-80(a_j-0.25)^2]-\exp[-80(a_j-0.75)^2], $$

$$ q_{3j}=\cos(5\pi a_j). $$

These are just templates. The latent variables choose how strongly each template affects the histogram. For each asset bin, define the score:

$$ r_j(z)=1.2z_1q_{1j}+z_2q_{2j}+0.8z_3q_{3j}. $$

Then a softmax turns the scores into a valid distribution:

$$ \mu_j=\frac{\exp(r_j(z))}{\sum_{l=1}^{60}\exp(r_l(z))}. $$

So the data are generated by the map:

$$ z \in \mathbb{R}^3 \rightarrow \mu \in \mathbb{R}^{60}. $$

This is the manifold point: the representation is 60-dimensional, but the generating structure is only 3-dimensional.

Now define three distributional shape features. First define basis functions:

$$ b_{1j}=\sin(8\pi a_j), $$

$$ b_{2j}=\exp[-120(a_j-0.72)^2]-\exp[-120(a_j-0.32)^2], $$

$$ b_{3j}=\cos(11\pi a_j). $$

Then aggregate them using the full distribution:

$$ s_1=\sum_{j=1}^{60}\mu_j b_{1j}, $$

$$ s_2=\sum_{j=1}^{60}\mu_j b_{2j}, $$

$$ s_3=\sum_{j=1}^{60}\mu_j b_{3j}. $$

The target is:

$$ y = \sin(5s_1)+s_2^2-0.8s_3+\varepsilon, $$

$$ \varepsilon \sim N(0,0.01^2). $$

So the structure is:

$$ z \rightarrow \mu \rightarrow (s_1,s_2,s_3) \rightarrow y. $$

The target still depends on distributional shape, not only on mean, variance, or skewness.

Baseline: moment approximation

A traditional approximation compresses the full histogram into a small number of hand-picked moments.

Compute mean, variance, and third central moment:

$$ m=\sum_j a_j\mu_j, $$

$$ v=\sum_j (a_j-m)^2\mu_j, $$

$$ s=\sum_j (a_j-m)^3\mu_j. $$

Use the feature vector:

$$ f(\mu)=\begin{bmatrix}1 & m & v & s & m^2 & v^2 & s^2 & mv & ms & vs\end{bmatrix}'. $$

The prediction is linear in these features:

$$ \hat y = \beta f(\mu). $$

The ridge estimator solves:

$$ \min_{\beta} \sum_{i=1}^n \left( y_i-\beta f(\mu_i) \right)^2 + \lambda \|\beta\|^2. $$

The first term is the usual squared prediction error. The second term,

$$ \lambda \|\beta\|^2, $$

penalizes large coefficients. This makes the fitted coefficients less sensitive to noisy or highly correlated features.

With feature matrix $F$ and target row vector $Y$, the closed-form ridge estimate is:

$$ \hat\beta = YF'(FF' + \lambda I)^{-1}. $$

Interpretation: the moment method maps $\mu \in \mathbb{R}^{60}$ into $f(\mu) \in \mathbb{R}^{10}$. This is simple and interpretable, but it may lose shape information that matters for the target.

Neural-network approximation

Now train a neural network on the full histogram. In the code, the input is scaled as:

$$ x = 60\mu \in \mathbb{R}^{60}. $$

The factor 60 is only a numerical scaling. It does not change the information in $\mu$.

Use a one-hidden-layer network:

$$ h = \tanh(W_1x+b_1), $$

$$ \hat y = W_2h+b_2. $$

The dimensions are:

$$ W_1 \in \mathbb{R}^{64\times 60}, \qquad b_1 \in \mathbb{R}^{64}, $$

$$ W_2 \in \mathbb{R}^{1\times 64}, \qquad b_2 \in \mathbb{R}. $$

Training solves the mean-squared-error problem:

$$ \min_\theta \frac{1}{n}\sum_{i=1}^n(\hat y_i-y_i)^2, $$

where:

$$ \theta=(W_1,b_1,W_2,b_2). $$

The hidden layer learns a representation of the distribution:

$$ \phi(\mu)=\tanh(W_1\mu+b_1). $$

So the neural-network approximation is:

$$ \mu \rightarrow \phi(\mu) \rightarrow \hat y. $$

The difference is:

• moment approximation: $\mu \rightarrow$ hand-picked moments $\rightarrow \hat y$.
• neural network: $\mu \rightarrow$ learned features from the full histogram $\rightarrow \hat y$.

Architecture used below

The next diagram shows the neural network used for this distribution-input example. The input layer is the full scaled histogram $x=60\mu\in\mathbb{R}^{60}$. The hidden layer applies the activation function $\tanh(\cdot)$, and the output layer uses the identity activation because this is a regression problem.

Structure of the example

The diagram below summarizes the two approximation routes. The moment method compresses the distribution into chosen statistics. The neural network uses the full histogram and learns features from it.

Results

We report train and test MSE for both methods. The test set contains new distributions generated from the same data-generating process but not used in training.

The comparison asks whether each approximation can recover the mapping from distributional shape to target value.

Diagnostic: did the hidden layer learn the latent structure?

In this toy example, we know the latent variables $z_i$ that generated each histogram $\mu_i$. After training the neural network, we can check whether the hidden layer contains this low-dimensional information.

Let

$$ h_i = \tanh(W_1x_i+b_1) $$

be the hidden-layer representation. A simple linear probe asks whether $z_i$ can be recovered from $h_i$:

$$ z_i \approx Ah_i+c. $$

If the out-of-sample $R^2$ is high, the hidden layer has learned information about the low-dimensional state behind the 60-dimensional histogram. This does not mean one neuron equals one latent variable; the representation can be rotated or transformed.

Interpretation

The comparison is the same train/test logic as before.

• If the moment approximation has high test MSE, its chosen moments are not rich enough for this target.
• If the neural network has low test MSE, it is using the full histogram to learn shape features that generalize to new distributions.

The point is not that moments are always bad. The point is narrower:

• if the policy or target depends on distributional shape, a small number of hand-picked moments may miss relevant information;
• a neural network can use the full histogram and learn shape features automatically;
• this gives a finite-dimensional approximation to an object that originally depends on an infinite-dimensional distribution state.

For HA models, the approximation logic is:

$$ g(k,z,\Phi) \approx g_\theta(k,z,\mu). $$