A neuron is a dot product and a threshold
Tonight's idea
An artificial neuron multiplies each input \(x_i\) by a weight \(w_i\), adds a bias \(b\), and compares with zero:
\(s = w_1 x_1 + w_2 x_2 + b\); output \(1\) if \(s > 0\), otherwise \(0\).
The sum is the dot product \(\mathbf{w} \cdot \mathbf{x}\), so the neuron fires when \(\mathbf{x}\) is on one side of the line \(\mathbf{w} \cdot \mathbf{x} + b = 0\): a neuron is a line with a yes side and a no side.
Show the neuron an input \(\mathbf{x}\) with correct answer \(t\). If its output \(o\) is wrong, apply the perceptron rule:
\(\mathbf{w} \leftarrow \mathbf{w} + (t - o)\,\mathbf{x}\), \(\quad b \leftarrow b + (t - o)\).
If it should have fired, \(t - o = 1\) and the weights move towards \(\mathbf{x}\), so \(s\) grows; if it fired wrongly, away. Repeat until all are right. If a line separates yes from no, the rule finds one in finitely many updates (the perceptron convergence theorem).
One neuron cannot learn XOR (fires when exactly one input is on): its yes and no points sit on crossing diagonals, and no line separates the pairs. Such data are not linearly separable; the rule circles forever.
Watch one: learning AND
AND fires only for \((1,1)\). Start from \(\mathbf{w} = (0, 1)\), \(b = -1\), presenting the inputs in the order \((0,0), (0,1), (1,0), (1,1)\).
First pass: sums \(-1, 0, -1, 0\), so it never fires. Only \((1,1)\) is wrong, \(t - o = 1\): \(\mathbf{w} = (1, 2)\), \(b = 0\).
Second pass: \((0,0)\) gives \(0\), right; \((0,1)\) gives \(2\) and fires wrongly: \(\mathbf{w} = (1, 1)\), \(b = -1\). Then \((1,0)\) gives \(0\) and \((1,1)\) gives \(1\), both right.
Third pass: sums \(-1, 0, 0, 1\). Only \((1,1)\) fires. Two updates. From \(\mathbf{w} = (0,0)\), \(b = 0\) the rule needs ten updates and ends at \(\mathbf{w} = (2,1)\), \(b = -2\): the theorem promises an answer, not a short road.
Your turn
- A neuron has \(\mathbf{w} = (2, 1)\), \(b = -2\). Compute \(s\) and the output for all four inputs \((0,0), (0,1), (1,0), (1,1)\). Which logic gate is it?
- Find weights and a bias for (a) NOT, a one-input neuron that fires when \(x = 0\) and not when \(x = 1\); (b) NOR, which fires only for \((0,0)\). Check all cases.
- Train OR (fires unless both inputs are 0) from \(\mathbf{w} = (0,0)\), \(b = 0\), inputs in the order \((0,0), (0,1), (1,0), (1,1)\), repeated. Write down \(\mathbf{w}\) and \(b\) after each update, and the final neuron.
- Prove that no neuron computes XOR. Write the two inequalities a yes point must satisfy (\(s > 0\)) and the two a no point must satisfy (\(s \le 0\)), then add them in pairs.
Stretch: two layers
Let two neurons read the inputs first: \(h_1\) is OR with \(\mathbf{w} = (1,1)\), \(b = -\tfrac12\), and \(h_2\) is AND with \(\mathbf{w} = (1,1)\), \(b = -\tfrac32\). Find weights and a bias for a third neuron, reading \((h_1, h_2)\), that computes XOR. Why does this not contradict question 4?
Answers
- \(s = -2, -1, 0, 1\); outputs \(0, 0, 0, 1\). AND. Note that \((1,0)\) gives \(s = 0\), which does not fire: the line \(2x_1 + x_2 = 2\) passes through that point, and the rule "fire if \(s > 0\)" puts it on the no side.
- (a) \(w = -1\), \(b = \tfrac12\): \(s = \tfrac12\) at \(x = 0\), \(s = -\tfrac12\) at \(x = 1\). (b) \(\mathbf{w} = (-1,-1)\), \(b = \tfrac12\): \(s = \tfrac12, -\tfrac12, -\tfrac12, -\tfrac32\). Many other answers work; any line in the right place does.
- Pass 1: \((0,1)\) should fire, does not: \(\mathbf{w} = (0,1)\), \(b = 1\). Pass 2: \((0,0)\) fires wrongly: \(b = 0\); then \((1,0)\) should fire, does not: \(\mathbf{w} = (1,1)\), \(b = 1\). Pass 3: \((0,0)\) fires wrongly: \(b = 0\). Pass 4: sums \(0, 1, 1, 2\), all correct. Four updates; final \(\mathbf{w} = (1,1)\), \(b = 0\).
- Yes points: \(w_2 + b > 0\) and \(w_1 + b > 0\). No points: \(b \le 0\) and \(w_1 + w_2 + b \le 0\). Adding the yes pair: \(w_1 + w_2 + 2b > 0\). Adding the no pair: \(w_1 + w_2 + 2b \le 0\). The same quantity cannot be both. No choice of \(w_1, w_2, b\) works.
Stretch. The hidden pair \((h_1, h_2)\) is \((0,0)\) for input \((0,0)\), \((1,0)\) for both \((0,1)\) and \((1,0)\), and \((1,1)\) for \((1,1)\). XOR should fire only at \((1,0)\): take weights \((1, -1)\) and \(b = -\tfrac12\), giving \(s = -\tfrac12, \tfrac12, -\tfrac12\). The output neuron is still a line, but it draws it in the hidden layer's coordinates, where the two yes inputs have been folded onto the same point. Stacking layers is the whole idea of a neural network.