SMOKE-Y Fullstack engineer: hardware to deep learing

home articles art bookmarks

non_linearity

Our monkey brain cannot comprehend non-linearity. During evolution, we never had the need to understand non-linearity. Hence, we are not able to wrap our heads around exponential behaviors such as compound interest, etc... Magic happens in non-linearity.

Computers are made out of semiconductor diodes. Semiconductor diodes are electrical components whose relation between the voltage applied across the diode is not linearly related to the current that passes through it. i.e. Ohm's law does not apply to semiconductor diodes.

Artificial neural networks are an entity which is modeled after our brain. One of the fundamental building blocks of neural networks is an activation function. This is a non-linear function that inputs a linear equation and outputs a single value between 0 and 1.

XOR is a boolean operation that cannot be modeled by a linear regression model. This is due to the fact that a single line cannot partition the input-output graph. You always need more than 1 neuron to train an XOR function.

[truth table]
0 ^ 0 = 0
0 ^ 1 = 1
1 ^ 0 = 1
1 ^ 1 = 0

[graph]
1| 1      0
 |
0| 0      1
 |__________
   0      1

This non-linear property gives rise to a lot of interesting results, such as swapping 2 numbers without a temp variable and saving space in a doubly-linked list by just storing 1 address. XOR is also heavily used in hashing and cryptography. To develop an intuition on the XOR operator, think of it as a bitwise "not equal to" operator.