- \n
- Crash course on Linear neural network models<\/li>\n
- Understanding network weights and biases<\/li>\n
- Bias and added network expressiveness<\/li>\n
- When Bias might be a bad thing<\/li>\n
- Bias value use in network design<\/li>\n
- PyTorch’s Linear Model example code<\/li>\n<\/ul>\n
TL;DR<\/strong> – It enables bias values within the network and defaults to
True<\/code> which is probably what you want.<\/p>\nCrash Course: The Linear Model<\/h2>\n
Deep learning makes use of multi-layer architectures inspired by the connectivity of neurons in the human brain. Essentially, this involves “nodes” and “layers” of a network such that all nodes in
Layer A<\/code> connect to all nodes inLayer B<\/code> whereLayer A<\/code> precedesLayer B<\/code> in calculations. This is a very vanilla model sometimes called a Multi-Layer Perceptron<\/em> and, more accurately, a multi-layer feedforward network. Generally, this requires at least 3 layers:<\/p>\n- \n
- Input Layer<\/li>\n
- Hidden Layer<\/li>\n
- Output Layer<\/li>\n<\/ul>\n
However, there are often multiple hidden layers and much more complex architectures in modern models such as Recurrent Neural Networks (RNN), Convolutional Nural Networks (CNN), and multi-attention head models like Transformers. For now, the only other concepts we need to know are weights<\/em> and biases, <\/em>and activation function<\/em>:<\/p>\n
- \n
- Weights<\/strong> are the numerical values associated with the connections between nodes (the lines in the image below) that are multiplied against the input values.<\/li>\n
- Biases<\/strong> are values that get added to the product of the weight and the input values<\/li>\n
- Activation Function<\/strong> – applied to the weighted sum of the input and bias (e.g. ReLU) often clamping into positive terms.<\/li>\n<\/ul>\n
Each of these components is relevant on a per-node level. That is; each node of a neural network is fed an input that is multiplied by a weight, added a bias, and fed into an activation function before being passed to the next node (usually). Below is a very simple fully-connected deep neural network, illustrating a four-layer fully-connected deep neural network:<\/p>\n
![\"deep](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/3516\/deep-learning-network-diagram.jpg\" "\\"deep-learning-network-diagram\\"")
Patterson, J. & Gibson, A. Deep Learning a Practitioner\u2019s Approach. Sebastopol: O\u2019Reilly Media, 2017.<\/figcaption><\/figure>\n In practice, there can be billions or even trillions of nodes in a layer. In the remainder of this article, we’ll discuss the Bias component, and how PyTorch’s Linear model handles its dynamics. You needn’t have deep PyTorch experience or familiarity to follow along but I don’t know why you’d be reading this if you didn’t.<\/p>\n
PyTorch Linear Model<\/h2>\n
To utilize a PyTorch Linear model (equivalent to one “layer” in the above image) one can import this class as such:<\/p>\n
# conventional PyTorch import syntax\r\nimport torch.nn as nn\r\n\r\n# creates the layer\r\nlinear_layer = nn.Linear(\r\n in_features=3,\r\n out_features=4,\r\n bias=True\r\n)\r\n\r\nprint(linear_layer)\r\n\r\nLinear(in_features=3, out_features=4, bias=True)<\/pre>\n
- Biases<\/strong> are values that get added to the product of the weight and the input values<\/li>\n
- Weights<\/strong> are the numerical values associated with the connections between nodes (the lines in the image below) that are multiplied against the input values.<\/li>\n
![\"y](\"https:\/\/www.alpharithms.com\/wp-content\/uploads\/3516\/y-f-wx-b.png\" "\\"y-f-wx-b\\"")
<\/p>\n