Neural network architecture

 a= f(wp+b)


Here is a single input neuron. 

p is the input. 

Here, w is the weight. The designer can use any value for the weight. Weight is also called 'offset'. 

The bias is also a weight with a constant input of 1. The bias can even be omitted or its value can be changed. 

w and b are scalar parameters. 

The designer chooses the activation function. Some learning algorithm helps to choose the values of w and b. 

wp+b  is called the net input. 

a is s function of the net input. This function is called the activation function or the transfer function. An example can be the sigmoid activation curve. 

Different types of activation functions:

They may be linear or non-linear. 

How to choose a transfer function? A neuron tries to solve some problem and the specifications of this problem help to choose the transfer function. 

The hard limit transfer function


The first figure shows the hardlimit transfer function. It helps to break the input into two categories. If the argument is less than zero then the output is zero. If the argument is zero or greater then the argument is one. These are the two categories. 

The second figure shows the transfer characteristics of the input-output characteristics of the single output neuron. 

Linear transfer function



In a linear transfer function, the input and output are equal. The equation forms a straight line. When a single input neuron uses the linear transfer function then it tilts and shifts the line horizontally. 


Log-sigmoid transfer function




The input can have any real value. The output has a value between zero and one. 


This is differentiable so it is used in multilayer networks and backpropagation. 


https://www.geeksforgeeks.org/activation-functions-neural-networks/


Linear activation function

The equation is like a straight line y=x. 

The final activation function of the last layer is a linear function of the first layer. It is true for any number of layers. 

The range is -∞ to +∞.

The linear activation function is used at the output layer only. 

Differentiating the function will bring a constant only. This will depend only on the input. 

The hidden layers can have non-linear functions as well. 

Multiple input neurons



The input vector is p. It is represented using either a vector or a matrix. Weight is also represented by using a matrix. The resultant is a scaler. 
If we have multiple neurons then even the output of the network would be a vector.  

Recurrent networks (RNN)

It uses data in the time series which is also called sequential data. They use training data to learn. Data from previous output influence both the input and output of these neural networks. It is different from traditional neural networks.  In traditional neural networks, the input and output are independent of each other. But, in RNN, they depend on each other. These neural networks make predictions on the basis of prior inputs. If the RNN is unidirectional RNN then it cannot account for future events. 


Delay

A Delay block is a simple building block. 
a(t) is the delay output. u(t) is the delay input. 
Assumption: Time is updated in discrete steps and takes up only integer values. 
a(t) = u(t-1) 

Output has to be initialized to t=0. 

Here, a(0) is not the weight but it is the initial condition. 

Integrator





Comments

Post a Comment

Popular posts from this blog

Diary -24th July

Image
  I tried learning Greek and Japanese. German is too easy and boring but I will get back to it as soon as I can. I wonder why not learn pretty much any real language on Duolingo? It will give me the variety I want.  The Greek letters sound just like Hindi letters. But the Greek letters are also like English letters. Here is my secret to Indian English. In Japanese gohan is rice. It sounds like go or cow or white and anna which means grain. Recipes I want to try: https://www.archanaskitchen.com/vegetarian-thukpa-recipe-by-saransh-goila-india-on-my-platter/amp https://www.cookwithmanali.com/thandai-powder/ Maggie masale noodles tastemaker and maggie masala e magic recipe https://hebbarskitchen.com/maggi-masala-powder-maggi-masala-magic/ Science and animals Hippos  https://www.usatoday.com/story/news/2022/09/27/what-do-hippos-eat/10303668002/
Read more

How to make clip art faces-Part 1: Pretty girl face with braided hair decorated with flowers

Image
  The use of circles makes a face look very amiable. And, this picture isn't as complicated as it seems.  I have used MS Word to make this clip art. But, you can use any software of your choice.  Let's begin. You will find all the shapes used in this picture in MS Word in this way: Insert>>Shapes>> Face Select an oval.  To change the color of the oval, right click on the oval, you will find 'fill' and 'outline' options.  Choose any color of your choice. I have played around a lot with colors. I must insist that you do the same according to your tastes and requirements. Hair Here is how we use the ovals to assemble the hair.  Like this, also use ovals to complete the other side.  Flowers Let us draw a single flower then we will copy and paste it again and again.  Start with a circle.  Then draw two more concentric circles of colors you like.  Now add the petals using the teardrop shape.  Use another color to draw an inner...
Read more

Object oriented programming in dart concepts

   No introduction, let us go straight to concepts :-) What is object oriented programming? It is a paradigm that involves classes and objects. First, we will understand paradigms in detail then we will understand how object oriented programming falls in this framework. Then we will explore what are classes and objects. After that we understand object oriented programming in Dart language. So, keep scrolling to get crystal clear concepts.  What is a paradigm? Paradigm is a method to solve a problem.  It may include some tools or coding languages. We can use any  programming language to solve a problem but that approach used to solve it is called a paradigm.  Types of paradigms Paradigms separate programming languages into different categories on the basis of features of various programming languages.  1. Imperative programming paradigm Statements change the state of the program.  They give commands to the computer. It  means this type of prog...
Read more