What is the Best Way to Use Autograd to do Gradient-Based Optimization?

Mar 28, 2022
4 min read

In machine learning, testing a prediction model is very essential. In the majority of circumstances, we assess the prediction models by maximizing different loss functions with respect to the model's parameters. We can use gradient descent to optimize these loss functions when going in-depth. Autograd is a Python tool that can assist us in optimizing numerous functions related to machine learning models or basic mathematics. In some cases, this package can be used in place of the well-known NumPy library. We will look at the Autograd package and see how it can be utilized in this article. Let us start with the definition of Autograd.

What is Autograd?

Autograd is a gradient-based optimization library. Autograd is a Python module that can help us discern between Numpy and Python code. Features like loops, ifs, recursion, and closures and be handled by the Autograd package. Apart from these, it can also have many step-wise derivatives of functions. Most of the time, we see that the Autograd package supports both the backpropagation and forward propagation methods, which are also known as reverse-mode differentiation and forward mode differentiation in mathematics, respectively. This means that this package can pick gradients from a function with scalar values.

We may mix and match these two ways using this package. This package's main goal is to enable us to execute gradient-based optimization. With this package, we may utilize gradients for the following operations and functions:

  • Mathematical operations - a variety of gradients are used to optimize the majority of mathematical operations.
  • A wide range of array manipulations can be performed using different gradients available.
  • Gradients are included in the package to aid in the completion of different matrix transformations.
  • Gradients for a wide range of linear algebra and Fourier transform algorithms are included in the package.
  • Modules are available for N-dimensional convolutions.
  • The use of complex numbers is permitted.

Using the following lines of code, we can install this package:

!pip install autograd

We are now ready to use this package for our work after installing it in the environment.

Autograd Implementations

Now we will have a look at some of the functions of the Autograd package.

Evaluating the hyperbolic tangent function's gradient

We will look at how to evaluate the gradient of the Tanh function in this implementation. Let us start by defining the function.

import autograd.numpy as agnp
def tanh(x):                 
     y = agnp.exp(-2.0 * x)
     return (1.0 - y) / (1.0 + y)

We used the module autograd.numpy, which is a NumPy wrapper in the autograd package, in the preceding code. We have also built a tan function. Take a look at the above-mentioned function's gradient.

from autograd import grad 
grad_tanh = grad(tanh)    


In the above code, we created a variable to contain the tanh function, and for evaluation, we imported a function named grad from the autograd package. Let us look at the difference in finite range.

(tanh(1.0001) - tanh(0.9998)) / 0.0002



We also have the option of differentiating the functions as many times as we like by calling a module called elementwise grad.

from autograd import elementwise_grad
x = agnp.linspace(-10, 10, 100)

In the preceding code, we called our module and constructed an array with random values ranging from -10 to 10. We will now try to draw derivatives for the previously given function.

plt.plot(x, agnp.tanh(x),
         x, elementwise_grad(agnp.tanh)(x),
         x, elementwise_grad(elementwise_grad(agnp.tanh))(x),
         x, elementwise_grad(elementwise_grad(elementwise_grad(agnp.tanh)))(x),
         x, elementwise_grad(elementwise_grad(elementwise_grad(elementwise_grad(agnp.tanh))))(x))        


We can see how the function varies for our defined x here.

Note that at the start of the implementation, we created a tanh function, and in the most recent iteration, we used the autograd function.

We looked at how to utilize autograd modules in the previous example. Let us have a look at how it may be applied to logistic regression.

Logistic regression loss function optimization

Let’s define a sigmoid function.

def function(x):
    return 0.5 * (agnp.tanh(x / 2.) + 1)

Defining a function for predictions:

def function2(w, i):
    return function(agnp.dot(i, w))

Defining loss function for training:

def loss_function(w):
    preds = function2(w, i)
    label_probabilities = preds * targets + (1 - preds) * (1 - targets)
    return -agnp.sum(agnp.log(label_probabilities))

Defining the weights and input:

i = agnp.array([[0.52, 1.12,  0.77],                
                [0.88, -1.08, 0.15],
                [0.52, 0.06, -1.30],
                [0.74, -2.49, 1.39]])
w = agnp.array([0.0, 0.0, 0.0])

Defining the target:

targets = agnp.array([True, True, False, True])

Defining gradient function for training loss:

training_gradient = grad(loss_function)

Optimization of weight using the gradient descent:

w = np.array([0.0, 0.0, 0.0])
print("loss in initial:", loss_function(w))
for i in range(100):
    weights -= training_gradient(w) * 0.01
print("loss after training:", loss_function(w))


loss in initial: 2.772588722239781
loss after training: 1.067270675787016

We have seen an example of logistic regression for weight optimization that we pushed in between utilizing the autograd package's components.