If you haven’t heard of it before, let me give you the lowdown:
TensorRT is an open-source deep learning inference optimizer and runtime system developed by NVIDIA that can significantly improve the performance of your models when running them on GPUs. It does this by using a technique called quantization to reduce the size of the model’s weights, which allows for faster execution times and lower memory usage.
Now, you might be wondering why we need TensorRT in the first place after all, PyTorch already has its own built-in optimizer that can handle most cases just fine. Well, there are a few reasons:
1) Speed: As I mentioned earlier, TensorRT is much faster than PyTorch’s default optimizer when it comes to inference time. This means you can run your models on GPUs at lightning speeds and get results almost instantly!
2) Memory usage: Another benefit of using TensorRT is that it can significantly reduce the amount of memory required by your model, which is especially important if you’re working with large datasets or running multiple experiments simultaneously.
3) Compatibility: Finally, TensorRT supports a wide range of deep learning frameworks and libraries, including PyTorch, Keras, Caffe, and more! This means that no matter what toolchain you prefer to use, you can still take advantage of its performance-boosting capabilities.
So, how do we go about installing TensorRT OSS on Ubuntu 20.04? Well, it’s actually pretty straightforward just follow these simple steps:
1) First, make sure that your system is up to date by running the following command in a terminal window:
# Update and upgrade the system to ensure it is up to date
$ sudo apt update && sudo apt upgrade
# The '$' symbol indicates that the following command should be run in a terminal window
# The 'sudo' command allows the user to run a command with administrative privileges
# 'apt' is a package management tool used to install, update, and remove software on Ubuntu
# 'update' is a command used to update the local package index
# '&&' is a logical operator that allows multiple commands to be executed in sequence
# 'upgrade' is a command used to upgrade all installed packages to their latest versions
# The '&&' operator ensures that the 'upgrade' command is only executed if the 'update' command is successful
2) Next, add the NVIDIA repository to your sources list so that you can install TensorRT and its dependencies. To do this, run:
# Download the libcudnn8 package from the NVIDIA repository
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/libcudnn8_7.5.2-1+cuda11.3_amd64.deb
# Install the libcudnn8 package using dpkg
$ sudo dpkg -i libcudnn8_7.5.2-1+cuda11.3_amd64.deb
# The above two commands add the NVIDIA repository to the sources list and install the necessary dependencies for TensorRT.
# This allows for the successful installation and usage of TensorRT on the system.
This will download and install the CUDA library, which is a prerequisite for TensorRT.
3) Now that we have CUDA installed, let’s proceed with adding the NVIDIA repository to our sources list:
# Download the CUDA repository package from the NVIDIA website using wget command
$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_11.3.2-1_amd64.deb
# Install the CUDA repository package using dpkg command with sudo privileges
$ sudo dpkg -i cuda-repo-ubuntu1804_11.3.2-1_amd64.deb
# The above commands will add the NVIDIA repository to our sources list, allowing us to access and install the CUDA library. This is a prerequisite for TensorRT.
This will download and install the NVIDIA repository, which contains all of the necessary packages for TensorRT OSS.
4) Finally, we can update our package list and install TensorRT using the following commands:
# This script updates the package list and installs the NVIDIA CUDA toolkit and TensorRT OSS.
# Update the package list
$ sudo apt-get update
# Upgrade existing packages
$ sudo apt-get upgrade -y
# Install the NVIDIA CUDA toolkit
$ sudo apt-get install nvidia-cuda-toolkit -y
# Install TensorRT OSS
$ sudo apt-get install tensorrt -y
# The -y flag automatically confirms any prompts during installation.
And that’s it! You should now have TensorRT OSS installed and ready to use. To test it out, you can run a simple example script like this:
# Import necessary libraries
import torch
from torch.onnx import load_model
import numpy as np
import onnxruntime as rt
# Load the model from PyTorch format
model = load_model('path/to/your/pytorch/model') # Load the PyTorch model from the specified path
# Convert it to ONNX format using TensorRT
input_names = ['input'] # Define the input names for the ONNX model
output_names = ['output'] # Define the output names for the ONNX model
trt_session = rt.InferenceSession('path/to/your/onnx/file', providers=['CUDAExecutionProvider']) # Create an ONNX runtime inference session using the specified ONNX file and CUDAExecutionProvider for GPU acceleration
# Define the input and output tensors for our script
input_shape = (1, 3, 224, 224) # Define the shape of the input tensor
input_data = np.random.randn(*input_shape).astype(np.float32) # Generate random input data with the specified shape
output_data = trt_session.run(None, {input_names[0]: input_data}) # Run the inference session with the input data and retrieve the output data
print('Output:', output_data) # Print the output data to verify that the script is working correctly
And there you have it a simple script for using TensorRT OSS with PyTorch on Ubuntu 20.04! Of course, this is just one example of what’s possible with TensorRT the real power comes from its ability to optimize your models and improve their performance in ways that were previously impossible. So why not give it a try today? Your data will thank you for it!