This thing is like a Swiss Army knife for image processing, but with way more features (and less knives).
To start: how do you use it? Well, let’s say you have an image that looks something like this:
And you want to preprocess it before feeding it into your model. Here’s how you do it with SiglipProcessor:
# Import necessary libraries
from PIL import Image # Importing Image module from PIL library for image processing
import requests # Importing requests library for making HTTP requests
from transformers import AutoProcessor, AutoModel # Importing AutoProcessor and AutoModel from transformers library for using pretrained models
import torch # Importing torch library for deep learning
# Load the image from a URL and convert it to a PIL Image object
url = "https://i.imgur.com/XZY12345.jpg" # Assigning the URL of the image to a variable
image = Image.open(requests.get(url, stream=True).raw) # Using requests library to get the image from the URL and converting it to a PIL Image object
# Preprocess the image using SiglipProcessor (which automatically resizes it to 224x224 pixels and normalizes its values between -1 and 1)
processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224") # Initializing the SiglipProcessor with a pretrained model
inputs = processor(images=image, return_tensors="pt") # Preprocessing the image using the SiglipProcessor and converting it to a PyTorch tensor
# Load the model from a pretrained checkpoint (which has already been trained on a large dataset of images) and use it to extract features from our preprocessed image
model = AutoModel.from_pretrained("google/siglip-base-patch16-224") # Initializing the model with a pretrained checkpoint
with torch.no_grad(): # Disabling gradient calculation for faster computation
# Get the textual features (i.e., the output of the last layer before the classification head) using the model's get_text_features() method
text_features = model(**inputs)[0] # Extracting the textual features from the preprocessed image using the model's get_text_features() method
And that’s it! SiglipProcessor takes care of all the heavy lifting for us, including resizing and normalization. It also supports a variety of other features like padding, cropping, and flipping (which can be useful if you have images with varying sizes or orientations).
But wait there’s more! SiglipProcessor is actually part of the larger SigLIP framework, which includes both image and text processing capabilities. This means that you can use it to preprocess your images before feeding them into a model like BERT (which is great for handling long sequences of text), or vice versa you can use it to preprocess your text data before feeding it into an image-based model like ResNet50 (which is good at recognizing patterns in visual data).
So if you’re looking for a powerful and flexible tool for preprocessing images and/or text, SiglipProcessor is definitely worth checking out. And the best part? It’s open source and available on GitHub!