Feature extraction

Feature extractors offers a standardized way to extract features from instances of the WildlifeDataset.

Feature extractors, implemented as classes, can be created with specific arguments that define the extraction properties. After instantiation, the extractor functions as a callable, requiring only a single argument—the WildlifeDataset instance. The specific output type and shape vary based on the chosen feature extractor. Output is FeatureDataset instance.

features.deep

DeepFeatures(model, batch_size=128, num_workers=1, device='cpu')

Extracts features using forward pass of pytorch model.

Parameters:

Name	Type	Description	Default
model	Module	Pytorch model used for the feature extraction.	required
batch_size	int	Batch size used for the feature extraction.	128
num_workers	int	Number of workers used for data loading.	1
device	str	Select between cuda and cpu devices.	'cpu'

__call__(dataset)

Extract features from input dataset and return them as a new FeatureDataset.

Parameters:

Name	Type	Description	Default
dataset	ImageDataset	Extract features from this dataset.	required

Returns:

Name	Type	Description
feature_dataset	FeatureDataset	A FeatureDataset containing the extracted features

ClipFeatures(model=None, processor=None, batch_size=128, num_workers=1, device='cpu')

Extract features using CLIP model (https://arxiv.org/pdf/2103.00020.pdf). Uses raw images of input ImageDataset (i.e. dataset.transform = None)

Parameters:

Name	Type	Description	Default
model		transformer.CLIPModel. Uses VIT-L backbone by default.	None
processor		transformer.CLIPProcessor. Uses VIT-L processor by default.	None
batch_size		Batch size used for the feature extraction.	128
num_workers		Number of workers used for data loading.	1
device		Select between cuda and cpu devices.	'cpu'

__call__(dataset)

Extract clip features from input dataset and return them as a new FeatureDataset.

Parameters:

Name	Type	Description	Default
dataset	ImageDataset	Extract features from this dataset.	required

Returns:

Name	Type	Description
feature_dataset	FeatureDataset	A FeatureDataset containing the extracted features

features.local

GlueFactoryExtractor(config, device=None, num_workers=1)

Base class for Gluefactory extractors.

Common configuration of extractors:

1. max_num_keypoints: Maximum number of keypoints to return.
1. detection_threshold: Threshold for keypoints detection (use 0.0 if force_num_keypoints = True).
1. force_num_keypoints: Force to return exactly max_num_keypoints keypoints.

Parameters:

Name	Type	Description	Default
config	dict	Configuration dictionary for the model.	required
device	None | str	Select between cuda and cpu devices.	None
num_workers	int	Number of workers used for data loading.	1

__call__(dataset)

Extract clip features from input dataset and return them as a new FeatureDataset. Gluefactory extractors requires with 3 channel RBG tensors scaled to [0, 1].

Parameters:

Name	Type	Description	Default
dataset	ImageDataset	Extract features from this dataset.	required

Returns: feature_dataset: A FeatureDataset containing the extracted features

SuperPointExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, **model_config)

Bases: GlueFactoryExtractor

Superpoint keypoints and descriptors.

• Paper: SuperPoint: Self-Supervised Interest Point Detection and Description
• Link: https://arxiv.org/abs/1712.07629

DiskExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, **model_config)

Bases: GlueFactoryExtractor

DISK keypoints and descriptors.

• Paper: DISK: learning local features with policy gradient
• Link: https://arxiv.org/abs/2006.13566

AlikedExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, **model_config)

Bases: GlueFactoryExtractor

ALIKED keypoints and descriptors.

• Paper: ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation
• Link: https://arxiv.org/abs/2304.03608

SiftExtractor(backend='opencv', detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, **model_config)

Bases: GlueFactoryExtractor

SIFT keypoints and descriptors.

features.memory

DataToMemory

Loads the dataset into memory for quicker access

Ideal for LOFTR, which operates directly on images, because loading images from storage can become a bottleneck when matching all query-database pairs, requiring n_query x n_database image loads.

__call__(dataset)

Loads data from input dataset into array and returns them as a new FeatureDataset.

Examples

Example - SuperPoint features

from wildlife_tools.features.local import SuperPointExtractor

extractor = SuperPointExtractor(backend='opencv', detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256)
features = extractor(dataset)

Example - Deep features

import timm
from wildlife_tools.features.deep import DeepFeatures

backbone = timm.create_model('hf-hub:BVRA/MegaDescriptor-T-224', num_classes=0, pretrained=True)
extractor = DeepFeatures(backbone, device='cuda')
features = extractor(dataset)