Skip to content

Reference features

features.deep

DeepFeatures(model, batch_size=128, num_workers=1, device='cpu', cache_path=None, config_tag=None)

Bases: FeatureCacheMixin, FeatureExtractor

Extracts features using forward pass of pytorch model.

Parameters:

Name Type Description Default
model Module

Pytorch model used for the feature extraction.

 required
batch_size int

Batch size used for the feature extraction.

 128
num_workers int

Number of workers used for data loading.

 1
device str

Select between cuda and cpu devices.

 'cpu'
cache_path str

Path for cached results. No caching for None.

 None
config_tag str

Free-form tag stored in the cache config. Reusing cache_path with a different tag raises an error. Changes of the model or of the image transform are not detected automatically, so encode them in the tag (e.g. "megadescriptor-L-384_resize384").

 None

lazy_load(model_factory, **kwargs) classmethod

Create the extractor without loading the model. The model is created by model_factory on its first use, so it is not loaded at all when all features are already cached.

Example 
import functools
import timm

extractor = DeepFeatures.lazy_load(
    functools.partial(timm.create_model, "hf-hub:BVRA/MegaDescriptor-L-384", num_classes=0, pretrained=True),
    device="cuda",
    cache_path="cache/MacaqueFaces/megadescriptor",
)

Parameters:

Name Type Description Default
model_factory Callable[[], Module]

Function without arguments returning the model. Use functools.partial instead of lambda if the extractor needs to be pickled.

 required
**kwargs

Remaining arguments of the class constructor (e.g. batch_size, device, cache_path, or processor for ClipFeatures and DinoFeatures).

 {}

Returns:

Name Type Description
extractor DeepFeatures

Extractor of the class on which the method is called.

ClipFeatures(model=None, processor=None, batch_size=128, num_workers=1, device='cpu', cache_path=None, config_tag=None)

Bases: DeepFeatures

Extract features using CLIP model (https://arxiv.org/pdf/2103.00020.pdf). Uses raw images of input ImageDataset (i.e. dataset.transform = None)

Parameters:

Name Type Description Default
model CLIPModel

Uses VIT-L backbone by default.

 None
processor CLIPProcessor | None

(CLIPProcessor, optional). Uses VIT-L processor by default.

 None
batch_size int

Batch size used for the feature extraction.

 128
num_workers int

Number of workers used for data loading.

 1
device str

Select between cuda and cpu devices.

 'cpu'
cache_path str

Path for cached results. No caching for None.

 None
config_tag str

Free-form tag stored in the cache config. Reusing cache_path with a different tag raises an error. Changes of the model or of the image transform are not detected automatically, so encode them in the tag (e.g. "clip-vit-large-patch14").

 None

features.local

LocalFeatureExtractor(max_num_keypoints=256, detection_threshold=0.0, force_num_keypoints=True, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1)

Bases: FeatureCacheMixin, FeatureExtractor

Base class for local feature extractors.

Common configuration of extractors:

1. max_num_keypoints: Maximum number of keypoints to return.
1. detection_threshold: Threshold for keypoints detection (use 0.0 if force_num_keypoints = True).
1. force_num_keypoints: Force to return exactly max_num_keypoints keypoints. Missing keypoints
    are padded with random locations and zero descriptors.

Each extracted feature is a dictionary with keypoints, keypoint_scores, descriptors and image_size.

Parameters:

Name Type Description Default
max_num_keypoints int

Maximum number of keypoints to return.

 256
detection_threshold float

Threshold for keypoints detection.

 0.0
force_num_keypoints bool

Force to return exactly max_num_keypoints keypoints.

 True
device str | None

Select between cuda and cpu devices.

 None
num_workers int

Number of workers used for data loading.

 1
cache_path str

Path for cached results. No caching for None.

 None
config_tag str

Free-form tag stored in the cache config. Reusing cache_path with a different tag raises an error. Changes of the image transform are not detected automatically, so encode them in the tag (e.g. "resize224").

 None
batch_size int

Number of images processed at once. Images in a batch must have the same size.

 1

DiskExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1, window_size=5, checkpoint='depth')

Bases: LocalFeatureExtractor

DISK keypoints and descriptors.

  • Paper: DISK: learning local features with policy gradient
  • Link: https://arxiv.org/abs/2006.13566

AlikedExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1, model_name='aliked-n16', nms_radius=2)

Bases: LocalFeatureExtractor

ALIKED keypoints and descriptors.

  • Paper: ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation
  • Link: https://arxiv.org/abs/2304.03608

XFeatExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1)

Bases: LocalFeatureExtractor

XFeat keypoints and descriptors. Match with MatchLightGlue(features="xfeat").

  • Paper: XFeat: Accelerated Features for Lightweight Image Matching
  • Link: https://arxiv.org/abs/2404.19174

DeDoDeExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1, descriptor='B', detector_weights='L-C4-v2')

Bases: LocalFeatureExtractor

DeDoDe keypoints and descriptors. Match with MatchLightGlue(features="dedodeb") for descriptor="B" and MatchLightGlue(features="dedodeg") for descriptor="G". The detection_threshold is not used.

  • Paper: DeDoDe: Detect, Don't Describe -- Describe, Don't Detect for Local Feature Matching
  • Link: https://arxiv.org/abs/2308.08479

DoGHardNetExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1, affnet=False)

Bases: LocalFeatureExtractor

Difference of Gaussians keypoints with HardNet descriptors. Match with MatchLightGlue(features="doghardnet"), or MatchLightGlue(features="dog_affnet_hardnet") for affnet=True.

  • Paper (HardNet): Working hard to know your neighbor's margins: Local descriptor learning loss
  • Link: https://arxiv.org/abs/1705.10872

KeyNetAffNetHardNetExtractor(detection_threshold=0.0, force_num_keypoints=True, max_num_keypoints=256, device=None, num_workers=1, cache_path=None, config_tag=None, batch_size=1)

Bases: LocalFeatureExtractor

KeyNet keypoints with AffNet shapes and HardNet descriptors. Match with MatchLightGlue(features="keynet_affnet_hardnet").

  • Paper (KeyNet): Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters
  • Link: https://arxiv.org/abs/1904.00889

features.memory

DataToMemory

Bases: FeatureExtractor

Loads the dataset into memory for quicker access

Ideal for LOFTR, which operates directly on images, because loading images from storage can become a bottleneck when matching all query-database pairs, requiring n_query x n_database image loads.

__call__(dataset)

Loads data from input dataset into array and returns them as a new FeatureDataset.