Matching based similarity scores
The similarity.pairwise module provides tools and methods for calculating pairwise matching similarity scores. At its core, the MatchPairs base class offers pairwise matching with support for batch processing, making it essential for neural network-based matching.
Specific implementations of of MatchPairs are:
MatchLightGlue: This class uses the LightGlue model, a lightweight neural matching that uses extracted SIFT, DISK, ALIKED or SuperPoint keypoints and descriptors.MatchLOFTR: This class uses the LOFTR (Local Feature TRansformer) model, which performs descriptor-free matching using directly pair of images.
Outputs from the matchers, such as confidence scores for local matches and keypoints, are processed using collectors from similarity.pairwise.collectors. In particular, the CollectCounts collector calculates matching similarity scores by counting significant matches based on given confidence thresholds.
Performance consideration: For the best performance LOFTR requires higher image resolution such as 512x512. However, this makes it about 5x slower than Lighglue with default values (256 keypoints and descriptors per image).
similarity.pairwise.base
MatchPairs(batch_size=128, num_workers=0, tqdm_silent=False, collector=None)
Base class for matching pairs from two datasets.
Any child class needs to implement get_matches method that implements processing of pair batches.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
batch_size |
int
|
Number of pairs processed in one batch. |
128
|
num_workers |
int
|
Number of workers used for data loading. |
0
|
tqdm_silent |
bool
|
If True, progress bar is disabled. |
False
|
collector |
Collector object used for storing results. By default, CollectCounts(thresholds=[0.5]) is used. |
None
|
__call__(dataset0, dataset1, pairs=None)
Match pairs of features from two feature datasets. Output for each pair is stored and processed using the collector.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset0 |
FeatureDataset
|
First dataset (e.g. query). |
required |
dataset1 |
FeatureDataset
|
Second dataset (e.g. database). |
required |
pairs |
ndarray | None
|
Numpy array with pairs of indexes. If None, all pairs are used. |
None
|
Returns:
| Type | Description |
|---|---|
|
Exact output is determined by the used collector. |
get_matches(batch)
Process batch and get matches of pairs for the batch. Implemented in child classes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
batch |
tuple
|
4-tuple with indexes and data from PairDataset. |
required |
Returns:
| Type | Description |
|---|---|
|
list of standartized dictionaries with keys: idx0, idx1, score, kpts0, kpts1. Length of list is equal to batch size. |
similarity.pairwise.lightglue
MatchLightGlue(features, init_threshold=0.1, device=None, **kwargs)
Bases: MatchPairs
Implements matching using LightGlue model correspondences. Introduced in: "LightGlue: Local Feature Matching at Light Speed"
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
features |
str
|
Features used for matching. Options: 'sift', 'superpoint', 'aliked', 'disk'. Must match extracted features from the dataset. |
required |
init_threshold |
float
|
Keep matches only over this threshold. Matches with lower values are not passed to the collector. |
0.1
|
device |
str
|
Device used for inference. Defaults to None. |
None
|
similarity.pairwise.loftr
MatchLOFTR(pretrained='outdoor', init_threshold=0.2, device=None, apply_fine=False, **kwargs)
Bases: MatchPairs
Implements matching pairs using LoFTR model correspondences. Introduced in: "LoFTR: Detector-Free Local Feature Matching with Transformers"
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pretrained |
str
|
LOFTR model used. |
'outdoor'
|
device |
str | None
|
Specifies device used for the inference. |
None
|
init_threshold |
float
|
Keep matches only over this threshold. |
0.2
|
apply_fine |
bool
|
Use LoFTR fine refinement of keypoints locations. Has no effect on confidence, but is faster without fine refinement. False by default. |
False
|
similarity.pairwise.collectors
CollectAll(**kwargs)
Collect the results without additional processing. Collected data is list of matcher results for each pair. Usefull for keypoint visualizations.
CollectCounts(grid_dtype='float16', thresholds=(0.5), **kwargs)
Collect count of significant matches given confidence thresholds. Output is stored in [n_query x n_database] grid.
If multiple thresholds are provided, returns a dictionary with each threshold as a key and the corresponding grid as value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
grid_dtype |
str
|
Data type of the output grid. |
'float16'
|
thresholds |
tuple
|
Confidence thresholds for counting. |
(0.5)
|
CollectCountsRansac(grid_dtype='float16', ransacReprojThreshold=1.0, method=cv2.USAC_MAGSAC, confidence=0.999, maxIters=100, **kwargs)
Bases: CollectCounts
Collect count of RANSAC inliers of fundamental matrix estimate. Output is stored in [n_query x n_database] grid.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
grid_dtype |
str
|
Data type of the output grid. |
'float16'
|
ransacReprojThreshold |
float
|
OpenCV RANSAC reprojection threshold. |
1.0
|
method |
Any
|
OpenCV RANSAC method. |
USAC_MAGSAC
|
confidence |
float
|
OpenCV RANSAC confidence. |
0.999
|
maxIters |
float
|
OpenCV RANSAC max iterations. |
100
|
Examples
Example - SuperGlue matching
Matches all pairs using the SuperGlue matcher with SuperPoint features and calculates similarity scores based on the count of significant matches at confidence thresholds of 0.25, 0.5, and 0.75.
import torchvision.transforms as T
from wildlife_tools.features.local import SuperPointExtractor
from wildlife_tools.similarity.pairwise.lightglue import MatchLightGlue
from wildlife_tools.similarity.pairwise.collectors import CollectCounts
from wildlife_tools.data.dataset import WildlifeDataset
transform = T.Compose([T.Resize([224, 224]), T.ToTensor()])
dataset_query = WildlifeDataset(metadata_query, transform=transform)
dataset_database = WildlifeDataset(metadata_database, transform=transform)
extractor = SuperPointExtractor()
matcher = MatchLightGlue(features='superpoint', collector=CollectCounts(thresholds=[0.25, 0.5, 0.75]))
output = matcher(extractor(dataset_query), extractor(dataset_database))
Example - LOFTR matching
Matches all pairs using the LOFTR matcher and calculates similarity scores based on the count of significant matches at confidence thresholds of 0.25, 0.5, and 0.75. Note that LOFTR operates directly on image pairs and requires no feature extraction.
from wildlife_tools.similarity.pairwise.loftr import MatchLOFTR
from wildlife_tools.similarity.pairwise.collectors import CollectCounts
from wildlife_tools.data.dataset import WildlifeDataset
transform = T.Compose([T.Resize([224, 224]), T.ToTensor()])
dataset_query = WildlifeDataset(metadata_query, transform=transform)
dataset_database = WildlifeDataset(metadata_database, transform=transform)
extractor = SuperPointExtractor()
matcher = MatchLightGlue(features='superpoint', collector=CollectCounts(thresholds=[0.25, 0.5, 0.75]))
output = matcher(dataset_query, dataset_database)