Similarity

Similarity subclasses provides a standardized method for calculating the similarity between two sets of features extracted by the same feature extractor.

Once instantiated, the Similarity object functions as a callable, expecting two arguments: query_features and database_features. The specific input type and shape depend on the chosen feature extractor.

The output of all Similarity objects is a dictionary. The keys represent properties of the similarity, such as the used threshold, while the values contain arrays with the shape n_query x n_database. In other words, each row in the array corresponds to one query image.

CosineSimilarity

Calculates cosine similarity between query and database features. Query should be 2D array with shape n_query x dim_embeddings and database should be 2D array with shapen_database x dim_embeddings. Output is dictionary with cosine key and value that contains 2D array with cosine similarities.

Example

In this context, query and database are 2D arrays of deep features.

from wildlife_tools.similarity import CosineSimilarity

similarity = CosineSimilarity()
sim = similarity(query, database)

Reference

Calculates cosine similarity, equivalently to sklearn.metrics.pairwise.cosine_similarity

Returns:

Name	Type	Description
dict		dictionary with cosine key. Value is 2D array with cosine similarity.

Source code in similarity/cosine.py

class CosineSimilarity(Similarity):
    '''
    Calculates cosine similarity, equivalently to `sklearn.metrics.pairwise.cosine_similarity`

    Returns:
        dict: dictionary with `cosine` key. Value is 2D array with cosine similarity.

    '''

    def __call__(self, query, database):
        return {'cosine': self.cosine_similarity(query, database) }

    def cosine_similarity(self, a, b):
        a, b = torch.tensor(a), torch.tensor(b)
        similarity = torch.matmul(F.normalize(a), F.normalize(b).T)
        return similarity.numpy()

MatchDescriptors

Calculates similarity between query and database as number of descriptor correspondences after filtering with ratio test. For each descriptor in query, nearest two descriptors in database are found. If their ratio of their distance is lesser than treshold, they are considered as valid correspondence. Similarity is calculated as sum of all correspondences.

Output is dictionary with key for each treshold. Values contains 2D array with number of correspondences.

Example

In this context, query and database are sets of SIFT descriptors with descriptor_dim = 128.

from wildlife_tools.similarity import MatchDescriptors

similarity = MatchDescriptors(descriptor_dim=128, thresholds=[0.8])
sim = similarity(query, database)

Reference

Calculate similarity between query and database as number of descriptors correspondences after filtering with Low ratio test.

Parameters:

Name	Type	Description	Default
descriptor_dim	int	dimensionality of descriptors. 128 for SIFT, 256 for SuperPoint.	required
thresholds	tuple[float]	iterable with ratio test tresholds. Should be in [0, 1] interval.	(0.5)
device	str	Specifies device used for nearest neigbour search.	'cpu'

Returns:

Name	Type	Description
dict		dictionary with key for each treshold. Values are 2D array with number of correspondences.

Source code in similarity/descriptors.py

class MatchDescriptors(Similarity):
    '''
    Calculate similarity between query and database as number of descriptors correspondences 
    after filtering with Low ratio test.

    Args:
        descriptor_dim: dimensionality of descriptors. 128 for SIFT, 256 for SuperPoint.
        thresholds: iterable with ratio test tresholds. Should be in [0, 1] interval.
        device: Specifies device used for nearest neigbour search.

    Returns:
        dict: dictionary with key for each treshold. Values are 2D array with number of correspondences.
    '''

    def __init__(
        self,
        descriptor_dim: int,
        thresholds: tuple[float] = (0.5, ),
        device: str = 'cpu',
    ):

        self.descriptor_dim = descriptor_dim
        self.thresholds = thresholds
        self.device = device


    def __call__(self, query, database):
        iterator = itertools.product(enumerate(query), enumerate(database))
        iterator_size = len(query)*len(database)
        similarities = {t: np.full((len(query), len(database)), np.nan, dtype=np.float16) for t in self.thresholds}

        index = get_faiss_index(d=self.descriptor_dim, device=self.device)
        for pair in tqdm(iterator, total=iterator_size, mininterval=1, ncols=100):
            (q_idx, q_data), (d_idx, d_data) = pair

            if (q_data is None) or (d_data is None):
                for t in self.thresholds:
                    similarities[t][q_idx, d_idx] = 0

            else:
                index.reset()
                index.add(q_data)
                score, idx = index.search(d_data, k=2)
                with np.errstate(divide='ignore'):
                    ratio = score[:, 0] / score[:, 1]
                for t in self.thresholds:
                    similarities[t][q_idx, d_idx] = np.sum(ratio < t)

        return similarities

MatchLOFTR

Uses LOFTR matching capabilities to calculate number of correspondences. Does not use descriptors and takes pair of greyscale image tensors instead. LOFTR implementation from Kornia is used.

Output is dictionary with key for each confidence treshold. Values contains corresponding 2D array with cosine similarities.

Reference

Calculate similarity between query and database as number of descriptors correspondences after filtering with Low ratio test.

Parameters:

Name	Type	Description	Default
pretrained	str	LOFTR model used. outdoor or indoor.	'outdoor'
thresholds	tuple[float]	Iterable with confidence tresholds. Should be in [0, 1] interval.	(0.99)
batch_size	int	Batch size used for the inference.	128
device	str	Specifies device used for the inference.	'cuda'
silent	bool	disable tqdm bar.	False

Returns:

Name	Type	Description
dict		dictionary with key for each treshold. Values are 2D array with number of correspondences.

Source code in similarity/loftr.py

class MatchLOFTR(Similarity):
    '''
    Calculate similarity between query and database as number of descriptors correspondences 
    after filtering with Low ratio test.

    Args:
        pretrained: LOFTR model used. `outdoor` or `indoor`.
        thresholds: Iterable with confidence tresholds. Should be in [0, 1] interval.
        batch_size: Batch size used for the inference.
        device: Specifies device used for the inference.
        silent: disable tqdm bar.

    Returns:
        dict: dictionary with key for each treshold. Values are 2D array with number of correspondences.
    '''

    def __init__(
        self,
        pretrained: str ='outdoor',
        thresholds: tuple[float] = (0.99, ),
        batch_size: int = 128,
        device: str ='cuda',
        silent: bool = False,
    ):
        self.device = device
        self.matcher = KF.LoFTR(pretrained=pretrained).to(device)
        self.thresholds = thresholds
        self.batch_size = batch_size
        self.silent = silent


    def __call__(self, query, database):
        iterator = batched(itertools.product(enumerate(query), enumerate(database)), self.batch_size)
        iterator_size = int(np.ceil(len(query)*len(database) / self.batch_size))
        similarities = {t: np.full((len(query), len(database)), np.nan, dtype=np.float16) for t in self.thresholds}

        for pair_batch in tqdm(iterator, total=iterator_size, mininterval=1, ncols=100, disable=self.silent):
            q, d = zip(*pair_batch)
            q_idx, q_data = list(zip(*q))
            d_idx, d_data = list(zip(*d))
            input_dict = {
                "image0": torch.stack(q_data).to(self.device),
                "image1": torch.stack(d_data).to(self.device),
            }
            with torch.inference_mode():
                correspondences = self.matcher(input_dict)

            batch_idx = correspondences['batch_indexes'].cpu().numpy()
            confidence = correspondences['confidence'].cpu().numpy()
            for t in self.thresholds:
                series = pd.Series(confidence > t)
                for j, group in series.groupby(batch_idx):
                    similarities[t][q_idx[j], d_idx[j]] = group.sum()
        return similarities