irspack.evaluation.EvaluatorWithColdUser

class irspack.evaluation.EvaluatorWithColdUser(input_interaction, ground_truth, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=1024)[source]

Bases: Evaluator

Evaluates recommenders’ performance against cold (unseen) users.

Parameters:

  • input_interaction (Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]) – The cold-users’ known interaction with the items.

  • ground_truth (Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]) – The held-out ground-truth.

  • offset (int) – Where the validation target user block begins. Often the validation set is defined for a subset of users. When offset is not 0, we assume that the users with validation ground truth corresponds to X_train[offset:] where X_train is the matrix feeded into the recommender class.

  • cutoff (int, optional) – Controls the number of recommendation. Defaults to 10.

  • target_metric (str, optional) – Optimization target metric. Defaults to “ndcg”.

  • recommendable_items (Optional[List[int]], optional) – Global recommendable items. Defaults to None. If this parameter is not None, evaluator will be concentrating on the recommender’s score output for these recommendable_items, and compute the ranking performance within this subset.

  • per_user_recommendable_items (Optional[List[List[int]]], optional) – Similar to recommendable_items, but this time the recommendable items can vary among users. Defaults to None.

  • masked_interactions (Optional[Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]], optional) – If set, this matrix masks the score output of recommender model where it is non-zero. If none, the mask will be the training matrix (input_interaction) it self.

  • n_threads (int, optional) – Specifies the Number of threads to sort scores and compute the evaluation metrics. If None, the environment variable "IRSPACK_NUM_THREADS_DEFAULT" will be looked up, and if the variable is not set, it will be set to os.cpu_count(). Defaults to None.

  • recall_with_cutoff (bool, optional) –

    This affects the definition of recall. If True, for each user, recall will be evaluated by

    \[\frac{N_{\text{hit}}}{\min( \text{cutoff}, N_{\text{ground truth}} )}\]

    If False, this will be

    \[\frac{N_{\text{hit}}}{N_{\text{ground truth}}}\]

  • mb_size (int, optional) – The rows of chunked user score. Defaults to 1024.

__init__(input_interaction, ground_truth, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=1024)[source]
Parameters:

  • input_interaction (Union[csr_matrix, csc_matrix]) –

  • ground_truth (Union[csr_matrix, csc_matrix]) –

  • cutoff (int) –

  • target_metric (str) –

  • recommendable_items (Optional[List[int]]) –

  • per_user_recommendable_items (Union[None, List[List[int]], csr_matrix, csc_matrix]) –

  • masked_interactions (Optional[Union[csr_matrix, csc_matrix]]) –

  • n_threads (Optional[int]) –

  • recall_with_cutoff (bool) –

  • mb_size (int) –

Methods

__init__(input_interaction, ground_truth[, ...])

get_score(model)

Compute the score with the cutoff being self.cutoff.

get_scores(model, cutoffs)

Compute the score with the specified cutoffs.

get_target_score(model)

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Attributes

get_score(model)

Compute the score with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated recommender.

Returns:

metric values.

Return type:

Dict[str, float]

get_scores(model, cutoffs)

Compute the score with the specified cutoffs.

Parameters:

  • model (BaseRecommender) – The evaluated recommender.

  • cutoffs (List[int]) – for each value in cutoff, the class computes the metric values.

Returns:

The Resulting metric values. This time, the result will look like {"ndcg@20": 0.35, "map@20": 0.2, ...}.

Return type:

Dict[str, float]

get_target_score(model)

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated model.

Returns:

The metric value.

Return type:

float