irspack.evaluation.Evaluator

class irspack.evaluation.Evaluator(ground_truth, offset=0, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=128)

Bases: object

Evaluates recommenders’ performance against validation set.

Parameters:

• ground_truth (Union[scipy.sparse.csr_matrix, scipy.sparse.csc_matrix]) – The ground-truth.

• offset (int) – Where the validation target user block begins. Often the validation set is defined for a subset of users. When offset is not 0, we assume that the users with validation ground truth corresponds to X_train[offset:] where X_train is the matrix feeded into the recommender class. Defaults to 0.

• cutoff (int, optional) – Controls the default number of recommendation. When the evaluator is used for parameter tuning, this cutoff value will be used. Defaults to 10.

• target_metric (str, optional) – Specifies the target metric when this evaluator is used for parameter tuning. Defaults to “ndcg”.

• recommendable_items (Optional[List[int]], optional) – Global recommendable items. Defaults to None. If this parameter is not None, evaluator will be concentrating on the recommender’s score output for these recommendable_items, and compute the ranking performance within this subset.

• per_user_recommendable_items (Union[None, List[List[int]], csr_matrix, csc_matrix]) – Similar to recommendable_items, but this time the recommendable items can vary among users. If a sparse matrix is given, its nonzero indices are regarded as the list of recommendable items. Defaults to None.

• masked_interactions (Optional[csr_matrix]) – If set, this matrix masks the score output of recommender model where it is non-zero. If none, the mask will be the training matrix itself owned by the recommender.

• n_threads (Optional[int]) – Specifies the Number of threads to sort scores and compute the evaluation metrics. If None, the environment variable "IRSPACK_NUM_THREADS_DEFAULT"` will be looked up, and if the variable is not set, it will be set to ``os.cpu_count(). Defaults to None.

• recall_with_cutoff (bool, optional) –

  This affects the definition of recall. If True, for each user, recall will be computed as

  \[\frac{N_{\text{hit}}}{\min(\text{cutoff}, N_{\text{ground truth}})}\]

  If False, this will be

  \[\frac{N_{\text{hit}}}{N_{\text{ground truth}}}\]

• mb_size (int, optional) – The rows of chunked user score. Defaults to 1024.

__init__(ground_truth, offset=0, cutoff=10, target_metric='ndcg', recommendable_items=None, per_user_recommendable_items=None, masked_interactions=None, n_threads=None, recall_with_cutoff=False, mb_size=128)

Parameters:

• ground_truth (Union[csr_matrix, csc_matrix]) –

• offset (int) –

• cutoff (int) –

• target_metric (str) –

• recommendable_items (Optional[List[int]]) –

• per_user_recommendable_items (Union[None, List[List[int]], csr_matrix, csc_matrix]) –

• masked_interactions (Optional[Union[csr_matrix, csc_matrix]]) –

• n_threads (Optional[int]) –

• recall_with_cutoff (bool) –

• mb_size (int) –

Return type:

None

Methods

__init__(ground_truth[, offset, cutoff, ...])
get_score(model)	Compute the score with the cutoff being self.cutoff.
get_scores(model, cutoffs)	Compute the score with the specified cutoffs.
get_target_score(model)	Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Attributes

n_users
n_items
masked_interactions

get_score(model)

Compute the score with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated recommender.

Returns:

metric values.

Return type:

Dict[str, float]

get_scores(model, cutoffs)

Compute the score with the specified cutoffs.

Parameters:

• model (BaseRecommender) – The evaluated recommender.

• cutoffs (List[int]) – for each value in cutoff, the class computes the metric values.

Returns:

The Resulting metric values. This time, the result will look like {"ndcg@20": 0.35, "map@20": 0.2, ...}.

Return type:

Dict[str, float]

get_target_score(model)

Compute the optimization target score (self.target_metric) with the cutoff being self.cutoff.

Parameters:

model (BaseRecommender) – The evaluated model.

Returns:

The metric value.

Return type:

float