Evaluation

RecDistillery evaluates teachers and students with top-k recommendation metrics. The common metric implementation lives in recdistill.evaluation and is used by the teacher and student evaluation scripts.

Teacher Evaluation

python scripts/recdistill/evaluate_teacher.py \
  --teacher-path results/teacher/<run>/artifacts/<teacher>_best.teacher

Student Evaluation

python scripts/recdistill/evaluate_students.py \
  --student-path results/student/<run>/artifacts/<student>_best.student

Distilled students use the same evaluator:

python scripts/recdistill/evaluate_students.py \
  --student-path results/recdistill/<run>/artifacts/<student>_best.distilled_student

Metrics

The evaluation module computes ranking metrics such as precision, recall, NDCG, and hit ratio over held-out validation or test interactions.

Function	Purpose
evaluate_teacher	Evaluates a TeacherState on validation and test splits.
evaluate_student	Evaluates a trained student model with the same ranking protocol.
evaluate_embeddings	Shared embedding/scorer evaluator used by teacher and student APIs.

Evaluate a serialized or imported teacher on validation/test splits.

The teacher may expose either user/item embeddings or a scorer-only representation such as a precomputed score matrix or top-k predictions. Training interactions are masked before ranking so the metrics are computed only over unseen candidate items.

Parameters:

Name	Type	Description	Default
teacher_state	TeacherState	Runtime teacher representation loaded from a .teacher artifact or an import adapter.	required
train_seen	dict[int, set[int]]	Mapping from user index to training items that must be removed from the ranked candidate set.	required
val_gt	dict[int, set[int]]	Validation ground-truth items keyed by user index.	required
test_gt	dict[int, set[int]]	Test ground-truth items keyed by user index.	required
top_k	int	Recommendation cutoff used by precision, recall, NDCG and hit ratio.	required
batch_size	int	Number of users evaluated per embedding-ranking batch.	required
device	device	Torch device used for score computation.	required
eval_val_only	bool	When True, skip the test split and return validation metrics only.	False

Returns:

Type	Description
dict[str, dict[str, float] | int]	A dictionary with split metrics and train-leakage counters. The metric
dict[str, dict[str, float] | int]	dictionaries contain users, precision, recall, ndcg, and hr.

Source code in recdistill/evaluation.py

def evaluate_teacher(
    teacher_state: TeacherState,
    train_seen: dict[int, set[int]],
    val_gt: dict[int, set[int]],
    test_gt: dict[int, set[int]],
    top_k: int,
    batch_size: int,
    device: torch.device,
    eval_val_only: bool = False,
) -> dict[str, dict[str, float] | int]:
    """Evaluate a serialized or imported teacher on validation/test splits.

    The teacher may expose either user/item embeddings or a scorer-only
    representation such as a precomputed score matrix or top-k predictions.
    Training interactions are masked before ranking so the metrics are computed
    only over unseen candidate items.

    Args:
        teacher_state: Runtime teacher representation loaded from a `.teacher`
            artifact or an import adapter.
        train_seen: Mapping from user index to training items that must be
            removed from the ranked candidate set.
        val_gt: Validation ground-truth items keyed by user index.
        test_gt: Test ground-truth items keyed by user index.
        top_k: Recommendation cutoff used by precision, recall, NDCG and hit
            ratio.
        batch_size: Number of users evaluated per embedding-ranking batch.
        device: Torch device used for score computation.
        eval_val_only: When `True`, skip the test split and return validation
            metrics only.

    Returns:
        A dictionary with split metrics and train-leakage counters. The metric
        dictionaries contain `users`, `precision`, `recall`, `ndcg`, and `hr`.
    """
    val_metrics, val_leaks = evaluate_embeddings(
        user_embeddings=teacher_state.user_embeddings,
        item_embeddings=teacher_state.item_embeddings,
        train_seen=train_seen,
        ground_truth=val_gt,
        top_k=top_k,
        batch_size=batch_size,
        device=device,
        scorer=teacher_state.scorer,
    )
    if eval_val_only:
        return {"val": val_metrics, "leaked_users_val": val_leaks}
    test_metrics, test_leaks = evaluate_embeddings(
        user_embeddings=teacher_state.user_embeddings,
        item_embeddings=teacher_state.item_embeddings,
        train_seen=train_seen,
        ground_truth=test_gt,
        top_k=top_k,
        batch_size=batch_size,
        device=device,
        scorer=teacher_state.scorer,
    )
    return {
        "val": val_metrics,
        "test": test_metrics,
        "leaked_users_val": val_leaks,
        "leaked_users_test": test_leaks,
    }

Evaluate a trained student model on validation and test splits.

The student must expose get_all_user_embeddings and get_all_item_embeddings. If it also implements score_items_for_user, that scorer is used for ranking.

Source code in recdistill/evaluation.py

def evaluate_student(
    model: torch.nn.Module,
    train_seen: dict[int, set[int]],
    val_gt: dict[int, set[int]],
    test_gt: dict[int, set[int]],
    top_k: int,
    batch_size: int,
    device: torch.device,
    eval_val_only: bool = False,
) -> dict[str, dict[str, float] | int]:
    """Evaluate a trained student model on validation and test splits.

    The student must expose `get_all_user_embeddings` and
    `get_all_item_embeddings`. If it also implements `score_items_for_user`,
    that scorer is used for ranking.
    """
    model.eval()
    user_embeddings = model.get_all_user_embeddings().detach()
    item_embeddings = model.get_all_item_embeddings().detach()
    scorer = model if hasattr(model, "score_items_for_user") else None
    val_metrics, val_leaks = evaluate_embeddings(
        user_embeddings=user_embeddings,
        item_embeddings=item_embeddings,
        train_seen=train_seen,
        ground_truth=val_gt,
        top_k=top_k,
        batch_size=batch_size,
        device=device,
        scorer=scorer,
    )
    if eval_val_only:
        return {"val": val_metrics, "leaked_users_val": val_leaks}
    test_metrics, test_leaks = evaluate_embeddings(
        user_embeddings=user_embeddings,
        item_embeddings=item_embeddings,
        train_seen=train_seen,
        ground_truth=test_gt,
        top_k=top_k,
        batch_size=batch_size,
        device=device,
        scorer=scorer,
    )
    return {
        "val": val_metrics,
        "test": test_metrics,
        "leaked_users_val": val_leaks,
        "leaked_users_test": test_leaks,
    }

Evaluate top-k recommendations from embeddings or a scorer.

Parameters:

Name	Type	Description	Default
user_embeddings	Tensor	User embedding matrix, or None when scorer can score users directly.	required
item_embeddings	Tensor	Item embedding matrix, or None for scorer-only teachers.	required
train_seen	dict[int, set[int]]	Training items keyed by user. These items are masked before ranking.	required
ground_truth	dict[int, set[int]]	Held-out target items keyed by user.	required
top_k	int	Recommendation cutoff.	required
batch_size	int	Number of users per embedding-ranking batch.	required
device	device	Torch device used for score computation.	required
scorer	TeacherScorer | None	Optional object implementing score_items_for_user.	None

Returns:

Type	Description
dict[str, float]	A pair containing the metric dictionary and the number of users whose
int	raw top-k list still contained a training item before final filtering.

Source code in recdistill/evaluation.py

def evaluate_embeddings(
    user_embeddings: torch.Tensor,
    item_embeddings: torch.Tensor,
    train_seen: dict[int, set[int]],
    ground_truth: dict[int, set[int]],
    top_k: int,
    batch_size: int,
    device: torch.device,
    scorer: TeacherScorer | None = None,
) -> tuple[dict[str, float], int]:
    """Evaluate top-k recommendations from embeddings or a scorer.

    Args:
        user_embeddings: User embedding matrix, or `None` when `scorer` can
            score users directly.
        item_embeddings: Item embedding matrix, or `None` for scorer-only
            teachers.
        train_seen: Training items keyed by user. These items are masked before
            ranking.
        ground_truth: Held-out target items keyed by user.
        top_k: Recommendation cutoff.
        batch_size: Number of users per embedding-ranking batch.
        device: Torch device used for score computation.
        scorer: Optional object implementing `score_items_for_user`.

    Returns:
        A pair containing the metric dictionary and the number of users whose
        raw top-k list still contained a training item before final filtering.
    """
    eval_users = sorted(user for user, items in ground_truth.items() if items)
    if not eval_users:
        return _metrics_at_k({}, ground_truth, top_k), 0

    recommendations, leaked_users = _build_topk_recommendations(
        user_embeddings=user_embeddings,
        item_embeddings=item_embeddings,
        users=eval_users,
        train_seen=train_seen,
        top_k=top_k,
        batch_size=batch_size,
        device=device,
        scorer=scorer,
    )
    return _metrics_at_k(recommendations, ground_truth, top_k), leaked_users