Skip to content

Validation leakage due to early stopping #25

@maryjis

Description

@maryjis

The validation set is used both for early stopping and for reporting/comparing results.
This causes validation leakage, since the model is effectively tuned on the validation data.

As a result, the reported validation performance is optimistically biased and likely higher than true generalization performance.

Suggestion:
Use a separate test set for final evaluation, or apply nested cross-validation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions