Evaluating Results#

mockdock provides a standalone evaluation engine, MDEvaluator, to compute standardized metrics across runs without requiring docking engines or GPU resources.

This page explains how to evaluate individual experiment runs, interpret the full suite of metrics, and aggregate multi-model benchmarking experiments.

Single Run Evaluation#

Tip

results.csv is generated automatically by MDOracle during scoring and saved in oracle.run_dir / "results.csv".

Python API#

from mockdock import MDEvaluator

# Initialize evaluator with target benchmark configuration
evaluator = MDEvaluator("CHK1")

# Compute metrics dictionary from a session CSV
metrics = evaluator.compute_metrics("my_run/results.csv")

# Print key summary metrics
print(f"Top 10 Mean Reward: {metrics['avg_top_10']:.3f}")
print(f"Filtered Top 10 Reward: {metrics['avg_top_10_filtered']:.3f}")
print(f"MedChem Pass Fraction: {metrics['fraction_medchem_pass']:.1%}")
print(f"Internal Diversity: {metrics['internal_diversity']:.3f}")

# Save metrics to JSON
evaluator.compute_metrics("my_run/results.csv", output_path="my_run/eval_metrics.json")

Command-Line Interface (CLI)#

You can evaluate any session CSV directly from the terminal:

python -m mockdock.evaluator my_run/results.csv \
  --benchmark CHK1 \
  --output my_run/eval_metrics.json

Standard Metric Suite#

MDEvaluator computes 22 standardized metrics across four main categories:

1. Generative Quality & Diversity#

Evaluates the structural validity, deduplication, and chemical variety of generated candidates.

Metric	Description
`validity`	Fraction of generated SMILES strings that parse into chemically valid RDKit molecules.
`uniqueness`	Fraction of valid molecules that are structurally unique.
`internal_diversity`	Average pairwise Tanimoto distance (using Morgan fingerprints, radius=2) among unique valid molecules.
`scaffold_diversity`	Fraction of unique Bemis-Murcko scaffolds among unique valid molecules.

2. Medicinal Chemistry & Drug-Likeness#

Applies medicinal chemistry filters and property distributions to assess clinical/lead viability.

Metric	Description
`mean_qed`	Mean Quantitative Estimate of Drug-likeness (QED).
`mean_sa`	Mean Synthetic Accessibility score (1 = easily synthesizable, 10 = difficult).
`fraction_lipinski`	Fraction of unique valid molecules satisfying all 4 Lipinski Rule-of-5 criteria (MW ≤ 500, LogP ≤ 5, HBD ≤ 5, HBA ≤ 10).
`fraction_pains_free`	Fraction of unique valid molecules with zero PAINS (Pan-Assay Interference) alerts.
`fraction_bms_free`	Fraction of unique valid molecules with zero BMS (Bristol Myers Squibb) structural alerts.
`fraction_medchem_pass`	Fraction passing all structural alert filters (PAINS, BMS) and property rules (Lipinski, QED ≥ 0.5, SA ≤ 4.0).

3. Novelty & Fragment Constraints#

Measures whether generated candidates retain the required pharmacophore while exploring novel chemical space.

Metric	Description
`fragment_incorporation`	Fraction of unique valid molecules containing the required target fragment substructure.
`novelty`	Fraction of unique valid molecules not present in the starting seed dataset.
`nonidenticality`	Fraction of generated molecules non-identical to their prompt/input parent.
`effective_novelty`	Fraction of molecules that are both novel and non-identical.
`snn`	Average maximum Tanimoto similarity to the nearest neighbor in the initial seed dataset.
`effective_yield_rate`	Fraction of all generated molecules that are valid, unique, contain the fragment, and are novel.

4. Optimization & Docking Performance#

Evaluates target-specific binding affinity optimization and oracle sample efficiency.

Metric	Description
`avg_top_1` / `10` / `100`	Mean bounded reward score of the top-1, top-10, and top-100 scoring molecules in the run.
`avg_top_1_norm` / `10` / `100`	Mean unbounded normalized docking score of the top-1, top-10, and top-100 molecules.
`avg_top_10_filtered`	Mean reward score of the top-10 molecules that satisfy all MedChem filters.
`auc_top_10`	Area under the running top-10 reward trajectory across cumulative oracle calls (normalized to [0.0, 1.0]).
`auc_top_10_filtered`	AUC of the running top-10 curve restricted to MedChem-passing compounds.
`valid_pose_rate`	Fraction of docked compounds whose docked fragment overlays the crystal structure within the RMSD threshold (≤ 2.0 Å).
`oracle_efficiency_80`	Number of oracle calls required to reach 80% of the final top-10 reward score (lower indicates faster learning).
`oracle_efficiency_100`	Number of oracle calls required to reach a top-10 reward of 1.0.

Multi-Experiment Aggregation & Plotting#

When running benchmark suites across multiple models, targets, and random seeds, you can aggregate all results and produce publication-ready comparison figures using scripts/analysis/analyze_experiments.py.

python scripts/analysis/analyze_experiments.py \
  --exps-dir exps/ \
  --output-dir analysis_results/

Generated Summary Artifacts:#

metrics_summary.csv: Per-model, per-benchmark detailed evaluation table.
metrics_summary_macro.csv: Macro-averaged metrics across all 7 benchmark targets.
Publication Figures: * Figure 1: Generative Quality & Diversity metrics across models. * Figure 2: Optimization & Docking Performance (Top-10 reward, MedChem-filtered Top-10, AUC). * Figure 3: Medicinal chemistry distributions (QED, SA, Lipinski, structural alert pass rates). * Figure 4: Cumulative learning trajectory curves over oracle calls.