Skip to content

OptimizationStatistics

Statistical utilities for analyzing optimization results.

Overview

OptimizationStatistics provides statistical functions for analyzing and comparing backtest results: confidence intervals, significance tests, correlations, and reporting.

#include "lrvx/backtest/optimization_stats.h"

Helper Functions (detail namespace)

namespace detail {
  double mean(const std::vector<double>& data);
  double variance(const std::vector<double>& data);
  double stddev(const std::vector<double>& data);
}

Basic statistical functions. Returns 0.0 for empty data.

extractMetric

template <typename ParamsT>
std::vector<double> extractMetric(
    const std::vector<OptimizationResult<ParamsT>>& results,
    RankMetric metric);

Extract specified metric values from all results.

Example: 

auto sharpes = extractMetric(results, RankMetric::SharpeRatio);
double avgSharpe = detail::mean(sharpes);

Class Definition

template <typename ParamsT, typename GridT>
class OptimizationStatistics {
public:
  struct ConfidenceInterval {
    double lower;
    double median;
    double upper;
  };

  static double permutationTest(
      const std::vector<double>& group1,
      const std::vector<double>& group2,
      size_t numPermutations = 10000,
      std::uint64_t seed = 42u);

  static double correlation(
      const std::vector<double>& x,
      const std::vector<double>& y);

  static ConfidenceInterval bootstrapCI(
      const std::vector<double>& data,
      double confidenceLevel = 0.95,
      size_t numSamples = 10000,
      std::uint64_t seed = 42u);

  static void printSummary(
      const std::vector<OptimizationResult<ParamsT>>& results);

  static bool generateReport(
      const std::vector<OptimizationResult<ParamsT>>& results,
      const std::filesystem::path& outputPath);
};

Methods

permutationTest

static double permutationTest(
    const std::vector<double>& group1,
    const std::vector<double>& group2,
    size_t numPermutations = 10000,
    std::uint64_t seed = 42u);

Two-sample permutation test for comparing group means. Returns a p-value, always strictly greater than zero: the p-value is (extreme + 1) / (numPermutations + 1), the standard add-one correction (the observed arrangement is itself one of the possible outcomes under the null).

The resample is seeded (default 42, matching whitesRealityCheck), so the same inputs always return the same p-value. Pass a different seed to draw an independent resample.

Example: 

// Compare two parameter configurations
std::vector<double> configA = {1.2, 1.5, 1.3};
std::vector<double> configB = {0.8, 0.9, 0.7};
double pValue = Stats::permutationTest(configA, configB);
// pValue < 0.05 suggests significant difference

correlation

static double correlation(
    const std::vector<double>& x,
    const std::vector<double>& y);

Pearson correlation coefficient between two vectors. Returns value in [-1, 1].

Example: 

auto sharpes = extractMetric(results, RankMetric::SharpeRatio);
auto returns = extractMetric(results, RankMetric::TotalReturn);
double r = Stats::correlation(sharpes, returns);

bootstrapCI

static ConfidenceInterval bootstrapCI(
    const std::vector<double>& data,
    double confidenceLevel = 0.95,
    size_t numSamples = 10000,
    std::uint64_t seed = 42u);

Bootstrap confidence interval for the mean. confidenceLevel is clamped to [0, 1]; numSamples == 0 returns {0.0, 0.0, 0.0} instead of resampling. Seeded the same way as permutationTest above: deterministic by default, override seed for an independent resample.

Example: 

auto sharpes = extractMetric(results, RankMetric::SharpeRatio);
auto ci = Stats::bootstrapCI(sharpes, 0.95);
// ci.lower, ci.median, ci.upper

printSummary

static void printSummary(
    const std::vector<OptimizationResult<ParamsT>>& results);

Print optimization summary to log. Shows total combinations, mean/stddev Sharpe, and best result details.

generateReport

static bool generateReport(
    const std::vector<OptimizationResult<ParamsT>>& results,
    const std::filesystem::path& outputPath);

Generate a Markdown report with a top-10 results table and statistics. Returns true on success; false (and an error logged) if outputPath could not be opened for writing, e.g. a nonexistent parent directory.

Example

using Stats = OptimizationStatistics<MAParams, MAGrid>;

// Run optimization
auto results = optimizer.runLocal();

// Quick summary
Stats::printSummary(results);

// Full report
Stats::generateReport(results, "optimization_report.md");

// Statistical analysis
auto sharpes = extractMetric(results, RankMetric::SharpeRatio);
auto ci = Stats::bootstrapCI(sharpes);
std::cout << "95% CI: [" << ci.lower << ", " << ci.upper << "]\n";

// Compare top vs bottom half
auto ranked = optimizer.rankResults(results, RankMetric::SharpeRatio);
size_t mid = ranked.size() / 2;
std::vector<double> topHalf, bottomHalf;
for (size_t i = 0; i < mid; ++i) {
  topHalf.push_back(ranked[i].sharpeRatio());
  bottomHalf.push_back(ranked[mid + i].sharpeRatio());
}
double pValue = Stats::permutationTest(topHalf, bottomHalf);

See Also