cc/cirrus_work: lib/python/cc/ccstats.py annotate

annotate lib/python/cc/ccstats.py @ 334:252cf1263ad0 trim

do merge in python working

author	Henry S. Thompson <ht@inf.ed.ac.uk>
date	Sun, 01 Mar 2026 15:22:10 +0000
parents	ca937be30ad7
children

rev	line source
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	1 #!/usr/bin/env python3
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	2 '''Rank self.correlation processing for a csv tabulation of self.counts by segment
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	3 First column is for whole crawl, then 100 columns for segs 0-99
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	4 Each row is self.counts for some property, e.g. mime-detected or tld
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	5
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	6 For example, assuming ALL.tsv has the whole-crawl warc-only self.counts
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	7 and s...tsv have the segment self.counts, ALL with self.counts in column 1,
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	8
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	9 tr -d ',' <ALL.tsv \|head -100 \| while read n m; do printf "%s%s\n" $n $(for i in {0..99}; do printf ",%s" $({ grep -w "w $m\$" s${i}.tsv \|\| echo NaN ;} \| cut -f 1 ) ; done ) ; done > all_100.csv
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	10
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	11 will produce such a file with
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	12 * 100 rows, one for each of the top 100 self.counts
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	13 * 101 columns, 0 for ALL and 1--100 for segs 0--99
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	14
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	15 Usage: python3 -i spearman.py name id
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	16 where name.csv has the input
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	17 '''
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	18
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	19 import numpy as np
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	20 from numpy import loadtxt
194 ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	21 import scipy
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	22 from scipy import stats
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	23 import statsmodels.api as sm
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	24 import matplotlib.pyplot as plt
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	25 import pylab
194 ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	26 from scipy.stats import _bootstrap
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	27
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	28 import sys, math, os.path, random
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	29
194 ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	30 RNG = np.random.default_rng()
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	31 try:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	32 SS
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	33 except NameError:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	34 SS = {}
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	35
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	36 class CCStats:
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	37 def __init__(self, year_month, name, file_base):
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	38 global SS
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	39 self.year_month = year_month
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	40 self.name = name
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	41 self.id = "%s %s"%(year_month,name)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	42 SS[self.id] = self
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	43 self.file = os.path.expanduser("~/results/CC-MAIN-%s/%s.csv")%(year_month,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	44 file_base)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	45 self.counts=loadtxt(self.file,delimiter=',')
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	46 self.N=self.counts.shape[1]-1
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	47 # "If axis=0 (default), then each column represents a variable, with
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	48 # observations in the rows"
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	49 # So each column is a sequence of self.counts, for whole crawl in column 0
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	50 # and for segments 0--self.N-1 in columns 1--self.N
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	51 self.corr=stats.spearmanr(self.counts,nan_policy='omit').correlation
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	52
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	53 self.all=self.corr[0][1:]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	54 self.all_s=stats.describe(self.all)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	55 self.all_m=self.all_s.mean
190 7492bd2f5423 can't add props to DescribeResult Henry S. Thompson <ht@inf.ed.ac.uk> parents: 185 diff changeset	56 self.all_sd=math.sqrt(self.all_s.variance)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	57
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	58 self.x=np.array([np.concatenate((self.corr[i][1:i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	59 self.corr[i][i+1:])) for i in range(1,self.N+1)])
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	60 # The above, although transposed, works because the correlation matrix
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	61 # is symmetric
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	62 self.xd=[stats.describe(self.x[i]) for i in range(self.N)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	63 self.xs=stats.describe(np.array([self.xd[i].mean for i in range(self.N)]))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	64 self.xm=self.xs.mean
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	65 self.xsd=np.sqrt(self.xs.variance)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	66
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	67 self.x_ranks=[stats.rankdata(-self.counts[:,i],method='average') for i in range(1,self.N+1)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	68
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	69 self.aa = self.ranks()
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	70 self.aa_by_all=self.aa[self.aa[:,1].argsort()]
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	71 self.iranks_by_all = [int(r) for r in self.aa_by_all[:,0]]
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	72
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	73 def qqa(self):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	74 # q-q plot for the whole crawl
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	75 sm.qqplot(self.all, line='s')
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	76 plt.gca().set_title('Rank self.correlation per segment wrt whole archive %s'%self.id)
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	77 plt.show()
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	78
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	79 def qqs(self):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	80 # q-q plots for the best and worst (by variance) segments
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	81 global xv, xworst, xbest
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	82 xv=[d.variance for d in self.xd]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	83 xworst=xv.index(max(xv))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	84 xbest=xv.index(min(xv))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	85 print(xbest,xworst)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	86 sm.qqplot(self.x[xbest], line='s')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	87 plt.gca().set_title('Best segment (least variance): %s'%xbest)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	88 plt.show()
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	89 sm.qqplot(self.x[xworst], line='s')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	90 plt.gca().set_title('Worst segment (most variance): %s'%xworst)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	91 plt.show()
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	92
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	93 def plot_x(self,sort=False,block=True,all_only=True,title=None):
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	94 # Make these two subplots, w. and w/o sorting
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	95 # See https://stackoverflow.com/questions/4700614/how-to-put-the-legend-outside-the-plot
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	96 # for legend hacking
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	97 if sort:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	98 aso=np.argsort(-self.all)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	99 plot_all=self.all[aso]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	100 plot_x=np.array([self.xd[i].mean for i in range(self.N)])[aso]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	101 else:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	102 plot_all=self.all
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	103 plot_x=[self.xd[i].mean for i in range(self.N)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	104 if title is None:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	105 l1='Rank self.correlation of segment self.x whole crawl'
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	106 l2='Mean of segment self.x whole crawl'
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	107 plt.legend(loc='best',fontsize='small')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	108 else:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	109 l1=l2=None
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	110 plt.plot(plot_all,'rx',label=l1)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	111 plt.plot([0,self.N-1],[self.all_m,self.all_m],'r',label=l2)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	112 if not(all_only):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	113 plt.plot(plot_x,'bx',label='Mean of rank self.correlation of each segment self.x self.all other segments')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	114 plt.plot([0,self.N-1],[self.xm,self.xm],'b',label='Mean of segment self.x segment means')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	115 plt.axis([0,self.N-1,0.85 if all_only else 0.8,1.0])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	116 plt.grid(True)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	117 if title is not None:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	118 plt.title(title)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	119 plt.show(block=block)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	120
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	121 def hist_x(self,align='mid'):
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	122 hist(self.xm,self.xsd,[self.xd[i].mean for i in range(self.N)],
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	123 'Mean of rank self.correlation of each segment self.x self.all other segments',
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	124 align)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	125
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	126 def hist_all(self,align='mid'):
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	127 hist(self.all_m,np.sqrt(self.all_s.variance),self.all,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	128 'Rank self.correlation of each segment self.x whole crawl %s'%self.id,
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	129 align)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	130
194 ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	131 def plot_ci(self,rhos=None,bootstrap=None,n=None,trim=None,conf=0.95,block=True):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	132 global cc
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	133 # rhos are (rank) correlation values
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	134 # If bootstrap is present, it should be the data from which rhos
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	135 # was derived via spearmanr, which will be used for bootstrapping, e.g. self.counts
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	136 if rhos is None:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	137 rhos = self.all
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	138 if n is None:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	139 n = rhos.shape[0]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	140 all_order=(-rhos).argsort()
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	141 rhos_s=rhos[all_order]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	142 if trim is None:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	143 l=len(rhos)
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	144 else:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	145 rhos_s=rhos_s[:trim]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	146 l=trim
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	147 if bootstrap is not None:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	148 all=bootstrap[:,0] # [all_order]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	149 rest=bootstrap[:,1:] # [all_order]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	150 cc=np.array([tuple(stats.bootstrap((all,
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	151 rest[:,i]),
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	152 sr, vectorized=bool(print(i)),
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	153 paired=True, method='bca',
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	154 random_state=RNG).confidence_interval)
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	155 for i in range(l)])
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	156 #cc=cc[(-rhos).argsort()].T
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	157 cc=cc.T
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	158 cc=[cc[0][all_order],cc[1][all_order]]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	159 else:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	160 cc=np.array([ci(r,n,conf) for r in rhos_s]).T
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	161 ue=cc[1]-rhos_s
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	162 le=rhos_s-cc[0]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	163 #for i in range(len(rhos)):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	164 #print(cc[i][0],rhos_s[i]-cc[i][0],rhos_s[i],cc[i][1],-rhos_s[i]+cc[i][1])
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	165 plt.errorbar(np.arange(l),rhos_s,yerr=[le,ue],fmt='o')
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	166 plt.title("Rank self.correlation of segments self.x whole archive %s\nwith confidence bars at %d%%"%(self.id,conf*100))
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	167 plt.show(block=block)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	168
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	169 def ranks(self):
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	170 # Combine segment measures:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	171 # segID,rank self.corr. wrt self.all,inverse variance, mean cross rank self.corr.,first disagreement
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	172 # convert to ranks, smallest value == highest rank
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	173 all_ranked=stats.rankdata(-self.all,method='average') # invert since
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	174 # large self.corr is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	175 x_variance_ranked=stats.rankdata([self.xd[i].variance for i in range(self.N)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	176 # small self.corr variance is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	177 x_mean_ranked=stats.rankdata([-(self.xd[i].mean) for i in range(self.N)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	178 # invert since
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	179 # large mean self.corr is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	180 fd_ranked=stats.rankdata([-first_diff(self.x_ranks[i]) for i in range(self.N)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	181 # invert since
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	182 # large first diff is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	183 return np.array([[i,
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	184 all_ranked[i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	185 x_variance_ranked[i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	186 x_mean_ranked[i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	187 fd_ranked[i]] for i in range(self.N)])
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	188
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	189 def hist(m,sd,hh,title,align):
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	190 sdd=[(i,m-(i*sd)) for i in range(-2,3)]
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	191 fig,hax=plt.subplots() # Thanks to https://stackoverflow.com/a/7769497
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	192 sdax=hax.twiny()
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	193 hax.hist(hh,color='lightblue',align=align)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	194 hax.set_title(title)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	195 for s,v in sdd:
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	196 sdax.plot([v,v],[0,18],'b')
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	197 sdax.set_xlim(hax.get_xlim())
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	198 sdax.set_ylim(hax.get_ylim())
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	199 sdax.set_xticks([v for s,v in sdd])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	200 sdax.set_xticklabels([str(s) for s,v in sdd])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	201 plt.show()
180 a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	202
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	203 def first_diff(ranks):
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	204 # first disagreement with baseline == {1,2,...}
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	205 for i in range(len(ranks)):
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	206 if ranks[i]!=i+1.0:
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	207 return i
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	208 return i+1
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	209
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	210 def ci(rho,n,conf=0.95):
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	211 # Courtesy of https://stats.stackexchange.com/a/18904
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	212 # rho is (rank) correlation, n is sample size
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	213 stderr=1.0/math.sqrt(n-3)
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	214 z=stats.norm.ppf(1.0-((1.0-conf)/2))
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	215 delta=z*stderr
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	216 lower=math.tanh(math.atanh(rho)-delta)
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	217 upper=math.tanh(math.atanh(rho)+delta)
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	218 return (lower,upper)
a20cf98f3a77 rename to avoid name clash with scipy.stats Henry S. Thompson <ht@inf.ed.ac.uk> parents: 179 diff changeset	219
194 ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	220 def sr(x,y):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	221 return stats.spearmanr(x,y)[0]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	222
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	223 def _percentile_along_axis(theta_hat_b, alpha):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	224 """`np.percentile` with different percentile for each slice."""
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	225 # the difference between _percentile_along_axis and np.percentile is that
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	226 # np.percentile gets _all_ the qs for each axis slice, whereas
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	227 # _percentile_along_axis gets the q corresponding with each axis slice
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	228 shape = theta_hat_b.shape[:-1]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	229 alpha = np.broadcast_to(alpha, shape)
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	230 percentiles = np.zeros_like(alpha, dtype=np.float64)
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	231 for indices, alpha_i in np.ndenumerate(alpha):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	232 if np.isnan(alpha_i):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	233 # See https://github.com/scipy/scipy/pull/14351/commits/e5f38dee638d63981e89b4630e2ea5bc3a59a51d
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	234 percentiles[indices] = np.nan
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	235 else:
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	236 theta_hat_b_i = theta_hat_b[indices]
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	237 percentiles[indices] = np.percentile(theta_hat_b_i, alpha_i)
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	238 return percentiles[()] # return scalar instead of 0d array
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	239
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	240 _bootstrap._percentile_along_axis = _percentile_along_axis
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	241
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	242 def explore_deltas(basis_ym, basis_names, target_ym, target_names, nn):
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	243 for n in nn:
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	244 for p in basis_names:
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	245 for q in target_names:
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	246 basis=SS[f'{basis_ym} {p}']
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	247 target=SS[f'{target_ym} {q}']
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	248 m = int((basis.N - n) / 2)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	249 best = stats.describe(target.all[np.array(basis.iranks_by_all[:n])]).mean
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	250 middle = stats.describe(target.all[np.array(basis.iranks_by_all[m:m+n])]).mean
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	251 worst = stats.describe(target.all[np.array(basis.iranks_by_all[-n:])]).mean
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	252 delta1 = best-worst
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	253 stdev = math.sqrt(target.all_s.variance)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	254 ds1 = delta1 / stdev
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	255 delta2 = abs(middle-target.all_m)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	256 ds2 = delta2 / stdev
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	257 print(n,p,q,'%6.4g'%delta1,'%4.2g'%ds1,'%6.4g'%delta2,'%4.2g'%ds2)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	258
182 45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	259 def heat_map(basis_ym, basis_names, target_ym, target_names, nn, outf = sys.stdout):
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	260 nn=nn if isinstance(nn,list) else list(nn)
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	261 with (outf if outf is sys.stdout else open(outf,'w')) as outfile:
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	262 print("\tN",file=outfile)
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	263 print("Predict","\t".join(["{:d}"]len(nn)).format(nn),sep="\t",file=outfile)
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	264 for p in basis_names:
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	265 for q in target_names:
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	266 basis=SS[f'{basis_ym} {p}']
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	267 target=SS[f'{target_ym} {q}']
185 7c5efd56ce17 change heatmap to by percentile Henry S. Thompson <ht@inf.ed.ac.uk> parents: 182 diff changeset	268 tmin,tmax = target.all_s.minmax
7c5efd56ce17 change heatmap to by percentile Henry S. Thompson <ht@inf.ed.ac.uk> parents: 182 diff changeset	269 twidth = tmax - tmin
182 45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	270 best = (stats.describe(target.all[np.array(basis.iranks_by_all[:n])]).mean
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	271 for n in nn)
185 7c5efd56ce17 change heatmap to by percentile Henry S. Thompson <ht@inf.ed.ac.uk> parents: 182 diff changeset	272 bestiles = (round((b - tmin) / twidth, 3) * 100 for b in best)
182 45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	273 print(f"{q} by {p}",
185 7c5efd56ce17 change heatmap to by percentile Henry S. Thompson <ht@inf.ed.ac.uk> parents: 182 diff changeset	274 "\t".join(['{:4.3g}']len(nn)).format(bestiles),
182 45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	275 sep="\t",file=outfile)
45120df952be add head_map fn Henry S. Thompson <ht@inf.ed.ac.uk> parents: 181 diff changeset	276
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	277 def predict(basis_ym, basis_name, target_ym, target_name, n):
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	278 basis = SS['%s %s'%(basis_ym, basis_name)]
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	279 target = SS['%s %s'%(target_ym, target_name)]
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	280 best = stats.describe(target.all[np.array(basis.iranks_by_all[:n])])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	281 tbest = stats.describe(target.all[np.array(target.iranks_by_all[:n])])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	282 m = int((basis.N - n) / 2)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	283 middle = stats.describe(target.all[np.array(basis.iranks_by_all[m:m+n])])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	284 tmiddle = stats.describe(target.all[np.array(target.iranks_by_all[m:m+n])])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	285 randind = list(range(100))
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	286 random.shuffle(randind)
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	287 rlist = np.array(randind[:n])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	288 randist = stats.describe(target.all[np.array(basis.iranks_by_all)[rlist]])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	289 trandist = stats.describe(target.all[rlist])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	290 worst = stats.describe(target.all[np.array(basis.iranks_by_all[-n:])])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	291 tworst = stats.describe(target.all[np.array(target.iranks_by_all[-n:])])
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	292 print(f"Predicting {basis.id} from {target.id}, window size {n}")
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	293 print(" Prediction Actual")
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	294 print(" mean stdev mean stdev value")
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	295 print(" Best %d %-6.4g %-6.3g %-6.4g %-6.3g %-6.4g"%(n, best.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	296 round(math.sqrt(best.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	297 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	298 tbest.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	299 round(math.sqrt(tbest.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	300 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	301 target.all_s.minmax[1]))
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	302 print(" Middle %d %-6.4g %-6.3g %-6.4g %-6.3g %-6.4g"%(n, middle.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	303 round(math.sqrt(middle.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	304 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	305 tmiddle.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	306 round(math.sqrt(tmiddle.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	307 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	308 target.all_m))
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	309 print("Randist %d %-6.4g %-6.3g %-6.4g %-6.3g %-6.4g"%(n, randist.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	310 round(math.sqrt(randist.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	311 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	312 trandist.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	313 round(math.sqrt(trandist.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	314 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	315 target.all_m))
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	316 print(" Worst %d %-6.4g %-6.3g %-6.4g %-6.3g %-6.4g"%(n, worst.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	317 round(math.sqrt(worst.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	318 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	319 tworst.mean,
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	320 round(math.sqrt(tworst.variance),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	321 4),
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	322 target.all_s.minmax[0]))
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	323
194 ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	324 def dump(file=sys.stdout):
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	325 for s in SS.values():
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	326 print("CCStats(%s,%s,%s)"%(repr(s.year_month),repr(s.name),
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	327 repr(s.file.split('/',7)[7][:-4])),
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	328 file=sys.stdout)
ca937be30ad7 implement alternative confidence measure using stats.bootstrap, Henry S. Thompson <ht@inf.ed.ac.uk> parents: 190 diff changeset	329
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	330 if __name__ == '__main__':
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	331 S = CCStats(*sys.argv[1:])
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	332
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	333 ### I need to review rows, e.g. self.counts[0] is an array of self.N+1 self.counts
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	334 ### for the most common label in the complete crawl,
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	335 ### from the complete crawl and all the segments
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	336 ### versus columns, e.g. self.counts[:,0] is an array of self.N decreasing self.counts
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	337 ### for all the labels in the complete crawl
181 0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	338
0058e523f649 add explore_deltas and predict analysis fns Henry S. Thompson <ht@inf.ed.ac.uk> parents: 180 diff changeset	339

Mercurial > hg > cc > cirrus_work

annotate lib/python/cc/ccstats.py @ 334:252cf1263ad0 trim