cc/cirrus_work: lib/python/cc/stats.py annotate

annotate lib/python/cc/stats.py @ 179:16d603447fbc

move to class with local vars instead of many globals

author	Henry S. Thompson <ht@inf.ed.ac.uk>
date	Fri, 24 Nov 2023 20:41:03 +0000
parents	a5d54736a77f
children

rev	line source
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	1 #!/usr/bin/env python3
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	2 '''Rank self.correlation processing for a csv tabulation of self.counts by segment
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	3 First column is for whole crawl, then 100 columns for segs 0-99
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	4 Each row is self.counts for some property, e.g. mime-detected or tld
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	5
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	6 For example, assuming ALL.tsv has the whole-crawl warc-only self.counts
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	7 and s...tsv have the segment self.counts, ALL with self.counts in column 1,
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	8
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	9 tr -d ',' <ALL.tsv \|head -100 \| while read n m; do printf "%s%s\n" $n $(for i in {0..99}; do printf ",%s" $({ grep -w "w $m\$" s${i}.tsv \|\| echo NaN ;} \| cut -f 1 ) ; done ) ; done > all_100.csv
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	10
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	11 will produce such a file with
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	12 * 100 rows, one for each of the top 100 self.counts
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	13 * 101 columns, 0 for ALL and 1--100 for segs 0--99
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	14
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	15 Usage: python3 -i spearman.py name id
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	16 where name.csv has the input
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	17 '''
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	18
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	19 import numpy as np
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	20 from numpy import loadtxt
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	21 from scipy import stats
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	22 import statsmodels.api as sm
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	23 import matplotlib.pyplot as plt
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	24 import pylab
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	25
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	26 import sys, math
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	27
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	28 class CCStats:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	29 def __init__(self, file, id):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	30 self.file = file
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	31 self.id = id
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	32 self.counts=loadtxt(file+".csv",delimiter=',')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	33 self.N=self.counts.shape[1]-1
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	34 # "If axis=0 (default), then each column represents a variable, with
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	35 # observations in the rows"
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	36 # So each column is a sequence of self.counts, for whole crawl in column 0
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	37 # and for segments 0--self.N-1 in columns 1--self.N
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	38 self.corr=stats.spearmanr(self.counts,nan_policy='omit').self.correlation
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	39
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	40 self.all=self.corr[0][1:]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	41 self.all_s=stats.describe(self.all)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	42 self.all_m=self.all_s.mean
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	43
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	44 self.x=np.array([np.concatenate((self.corr[i][1:i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	45 self.corr[i][i+1:])) for i in range(1,self.N+1)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	46 # The above, although transposed, works because the self.correlation matrix
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	47 # is symmetric
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	48 self.xd=[stats.describe(self.x[i]) for i in range(self.N)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	49 self.xs=stats.describe(np.array([self.xd[i].mean for i in range(self.N)]))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	50 self.xm=self.xs.mean
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	51 self.xsd=np.sqrt(self.xs.variance)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	52
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	53 self.x_ranks=[stats.rankdata(-self.counts[:,i],method='average') for i in range(1,self.N+1)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	54
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	55 self.aa=ranks()
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	56 self.aa_by_all=self.aa[self.aa[:,1].argsort()]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	57
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	58 def qqa(self):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	59 # q-q plot for the whole crawl
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	60 sm.qqplot(self.all, line='s')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	61 plt.gca().set_title('Rank self.correlation per segment wrt whole archive %s'%id)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	62 plt.show()
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	63
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	64 def qqs(self):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	65 # q-q plots for the best and worst (by variance) segments
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	66 global xv, xworst, xbest
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	67 xv=[d.variance for d in self.xd]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	68 xworst=xv.index(max(xv))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	69 xbest=xv.index(min(xv))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	70 print(xbest,xworst)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	71 sm.qqplot(self.x[xbest], line='s')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	72 plt.gca().set_title('Best segment (least variance): %s'%xbest)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	73 plt.show()
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	74 sm.qqplot(self.x[xworst], line='s')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	75 plt.gca().set_title('Worst segment (most variance): %s'%xworst)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	76 plt.show()
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	77
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	78 def plot_x(sort=False,block=True,all_only=True,title=None):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	79 # Make these two subplots, w. and w/o sorting
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	80 # See https://stackoverflow.com/questions/4700614/how-to-put-the-legend-outside-the-plot
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	81 # for legend hacking
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	82 if sort:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	83 aso=np.argsort(-self.all)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	84 plot_all=self.all[aso]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	85 plot_x=np.array([self.xd[i].mean for i in range(self.N)])[aso]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	86 else:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	87 plot_all=self.all
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	88 plot_x=[self.xd[i].mean for i in range(self.N)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	89 if title is None:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	90 l1='Rank self.correlation of segment self.x whole crawl'
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	91 l2='Mean of segment self.x whole crawl'
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	92 plt.legend(loc='best',fontsize='small')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	93 else:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	94 l1=l2=None
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	95 plt.plot(plot_all,'rx',label=l1)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	96 plt.plot([0,self.N-1],[self.all_m,self.all_m],'r',label=l2)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	97 if not(all_only):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	98 plt.plot(plot_x,'bx',label='Mean of rank self.correlation of each segment self.x self.all other segments')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	99 plt.plot([0,self.N-1],[self.xm,self.xm],'b',label='Mean of segment self.x segment means')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	100 plt.axis([0,self.N-1,0.85 if all_only else 0.8,1.0])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	101 plt.grid(True)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	102 if title is not None:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	103 plt.title(title)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	104 plt.show(block=block)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	105
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	106 def hist_x(align='mid'):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	107 hist(self.xm,self.xsd,[self.xd[i].mean for i in range(self.N)],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	108 'Mean of rank self.correlation of each segment self.x self.all other segments',
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	109 align)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	110
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	111 def hist_all(align='mid'):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	112 hist(self.all_m,np.sqrt(self.all_s.variance),self.all,
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	113 'Rank self.correlation of each segment self.x whole crawl %s'%id,
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	114 align)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	115
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	116 def hist(m,sd,hh,title,align):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	117 sdd=[(i,m-(i*sd)) for i in range(-2,3)]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	118 fig,hax=plt.subplots() # Thanks to https://stackoverflow.com/a/7769497
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	119 sdax=hax.twiny()
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	120 hax.hist(hh,color='lightblue',align=align)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	121 hax.set_title(title)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	122 for s,v in sdd:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	123 sdax.plot([v,v],[0,18],'b')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	124 sdax.set_xlim(hax.get_xlim())
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	125 sdax.set_ylim(hax.get_ylim())
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	126 sdax.set_xticks([v for s,v in sdd])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	127 sdax.set_xticklabels([str(s) for s,v in sdd])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	128 plt.show()
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	129
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	130 def ci(rho,n,conf=0.95):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	131 # Courtesy of https://stats.stackexchange.com/a/18904
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	132 # rho is (rank) self.correlation, n is sample size
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	133 stderr=1.0/math.sqrt(n-3)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	134 z=stats.norm.ppf(1.0-((1.0-conf)/2))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	135 delta=z*stderr
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	136 lower=math.tanh(math.atanh(rho)-delta)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	137 upper=math.tanh(math.atanh(rho)+delta)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	138 return (lower,upper)
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	139
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	140 def plot_ci(rhos,n,trim=None,conf=0.95):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	141 # rhos are (rank) self.correlation values
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	142 rhos_s=rhos[(-rhos).argsort()]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	143 if trim is None:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	144 l=len(rhos)
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	145 else:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	146 rhos_s=rhos_s[:trim]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	147 l=trim
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	148 cc=(np.array([ci(r,n,conf) for r in rhos_s])).T
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	149 ue=cc[1]-rhos_s
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	150 le=rhos_s-cc[0]
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	151 #for i in range(len(rhos)):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	152 #print(cc[i][0],rhos_s[i]-cc[i][0],rhos_s[i],cc[i][1],-rhos_s[i]+cc[i][1])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	153 plt.errorbar(np.arange(l),rhos_s,yerr=[le,ue],fmt='o')
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	154 plt.title("Rank self.correlation of segments self.x whole archive %s\nwith confidence bars at %d%%"%(id,conf*100))
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	155 plt.show()
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	156
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	157 def first_diff(ranks):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	158 # first disagreement with baseline == {1,2,...}
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	159 for i in range(len(ranks)):
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	160 if ranks[i]!=i+1.0:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	161 return i
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	162 return i+1
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	163
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	164 def ranks():
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	165 # Combine segment measures:
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	166 # segID,rank self.corr. wrt self.all,inverse variance, mean cross rank self.corr.,first disagreement
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	167 # convert to ranks, smallest value == highest rank
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	168 all_ranked=stats.rankdata(-self.all,method='average') # invert since
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	169 # large self.corr is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	170 x_variance_ranked=stats.rankdata([self.xd[i].variance for i in range(self.N)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	171 # small self.corr variance is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	172 x_mean_ranked=stats.rankdata([-(self.xd[i].mean) for i in range(self.N)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	173 # invert since
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	174 # large mean self.corr is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	175 fd_ranked=stats.rankdata([-first_diff(self.x_ranks[i]) for i in range(self.N)])
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	176 # invert since
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	177 # large first diff is good
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	178 return np.array([[i,
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	179 all_ranked[i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	180 x_variance_ranked[i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	181 x_mean_ranked[i],
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	182 fd_ranked[i]] for i in range(self.N)])
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	183
a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	184 if __name__ == '__main__':
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	185 S = CCStats(*sys.argv[1:])
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	186
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	187 ### I need to review rows, e.g. self.counts[0] is an array of self.N+1 self.counts
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	188 ### for the most common label in the complete crawl,
179 16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	189 ### from the complete crawl and all the segments
16d603447fbc move to class with local vars instead of many globals Henry S. Thompson <ht@inf.ed.ac.uk> parents: 177 diff changeset	190 ### versus columns, e.g. self.counts[:,0] is an array of self.N decreasing self.counts
177 a5d54736a77f renamed from spearman.py Henry S. Thompson <ht@inf.ed.ac.uk> parents: diff changeset	191 ### for all the labels in the complete crawl

Mercurial > hg > cc > cirrus_work

annotate lib/python/cc/stats.py @ 179:16d603447fbc