This is the directory containing code for calculating several metrics, namely 
remaining uncertainty and misinformation, for the performance of several baseline 
models. 

########################################################################
########################################################################

If you use this code in your work please cite:

W.T. Clark and P. Radivojac. Information-theoretic evaluation of predicted 
ontological annotations. Bioinformatics, 29(13):i53-i61, 2013.

########################################################################
########################################################################

This code is in MATLAB.  Sorry for those of you without a license.  If you convert this 
code to a language such as perl or python and can demonstrate its validity I will 
either post it on this site, or link to your code elsewhere. 

########################################################################
########################################################################

please contact Wyatt Clark ( wyatt.clark@yale.edu ) with any questions or
concerns

########################################################################
########################################################################

The folder is organized into three subdirectories described below:

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 

DATA: contains data for running experiments and predictions of baseline models 
	(found in PREDICTIONS/):

	CCO_FOLDS: testing folds for cross validation
	CCO_IA: information accretion values, and conditional probabilities of terms 
		given parents (CPGP).
	CCO_LEAFER: leaf annotations
	CCO_Protein_IC: information content of annotations associated with proteins
	CCO_Sequence_DATA: Sequences, ID’s and accessions associated with 
		proteins
	CCO_TERM_INFORMATION: information content of each term in the ontology 
		calculated directly from the data (used for calculating Resnik and 
		other traditional IC metrics).
	CCO_TESTING_LABEL: label matrix permuted in order of testing folds.
	CCO_TESTING_LAFER: leaf matrix permuted according to testing folds.
	CCO_TSM: term (columns) sequence (rows) map 
	linear_CCO: linearized ontology.
	linear_propogated_CCO: (sic) propagated ontology.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 

etc: contains code for calculating conditional probabilities, IA, and IC; as well as 
script for calculating protein annotation IC from IA.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 

Analysis: contains code in several folders for calculating evaluation metrics:

	IA_Weighted_Precision_Recall: calculating precision, recall AND specificity 
		using information accretion values and weighting each protein 
		according to information content of annotations when averaging. 
	Jaccard_Cosine_Dice: code for calculating jaccard, cosine, and dice metrics.
	MICA_Based: code for calculating several traditional information content 
		metrics using the most informative common ancestor.  Employs an all 
		pair averaging method for averaging over multiple data points.
	MICA-Based_Max_Avg: same as above, but employs max averaging scheme of 
		averaging.
	Precision_Recall: calculates precision and recall using the ontology 
		(intersection of graphs for true and predicted terms).  Differs from 
		CAFA and FANN-GO paper in the sense that the root term is always an 
		annotation (case for all metrics).  This eliminates selection bias 
		between metrics at a given threshold.
	Precision_Recall_Verspoor: calculates pairwise precision and recall and 
		averages using max averaging scheme.
	SS_Bayesian_Network: calculated remaining uncertainty and misinformation.  
		Also calculates weighted averages.

########################################################################
########################################################################
