modelsMutationTaster2 still uses 3 different models, one for alterations that do not cause any amino acid ( without_aee), one for simple substitutions ( simple_aae), and one for those changes that cause more complex changes in the aa sequence of the resulting peptide, such as a frameshift or a shifted start ATG ( complex_aae).
training setsThe following table shows the composition of the data sets used to train the classifier. We used all available alterations that fulfilled our criteria (status 'disease mutation' in HGMD Pro or polymorphism confirmed by at least 4 carriers of this genotype in the 1000 Genomes Project (TGP) - variants that appeared in both groups were discarded). For the web-version, the classifier was trained with all alterations suitable for the given model; alterations of the less frequent type were fed several times into the training to reach equal frequencies of disease mutations and polymorphisms. See below for the composition of the cross-validation data sets.
|model||n (polymorphisms)||n (disease mutations)||comments|
|without_aae||6807269||122238||each disease mutation was used 55 times in the training of the web version (56 times in the cross-validation)|
|simple_aae||20967||151542||each polymorphism was used 7 times in the training of the web version and in the cross-validation|
|complex_aae||2340||123213||each polymorphism was used 52 times in the training of the web version (57 times in the cross-validation)|
results of the cross-validation
|1||1800 known disease mutations from HGMD Pro (disease state = DM [disease mutation])|
|1800 harmless polymorphisms from the 1000 genomes project (each of the 3 possible genotypes found in at least 50 samples)*|
|2||1800 known disease mutations from ClinVar (disease state = pathogenic) [not with MutationTaster1]|
|1800 harmless polymorphisms from the 1000 genomes project (each of the 3 possible genotypes found in at least 50 samples) [not with MutationTaster1]*|
We submitted these alterations to the web interfaces of PolyPhen-2 (HumVar and HumDiv), SIFT, PROVEAN, MutationTaster2, and MutationTaster1 by entering the DNA change via the chromosomal position. Since these variants (on DNA level) might cause different substitutions in different transcripts, we extracted the one result corresponding to the amino acid exchange from the testset. If there were several results for the amino acid exchange in question, we used the first result. Since some variants could not be analysed by all programs (or at least did not return the required amino acid substitution), we randomly selected 1300(ClinVar)/1100(HGMD) disease mutations and 1300(ClinVar)/1100(HGMD) polymorphisms out of the 2381 (HGMD) or 2814 (ClinVar) variants for which all programs gave predictions. We then compared the predictions for these 2600/2200 test cases.
results of the cross-comparison
|1000 genomes and HGMD Pro|
|1000 genomes and ClinVar|
MutationTaster2 displays automatic predictions for known harmless polymorphisms from the 1000 Genomes Project and known disease mutations from NCBI ClinVar. For the comparison of MutationTaster2 with the other tools, we did not consider the automatically displayed prediction (which is per se correct - or at least the same as our prior) but the actual prediction made by the classifier, which is reflected by the probability-value. (An automatic prediction with a probability below 0.5 means that MutationTaster2 would predict the other case if it would not consider the known outcome of this variant.) The performance of MutationTaster on real life data, where known polymorphisms / disease mutations are recognised, is hence even better than the accuracy shown here. Please see our example using a real exome to evaluate the use of MutationTaster!
These results suggest a bias towards mutations with a more obvious
effect on the protein in ClinVar (as of February 2013) because all
programs perform better on the ClinVar data set. See the ClinVar set and the single results.
For copyright reasons, we are unable to reproduce the list of disease mutations obtained from HGMD Professional as text, but we can offer the predictions for the disease mutations and polymorphisms as images. We also provide the results as a comprehensive text file, but here the identities of the HGMD disease mutations are replaced by sequential numbers.
We also provide detailed statistics about the consensus among the different tools for the HGMD data set.
This plot shows the receiver operating characteristics for the
comparison using the HGMD/TGP dataset. Please note that these plots
are intended to set a threshold to discriminate signals from noise -
or, in case of score-based predictiors, to find an optimal cut-off
value between disease mutations and polymorphisms.
MutationTaster2 does not return a 'score' but only a boolean prediction (disease causing or not) plus a confidence score for this prediction. This kind of plot is hence not very useful to determine the performance of MutationTaster2 (it would indeed be very useful to determine cut-off values for continuous values, e.g. for predictions only based on PhyloP or PhastCons).
We know, however, that in many articles ROC curves are used to compare tools such as MutationTaster2 and hence include these curves to show that MutationTaster2 has, as its predecessor, a higher area under curve (AUC).
Non-synonymous, synonymous, and non-coding variants
only variants leading to single amino acid substitutions
dataMutationTaster's results can be interactively inspected: http://doro.charite.de/temp/vcf_7890_1559988265/progress.html.