Reliability and Validity Analysis in the Field of Tumor; Common Mistakes

Reliability and Validity Analysis in the Field of Tumor; Common Mistakes

 

Siamak Sabour

 

Siamak Sabour, Safety Promotion and Injury Prevention Research Center, Shahid Beheshti University of Medical Sciences, Tehran, I.R. Iran

Siamak Sabour, Department of Clinical Epidemiology, School of Dentistry, Shahid Beheshti University of Medical Sciences, Tehran, I.R. Iran

Correspondence to: Siamak Sabour, MD, MSc, DSc, PhD, Safety Promotion and Injury Prevention Research Center, Shahid Beheshti University of Medical Sciences, Tehran, I.R. Iran.

Email: s.sabour@sbmu.ac.ir

Telephone: +98-21- 22421814              

Received: January 16, 2014            Revised: February 5, 2014

Accepted: February 10, 2014

Published online: April 18, 2014

 

ABSTRACT

Reliability (precision) and validity (accuracy) are two important methodological issues in all fields of researches. The reliability is being assessed by inappropriate tests which all of them are among common mistakes. For quantitative variable Intra Class Correlation Coefficient (ICC) and for qualitative variables weighted kappa should be used. Sensitivity, specificity,)positive predictive value (PPV), negative predictive value (NPV), likelihood ratio positive and likelihood ratio negative as well as diagnostic accuracy and odds ratio are among the tests to evaluate the validity of a single test compared to a gold standard. For reliability and validity analysis, appropriate tests should be applied by clinical researchers. Otherwise, misdiagnosis and mismanagement of the patients in routine clinical care cannot be avoided using inappropriate tests to assess reliability and validity.

 

© 2014 ACT. All rights reserved.

 

Key words: Reliability, Validity, Diagnostic tests, Mistake

 

Sabour S. Reliability and Validity Analysis in the Field of Tumor; Common Mistakes. Journal of Tumor 2014; 2(4): 122-124 Available from: URL: http://www.ghrnet.org/index.php/JT/article/view/652

 

ADVANCES IN KNOWLEDGE

1. Reliability (precision) and validity (accuracy) are two important methodological issues in all fields of researches.

2. The reliability is being assessed by inappropriate tests which all of them are among common mistakes and is being published by high impact journals.

3. As a take home message, for reliability and validity analysis, appropriate tests should be applied by clinical researchers.

 

IMPLICATION FOR PATIENT CARE

Misdiagnosis and mismanagement of the patients in routine clinical care cannot be avoided using inappropriate tests to assess reliability and validity.

 

RELIABILITY AND VALIDITY ANALYSIS IN THE FIELD OF TUMOR; COMMON MISTAKES

Reliability (precision) and validity (accuracy) are two completely different and important methodological issues in all fields of researches. Reliability (repeatability or reproducibility) is being assessed by different statistical tests such as Pearson, least square and paired t test which all of them are among common mistakes in reliability analysis (Figure 1, 2)[1] and is being published by high impact journals[2-8].

     Briefly, for quantitative variable Intra Class Correlation Coefficient (ICC) and for qualitative variables weighted kappa should be used with caution because kappa has its own limitation too[1,9-40].

    It is crucial to know that there is no value of kappa that can be regarded universally as indication good agreement. Two important weaknesses of k value to assess agreement of a qualitative variable are as follow: It depends upon the prevalence in each category which means it can be possible to have different kappa value having the same percentage for both concordant and discordant cells! Figure 3 shows that in both (a) and (b) situations the prevalence of concordant cells are 80% and discordant cells are 20%, however, we get different kappa value (0.38 and 0.60) respectively. Kappa value also depends upon the number of categories which means the higher the categories, the lower the amount of kappa value[9-40].

    Sensitivity [Percent with the disease who test positive, True Positives / (True Positives + False Negative)], specificity [Percent healthy who test negative, True Negatives / (True Negatives + False Positive)] positive predictive value (PPV), [Percent of positive tests who actually are diseased, True Positives / (True Positives + False Positive)], negative predictive value (NPV) [Percent of negative tests who are healthy, True Negatives / (True Negatives + False Negative)], likelihood ratio positive and likelihood ratio negative as well as diagnostic accuracy [(both true positive and true negative results / total)* 100]  and odds ratio (true results / false results) preferably more than 50, are among the tests to evaluate the validity of a single test compared to a gold standard[9-40].

    As a take home message, for reliability and validity analysis, appropriate tests should be applied by researchers. Otherwise, misdiagnosis and mismanagement of the patients cannot be avoided.

 

CONFLICT OF INTERESTS

There are no conflicts of interest with regard to the present study.

 

REFERENCES

1    Lawrence I, Kuei Lin. A Concordance Correlation Coefficient to Evaluate Reproducibility. BIOMETRICS 1989 March; 45: 255-268

2    Jain V, Duda J, Avants B, Giannetta M, Xie SX, Roberts T, Detre JA, Hurt H, Wehrli FW, Wang DJ. Longitudinal Reproducibility and Accuracy of Pseudo-Continuous Arterial Spin-labeled Perfusion MR Imaging in Typically Developing Children. Radiology 2012 May; 263(2): 527-536

3    Albarakati SF, Kula KS, Ghoneima AA, The reliability and reproducibility of cephalometric measurements: a comparison of conventional and digital methods. Dentomaxillofac Radiol 2012 Jan; 41(1): 11-17

4    Lai V, Tsang WK, Chan WC, Yeung TW. Diagnostic accuracy of mediastinal width measurement on posteroanterior and anteroposterior chest radiographs in the depiction of acute nontraumatic thoracic aortic dissection. Emerg Radiol 2012 Mar 14.

5    Ling LF, Obuchowski NA, Rodriguez L, Popovic Z, Kwon D, Marwick TH. Accuracy and Interobserver Concordance of Echocardiographic Assessment of Right Ventricular Size and Systolic Function: A Quality Control Exercise. J Am Soc Echocardiogr 2012 Apr 26

6    Sato T, Tsujino I, Ohira H, Oyama-Manabe N, Yamada A, Ito YM, Goto C, Watanabe T, Sakaue S, Nishimura M. Validation study on the accuracy of echocardiographic measurements of right ventricular systolic function in pulmonary hypertension. J Am Soc Echocardiogr 2012 Mar; 25(3): 280-286

7    Maislin G, Ahmed MM, Gooneratne N, Thorne-Fitzgerald M, Kim C, Teff K, Arnardottir ES, Benediktsdottir B, Einarsdottir H, Juliusson S, Pack AI, Gislason T, Schwab RJ. Single Slice vs. Volumetric MR Assessment of Visceral Adipose Tissue: Reliability and Validity Among the Overweight and Obese. Obesity (Silver Spring) 2012 Mar 7. doi: 10.1038/oby.2012.53

8    Soviero VM, Leal SC, Silva RC, Azevedo RB. Validity of MicroCT for in vitro detection of proximal carious lesions in primary molars. J Dent 2012 Jan; 40(1): 35-40

9    Jeckel. J.F, Katz. D.L, Elmore, J.G, Wild, D.M.G, Epidemiology, Biostatistics and Preventive Medicine, 3rd edition. 2007, SAUNDERS, Elsevier, Philadelphia, PA, United State

10   Kenneth J. Rothman, Sander Greenland, Timothy L. Lash. Modern Epidemiology, 4th edition. 2010. Lippincott Williams & Wilkins, Baltimore, United States

11   Szklo M, Nieto. F.J, Epidemiology beyond the basics, 2nd edition, 2007, Jones and Bartlett Publisher, Manhattan, new York, United State

12   Sabour S, Dastjerdi EV. Reliability of four different computerized cephalometric analysis programs: a methodological error. Eur J Orthod 2013 Dec; 35(6): 848. doi: 10.1093/ejo/cjs074. Epub 2013 Oct 16.

13   Sabour S. Reliability and accuracy of skeletal muscle imaging in limb-girdle muscular dystrophies. Neurology 2013 Jun 11; 80(24): 2275. doi: 10.1212/WNL.0b013e318299ef6b

14   Sabour S, Ghassemi F. Predictive value of confocal scanning laser for the onset of visual field loss. Ophthalmology 2013 Jun; 120(6): e31-2. doi: 10.1016/j.ophtha.2013.01.055.

15   Sabour S, Kermani H. A comparison between dental measurements taken from CBCT models and those taken from a digital method: a query about methodology. Eur J Orthod 2013 Oct; 35(5): 714-715. doi: 10.1093/ejo/cjt022. Epub 2013 May 29

16   Sabour S. Interlaboratory and interstudy reproducibility of a novel lateral-flow device: a statistical issue. J Clin Microbiol 2013 May; 51(5): 1652. doi: 10.1128/JCM.00111-13

17 Sabour S, Ghassemi F. Interrater reliability of intensive care unit electroencephalogram revised terminology: pitfalls and challenges of using kappa value. J Clin Neurophysiol 2013 Apr; 30(2): 210. doi: 10.1097/WNP.0b013e31827edcca.

18  Sabour S. A quantitative assessment of the accuracy and reliability of O-arm images for deep brain stimulation surgery. Neurosurgery 2013 Apr; 72(4): E696. doi: 10.1227/NEU.0b013e318282d66e.

19  Sabour S. Single slice vs. volumetric MR assessment of visceral adipose tissue: reliability and validity among the overweight and obese. Obesity (Silver Spring) 2013 Jan; 21(1): 6-7. doi: 10.1002/oby.20093.

20 Sabour S, Ghorbani Z. Developmental fluoride neurotoxicity: clinical importance versus statistical significance. Environ Health Perspect 2013 Mar; 121(3): A70. doi: 10.1289/ehp.1206192.

21    Sabour S. Effect of Training Status on Reliability of 1RM Testing in Women, Methodological Mistake. J Strength Cond Res 2013 Feb 25. [Epub ahead of print]

22   Sabour S, Dastjerdi EV, Moezizadeh M. Accuracy of peri-implant bone thickness and validity of assessing bone augmentation material using cone beam computed tomography--is this correct? Clin Oral Investig 2013 Sep; 17(7): 1785. doi: 10.1007/s00784-013-0944-0. Epub 2013 Feb 21.

23    Accuracy of linear intraoral measurements using cone beam CT and multidetector CT: methodological mistake. Sabour S, Kermani H, Dentomaxillofac Radiol. 2013;42(4):20130048. doi: 10.1259/dmfr.20130048. Epub 2013 Feb 18. No abstract available.

24    Sabour S, Moezizadeh M, Dastjerdi EV. Reliability of shade selection using an intraoral spectrophotometer: common mistakes in reliability analysis. Clin Oral Investig 2013 Apr; 17(3):1025. doi: 10.1007/s00784-013-0930-6. Epub 2013 Feb 13.

25   Sabour S, Vahid-Dastjerdi E,Validity of the bolton index using cone-beam computed tomography (CBCT); methodological mistake. Med Oral Patol Oral Cir Bucal 2013 Sep 1; 18(5):e822-3

26    Sabour S, Dastjerdi EV. Validity of a visual scoring method for masticatory ability using test gummy jelly: methodological mistake. Gerodontology 2013 Mar; 30(1): 85. doi: 10.1111/ger.12007.

27   Sabour S, Dastjerdi EV. Reliability of assessment of nasal flow rate for nostril selection during nasotracheal intubation: common mistakes in reliability analysis. J Clin Anesth 2013 Mar; 25(2):162. doi: 10.1016/j.jclinane.2012.10.006. Epub 2013 Jan 16.

28    Sabour S, Dastjerdi EV. Reliability of implant surgical guides based on soft-tissue models: a methodological mistake. J Oral Implantol 2012 Dec; 38(6): 805. doi: 10.1563/AAID-JOI-D-12-00176.

29    Sabour S, Ghassemi F, Ocular duction studies: statistical issues. Ophthalmology 2013 Jan; 120(1): 222-223. doi:10.1016/j.ophtha.2012.08.020.

30   Sabour S. Recovery, dependence or death after discharge. J Gen Intern Med 2013 Mar; 28(3): 342. doi: 10.1007/s11606-012-2311-2.

31   Sabour S, Dastjerdi EV, Moezizadeh M, Geha H. Effect of JPEG compression on the diagnostic accuracy of periapical images; mistakes and misinterpretations. Dent Traumatol 2013 Jun; 29(3): 251. doi: 10.1111/edt.12014. Epub 2012 Nov 6

32   Sabour S. Reliability and repeatability of toe pressures measured with laser Doppler and portable and stationary photoplethysmography devices. Ann Vasc Surg 2012 Nov; 26(8): 1167. doi: 10.1016/j.avsg.2012.05.008.

33     Sabour S. Validity of chronic obstructive pulmonary disease diagnoses in a large administrative database: The rule of thumb in validity analysis of a test. Can Respir J 2012 Sep-Oct; 19(5):331; author reply 331.

34    Sabour S. Commentary on “Reliability of two different presurgical preparation methods for implant dentistry based on panoramic radiography and cone-beam computed tomography in cadavers”. J Periodontal Implant Sci, 2012 Aug; 42(4): 144. doi: 10.5051/jpis.2012.42.4.144. Epub 2012 Aug 31.

35   Sabour S, Ghassemi F. The reproducibility of measurements of differential renal function in paediatric 99mTc-MAG3 renography: is this correct? Nucl Med Commun 2012 Dec; 33(12): 1311; author reply 1311-2. doi:10.1097/MNM.0b013e328359453a.

36   Sabour S. Reliability of subjective, linear, ratio, and area cephalometric measurements in assessing adenoid hypertrophy among different age groups. Angle Orthod 2012 Sep; 82(5): 948; author reply 949-50. doi: 10.2319/0003-3219-82.5.948.

37     Sabour S, Dastjerdi EV. Reliability of Soft Tissue Model Based Implant Surgical Guides; A Methodological Mistake. J Oral Implantol 2012 Aug 20. [Epub ahead of print]

38    Sabour S, Ghassemi F. Accuracy, validity, and reliability of the infrared optical head tracker (IOHT). Invest Ophthalmol Vis Sci 2012 Jul 13; 53(8): 4776. doi: 10.1167/iovs.12-10324.

39   Sabour S, Ghassemi F. Validity of self-reported eye disease and treatment: is that correct?! Br J Ophthalmol 2012 Oct; 96(10): 1359. Epub 2012 Jul 11

40    Sabour S, Ghassemi F. Reliability and validity of conjunctival ultraviolet autofluorescence measurement. Br J Ophthalmol 2012 Sep; 96(9): 1271; author reply 1271. doi: 10.1136/bjophthalmol-2012-302087. Epub 2012 Jun 13

 

Peer reviewers: Siamak Sabour, MD, PhD, Community Oral  Health, Clinical Epidemiology,  School  of  Dentistry,  Chamran highway, Tabnak Blv. Velenjac, Tehran, Iran; Siamak Sabour, MD, PhD, Community Oral  Health, Clinical Epidemiology,  School  of  Dentistry,  Chamran highway, Tabnak Blv. Velenjac, Tehran, Iran.

 

 

 

 

 

 

 

 

 

 

 

 

 

Refbacks

  • There are currently no refbacks.