Analysis of undergraduate pharmacology question papers at Goa medical college as regards to their content areas

INTRODUCTION

Undergraduate medical education aims at sculpting the future physicians who have to deal with human ailments and their lives. Assessment forms a very important tool to ascertain what they have learnt. Students usually concentrate their studies on what is asked or what could be asked in the examinations. As teachers it is our duty to guide them towards proper learning and to apply this in practice. In this way we also indirectly protect the public from improper practices of the future physicians.¹ As such our examination questions should be in proper direction.

Assessment should match the contents of the course and should provide proportional weightage to each of the content.² We know that students learn what is asked in examinations and hence it is important to ask the right questions. Thus a question paper in the form of a written examination forms a very important tool of the assessment.

Validity and reliability are vitally important attributes in assessment. Irrespective of the type of assessment tool, validity of content should always be carried out. It should include at least the content validity and the construct validity, because other measures of validity like concurrent and predictive validity, though relevant but can be more difficult to determine.³

Written examination mainly aims at assessing the knowledge where as viva voce and practical are clinically oriented.⁴ This study is a question paper based study and intends to throw light on the weightage given to different content areas in 2nd MBBS pharmacology university examinations and also to correlate the weightage with time allocated to them in the form of teaching hours.

Aims and objectives

1. To analyze weightage given to different topics and subtopics in 2nd MBBS pharmacology university written theory examinations.

2. To correlate the topic and subtopic weightage with the teaching hours allotted to each.

METHODS

In this retrospective study we evaluated the pharmacology university theory examination question papers of II^nd MBBS students at our institute .The examinations are carried out twice in a year: January and July. The students have to answer two question papers: paper I and paper II, each being of 40 marks .There are two questions in each paper containing short notes where there is an option to answer any three out of four short notes asked. Thus for analysis, total marks per paper with these optional questions became 44. We analyzed question papers from the year 2000 to 2016, total being 64 papers, 32 papers of paper I and 32 papers of paper II. We analyzed all the questions for allocation of marks, their frequency of occurrence and weightage to content areas as regards to the syllabus and expressed them as percentage. Weightage of content areas was also compared to the time allotted to various topics in the form of lectures.

RESULTS

The syllabus was divided into two papers, paper I and paper II as illustrated in Table 1.

Overall weightage of the topics in pharmacology question papers over the years was as follows: Chemotherapy (16.74%), ANS (12.65%), CNS and LA(11.87%), Endocrine (11.19%), CVS (11.16%), General Pharmacology (7.60%), Blood (6.40%), RS(4.76 %), Autacoids (4.55%), GIT (3.91%), NSAIDs, Gout, RA (3.38%), Kidney (2.91%), miscellaneous (2.88%). Table 2 and 3 illustrate weightage of topics in paper I and paper II.

Certain topics were given more weightage consistently over the others. Autonomic nervous system (25.28%) and cardiovascular system (22.30%) in paper I and chemotherapy (33.50%) and Central Nervous System (23.76%) in paper II were the maximum weightage areas (table 2 and 3). Subtopic weightage distribution of these major content areas is given in table 4, 5, 6 and 7.

Seven instances were observed where questions from paper I were asked in paper II or vice versa. Questions were from the following topics- Autacoids (2), ANS(1), NSAIDS(1), Gout(1), Endocrine(1) and RA(1).

Weightage of content areas over 15 years is illustrated in Table 8 and 9. It was observed that some aspects remained uncovered in a few papers- Autacoids, kidney, GIT, NSAIDs, miscellaneous. These drugs are of complete clinical importance for a general practitioner who often has to treat very common ailments like fever, allergies, insect bites, acidity, diarrhoeas etc. Thus students will not lay stress on learning to treat these common ailments.

Distribution of marks was observed to be in proportion to the lecture time allotted to the topics as seen in Table 2 and 3, and Figure 1 and 2.

Positive correlation was observed with correlation coefficient in paper I: 0.95 and paper II: 0.98.

DISCUSSION

Assessment is the process of gathering information on students learning. Evaluation is the process of analyzing, reflecting upon and summarizing assessment information and making judgments and decisions based on the information collected.

Assessment and evaluation are important components of teaching and learning. There is a profound and well established link between the quality of assessment and evaluation to the students’ performance in educational process. Research consistently shows that monitoring and feedback are essential to improve students’ learning. What is assessed and evaluated, how it is done and how results are communicated send a clear message to students about what is really valued, what is worth learning and how it should be learnt, what elements of study are most important and how well students are expected to learn.

Assessment can be classified in many ways, one of it is as formative or summative. An educator’s view would be that formative assessment has important role in learning process because it is conducted with primary purpose of providing feedback to students and teachers. The ability of assessment to discriminate effectively between good and poor candidates, as well as the fidelity of the assessment is also an important consideration in evaluating the assessment tool.³

The assessment results help in improving the quality of medical education by providing data on students’ achievements; hence for the validity of the assessment it is important to have a proper coverage of the curriculum.

A good assessment should have the following attributes as mentioned by Van der Vleuten.et al –validity, reliability, educational impact of assessment and feasibility, costs and acceptability of the tests. The validity of the assessment is said to be the degree of accuracy with which it measures the aspect which is to be assessed. However the contemporary view of validity is to make a more direct assessment of clinical competence by increasing the authenticity of assessment.⁵ One of the better, time tested analytic guidelines for such assessment systems is Miller’s pyramid. Miller conceptualized clinical competence in the shape of a pyramid which has four levels, base of which is formed by ‘Knows’ followed by ‘knows how’, ‘shows how’ and ‘does’. This shows that the upper two levels which represent performance should be supported by a wide and solid foundation of knowledge represented by the lower two levels. This is often a neglected fact since it is assumed that physician should be assessed on practical skills and knowledge is of academic interest only. Knowledge is the best predictor of clinical competence. The onus is then placed on the educator to find the degree of knowledge the student has imbibed. The most convenient means of achieving this is the written assessment.

This study aims at analyzing the content validity of the theory written examination in Pharmacology in University examinations at Goa Medical College wherein it was observed that the syllabus was defined but marks allotted to each topic were not defined. There are no set guidelines for distribution of marks to various topics, and hence paper setters used their own judgment to set the papers. Autonomic nervous system and cardiovascular system in paper I and chemotherapy and central nervous system in paper II were the topics which received high weightage consistently. While certain aspects like Autacoids, kidney, GIT, NSAIDs and miscellaneous remained uncovered in a few papers. These topics include a few commonly prescribed drugs in clinical practice like analgesics, diuretics, H₂ blockers etc. It is important that while setting a paper the topics should be selected based on practical importance, and rare and unusual topics can be avoided to lessen students’ burden of learning.

Thus consistency can be ensured if there are set guidelines for marks distribution to the topics, based on which papers can be set. Similar studies done in other subjects in Rajasthan and Pondicherry also impress upon the requirement of guidelines for proper distribution of weightage to the content areas.^6,7 Allocation of weightage to various topics usually depends on two criteria (1) the perceived impact/importance of a topic in terms of its impact on health (2)The frequency of occurrence of a particular disease or the health problem.⁸

Blueprinting and good sampling of the content is very helpful to ensure content representation. A blueprint provides a template for the question paper setter and the examiner to assess all that is expected from a student at the end of a learning session. It specifies the content areas topics, the domains of learning and the appropriate methods or tools of assessment. Therefore it serves as a reference framework for the question paper setter to prepare questions according to the accepted norms and guidelines.⁹

CONCLUSION

For consistency of validity in the assessment, we suggest that it should be based on structured guidelines. Methods like test blueprinting and table of specifications may be used. Frequent analysis of methods of teaching and assessment should be carried out by the whole faculty involved in teaching and assessment in the department, look for any lacunae and improve upon these lacunae. Thus teaching and assessment should be periodically reviewed and improved upon.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the institutional ethics committee

REFERENCES

1. 1. Ronald ME. Assessment in Medical Education. N Engl J Med. 2007;356:387-96.
2. 2. Tejinder S. Basics of assessment. In: Tejinder S, Anshu, editors. Principles of assessment in medical education. India: Jaypee Brothers Medical Publishers; 2012: 1-13.
3. 3. Edward J P, Peter GD. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Medical Education. 2007;7:49.
4. 4. Anshu. Assessment of knowledge:Written assessment. In: Tejinder S, Anshu, editors. Principles of assessment in medical education. India: Jaypee Brothers Medical Publishers; 2012: 70-79.
5. 5. Tejinder S. What to assess. In: Tejinder S, Anshu, editors. Principles of assessment in medical education. India: Jaypee Brothers Medical Publishers; 2012: 14-24.
6. 6. Rohin G, Dhiraj S, Sushila S, Neha D. Analytical study of written examination papers of undergraduate anatomy: Focus on its content validity. IJBAMR. 2013;2(8):1110-6.
7. 7. Ananthakrishnan N, Ananthakrishnan S, Oumachigui A. MBBS examinations--are we asking the right questions. J Postgrad Med. 1993;39:31.
8. 8. Sylvain C, Wayne W, Kevin M, Twelve tips for blueprinting. Med Teach. 2009;31:322–4.
9. 9. Adkoli BV, Deepak KK. Blueprinting in assessment. In: Tejinder S, Anshu, editors. Principles of assessment in medical education. India: Jaypee Brothers Medical Publishers; 2012: 215-213.