Open AccessEnhancing Course Evaluation with Achievement Pathways in Outcome-Based Education Frameworks
View Full Article|
Download PDFAbstract:
Course evaluation, a critical component for the implementation of outcome-based education (OBE), provides substantial data support. The reliability, validity, and discriminative power of evaluation results are significantly influenced by the choice of course evaluation methods. An effective course evaluation method identifies weak links in the teaching process, offering a foundation and reference for continuous course improvement. This study introduces a course evaluation method based on achievement pathways, establishing the supportive relationship among course-related graduation requirement indicators, course objectives, and achievement pathways. Grounded on formative assessment, a system to quantify the achievement of teaching objectives in courses is constructed. The method has been applied to courses, such as Data Visualization and Software Engineering, at the Beijing Institute of Petrochemical Technology. Practice demonstrates that this method is capable of identifying weaknesses in the course implementation process, providing theoretical foundation and reliable assurance for ongoing course improvement.1. Introduction
In the OBE model, the learning outcomes of students are considered the most powerful and direct evidence for evaluating the effectiveness of talent cultivation. Educators are required to have a clear understanding of the capabilities and levels students should achieve upon graduation, and then seek to design appropriate educational methods and teaching means to ensure these anticipated goals are met. As the fundamental carrier of student learning, courses play a crucial role in supporting the achievement of graduation requirements and realizing training objectives. The student-centered philosophy of OBE emphasizes that teaching objectives precede teaching content, advocating for reverse instructional design based on the evaluation of course objective achievement to guide continuous improvement in teaching activities. In this process, course evaluation serves as an essential support for continuous course improvement and ensuring the quality of professional talent cultivation, holding significant importance.
Research on course evaluation typically employs either summative or formative evaluation methods to assess course objectives. Summative evaluation, often presented in the form of exams, tests, or reports, predominantly assesses knowledge content and is more readily quantified through objective scores. In contrast, formative evaluation assesses not only knowledge but also the soft skills students exhibit in their daily learning processes, such as performance, attitude, abilities, and literacy. Summative evaluation is conducted after the course completion for continuous improvement at the course level, while formative evaluation allows for timely feedback adjustments, fostering maximized teacher and student growth. This study combines summative and formative evaluations, proposing a quantifiable course objective evaluation system based on the achievement pathways of courses. This system effectively addresses the measurability issue in quantifying formative evaluations, enabling teachers and students to monitor the achievement status of teaching objectives in real-time throughout the teaching process. As a result, it facilitates instructional feedback and strategy adjustments for continuous course improvement.
2. Methodology
In OBE, course evaluation is identified as an activity that assesses the implementation process and outcomes of a course based on its objectives, serving instructional decision-making. It is a process where the real or potential effects of teaching activities are judged. Comprehensive and holistic evaluations of the entire course implementation process are conducted, positioning course evaluation as the logical endpoint of the teaching process, yet also as the starting point for new teaching processes. Teaching, being a complex process activity, does not guarantee immediate improvements through course evaluation, but it allows for a deeper understanding of the relationship between student emotional needs and their agency (Bacchus et al., 2020). Course evaluation clarifies the direction of course teaching design, guides teaching activities towards more effective approaches, and indicates the direction of effort for both teachers and students, thus holding significance in guiding teaching practices.
Yang et al. (2021) conducted a multivariate linear regression on evaluative indicators for student learning development, theoretically constructing an evaluative indicator system for student learning engagement and improving the evaluation indicator system using the fuzzy comprehensive evaluation method. Ma et al. (2022). utilized the Analytic Hierarchy Process (AHP) based on course assessment scores, assigning weight vectors to different course objectives for corresponding chapters, calculating the achievement levels of different chapters, and thereby obtaining the overall course achievement level. Xu et al. (2020), on the basis of analyzing the support relationship between course teaching content and student capability achievement, employed course assessment score analysis and rubric analysis methods to calculate achievement levels, forming a forward implementation, reverse evaluation teaching evaluation method. Yuan (2019) applied the Delphi method, using controlled feedback group communication as an expert opinion collection method to avoid the weaknesses of subjective weighting, establishing the relationship between graduation requirement indicators and supporting courses. Wang et al. (2024) proposed a "dual-thinking union" multi-granularity graduation requirement evaluation method addressing the granularity issue of evaluation for graduation requirements in broad-category enrollment majors. This method evaluates from a micro-level according to course groups, from a macro-level according to individual students, objectively using survey questionnaires for evaluation, and subjectively using achievement level calculations for evaluation.
Yan et al. (2023) emphasized the dual role of students as both subjects of classroom teaching and evaluation, advocating for an evaluation of both teaching and learning quality. On this basis, a student evaluation of teaching indicator system was constructed to promote the digital transformation of classroom teaching quality through student evaluations. Li & Hou (2023) developed an improvement algorithm for course objectives and graduation requirements achievement levels based on the conversion of "qualitative judgment weights to quantitative composite weights" and "evaluation object decomposition into corresponding support relationships", achieving the calculation of course objective achievement levels. Zhang (2021) utilized factor analysis to analyze the achievement of student graduation requirements, extracting eight indicators of graduation requirement achievement. The factor score coefficient matrix was improved based on the explanatory significance of the extracted indicators, and the entropy weight method was applied to determine the weights of each indicator and calculate the graduation requirement achievement levels. Wang et al. (2023) constructed a basic process and evaluation method system for course objective achievement evaluation. Using the course of deformation monitoring and data processing as an example, the relationship among graduation requirements, course objectives, and course content was established to analyze the course objective achievement. Li (2023) established a model for the quantitative evaluation of the achievement levels of course teaching objectives through establishing the match between evaluation content and core competency indicators, encompassing the entire course teaching process, and setting a unified quantitative evaluation standard, implemented in the teaching of engineering drawing courses.
Li (2021) began with the analysis of the indicator-course corresponding matrix, applying a graded and hierarchical method to determine the support degree of courses for indicators. Based on teachers’ qualitative evaluations of indicators and students’ self-evaluations, combined with the operation rules of qualitative data, a qualitative evaluation of the achievement of course indicators was conducted. Biaer (2019) employed an open-ended survey questionnaire to understand the learning experiences of students and peers within learning communities, capturing content not accessible through quantitative analysis, used to assess learning outcomes. Kilgour et al. (2020) proposed a joint co-construction method, allowing students and teachers to collaboratively design course evaluation criteria. Data collection was achieved through an improved Delphi method, questionnaires, focus groups, and interviews, facilitating the assessment of learning outcomes. Hanefar et al. (2022) noted the transition from summative to formative evaluation in course evaluation, collecting data through surveys and semi-structured interviews to investigate the impact of formative evaluation on teaching practices. Ana et al. (2020) used electronic standards to assess students’ skills and abilities, enabling learners to present their learning situations in a real and interpretable manner, while also allowing students to monitor their progress and weaknesses. Yulianto (2022) applied a qualitative descriptive method in the Merdeka Belajar course, integrating diagnostic, formative, and summative assessment modes, considering assessment as an inseparable part of learning, not only to determine the achievement of learning outcomes but more importantly to enhance students’ abilities during the learning process.
Tractenberg (2021) discussed an Assessment and Evaluation Reporting (AER) method based on semi-qualitative analysis, through which teachers could evaluate the outcomes of educational assessments to ensure these outcomes foster learning or embody the student-centered educational philosophy. This serves as a means to improve instructional design and optimize teaching measures. Sasipraba et al. (2020), drawing on the characteristics of projects at Sathyabama Institute of Science and Technology, categorized students into groups, defining distinct COs-POs mapping sets for each group. The performance of students in capstone projects was discussed, and a capstone project evaluation framework was established to assess learning outcomes.
Existing research has investigated course evaluation methods from both quantitative and qualitative perspectives, with some developing course evaluation systems and others assessing the evaluation framework. While graduation requirement indicators are supported by the professional course system, they are not synonymous with teaching objectives, as different teaching objectives and design philosophies may vary. A mere superficial structural connection between graduation requirement indicators and teaching objectives is insufficient; an intrinsic substantive link is also required to facilitate the transformation of talent cultivation models and teaching methods (Meng et al., 2021). This study begins by discussing the relationship between course-related graduation requirement indicators, course objectives, and achievement pathways. A quantitative course evaluation method based on achievement pathways is proposed, combined with a survey questionnaire to test the method’s effectiveness and reasonableness. The practical application of this method in courses is also elaborated.
3. The Course Evaluation Method Based on Achievement Pathways
The course evaluation method based on achievement pathways is constructed under the constraint of the course’s support for graduation requirement indicators, based on course objectives and achievement pathways, following a scientific method to create a specific variable system for evaluating the achievement of course objectives. Starting from student sample data, this method assesses and scores students’ performance throughout the course learning process, using process assessment materials to quantitatively calculate the achievement of each course objective and graduation requirement indicator.
Graduation requirement indicators supported by the course from various dimensions delineate the knowledge, skills, and competencies required to master the course. The teaching objectives of the course provide close support for the graduation requirement indicators. Under the constraint of these indicators, and combined with the content characteristics of the course, the teaching objectives to be achieved by the course are formulated. Thus, an evaluative relationship between course objectives and graduation requirement indicators is established.
The establishment of course objectives offers crucial support for the design of course content, the implementation of teaching processes, and the evaluation of the course. The implementation of the teaching process relies on a series of specific and effective teaching segments, namely the pathways to achieve course objectives. Each course objective may be supported by multiple achievement pathways, each representing one or some course objectives from different perspectives. All achievement pathways of a course collectively represent all course objectives, thereby establishing an evaluative relationship between achievement pathways and course objectives.
The course evaluation method based on achievement pathways encompasses the construction of course objectives, the establishment of evaluation weights, and the calculation of achievement levels. The construction of course objectives is grounded on the course’s support for graduation requirement indicators, determining the teaching objectives the course aims to achieve. A series of reasonable and effective achievement pathways are designed to achieve these teaching objectives, clarifying each pathway’s support for the course objectives, thereby establishing an evaluative relationship among achievement pathways, course objectives, and graduation requirement indicators. Based on this evaluative relationship, the achievement of course objectives by student sample data within the achievement pathways is assessed, further evaluating the achievement of course objectives in relation to graduation requirement indicators, thus obtaining the evaluation results of the course based on achievement pathways.
The Bloom’s Taxonomy of Educational Objectives divides educational objectives into the dimensions of knowledge and cognitive processes (Anderson, 2018). The formulation of course objectives necessitates a comprehensive consideration of the retention and transfer of student learning. On the basis of providing essential knowledge and cognitive processes, emphasis is placed on the cultivation and training of students’ constructivist learning abilities. The development of student capabilities is systematic; graduation requirements and capabilities do not exist in isolation but are interconnected within a systematic whole according to certain logical relationships. Therefore, graduation requirement indicators and course objectives are intricately linked, supporting each other mutually.
Assuming the number of graduation requirement indicators supported by the course is denoted as $m$, with $m \geq 1$, the set of related graduation requirement indicators is defined as $A=\left\{a_1, \ldots, a_m\right\}$, the number of course objectives is denoted as $n$, with $n \geq 1$, the set of course objectives is defined as $B=\left\{b_1, \ldots, b_n\right\}$, with $n \geq m$, the support relationship between related graduation requirement indicators and course objectives can be expressed as $f_1$, then $f_1: A \rightarrow \pi_B$, where $\pi_B=\left\{B_1, \ldots, B_u\right\}$, with $u \geq 1$, and $B_1, \ldots, B_u$ represents a partition of the set $B$ of course objectives.
Achievement pathways facilitate the realization of course objectives, which in turn support the achievement of graduation requirement indicators. Within this two-tier support relationship, the contribution of each achievement pathway to different course objectives and the contribution of each course objective to different graduation requirement indicators vary. If evaluation weights are used to express this contribution, then the evaluation weights of different achievement pathways towards course objectives and the evaluation weights of different course objectives towards graduation requirement indicators can differ.
The evaluation weights of course graduation requirement indicators. Each course contributes differently to the professional graduation requirements; hence, the support of different courses to the professional graduation requirements can have varying weights, meaning the evaluation weights of different course graduation indicators themselves can be different. Following a method similar to that described in the study by Gregori-Giralt & Menéndez-Varela (2021), and adhering to Messick’s unified construct validity theory, an expert panel is assembled. The evaluation weights of the course graduation requirement indicators are quantified by several domain experts based on the professional educational philosophy, training objectives, course system, and the contribution level of courses to graduation requirements, followed by mean calculation to determine quantitative evaluation weights.
The evaluation weights of course objectives towards graduation requirement indicators. The evaluation weights of course objectives towards graduation requirement indicators are quantified by a course expert group based on the course graduation requirement indicators and the status and role of the course, followed by mean calculation, and finally confirmed for reasonableness by the profession.
The evaluation weights of achievement pathways towards course objectives. The evaluation weights of the course achievement pathways themselves, as well as the evaluation weights of achievement pathways towards course objectives, are quantified by the course instructors based on the course objectives and achievement pathways, followed by mean calculation, and finally confirmed for reasonableness by the course expert group.
Assuming there are $p$ achievement pathways, with $p \geq 1$, the set of achievement pathways is defined as $C=\left\{c_1, \ldots, c_p\right\}$, with $p \leq m$, the support relationship between achievement pathways and course objectives is expressed as $f_2$, then $f_2: B \rightarrow 2^C$, where each element in $2^C$ is a non-empty subset of set $C$. The evaluation weight is represented by $W$, leading to the following conclusions:
(a) If $W_C\left(c_i\right)$, with $a_i \in A$, represents the evaluation weight of the $i$-th related graduation requirement indicator, then $W_A\left(a_i\right) \in[0,1]$, and the evaluation weight of this course within the entire professional course system is $\sum_{i=1}^m W_A\left(a_i\right)$.
(b) If $W_C\left(c_i\right)$, with $c_i \in C$, represents the evaluation weight of the $i$-th achievement pathway, then $W_C\left(c_i\right) \in[0,1]$, and $\sum_{i=1}^p W_C\left(\mathrm{c}_i\right)=1$.
(c) If $W_C^B(i, j)$ represents the evaluation weight of the $j$-th course objective within the $i$-th achievement pathway, for $\forall c_i \in C$ and $\forall b_i \in B$, the following three conditions should be met:
① $W_C^B\left(c_i, b_j\right) \in[0,1]$; ② $\sum_{j=1}^n W_C^B\left(c_i, b_j\right)=1$; ③ $\sum_{j=1}^n \sum_{i=1}^p W_C^B\left(c_i, b_j\right)=p$
The determination of evaluation weights signifies the establishment of proportions for all assessment content during the course implementation process. Based on the scores obtained by student samples on various achievement pathways during the teaching implementation process, and considering the support relationship of achievement pathways towards course objectives, the achievement levels of each course objective within the achievement pathways can be calculated. Furthermore, based on the support relationship between course objectives and related graduation requirement indicators, the achievement level of each related graduation requirement indicator within each achievement pathway can be calculated. On this basis, by summarizing all achievement pathways, the achievement levels of each related graduation requirement indicator supported by course objectives can be determined. Finally, the final achievement level of the course can be calculated based on the weights of the related graduation requirement indicators and the achievement level of each graduation requirement indicator, as shown in Algorithm 1.
Algorithm 1: Algorithm for calculating achievement levels
Input: Set $A$ of related graduation requirement indicators, weight factors $W_A\left(\mathrm{a}_i\right)$ of course-related graduation requirement indicators, set $B$ of course objectives, set $C$ of achievement pathways, weight factors $W_c\left(c_i\right)$ of achievement pathways, weight factors $W_c^B\left(c_i, b_j\right)$ of the $k$-th course objective in the $j$-th achievement pathway, scores $q$ of students in the sample set $S$ Output: Course achievement level $T$ (a) Based on the mapping relationship between $f_1$ and $f_2$ (b) Calculate the weight $W_c^A\left(c_i, a_l\right)$ of the $l$-th graduation requirement indicator in the $j$-th achievement pathway (c) For $i$= 1 to $q$ do // For the size of each student sample (d) For $j$= 1 to $p$ do // For each achievement pathway (e) For $k$= 1 to $n$ do // For each course objective (f) Calculate the achievement situation $T_i\left(c_j, b_k\right)$ of the corresponding course objective based on the score and weight factor $W_c^B\left(c_j, b_k\right)$ of each sample in the $k$-th course objective of the $j$-th achievement pathway (g) For $l$= 1 to $m$ do // For each graduation requirement indicator Calculate the achievement situation $T_i\left(a_i\right)$ of the $l$-th graduation requirement indicator in the $j$-th achievement pathway for each sample based on $T_i\left(c_j, b_k\right)$ and $W_c^A\left(c_j, a_l\right)$ (h) For $j$= 1 to $m$ do // For each graduation requirement indicator (i) Calculate the achievement situation $T_i\left(a_j\right)$ of the $j$-th graduation requirement indicator in all achievement pathways for each sample (j) For $j$= 1 to m do // For each graduation requirement indicator (k) Calculate the mean achievement level $\operatorname{ave}\left(b_j\right)=\sum_{i=1}^q T_i\left(a_j\right) / q$ of the $j$-th graduation requirement indicator (l) $T+=\operatorname{ave}\left(b_j\right) * W_A\left(a_j\right)$ (m) Output the course achievement level $T=T / \sum_{i=1}^m W\left(a_i\right)$ |
Students, being the primary agents of classroom and course learning, play a crucial role in the student-centered OBE educational philosophy, which emphasizes the importance of conducting teaching from the perspectives and stances of students. Student participation in assessing their learning positively impacts addressing student needs and implementing improvements (Montenegro & Jankowski, 2020). To evaluate the effectiveness and rationality of the course evaluation method based on achievement pathways, this study employed a survey questionnaire approach. By designing and distributing questionnaires to students who selected the course, opinions on the rationality of the setting of course objectives and evaluation weights were obtained, along with self-evaluations of the achievement of course objectives by the students. Through self-assessment, student sample data were collected and processed by mean calculation to obtain a qualitative understanding of the students’ mastery of course objectives across different achievement pathways, as shown in Figure 1. Figure 2 shows the survey statistics on the achievement of course objectives. The survey results indicated that approximately 90% of students deemed the setting of course objectives and evaluation weights as reasonable; 80%-90% of students found the results of the achievement level calculation based on achievement pathways to be at least equal to their expectations, suggesting that the course evaluation method based on achievement pathways is both reasonable and effective.
4. Application Case
This section illustrates the practical application of the course evaluation method based on achievement pathways, using the Data Visualization course offered in the big data major of the Beijing Institute of Petrochemical Technology as an example. Data Visualization is a compulsory course for the big data major, with an evaluation weight supporting graduation requirements of 1.2. The number of graduation requirement indicators supported by this course is , and their corresponding evaluation weights are as shown in Table 1.
Graduation requirements | Graduation Requirement Indicators | Evaluation Weight of the Graduation Requirement Indicator |
|---|---|---|
1: Engineering knowledge | 1.5 Mastery of fundamental data science knowledge applicable to data management and analysis. | 0.2 |
3: Design/development of solutions | 3.4 Capability to decompose and refine complex engineering problems in big data for specific needs, with design and implementation skills in large-scale data analysis, intelligent computing, and visualization, reflecting innovation. | 0.2 |
4: Research | 4.3 Ability to analyze and interpret data, obtaining reasonable and effective conclusions through information synthesis. | 0.2 |
5: Use of modern tools | 5.1 Proper selection of modeling and simulation tools for simulation and analysis in data science and big data engineering projects. | 0.3 |
5.3 Development of appropriate technologies and resources, and the use of software development tools for data acquisition, processing or analysis, or for the development of big data application systems. | 0.2 | |
12: Lifelong learning | 12.2 Ability to track the development dynamics and industry demands in the fields of data science and big data technology, with the capacity for continuous learning and adaptation. | 0.1 |
The course objectives correspond with the graduation requirement indicators, with an evaluation weight of 1 for each, as indicated in Table 2.
Course Objectives | Graduation Requirement Indicators | Evaluation Weight of Course Objectives Towards the Graduation Requirement Indicator |
|---|---|---|
1: Ability to accumulate basic principles and application technologies of Data Visualization for use in data management and analysis, fostering a factual and scientific attitude. | 1.5 | 1 |
2: Ability to decompose and refine complex engineering problems in big data for specific needs, with innovative design and implementation capabilities in large-scale Data Visualization. | 3.4 | 1 |
3: Ability to analyze and interpret data visually, obtaining reasonable and effective conclusions through information synthesis, enhancing national pride, and cultivating patriotism. | 4.3 | 1 |
4: Ability to properly select visualization modeling tools for the simulation analysis of big data engineering projects. | 5.1 | 1 |
5: Ability to properly use software development tools for Data Visualization analysis in application systems. Spreading Chinese culture, firmly believing in the superiority of the socialist system of China. | 5.3 | 1 |
6: Ability to continuously explore new methods and technologies in Data Visualization. | 12.2 | 1 |
The number of achievement pathways for the course is $p=4$, which include regular assignments, classroom quizzes, experimental assessments, and final exams. The support relationship of achievement pathways towards course objectives along with their evaluation weights are shown in Table 3.
Achievement Pathways | Course Objectives | Evaluation Weights |
Regular grades | Course objective 2 | 0.2 |
Course objective 3 | 0.3 | |
Course objective 4 | 0.2 | |
Course objective 6 | 0.3 | |
Classroom quizzes | Course objective 1 | 0.5 |
Course objective 2 | 0.5 | |
Experiments | Course objective 2 | 0.2 |
Course objective 3 | 0.2 | |
Course objective 4 | 0.3 | |
Course objective 5 | 0.3 | |
Final exams | Course objective 1 | 0.2 |
Course objective 2 | 0.3 | |
Course objective 3 | 0.4 | |
Course objective 6 | 0.1 |
By comparing each student sample’s score for each course objective on each achievement pathway with the full score of the course objective on that pathway, the achievement situation of each student sample for each course objective within each achievement pathway can be calculated, as shown in Table 4. For example, if student sample 310811’s regular grade score for Course Objective 2 is 1.8 out of a full score of 2, then the achievement of this student’s regular grade for Course Objective 2 is calculated as 1.8/2 = 0.9. Using the same method, the achievement of this student’s regular grades towards each course objective can be calculated.
Major | Big Data Major | |||||||||
Course objective | 2 | 3 | 4 | 6 | Regular grades (full score: 10 points) | |||||
Evaluation weight | 0.2 | 0.3 | 0.2 | 0.3 | ||||||
Name/Score/Statistics | Score (full score: 2 points) | Achievement value of course objective 2 | Score (full score: 3 points) | Achievement value of course objective 3 | Score (full score: 2 points) | Achievement value of course objective 4 | Score (full score: 3 points) | Achievement value of course objective 6 | Score (full score: 10 points) | Achievement value of regular grades |
310808 | 2.00 | 1.00 | 2.50 | 0.83 | 2.00 | 1.00 | 2.80 | 0.93 | 9.30 | 0.93 |
310809 | 1.90 | 0.95 | 3.00 | 1.00 | 1.50 | 0.75 | 2.70 | 0.90 | 9.70 | 0.97 |
310810 | 2.00 | 1.00 | 2.50 | 0.83 | 2.00 | 1.00 | 3.00 | 1.00 | 10.00 | 1.00 |
310811 | 1.80 | 0.90 | 3.00 | 1.00 | 2.00 | 1.00 | 3.00 | 1.00 | 10.00 | 1.00 |
310812 | 2.00 | 1.00 | 2.80 | 0.93 | 1.50 | 0.75 | 2.90 | 0.97 | 9.20 | 0.92 |
310813 | 2.00 | 1.00 | 3.00 | 1.00 | 2.00 | 1.00 | 2.50 | 0.83 | 9.50 | 0.95 |
310814 | 2.00 | 1.00 | 3.00 | 1.00 | 2.00 | 1.00 | 2.80 | 0.93 | 9.80 | 0.98 |
310815 | 2.00 | 1.00 | 3.00 | 1.00 | 2.00 | 1.00 | 3.00 | 1.00 | 10.00 | 1.00 |
… | … | … | … | … | … | … | … | … | … | … |
Class average | 1.98 | 0.94 | 2.93 | 0.95 | 1.97 | 0.96 | 2.95 | 0.93 | 9.83 | 0.98 |
Achievement value | 0.94 | 0.95 | 0.96 | 0.93 | 0.98 | |||||
Subsequently, the achievement for each student sample towards each course objective is calculated. For example, the achievement of Course Objective 1 for student 310811 is calculated as (0.98*0.5+0.75*0.2) / (0.5+0.2)=0.91. The calculation results for the student samples are illustrated in Table 5, and through mean calculation, the class average achievement value for Course Objective 1 is determined to be 0.89.
Major | Big Data | |||||||||||||||||||
Course objective | 1 | 2 | 3 | 4 | 5 | 6 | ||||||||||||||
Evaluation weight | 0.5 | 0.2 | 0.7 | 0.2 | 0.5 | 0.2 | 0.3 | 1.2 | 0.3 | 0.2 | 0.4 | 0.2 | 0.2 | 0.3 | 0.5 | 0.3 | 0.3 | 0.3 | 0.1 | 0.4 |
Name/Score/Statistics | Classroom quiz achievement | Final exam achievement | Achievement of course objective 1 | Regular grade achievement | Classroom quiz achievement | Experiment achievement | Final exam achievement | Achievement of course objective 2 | Regular grade achievement | Experiment achievement | Final exam achievement | Achievement of course objective 3 | Regular grade achievement | Experiment achievement | Achievement of course objective 4 | Experiment achievement | Achievement of course objective 5 | Regular grade achievement | Final exam achievement | Achievement of course objective 6 |
310808 | 0.9 | 0.675 | 0.84 | 1 | 1.00 | 0.75 | 0.67 | 0.88 | 0.83 | 0.88 | 0.68 | 0.77 | 1.00 | 0.83 | 0.90 | 0.83 | 0.83 | 0.93 | 0.65 | 0.86 |
310809 | 1 | 0.75 | 0.93 | 0.95 | 0.96 | 0.50 | 0.77 | 0.83 | 1.00 | 0.50 | 0.75 | 0.88 | 0.75 | 0.50 | 0.60 | 0.33 | 0.33 | 0.90 | 0.70 | 0.85 |
310810 | 0.9 | 0.85 | 0.89 | 1 | 1.04 | 0.88 | 0.83 | 0.95 | 0.83 | 0.75 | 0.85 | 0.82 | 1.00 | 0.83 | 0.90 | 0.92 | 0.92 | 1.00 | 0.80 | 0.95 |
310811 | 0.98 | 0.75 | 0.91 | 0.90 | 1.00 | 0.88 | 0.73 | 0.90 | 1.00 | 0.88 | 0.75 | 0.86 | 1.00 | 0.83 | 0.90 | 0.83 | 0.83 | 1.00 | 0.70 | 0.93 |
310812 | 0.9 | 0.8 | 0.87 | 1.00 | 0.94 | 0.88 | 0.80 | 0.90 | 0.93 | 0.88 | 0.80 | 0.86 | 0.75 | 0.92 | 0.85 | 1.00 | 1.00 | 0.97 | 0.80 | 0.93 |
310813 | 1 | 0.75 | 0.93 | 1.00 | 0.96 | 0.88 | 0.77 | 0.90 | 1.00 | 0.88 | 0.75 | 0.86 | 1.00 | 0.92 | 0.95 | 0.92 | 0.92 | 0.83 | 0.70 | 0.80 |
310814 | 0.96 | 0.65 | 0.87 | 1.00 | 0.96 | 0.88 | 0.67 | 0.88 | 1.00 | 0.75 | 0.63 | 0.78 | 1.00 | 0.83 | 0.90 | 0.92 | 0.92 | 0.93 | 0.70 | 0.88 |
310815 | 0.9 | 0.85 | 0.89 | 1.00 | 0.90 | 0.88 | 0.87 | 0.90 | 1.00 | 1.00 | 0.88 | 0.94 | 1.00 | 0.92 | 0.95 | 1.00 | 1.00 | 1.00 | 0.90 | 0.98 |
… | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … |
Class average | 0.94 | 0.76 | 0.89 | 0.98 | 0.97 | 0.81 | 0.76 | 0.89 | 0.95 | 0.81 | 0.76 | 0.83 | 0.94 | 0.82 | 0.87 | 0.84 | 0.84 | 0.95 | 0.74 | 0.90 |
Achievement value | 0.89 | 0.89 | 0.83 | 0.87 | 0.84 | 0.9 | ||||||||||||||
Finally, the course’s achievement level is calculated based on the support of course objectives towards graduation requirement indicators and the evaluation weights of the graduation requirement indicators as follows: (0.84*0.2+0.83*0.2+0.81*0.2+0.83*0.3+0.79*0.2+0.88*0.1)/1.2=0.82. Table 6 shows the calculation results for achievement in Data Visualization.
Major | Big Data | |||||||
Course objective | 1 | 2 | 3 | 4 | 5 | 6 | Σ | |
Graduation requirement indicator | 1.5 | 3.4 | 4.3 | 5.1 | 5.3 | 12.2 | ||
Indicator evaluation weight | 0.2 | 0.2 | 0.2 | 0.3 | 0.2 | 0.1 | 1.2 | |
Achievement pathway | The evaluation weight and achievement status of course objectives based on achievement pathway | score | ||||||
Method | Evaluation weight | |||||||
Regular grades | 0.1 | 0.2 | 0.3 | 0.2 | 0.3 | 9.83 | ||
0.94 | 0.95 | 0.96 | 0.93 | |||||
Classroom quiz | 0.1 | 0.5 | 9.36 | |||||
0.5 | 0.84 | |||||||
Experiment assessment | 0.2 | 0.86 | 0.2 | 0.2 | 0.3 | 0.3 | 16.99 | |
0.77 | 0.75 | 0.74 | 0.79 | |||||
Final exam | 0.6 | 0.2 | 0.3 | 0.4 | 0.1 | 39 | ||
0.78 | 0.76 | 0.74 | 0.74 | |||||
Weight | 1 | 0.7 | 1.2 | 0.9 | 0.5 | 0.3 | 0.4 | 75.18 |
Achievement | 0.84 | 0.83 | 0.81 | 0.83 | 0.79 | 0.88 | 0.82 | |
From the calculation results of the achievement level, it is observed that in all the achievement pathways of the Data Visualization course, the overall achievement levels of course objectives in the experiment assessment and final exam segments are relatively low; the lowest achievement is for Course Objective 5, followed by Course Objective 3, suggesting targeted improvements could be made subsequently.
This section discusses the practical application of the course evaluation method based on achievement pathways, using the Software Engineering course from the computer science major of the Beijing Institute of Petrochemical Technology as an example. Software Engineering is a compulsory course for the computer science major, with an evaluation weight of 0.6 for supporting graduation requirements. The number of graduation requirement indicators supported by this course is $m=5$, and their corresponding evaluation weights are shown in Table 7. The evaluation weights of course objectives towards graduation requirement indicators are presented in Table 8.
Graduation Requirements | Graduation Requirement Indicators | Evaluation Weight of the Graduation Requirement Indicator |
1: Engineering knowledge | 1.6 Mastery of foundational computer software knowledge and database principles, applicable in software design and database modeling, and for modeling and designing software and database applications. | 0.2 |
2: Problem analysis | 2.3 Capability to describe requirements, analyze, and model computer software modules and systems. | 0.2 |
3: Design/development of solutions | 3.5 Capability to decompose and refine complex engineering problems for specific needs, with skills in designing, implementing, and integrating software systems, reflecting innovation. | 0.2 |
5: Use of modern tools | 5.1 Proper selection of modeling tools and technological resources for simulation and analysis in computer engineering projects, and understanding their limitations. | 0.2 |
11: Project management | 11.1 Mastery of engineering project management methods, understanding of economic and management influencing factors. | 0.2 |
Course Objectives | Graduation Requirement Indicators | Evaluation Weight of Course Objectives Towards the Graduation Requirement Indicator |
|---|---|---|
1: Ability to apply the fundamental principles of Software Engineering, analyze the relevant elements involved in the software development process, and select an appropriate software process model. | 1.6 | 1 |
2: Capability to establish a reasonable requirements model using structured software analysis methods or object-oriented software analysis methods, based on software requirements engineering principles. | 2.3 | 1 |
3: Analysis of the software requirements model, derivation of a reasonable software design model based on software design principles, completion of software preliminary and detailed design, and optimization. | 3.5 | 1 |
4: Ability to use different modeling tools to realize various model designs throughout the software lifecycle; concurrently, ability to use different testing techniques to complete software test case design. | 5.1 | 1 |
5: Ability to reasonably organize and control the technical points, cost, risk, and quality in the software development process, possessing basic software project management qualities and capabilities. | 11.1 | 1 |
The number of achievement pathways for the course is $p=4$, which include regular grades, classroom quizzes, experiment assessments, and final exams. The support relationship of achievement pathways towards course objectives and their evaluation weights are shown in Table 9.
Achievement Pathways | Course Objectives | Evaluation Weights |
Regular grades | Course objective 1 | 0.2 |
Course objective 2 | 0.4 | |
Course objective 3 | 0.4 | |
Classroom quizzes | Course objective 1 | 0.4 |
Course objective 2 | 0.2 | |
Course objective 3 | 0.4 | |
Experiment assessment | Course objective 1 | 0.1 |
Course objective 2 | 0.1 | |
Course objective 3 | 0.1 | |
Course objective 4 | 0.4 | |
Course objective 5 | 0.3 | |
Final exams | Course objective 1 | 0.2 |
Course objective 2 | 0.4 | |
Course objective 3 | 0.4 |
The achievement for each student sample within each achievement pathway towards each course objective is calculated, as shown in Table 10. For instance, if student sample 311112 scored 3.5 for Course Objective 1 in the classroom quiz, with a full score value of 4, then the achievement of this student for Course Objective 1 in the classroom quiz is calculated as 3.5/4 = 0.875. Using the same method, the achievement of this student for each course objective in the classroom quiz can be calculated.
Major | Computer Science | |||||||
Course objective | 1 | 2 | 3 | Classroom quiz (full score: 10 points) | ||||
Evaluation weight | 0.4 | 0.2 | 0.4 | |||||
Statistics/Score/Name | Score (full score: 4 points) | Achievement value of course objective 1 | Score (full score: 2 points) | Achievement value of course objective 2 | Score (full score: 4 points) | Achievement value of course objective 3 | Score (full score: 10 points) | Classroom quiz achievement value |
310807 | 3.5 | 0.875 | 2 | 1 | 3.4 | 0.85 | 8.9 | 0.89 |
310808 | 3.5 | 0.875 | 2 | 1 | 3.7 | 0.925 | 9.2 | 0.92 |
310809 | 4 | 1 | 2 | 1 | 3.9 | 0.975 | 9.9 | 0.99 |
310810 | 4 | 1 | 1.5 | 0.75 | 3.8 | 0.95 | 9.8 | 0.98 |
310811 | 4 | 1 | 2 | 1 | 3.6 | 0.9 | 9.6 | 0.96 |
310812 | 3.5 | 0.875 | 2 | 1 | 3.3 | 0.825 | 8.8 | 0.88 |
310813 | 4 | 1 | 2 | 1 | 3.7 | 0.925 | 9.7 | 0.97 |
310814 | 3.5 | 0.875 | 2 | 1 | 3.7 | 0.925 | 9.2 | 0.92 |
310815 | … | … | … | … | … | … | … | … |
Class average | 3.7 | 0.93 | 1.92 | 0.96 | 3.6 | 0.91 | 9.2 | 0.92 |
Achievement value | 0.93 | 0.91 | 0.91 | 0.92 | ||||
Subsequently, the achievement for each student sample towards each course objective is calculated. For example, the achievement of Course Objective 1 for student 311112 is calculated as (0.75*0.2+0.875*0.4+0.75*0.1+0.85*0.2)/(0.2+0.4+0.1+0.2)=0.83. The calculation results for the student samples are shown in Table 11, and after mean calculation, the class average achievement value for Course Objective 1 is determined to be 0.83.
Major | Computer Science | ||||||||||||||||||
Course objective | 1 | 2 | 3 | 4 | 5 | ||||||||||||||
Evaluation weight | 0.2 | 0.4 | 0.1 | 0.2 | 0.9 | 0.4 | 0.2 | 0.1 | 0.4 | 1.1 | 0.4 | 0.4 | 0.1 | 0.4 | 1.3 | 0.4 | 0.4 | 0.3 | 0.3 |
Name/Score/Statistics | Regular grade achievement | Classroom quiz achievement | Experiment achievement | Final exam achievement | Achievement of course objective 1 | Regular grade achievement | Classroom quiz achievement | Experiment achievement | Final exam achievement | Achievement of course objective 2 | Regular grade achievement | Classroom quiz achievement | Experiment achievement | Final exam achievement | Achievement of course objective 3 | Experiment achievement | Achievement of course objective 4 | Experiment achievement | Achievement of course objective 5 |
310808 | 0.75 | 0.875 | 0.75 | 0.55 | 0.76 | 0.75 | 1.00 | 0.75 | 0.55 | 0.72 | 0.80 | 0.85 | 0.75 | 0.58 | 0.74 | 0.75 | 0.75 | 0.67 | 0.67 |
310809 | 0.75 | 0.875 | 0.75 | 0.75 | 0.81 | 0.88 | 1.00 | 0.75 | 0.75 | 0.84 | 0.93 | 0.93 | 0.75 | 0.73 | 0.85 | 0.75 | 0.75 | 0.83 | 0.83 |
310810 | 1 | 1 | 1.00 | 0.85 | 0.97 | 0.88 | 1.00 | 1.00 | 0.85 | 0.90 | 0.88 | 0.98 | 1.00 | 0.86 | 0.91 | 0.94 | 0.94 | 0.92 | 0.92 |
310811 | 0.75 | 1 | 0.75 | 0.70 | 0.85 | 0.88 | 0.75 | 0.50 | 0.73 | 0.76 | 0.83 | 0.95 | 0.75 | 0.73 | 0.83 | 0.69 | 0.69 | 0.67 | 0.67 |
310812 | 0.75 | 1 | 0.75 | 0.75 | 0.86 | 0.75 | 1.00 | 0.75 | 0.78 | 0.80 | 0.83 | 0.90 | 0.75 | 0.78 | 0.83 | 0.69 | 0.69 | 0.67 | 0.67 |
310813 | 0.75 | 0.875 | 0.75 | 0.85 | 0.83 | 0.75 | 1.00 | 0.75 | 0.85 | 0.83 | 0.93 | 0.83 | 0.75 | 0.85 | 0.86 | 0.75 | 0.75 | 0.75 | 0.75 |
310814 | 0.75 | 1 | 1.00 | 0.80 | 0.90 | 0.88 | 1.00 | 1.00 | 0.78 | 0.87 | 0.88 | 0.93 | 1.00 | 0.78 | 0.87 | 0.88 | 0.88 | 0.75 | 0.75 |
310815 | 1 | 0.875 | 0.75 | 0.65 | 0.84 | 0.88 | 1.00 | 0.75 | 0.65 | 0.80 | 0.88 | 0.93 | 0.75 | 0.65 | 0.81 | 0.75 | 0.75 | 0.75 | 0.75 |
… | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … |
Class average | 0.77 | 0.93 | 0.78 | 0.74 | 0.83 | 0.77 | 0.96 | 0.78 | 0.74 | 0.79 | 0.8 | 0.91 | 0.8 | 0.91 | 0.81 | 0.77 | 0.77 | 0.77 | 0.77 |
Achievement value | 0.83 | 0.79 | 0.81 | 0.77 | 0.77 | ||||||||||||||
Finally, the course’s achievement level is calculated based on the support of course objectives towards graduation requirement indicators and the evaluation weights of the graduation requirement indicators, as shown in Table 12 (0.78*0.1+0.85*0.1+0.8*0.4)/0.4=0.8.
Major | Computer Science | ||||||
Course objective | 1 | 2 | 3 | 4 | 5 | ||
Graduation requirement indicator | 1.6 | 2.3 | 3.5 | 5.1 | 11.1 | ||
Indicator evaluation weight | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 1 | |
Achievement pathway | Evaluation weights of achievement pathways towards course objectives and the achievement | Grades | |||||
Method | Evaluation weight | ||||||
Regular grades | 0.1 | 0.2 | 0.4 | 0.4 | 8.3 | ||
0.81 | 0.82 | 0.84 | |||||
Classroom quiz | 0.1 | 0.4 | 0.2 | 0.4 | 9.2 | ||
0.93 | 0.91 | 0.91 | |||||
Experiment assessment | 0.2 | 0.1 | 0.1 | 0.1 | 0.4 | 0.3 | 15.5 |
0.78 | 0.78 | 0.8 | 0.77 | 0.77 | |||
Final exam | 0.6 | 0.2 | 0.4 | 0.4 | 44.4 | ||
0.74 | 0.74 | 0.74 | |||||
Weight | 1 | 0.9 | 1.1 | 1.3 | 0.4 | 0.3 | 77.4 |
Achievement | 0.83 | 0.79 | 0.81 | 0.77 | 0.77 | 0.8 | |
From the achievement level calculation results, it is observed that among all the achievement pathways, the overall achievement level of course objectives in the final exam segment is comparatively low, followed by the achievement level in the experiment assessment segment; Course Objectives 4 and 5 have the lowest achievement levels, suggesting that these areas could be targeted for improvement in subsequent efforts.
5. Conclusion
This study proposes a quantitative calculation method for course achievement under the OBE model, based on achievement pathways. It constructs a quantitative constraint relationship among course-related graduation requirement indicators, course objectives, and achievement pathways, along with an evaluation algorithm for course achievement based on achievement pathways. This method has been practically applied in several courses within the big data and computer science majors of Beijing Institute of Petrochemical Technology, including Data Visualization, Software Engineering, C# Programming and Applications, Engineering Project Practice, showing positive outcomes. By calculating the achievement levels of various course objectives within different achievement pathways and the achievement levels of graduation requirement indicators, weaknesses in the course implementation process can be identified, providing a theoretical basis and reliable assurance for the continuous improvement of courses.
The data used to support the research findings are available from the corresponding author upon.
The authors declare no conflict of interest.
Views: 366