Solving the Achievement Gap in Programming

Learning to program is difficult: 30-60% of students in university introductory programming courses fail. This is associated with a belief that only students with a natural “aptitude” or talent for programming are able to truly succeed in learning to program, while students without this aptitude are doomed for failure or at best mediocrity in programming. Although studies have shown that this programming achievement gap based on natural talent does exist in introductory courses, recent work has pointed towards pair programming as a possible solution for helping bridge this gap.

Saeed Dehnadi and Richard Bornat, computer science professors at UK’s Middlesex University, spearheaded the initial work to quantify and prove the existence of this programming achievement gap. They designed a test that asked logic based, simple programming questions designed to quantify what sort of problem-solving methods, or “mental models” their students used when faced with programming questions they had never encountered before. Dehnadi and Bornat’s test focused on a specific programming concept, “variable assignment” in which math-like variables such as “x” and “y” change throughout a set of programming instructions. They analyzed their students’ results and quantified all the different possible problem-solving methods, or mental models, their students used: while right to left copy was the correct model to use, students also incorrectly used other mental models such as left to right copy.

Dehnadi and Bornat found that their students fell into two main groups: those who used a consistent mental model during the test, and those who were inconsistent in applying mental models. The consistent group did not necessarily apply the correct mental model for the problems, rather, they chose and stuck with one mental model (which may have been incorrect!) for the entire exam. In contrast, the inconsistent group used different problem-solving strategies for different questions even though they were faced with essentially the same question the entire exam, slightly changed each time. Tracking both groups throughout their programming course, they found that only 22% of the consistent group failed while 52% of the entire class failed the class. Aggregating their results with experiments conducted at six other schools, they noted that this programming achievement gap persisted through multiple trials. The difference between both groups, even though neither group had programming experience, seemed to support the popular theory that only those born with an aptitude in programming could succeed, which runs contrary to the popular mantra by educators in America that “all children can learn”.

However, Alex Radermacher of North Dakota State University theorized that the results observed by Dehnadi and Bornat were only a byproduct of how programming was taught at most universities. Notably, Michael Caspersan of University of Aarhus was unable to replicate Dehnadi’s programming achievement gap and theorized that perhaps differences in the way his course was taught (instructors, materials, etc.) with the schools Dehnadi analyzed could explain his results. Rademacher similarly believed that using a new teaching strategy that integrated pair programming, in which peers collaborate to solve a programming assignment, would allow students with inconsistent mental models to improve into a consistent and correct one, and thus succeed and overcome the “hump” that the inconsistent group experienced in Dehnadi and Bornat’s experiments. Radermacher used Dehnadi’s test to similarly group his students and track their progress throughout the semester. He showed that pair programming helped his inconsistent group migrate mental models and succeed in his course better than the inconsistent group in Dehandi’s experiments.

Radermacher’s results, although only preliminary, are interesting as a possible solution to solving the programming achievement group, especially because of the disdain some experts hold about pair programming as useless and ineffective. The next important question to ask, as Radermacher himself poses, is hypothesizing and testing specific programming pairs for maximizing the inconsistent group’s success in the course. Studies in group theory and learning seem to indicate that pairing an inconsistent student with another inconsistent student would be more effective at improving each student’s programming ability than pairing an inconsistent student with a consistent student. This seems strange at first: one would expect that pairing a student bad at programming with one good at programming would allow the inconsistent student to learn from the better student. However, Michelene T. H. Chi from University of Pittsburgh shows that some incorrect mental models are robust misconceptions in which a typical teacher-student or better student-worse student interaction is ineffective at teaching the student. Instead, Chi says, students with robust misconceptions need to “step outside of the box” so to speak and look at the problem in a completely different way in order to learn the concept effectively.

An inconsistent-inconsistent group, in contrast, pairs together students of a similar skill level. As Christopher Hundhausen from Washington State University notes, this allows each student to feel more comfortable and effective in the group as opposed to an inconsistent-consistent group, in which the inconsistent student may feel inferior and ineffective in the group. As Litrico and Choi notice in their studies on small groups, the higher comfort and “self-efficacy” in an inconsistent-inconsistent programming pair increases the ability of the pair to learn more effectively and quickly than an inconsistent-consistent pair and certainly better than each student by himself.

What I’m proposing is a paradigm shift in programming education: one focused on pair based learning that groups together people of similar skill levels in order to utilize the benefits of a functional and collaborative group. Such a pairing is effective at closing the programming achievement gap because it allows weaker students to overcome the main barrier towards their success, mental model consistency, which is dealt with ineffectively in many classrooms today. This shift would be an improvement on our current education system still primarily focused on the student-teacher relationship, potentially allowing those without an “aptitude” for programming to overcome the programming achievement gap.

Note: I originally wrote this paper for my writing seminar class. This is reprinted with permission from my professor (thanks a lot!). The assignment asked us to “communicate science to the public” by making our longer research papers (mine was ~14 pages) accessible to a more general audience. As a result, this article is shorter and is less rigorous academically than my original research paper (i.e. my research paper delves deeper into academic sources about group theory and mental models to make a stronger argument on why an inconsistent-inconsistent pair is superior to an inconsistent-consistent pair at improving mental model consistency in programming). Sources for this article and my longer research paper are included below.

Works Cited Dehnadi, S. and Bornat, R. 2006. The camel has two humps. Little PPIG, 1-21. Chi, M. T. 2008. Three types of conceptual change: Belief revision, mental model transformation, and categorical shift. International handbook of research on conceptual change. 61-82. Thomas, M.D., and Bainbridge, W. L. 2001. ‘All Children Can Learn’: Facts and Fallacies. Phi Delta Kappan 82, no. 9. 660-662. Radermacher, A., Walia, G. & Rummelt, R. 2012. Improving student learning outcomes with pair programming. Proceedings of the ninth International conference on Computing education research (ICER ’12). 87-92. Caspersen, M. E., Larsen, K. D., and Bennedsen, J. 2007. Mental models and programming aptitude. ITiCSE ’07: Proceedings of the 12th annual SIGCSE conference on Innovation and technology in computer science education. 206-210. Dehnadi, S., Bornat, R., and Adams, R. 2009. Meta-analysis of the effect of consistency on success in early learning of programming. PPIG. 1-13. P. N. Johnson-Laird. 2013. Mental models and cognitive change, Journal of Cognitive Psychology, 25:2, 131-138. Lou, Y., Abrami, P., and d’Apollonia, S. 2001. Small Group and Individual Learning with Technology: A Meta-Analysis. Review of Educational Research, Vol. 71, No. 3. 449-521. Bender, L., Walia, G., Kambhampaty, K., Nygard, K. E., and Nyhard, T. E. 2012. Social sensitivity and classroom team projects: an empirical investigation. Proceedings of the 43rd ACM technical symposium on Computer Science Education, SIGCSE ’12. 403-408. Hundhausen, C.D., Agarwal A., and Agarwal, P. 2013. Talking about code: Integrating pedagogical code reviews into early computing courses. ACM Trans. Comput. Educ. 13, 3, Article 14 (August 2013). 1-28. Mohamadi, F.S., Asadzadeh H., Ahadi, H., and Jomehri, F. 2010. Testing Bandura’s Theory in School. International Conference on Education and Educational Psychology, ICEEPSY 2010. 426-435. Litrico, J.B., and Choi, J. N. 2013. A look in the Mirror: Reflected Efficacy Beliefs in Groups. Small Group Research 44, no. 6. 658-679.