Friday, May 27, 2011

Book Review: Academically Adrift

Arum, Richard and Jospia Roksa. Academically Adrift: Limited Learning on College
. Chicago: The University of Chicago Press, 2011.

Much has changed in American higher education in the last half-century. Fifty years ago, U. S. colleges and universities lead the world in undergraduate graduation rates. Students averaged 40 hours per week--the equivalent of a full-time job--attending classes and studying (Arum and Roska, 3). A majority of these students either paid their own way through school or relied on scholarships and grants (money that need not be paid back except in terms of the graduate's contribution to society) to cover costs, and a college degree was seen as one of several viable post-high school paths to career success.

Today, America has embraced the "college for all" mentality. As the U.S. looks to (re)gain its world leadership in the medical, scientific, and technological sectors, policy makers have introduced a variety of initiatives designed to improve the quality of education at all levels, to encourage young people to study math and science, and to raise the overall educational level of citizens. Moreover, in the wake of the recent recession, young people who might previously have gone straight from high school to the work force have decided that they need a college degree in order to remain competitive in the tightening job market. A college degree is now seen by many as a necessity.

Yet, despite an increase in access to higher education in this country, America no longer leads the world in college graduation rates (53). Figures vary by study, but a government study in 2005 found that only 54 percent of students who enrolled in 4-year colleges in 1997 had completed a bachelor's degree by 2005. In another dramatic shift, today's college student spends an average of just 27 hours per week attending classes and studying--less time than s/he probably spent attending classes in high school (3)--and most likely funds his/her education at least partially through student loans and on- or off-campus employment.

With so much on the line, parents, policymakers, and employers are beginning to ask just how much value students are getting for their dollar, and how effective the reform and accountability initiatives have been. What, exactly, should students be learning in college, and how well are they learning it?

These two questions are the subject of Academically Adrift. Sociologists Arum and Roksa surveyed educators and employers and compared courses of study across a wide range of disciplines. They concluded that the primary academic goal of a college education was the acquisition of critical thinking and writing skills. They then noted that most standardized tests of college-learning measure major content rather than skills. In order to find the answer to the second question, therefore, Arum and Roksa designed their own test of college learning--the Collegiate Learning Assessment (CLA)--which could be administered to students of any discipline and which focused on critical thinking and writing. The test is open-ended (rather than multiple choice) and consists of two "writing tasks" (make an argument and break an argument) and a "performance" task that asks students to read several different texts and respond to a writing prompt by analyzing the texts, synthesizing the information, and producing a written solution to a practical problem.

Arum and Roksa administered the CLA to approximately 1100 students at 24 institutions of higher education at the beginning of their freshman year and again at the end of their sophomore and compared the results. After controlling for several demographic variables, they concluded that the average student's CLA score only improves by 7 percent in his/her first two years of college. They also found that 45 percent of students who took the test saw no statistically significant gains in critical thinking and writing skills at all in the same two years (36).

Arum and Roksa next turned their attention to institutional and cultural factors that might explain the "limited learning" demonstrated by the CLA. They concluded that none of the many parties involved in higher education--students, faculty, administration--has really made student academic development the highest priority. For students and parents, social development and "credentialing" were the top priorities, and while the latter might sound like academic development, today's college students have become consumers of educational services, seeking the highest (GPA) return for the least possible (time) investment (16). True "learning" takes a back seat. Research and student retention are the top priorities for college faculty and administrators. The technological advances of the post-World War II era fueled a shift within the academy from teaching to research, and most faculty now know that research and publication (activities external to the university itself) weigh more heavily in promotion and tenure decisions than student course evaluations do--and that better course evaluations are often linked to lower expectations and lighter assignment loads (13). Additionally, although administrators are interested in increasing undergraduate student retention, their efforts in this department tend to focus more on developing a close-knit student body, offering attractive residence halls, and improving student services rather than on increasing academic standards (135-6).

Few people, it seems, are really interested in teaching and learning. Yet the authors' research suggests that helping students develop critical thinking skills and good study habits contributes at least as much to retention as creating a positive social environment (ibid.). Based on their research, Arum and Roksa offer several practical suggestions for restoring the culture of learning on college campuses. For example, they note that students whose teachers require them to read more than 40 pages per week and write at least 20 pages over the course of the semester see the biggest gains in their CLA scores (94-5). They also urge adminstrators to train future faculty (i.e., graduate students) in good teaching techniques as well as in research skills.

As thought-provoking as this book is, the research behind it is limited. Only 24 schools participated in the study, and the students at these schools self-recruited. Thus, the study group likely contained a higher than desirable percentage of well-organized and highly motivated students. Additionally, over half the sample reported a high school GPA of 3.7 or higher (Methodological Appendix). Perhaps these students saw such small gains on their CLA scores because they came to college already in possession of the skills that the test measures. If the sample had included a higher percentage of "underprepared" students (those with the potential for greater gains in critical thinking and writing skills), would the test results have been more encouraging? Would the authors' conclusions and suggestions have changed?

In spite of its limitations, Academically Adrift is an interesting read that offers some good advice for those who work in the field of higher education, and the CLA is an instrument that has the potential to change the way we educate our young adults for the better. As much as this reviewer dislikes standardized testing, she believes that administering the assessment to more first- and second-year college students would offer some meaningful and much-needed insight into the education reform process. This book is highly recommended.

Tuesday, May 10, 2011

End-of-Term Assessment: What I've Learned in the Last Two Years

School's out! My students have just finished their final exams, and I thought it would be appropriate to evaluate my own progress after two years of teaching, especially since I made some significant changes this year. Here's a review of what worked, what didn't, and what I've learned.

What worked:

1. Administering the reading quizzes as soon as the clock indicates that class time has begun seems to have eliminated tardiness, as I had hoped. Either that or this year's class was just more concerned with punctuality than last year's.

2. I have finally found a paper-grading rubric that yields grades I am (usually) happy with, but it works because it grades on a 4-point (i.e. GPA) scale, rather than on the 100-point (percentage) scale that students are used to. Explaining it to unhappy students has been difficult, but I can't figure out how to convert it.

3. This one's really basic: putting the desks in a circle went a long way to eliminating distracting side conversations. Last year, my classroom layout made circling desks impossible, and I had to resort to docking participation points for classroom distractions instead. Preventing the distractions in the first place was much more effective.

4. Providing students with a "creative option" on some assignments also worked out well. Not many students chose this option, but the assignments of those who did were exceptionally good. I'm not entirely sure why, but I know it's not because these students are the best ones in the class. I'd like to think it was because the creative option required them to think about the content on a deeper level in order to transform it into something new. I just hope it wasn't bias on my part (as in, "Oh boy--a break from the monotony! This is great work!").

5. Switching the focus of the final exam from course content to critical reading, thinking, and writing skills (with specific reference to texts read for class; basically, I gave them short passages from course texts and authors and a series of reading comprehension questions for each passage) worked out well, although there's still room for improvement. Last year I discovered how difficult it was to create a content-focused exam for a discussion course, especially when the discussion didn't always go the same direction in both sections. Sure, students were responsible for all the assigned reading, but for many of them, this was their first exposure to primary sources, and I didn't think it was fair to put something on the exam that I wasn't absolutely sure we talked about in class. This year I reasoned that since the main objectives of the course were to teach students to read a variety of genres of theological literature critically, to interact with them intelligently, and to write persuasively, the final exam should test those skills more than course content. The students will have to take additional theology courses at CUA, anyway, and those courses will focus on content mastery. The exam worked well, although since I didn't think to give a pretest at the beginning of the semester, I have no hard evidence to show how much progress the students made in acquiring these skills. Also, in hindsight, I think that I should have included an objective (matching or fill in the blank) section testing the students on their grasp of essential terms that they will run across later on.

What didn't work/needs improvement:

1. Reading quizzes didn't do enough to convince students to read in preparation for class. Many students were content to take their chances by guessing on the questions. I could eliminate guesswork by making the questions short answer or fill in the blank, but it was difficult to come up with questions that were easy enough to ensure that the students who really did the reading would be able to remember the answers but difficult enough to make blind guessing unprofitable. I could also weight the quizzes more heavily than I currently do.

2. Related to #1, reducing the length of reading passages also did not result in increased class participation. If anything, the number of students who read the texts actually declined (as the book Academically Adrift, which will be the subject of my next post, confirms).

3. The class session on speed reading was a bust this year. I had one student tell me "please don't assign speed reading ever again," even though I didn't have an assignment connected with the lesson and had no plans to grade anyone on their mastery of the techniques. I guess the latter didn't come across clearly in class. I think I will (mostly) take her advice, though. The "above-the-line" technique is easy to teach and master and has the potential to double reading speed instantly, so I think I will continue to teach it in class. But the full technique takes more than one session to learn and really needs to have homework attached to it for students to master. I may offer to teach it as a workshop series, but I don't think I'll attempt it again as a stand-alone lesson in a theology course, even if one of the major goals is to teach reading skills.

4. I need to improve my ability to rephrase the questions I ask in class discussions. I have this problem in Sunday School, too--I know what's in my head, but I can't seem to express it in a way that gets others there.

5. The rubric I used to grade the group presentations graded too highly. I'm pretty sure this is because I reverted to a percentage rubric (i.e. content is worth up to 55 points out of 100, observing time limits is worth up to 10 points, etc.) instead of using the GPA/"quality" system that worked so well for the written assignments. But GPA rubrics only work--I think--in cases where all grading criteria are weighted equally, and I don't think I want to weight observance of time limits equally with mastery of content. I'm not sure how to fix this.

6. The critical bibliographical essay assignment (honors section only) was a good idea, and the students seemed to appreciate the challenge. Most papers were good, and a few were excellent, but others had serious deficiencies relating either to poor or uninformed source selection or to an over-reliance on personal opinion instead of critical analysis. I'll definitely use this assignment again, but I need to provide more supervision of students' source choices (i.e., requiring them to submit abstracts for all three sources instead of only one) in order to make sure they don't paint themselves into a corner. Several students left my class still confused about the appropriateness of sources for academic papers, and that's unacceptable.

Other things I've learned:

1. Teaching is every bit as exhausting as being a student, although it's a different kind of tired. I was surprised at how much it took out of me last year. An experienced teacher explained that teaching is a kind of performance--inspiring and engaging students takes a fair amount of energy, both physical and mental. I have a new appreciation for elementary and preschool teachers--I can not imagine how much more tiring their jobs must be than mine.

2. Some students still see only what they wish the text says instead of what it actually says, no matter how you try to steer the discussion ("Is that really what Augustine is saying here?") or how many times you make this comment on their papers. I'm still trying to figure out how to get through to these students.

3. Some students just don't follow directions, even when they have an instruction sheet for an assignment in front of them. No one should have lost "format" points on papers, but about a third of one class consistently did, in spite of my repeated red inking and comments, both on papers and in class.

4. Related to #2 and #3, some students don't read the comments I make on papers and thus don't learn as much as they could from the experience. I was amazed at the number of "final" annotated bibliographies that I received that hadn't incorporated a single correction from the draft, although in hindsight (we all know this generation doesn't read much) I shouldn't have been so surprised.

5. Reducing student workload does not improve the quality of students' work. If anything, it has the opposite effect, as students either forget about or push to the side assignments that are less time-consuming. The old adage, "if you want a job done, give it to a busy person" applies here as well.

My faculty mentors have said that it takes two revisions of a course to get it exactly the way you want it, and this was only my second year (one revision). I tried several things differently this year, and for the most part, I would say that the overall experience was better. But there's still plenty of room for improvement, and I do think that one more revision would do the trick. So it was with some reluctance that I turned down the opportunity to return for a third year. The teacher part of me wants one more chance to get it right. The Ph.D. candidate part of me, however, knows that hiring committees consider how long it took an applicant to finish his/her degree, and that time is not on my side. I've been a student too long already, and teaching cuts heavily into writing time. I've recorded my ideas for next time here, and the sooner I finish the dissertation, the sooner "next time" will arrive.