My Design for Organic Exams

October 21st, 2014

ed_academic_bigIt’s fall semester at SLU, and I’m teaching organic chemistry for our majors once again. Last year, I was focused on just getting through the course. I was so consumed by producing slides, homework assignments, practice exams, and real exams on a tight schedule that I didn’t get to think hard about the finer points of design until the class was over. Now, on the second pass, I can focus on making improvements rather than creating everything from scratch.

At SLU, instructors don’t get assigned TAs for proctoring and grading unless the class has greater than 30 students. In the spring, the majors’ class fell below this number and I had to think about designing exams to make grading as efficient as possible without eroding their effectiveness as tools for teaching and thoughtful evaluation. The system I settled on was different from my experience in both college and grad school, so I thought I’d share.

I’ve settled into a system in which quizzes and exams generally have four or five sections/problems picked from the following varieties: multiple choice, mechanism, synthesis, A+B reactions, explanations, and calculations. While I’ve never been a huge fan of multiple choice, it allows me to test a variety of straightforward points without the strain of having to grade a wide assortment of free-response questions. My multiple-choice questions are typically five choices, with students getting +5 points for a correct response and +2 points for leaving a question intentionally blank. This scoring system introduces a penalty for guessing; students need to recognize that sometimes the best option in life is to admit “I don’t know.”

In my system, if there is an exam on Wednesday, the students have a quiz due on the previous Friday that covers the same material. Each quiz contributes 2% to the final grade in the course, while each exam contributes 20%. The quiz has the same length, format, and answer sheet as the exam, except the problems are harder to compensate for the fact that I give students 48 hours for the quiz and allow them to talk with each other about the answers. I like how this approach requires the class to be familiar with the exam material almost a full week before the exam, and that students have an incentive to form study groups and debate answers to hard problems. I never learned as much in organic chemistry as when I was trying to defend my proposed answers to classmates. Finally, having what is essentially a hard practice exam due on the Friday before the “real” exam means that I can return it graded and marked with comments by Monday. In many cases, these quizzes serve as wake-up calls to students while there is still time to fix issues that need attention.

In every chemistry class I’ve taken, students either wrote their exam answers directly on the problem booklet or in small blue exam books. I find these response media are murder to grade because you spend so much time flipping through pages, hunting for answers, and flipping back to write the subtotal for that problem on the cover page. And, of course, there are always two or three students who get creative and write their answers out of order or on the back side of pages.

To solve this problem, I started writing exams such that students have to place all of their answers on a single letter-sized piece of paper. I draft an answer sheet for each exam on which it is clear where each answer should be written. When the answer is a single word or structure, I’ll typically draw a box for it. The use of a single sheet minimizes the burden of flipping, while the answer boxes (located at the same spot for every student) minimize the burden of hunting. Here’s a sample answer sheet typical of one of my orgo exams:

sample_exam_answer_sheet

I copy answer sheets onto 65-lb. card stock so that both sides of the sheet can be used without the pen ink bleeding through to the other side. I find that the smooth sheets sold under the Neenah brand are better than sheets of the less expensive Staples brand, which have an annoying coarse texture on one side. The card stock also gives a regal quality to the answer sheets. These puppies are suitable for framing and will withstand decades of wear if pinned to the refrigerator door of a proud parent.

Another nice thing about having students limit their answers to a single sheet is that the entire stack for the class can be scanned without hassle. Most modern office copiers are capable of sheet-feed scanning, so if your exams have no staples, you can scan the entire stack of paper all at once and e-mail the data to yourself as a single PDF. What a wonderful miracle of technology.

I scan the entire set of answer sheets both before and after I’ve graded them. In doing so, I have a permanent record of performance that I can access for eternity. If there is an issue over improper grading or a student attempting to cheat by altering answers upon return of the graded copy, I can refer to the electronic file. If a student needs a letter of recommendation two years down the road, I can open up the file and make specific comments about his/her performance. And if I want to mine data in the future for some pedagogical purpose or project, all of the data will be available for analysis.

A final important feature of each answer sheet is that I have a line for students to write their names at the top of the back side. This makes returning exams very simple, because I can fan them in columns in alphabetical order on a table and have students come in groups to pick them up. With only the top inch of the back page of each sheet exposed, students cannot see each other’s grades. Of course, as the columns thin out, I adjust the sheets to prevent greater exposure.

As I discussed before, I allow students to bring handwritten notes into exams. They serve to emphasize that organic chemistry is about analysis—not memorization. Also, having students organize the information of each unit onto a single sheet of paper forces them to make connections about the material in their minds ahead of the exam. This semester, I’ve started collecting, scanning, and returning these note sheets as well. I am fascinated at how students organize their thoughts, and I plan to use this information in my future letters of recommendation when appropriate. How an individual organizes her thoughts on paper gives you an interesting window into her mind.

So, that’s the system. Are there drawbacks? Yes. I tend not to write long mechanisms or synthesis problems that go over four steps, because there isn’t enough space to do so. With that said, I’m not sure if these problems offer much added benefit relative to alternatives with more concise answers.

One thing that I’ve found surprising is that even though the answers for the exam all fit on one page, there are students who still feel they need three hours to finish. I thought that offering four shorter exams instead of three longer ones would make time a non-factor, but often at the end of two hours, I still find myself nagging for all of the papers to be turned in.

Anyway, I like how this system works and will continue to modify it to work better for both the students and me. It is ironic that as a teacher, I feel that I am the one in the classroom who has the most to learn.


33 Responses to “My Design for Organic Exams”

  1. MarkO Says:

    Hey Paul – interesting approach. I don’t know the US system at all – how many summative assessments do students get in one course? It seems like they are getting 4 exams – plus labs, plus a final? I’m just curious – we are limited to no more than 4 summative assessment items per unit (a unit is one eighth of an annual load). The balance of assessment vs workload vs actual student learning is an unsolved problem I think.

  2. Paul Bracher Says:

    @MarkO: The school does not set a limit to the number of assignments I give, but here’s my breakdown: 14 30-minute homework assignments online (5%, drop lowest 2), 6 take-home quizzes (10%, drop lowest score), 4 2-hour exams (60%, drop lowest score), and 1 two-hour final exam (25%). Our semester is about 3.5 months long.

  3. Scott Hartley Says:

    I like the idea of a pre-exam assignment a lot. I have 130 students right now, so the prospect of the added grading is a little daunting, but I wonder if even an all-MC assignment or an additional assignment through online homework would be worth it. Could be an interesting Chem Ed study (if it hasn’t been done already).

    The worksheet is a great idea. I did something similar with a spec id class and it saved my sanity, but I never really thought to do something this concise for sophomore organic.

  4. Paul Bracher Says:

    @Scott: I will eventually be assigned the 250-student non-majors class, and my plan is exactly what you said: to make the quizzes all multiple choice so they can be machine graded.

  5. John Arnold Says:

    A number of the large courses at UC Berkeley are using an awesome program called Gradescope (https://www.gradescope.com/about) to help alleviate some of the problems associated with grading hand-written exams (and homeworks).
    We use this in Chem 1a for example to grade four exams a semester (three midterms and a final) for 1300 students. I can highly recommend it and would never go back to using Scantron/multiple choice tests, which I detest.

  6. Josh VW Says:

    This is all kinds of brilliant – I love the idea of having a hard-but-low-points quiz before the important exam. For the answer sheet, is the idea that they work out the answer in their blue books and then rewrite it in concise/correct form on the sheet? Do you get any crossword-in-pen types who try to just do it straight on the answer sheet?

  7. azmanam Says:

    Really neat insight. I’ve struggled with every one of the points you’ve mentioned. Thanks for the analysis.

    And if I want to mine data in the future for some pedagogical purpose or project, all of the data will be available for analysis.

    No, it won’t. At least here at my school, you need IRB approval BEFORE collecting any data whatsoever – even if your research will be IRB exempt. Maybe your school interprets ‘exempt’ differently, but that’s how it is here, fyi.

  8. Keith Wampler Says:

    As a SLU alumnus who had the very unfortunate experience of taking the majors’ organic course with one of your predecessors, whom is no longer with the department, I am extremely happy to see this level of thoughtful preparation go into the exam generation process. I feel as though my time at SLU (2001-2005) was during a liminal period for the department and things appear to progressing in the proper direction. Keep up the good work!

  9. CCChem Says:

    Yes, you’ll have all this wonderful data blah, blah…..New professors…..gotta love them. =0}

  10. dvizard Says:

    That sounds great – I would be interested in how well these mechanisms work in reality? I have seen quite a few well-intentioned ideas to give students an incentive to study early, discuss among each other etc, but quite often the students are clever at finding a way how to complete tasks with the minimum amount of work possible – for example, I imagine quite a few students put no effort at all into the 2% quizzes and/or copy answers from the one friend that actually did the work… What are your experiences?

  11. Neil Says:

    (As a UK based academic) If I presented this schedule it would be thrown back at me for overassessment of the students.

    In one unit (which represents 1/6th of the year and ~3.5 hours per week contact time over 11 weeks) we are only allowed one 2 h exam and one 40 minute mid-term test only for summative assessments.

    Use of formative assessments is encouraged however – e.g. lecture wrap-up “self-tests”, problems with on line answers, workshops with model answers etc. In this way students can monitor their own progress as the course develops and the staff don’t have large amounts of marking. Do students do the self assessments is always my worry, but of they dont and they fail the final exam then whose fault is it?

  12. Kevin Says:

    Gotta love the comments from our UK colleagues–they have a formula for everything (just curious, do you get student evaluations as your sole determinant of your teaching ability?). I’m sitting here watching my students take a 50 minute midterm in class (the only one they will have, besides the final). They are juniors and seniors and probably hate exams as much as I did. For the best courses I took as an undergrad or grad student, I didn’t understand the material until a couple years later…

  13. Neil Says:

    reply to Kevin .

    Well I guess it depends on how you believe your teaching ability can be measured? Of course we get student feedback from questionnaires, but unless the course is exceptional or very poor, most of the numerical evaluations tend to an average. I always look for the free comments section. These evaluations are always done before the final assessment, so should not be coloured by a “poor mark = I was taught badly” but still take up is only around 50% of the class – those with strong opinions either way.

    Staff also have to undergo annual peer observation of teaching activities – mine was done yesterday :-)
    and we get a grilling by our peers at the end of year exam board meetings when the marks are ratified (and post exam module evaluation by the teaching team. All of these reports are reviewed by a departmental committee). So if you teach badly and the students do poorly then it comes to light and the Head of Department camn send you on some course of “retraining”. High exam results amongst some of my colleagues, I am sure, come from teaching to the test or at least little variation in the assessments (students can get to see past papers and so can question spot). These are not so easy to spot.

    I also agree with you that I only started to understand as a grad student, and *really* started to understand when I had to teach. Maybe there is a lesson in that – get the students to teach each other!

  14. EW Says:

    As a grad student at a large university where TA’s do all the exam grading for 600+ student O1/O2 classes, this 1-page grading system would probably save us a ton of time in page-flipping alone. Would likely also save the department a bit in paper/copying costs.

    Interesting and good idea, if I get an academics job in the future, I’ll have to see if I can implement something like it for my organic classes. Thanks for sharing!

  15. mica Says:

    azmanam Says:

    “No, it won’t. At least here at my school, you need IRB approval BEFORE collecting any data whatsoever – even if your research will be IRB exempt. Maybe your school interprets ‘exempt’ differently, but that’s how it is here, fyi.”

    Great point (for those unfamiliar with IRB, cf., Facebook human experiments; they should be shut down for what they did, but law is for small people like Paul and us…), and — at least at my place– you have to go through IRB chair to get “the “exempt” nod. Otherwise, it would be open to interpretation and you could claim “I thought that me injecting crap into people…”.

    But it is likely that in Paul’s case he could get a nod to use available accumulated data which was obtained for other purposes. It should not be difficult to apply for exemption post collection, but he certainly should have it before any use of data to make conclusions that are not strictly related to grading.

  16. Neil Says:

    I’ve not done anything like this, but in the UK I think Paul would be able to use this data if he was able to uncouple completely the data from the student ID. I.e. the data would just be an anonymous pile of numbers.

    If he was also keeping the records for writing letters of recommendation (and therefore he knew whose data was whose) then I guess this would not be possible.

  17. Paul Bracher Says:

    So, let me get this straight…

    If I used the same multiple-choice question on two exams except I inverted the order of answers (a to e vs. e to a) and wanted to report statistics on the difference in percentage of students who got the correct answer (from year 1’s exam to the year 2’s exam), I would need approval of a review board? (But if I didn’t want to study anything, just invert the answers because I felt like not copying an old exam verbatim, I would not need approval of the review board?)

  18. CCChem Says:

    Paul, if you want to publish those results, yes.

  19. Tiger Chem Says:

    As one of the Beleaguered teaching assistants for the first semester organic course at an Ivy I wholeheartedly support this condensed answer page. Our professor gives open book tests (?) and takes this as an excuse to ask questions that are not only open ended but also way beyond the abilities of a sophomore student (eg justify why 3,5-dimethyl-4-nitrophenol is less acidic than 4-nitrophenol but more acidic than 4-trifluoromethylphenol, all of this on the first organic exam that they ever take!). What this ends up being is a recipe for students copying a whole bunch of nonsense from their notes/books and we TAs having to haphazardly partial credit to their mini-essays. We had 18 TAs to grade ~250 exams and it still took us 5.5 hours. In the end they all get A’s (regardless of understanding) so I guess that I shouldn’t take it too seriously…

  20. Paul Bracher Says:

    @Tiger Chem: It always bothered me when professors who had no intention of grading would write exams that were murder to grade. At NYU, the first thing we did when we got to the grading room was offer guesses as to which problem would be responsible for keeping us from going home early that night. And the organic exams were always on Friday afternoons.

    And who on Earth would ever do this…

    (eg justify why 3,5-dimethyl-4-nitrophenol is less acidic than 4-nitrophenol but more acidic than 4-trifluoromethylphenol, all of this on the first organic exam that they ever take!).

    Oops. I did. And it was every bit the disaster you implied. Live and learn.

  21. Tiger Chem Says:

    @ Paul: I’ve actually written a test with this exact question before, the biggest difference being that it was a grad PhysOrg course instead of organic I (or “orgo,” as these kids call it)! I think what this boils down to is one of the biggest difficulties that we, relatively talented (ie graduate students and beyond) chemistry thinkers, can have: putting ourselves into the shoes of the average organic student. I always thought that organic was obvious but what I saw time and time again in undergrad with my classmates (and time and time again with my recitation students now) is that things that we just “got” and take for granted can take a LOT of work for other people. Organic I (and especially the first midterm in Organic I) is the opportunity to teach and reinforce the fundamentals to solve this kind of problem in the future; asking such a busy question now was (at least for us) was seriously jumping the gun. Just my two cents…

    Anyway, teaching is something that I’m pretty passionate about and I really appreciate you taking the time to share your experience and (perhaps most importantly AND humourously) your own learning experiences. I was happy to see this post and hope that you’re able to eke out the time in the future to give us some more updates.

  22. Jimmy Says:

    Dear Chemistry People,
    I have an unrelated question for all:
    Frequently young assistant professors (and professors) in general bemoan the low salaries of academics?
    Is this for real? I know a number of young professors whose real starting salaries are about $100k because their listed salary is only 9 months of the year, and they add to that from their grants during the summer months.

    Also, perhaps there are difference between research-oriented professors and education-oriented professors. Do the latter make less?

  23. Scott Says:

    @Jimmy. Academic definitely make less than their counterparts in industry; whether that’s low is really up to you. I teach at a midsized us public institution. I’ve yet to clear 100k (total) despite being at this for 7 years. I personally think it’s worth it, but you might not. Profs at larger R1 institutions would definitely make more, profs at smaller institutions definitely less. I would guess that starting salaries range from about 60-110k depending on the prestige of the institution.

  24. Jimmy Says:

    @Scott
    Thank you for your reply!
    Does your salary include the summer pay part as well? That’s mostly what I’m curious about.
    Industrial chemists very roughly start out making ~90k. So, if the summer pay makes academics earn the same, then they really have nothing to complain about. And definitely most of the R1 schools do end up paying assistant profs ~70k to start, so when you factor in the summer pay they are doing just fine.

    I don’t really have sympathy for the assistant profs at R1 institutions; they are doing it by choice, and their pay seems equal once you factor in that summer pay. I am more curious about less competitive schools, let’s say rank >20. I respect what they do (well, most of the time; a lot of academics do research that really has little impact, especially in inorganic chemistry which is my field) and their choice… and they have the very important responsibility of educating the future’s scientists which is possibly more important than research. In return they get great job flexibility and security once they get tenure… and eventually the salaries get decent even without the summer pay.

    My pity is more for the assistant profs (and especially adjuncts) whose primary focus is education, and I suspect their pay is quite low. The importance of education is terribly low in modern academic institutions, and this will impact future generations. I saw this at my alma mater (a highly ranked midwestern school) where the profs often despised teaching had negative reactions to their grad students spending too much time teaching. Everything was research research research… and to some degree slave-like labor…. not to mention the bigotry and general rights violations that are well known in graduate school.

    So, i’m curious to better understand how salary fits into all of this, and if changing salaries in certain areas could change that.

  25. Scott Says:

    @Jimmy

    My estimate of my own salary includes an additional 1/9-2/9 of my 9-month salary for summer pay. Remember, the amount you can pay yourself is dictated by the amount you make during the academic year. It’s also soft money: in some ways, it’s almost better to think of it as a nice bonus because there are no guarantees, especially in this funding climate.

    Of course, you can often get summer salary by teaching summer courses, even if you’re not research-active.

    I’m really not an expert in this, so please take these comments for what they’re worth. My personal impression is that a tenure-track job at a college of reasonable quality, regardless of its research standing, is likely a pretty good job. Relatively low paying, but there are real benefits to the job security (once you get tenure), flexibility, and satisfaction. The health/retirement benefits are probably pretty good as well. The same is true for *permanent* teaching positions at good quality colleges. The pay is likely even lower (permanent teaching-only faculty at my school make on the order of $45k + benefits to start), but if it’s what you’re in to it’s a good job.

    So, you can lead a comfortable, happy life if you want to focus on teaching and can land a job at a good teaching-oriented institution. There are many great colleges that value undergrad teaching above all else. They are often not the really huge schools, either in terms of population or reputation.

    The problem I think you’re getting at near the end of your post is the problem of temporary teaching faculty (visiting professors and instructors). In theory, these positions are fine for a year or two, but they have, in some cases, turned into permanent jobs and some schools have become reliant on these sorts of faculty to keep costs down. This is where you run into problems of teaching faculty really being exploited (teaching many courses, no benefits, very low pay).

  26. Jimmy Says:

    @Scott
    I think the problem is more that it’s a pyramid scheme, and I think it’s a terribly big problem. It’s a conspiracy theory level problem.

    What is more important in higher education: research or education? They should be of equal importance.

    However, research is by far given a large priority at most schools… at least my observation from the R1 school I attended, and some stories I have heard about others.

    Who does most of the teaching? Adjuncts, grad students, and assistant profs.
    Who gets paid the most? Tenured profs, and department heads.
    Who is most qualified to give the teachings? This is obviously quite variable, but the tenured profs I think should have a lot to contribute.
    Many bigger R1 schools have a general trend towards larger classrooms, larger amounts of adjuncts. Possibly even larger numbers of grad students to fuel the teaching.

    I am curious what brings in more money towards schools; tuition or research grants; I would guess the former, but am not certain.

    If we have a lot of money coming in from tuition, and relatively small cuts-of-the-pie going towards the educators (not tenured profs, but the lower ranked people) then there is an imbalance. And that imbalance is not going to favor high quality education.

    Anyways, I kind of just rattled off that rant without making sure all of the tenants were well fortified, but I think the overall concept is an important one for discussion.

  27. Scott Says:

    @Jimmy
    Agree with most of your points, but I think it’s important to keep in mind that not all schools are R1 schools. At my own school the vast majority of classes are taught by tenure-track faculty, some are taught by lecturers (permanent teaching faculty), and a minority are taught by 1 or 2 visiting faculty each year (we have 20+ permanent faculty, so this is a fairly low proportion). The temporary faculty are typically brought in to cover for a faculty member on sabbatical or if someone leaves before they can be replaced. We never have grad students teach lecture courses: they work in teaching labs (under faculty supervision) or run recitations. Tenured faculty actually teach a higher load, typically, than the pretenure assistant profs (because they’re also expected to be setting up their research programs). I should point out that I work in a department with a PhD program and run a research group; it’s just that we value balance between the teaching and research sides. My impression from Paul’s post above is that the same is true at SLU.

  28. Untenured Prof Says:

    @Jimmy

    I think you are being overly optimistic about the salary benefits of being an Assistant Professor – even factoring in summer salary. Firstly, summer salary is very difficult to get and (typically if we are talking about a research school) requires securing grant funding which is at an historical low. Approximately 10% of grants get funded, roughly speaking, by the federal agencies which allow summer salary. There are often private foundations and other sources of money one can apply for, but often these explicitly state that faculty salary is not allowable on the grant. If one makes $65k per 9 months and is lucky enough to secure an NSF grant, which allows for 2 months faculty salary maximum but more than likely will correspond to 1 summer month. The reason I say that is that every dollar I take out in faculty salary is a dollar away from my students stipends or research expenses. So lets assume you make $65k and have 1 summer month salary, this brings you to a total of ~$72K. Not a terrible salary by any stretch of the imagination but certainly well below the $90k you claim for industrial chemists… which I also think is a low-ball number for a PhD level chemist in industry as most of my colleagues that have gone on to such positions command 6 figure salaries.

  29. TS-CI Says:

    I wonder if memorization helps though. I find that a lot of academics downplay the value of rote and play up the value of higher level thinking. Like things need to be fancy. Or like they don’t believe enough in progressive learning.

  30. TS-CI Says:

    I really liked how we learned things in high school (even high school AP). Daily homework, graded, and a reasonable amount of the total grade. Homework which included mostly easy or medium problems (only a few hard ones and those often for extra credit). Tests every week or two.

    I also like how learning is done in sports, in the military (including technical topics in the military). With lots of progression and demonstration of mastery. And attention.

    And I say this with a chem “union card”, 1580 SAT, 2370 GRE. Not bragging, think kindness/courage are more important than brain cells…but just want to preempt any “you can’t handle” rebuttal.

    I’m NOT impressed with the pedagogy at the average American research university. I went to undergrad at a military academy and NEVER had a lecture hall class, always had access to my profs in the afternoon, etc.. etc. I think kids at “colleges” are getting ripped off. My “trade school” was better.

    P.s. I’m also heretical enough to think lecture teaching ITSELF is a bad idea. People learn more from doing than listening. There should be lots of homework going on DURING class. This is how sports work.

    I really like “programmed learning”. It’s progressive and you get practice and aren’t bored/distracted. Wonder why it has not taken off more. It was all the rage and never caught on (were universities and teachers threatened?) I guess it sort of has caught on in terms of IT-based learning.

    Here is a gem of a book that teaches rules of the road. Better tnan any other text, than a class, than reading the rules themselves.

    http://www.amazon.co.uk/A-Seamans-Guide-Rule-Road/dp/0948254580

    I also had wonderful experience learning accounting (literally) during a transatlantic flight, by working through a fill in the blank workbook.

  31. milli Says:

    good one

  32. fred Says:

    Hola! I’ve been following your web site for a while now and finally got the bravery to go ahead
    and give you a shout out from Houston Tx!
    Just wanted to say keep up the excellent work!

  33. topodinpro Says:

    This post is in fact a pleasant one it helps new web people, who are wishing for blogging.


Leave a Reply

*