What We Measure


I listened to Dr. Susan Embretson speak at UCLA last Friday.  Lots of what she said went over my head, which might be because she’s just a lot smarter than me – a situation I’m fortunate enough to be in pretty much every day of the week at UCLA and at home. But one thing I caught was that she’s working on new ways of generating test items automatically.

Her work is making it easier and cheaper to develop reliable new tests. Efficiency is always good. And research in the area of item response theory and computer adaptive testing is leading to exciting developments in diagnostically measuring student progress and providing each student with more appropriate and quicker feedback. Exciting may seem too strong a word for this extremely complex and math-heavy research, but you can check out the School of One to see what this type of individualized education could look like.

But, on the other hand, for most teachers and students, the effects so far seem to be more testing, and thinking about more testing made me think about the bigger picture of educational research. I began to wonder, not for the first time, whether a large part of this progress and this important work is rather beside the point as far as students and teachers are concerned.

Maybe we need to step back and think about our schools from a variety of perspectives. What are the problems that other people might wish we were working on?

From students’ point of view, one of the biggest problems is the lack of jobs. Huge numbers of black and brown teenagers are unemployed. But, partly due to our research emphasis on measurement and our policy emphasis on accountability, vocational programs have been cut or curtailed sharply.

From parents’ perspective, one of the biggest problems has got to be our children’s health.  The growing epidemic of obesity and diabetes seems highly relevant, yet the testing/accountability mania has led schools to cut PE, health, and recess.

From the planet’s perspective, one of the biggest problems is the epidemic of waste caused by humans.  Yet every day our schools teach our children to throw away huge amounts of food because lunch and recess times are too short and play time is too precious to be spent eating.

We are measuring carefully, but the things we’re measuring are not necessarily the things we care about. Undoubtedly, researchers are concerned about and probably even working to end unemployment, obesity, and global warming, so if we truly care about these and other issues, we need to start thinking a little less about how to measure and a little more about what we  measure.


Eat Your Vegetables


I’ve been thinking a lot lately about purpose: What’s the purpose of this blog? Who is the audience? And is that audience interested in that purpose?

It’s a tricky line because this blog is intended to jumpstart a conversation that everyone seems to agree is a good idea, but that no one seems to have much of a personal stake in having. The point is to get educators from all different fields talking directly to one another – teachers talking to researchers, researchers to teachers, quantitative researchers to qualitative, scholar activists to policy analysts, etc., Everyone I’ve talked to (and yes, that is an unscientific sample) seems to agree that this dialogue, this conversation, is a worthy goal. Yet everyone also seems to agree that this conversation is largely silent as of now.

On a personal level, this blog has helped me start lots of interesting and useful conversations, but on a broader level, it has spread a bit more haphazardly. This conversation seems to fall into roughly the same category as reading up on the candidates for the local city council race. We all know we should do these things, and we all have the vague sense that, in the long run, it will be good for us and for our community if we carve out the time to get these things done. But, on a day-to-day, minute-to-minute basis, we often have more pressing concerns, responsibilities, and interests.

Teachers, for instance, generally want practical advice. They want ideas that they can use to tweak tomorrow’s lesson plan and get better results. Philosophical debates on value-added models, Teach For America, massive open online courses, and iPads (link to these) might be interesting, but on a daily basis, teachers often feel powerless to impact these debates. The problem is, if teachers don’t pay attention (and they often don’t), then folks are having a conversation about changing teaching and learning while the most informed people are out of the room. I think this often leads to bad policy decisions, and also to less informed teachers who are likely to have difficulties implementing these policies.

Researchers are focused on practical concerns, too. Only their practical concerns revolve around advancing their own particular niche of expertise, working on publications, and furthering their careers. In a sense, it’s the same concern. They want to tweak their models and get better results. This seems to leave researchers having technical debates that almost no one else is listening to, so that, even when and if the technical questions get answered, those answers seem to have little to no bearing on how the policies are implemented.

So, if we agree that this conversation is worth having, and we acknowledge that it is largely not being had, then perhaps we need a new set of incentives to help us all begin to take part a bit more enthusiastically. Perhaps we should look at this blog less as an exercise in civic duty, and more as exercise. Less as a community clean-up and more as dusting our bookshelves. In other words, instead of focusing on why this conversation is worth having, we should all think a bit about what it can do for us. As teachers, we imagine a day when district and state officials might heed our counsel. As graduate students and even as tenured professors, we dream of being able to influence public policy. In order for our voices to be heard in these larger debates, we must train ourselves to speak and write convincingly. This blog, then, can be our training ground as well as our slowly expanding megaphone. As we refine our thinking and add additional thinkers, our platform will expand along with us, and so, when the opportunity comes for some or all of us to push that needle and sway the opinions and decisions that matter, we will be ready. Students in the teacher education program, comparative education, social research methodologies, and development psychology will speak directly to one another. Students from UCLA will get a glimpse into the education at Penn State and ASU, and vice versa. Teacher trainers will speak to researchers. And we will all learn from one another and begin to grow the community we will need in order to make the changes we all dream about.

Eat your vegetables, hone your skills. Write. The day will soon come, and may have already arrived, when your voice will be called upon.

Before I conclude, there might be one other thing holding people back. Fear of the unknown, or at least reticence and something that I think I might describe as a general tendency among educators toward a somewhat overdeveloped sense of humility and caution. Researchers are inclined to be careful about what they say. Papers tend to end with caveats and limitations rather than talking points and exhortations. But new styles can be learned. So if you’re not sure that you’re ready to write in this genre or this style, then this is the perfect place to start. When you have an idea that is half-way ready, send it to teachingdiablogue@gmail.com and we’ll be happy to help you polish it up to share more widely.