Monday, January 26, 2009

The Statistics of Homeschooling

Since 1999, the famous Rudner study has been touted as finally answering the academic question about homeschooling once and for all. Meanwhile, the original critique of that study has been largely ignored.

Subsequent critiques by homeschoolers themselves have come to the fore as well:

1. "Embarrassing and Dangerous" (This series was written by a homeschooling leader in Wisconsin; part two is an excellent dissection of the testing methods--taken from the Home Education Magazine).
2. "The Fraser Study: Puffing Up Homeschooling and Selling Our Freedoms" (This series was also in the Home Education Magazine, with multiple links to other articles critiquing NHERI and other research leaders).

This means that homeschoolers are not dominating in academics--or at the very best we do not statistically know how they are doing.

This may be quite shocking to some readers. I can understand; I was amazed as well. But facts are facts. I have no ax to grind since I will soon be a homeschooling dad.

What the Numbers Reveal

The Rudner study showed that homeschoolers in the Bob Jones University (BJU) locale who took the test did very well on that standardized test.

That's it.

What the Numbers Do Not Reveal

The fundamental problem with the Rudner study is a problem admitted by the author himself: " should be noted that it was not possible within the parameters of this study to evaluate whether this sample is truly representative of the entire population of home school students."

If his study is not "truly representative" in any conclusive sense, then why bother with it at all? Admittedly, researchers are about knowing. But after the knowing is accomplished and the caveats written, such a study reveals very little. Another answer may be that the study was funded by an organization already in favor of homeschooling. Many homeschooling groups touted the study, even to this day, without admitting any of the caveats the author himself wrote (at least the full report is there). Too often conservative Christians blame the liberal left for "tweaking" data in their favor--who's to say such a thing cannot happen amongst conservatives?

"This was not a controlled experiment,'' and it "does not demonstrate that home schooling is superior to public or private schools, and the results must be interpreted with caution" are some serious caveats. And these carefully chosen words were in the opening paragraph of the report. The fact that they were not touted as loudly as the finding itself points to sloppy research or self-interest--either is not something commendable.

Through all the newspapers, journals, tv reports, lectures, postings, advertisements and word-of-mouth I encountered, I never once heard these caveats.

Lone Ranger?

Perhaps some distraught readers out there think that this is a non-issue or a smokescreen brought up by an unskilled one-man army. Besides the fact that I have an extensive math background (B.S. Electrical Engineering), scholars themselves have pointed out the limitations of the studies and even some homeschooling leaders themselves are uncomfortable with this loose usage of statistics.

One interesting read is from Professor Milton Gaither who has written a virtually definitive book on the history of modern homeschooling in America.

What About Those Experts?

Shorty after Rudner published his finely nuanced report, a peer review was published analyzing the usage and methods empoloyed--a not untypical activity in this field--by Welner and Welner.

Another expert writes in Education and Urban Society. Similar concerns were discovered in other education magazines.

And most interesting of all, Bruce Ray of the NHERI (homeschooling advocate group) notes modest academic numbers for homeschoolers. The latest work by Brian Ray and Bruce Eagleson, State Regulation of Homeschooling and Homeschoolers’ SAT Scores, notes in the introductory background information, that there are mixed results on the testing advantages of homeschoolers (SAT and ACT) (two studies show virtually no statistical advantage and two more show some advantage). As for college exams: "The few studies done on home-educated students’ performance on college-admissions tests suggest they score about as well as do those who are not homeschooled."

The Devil is in the Details...

The first analysis of Rudner's first study was accomplished by the team Welner and Welner:

"Some researchers, in fact, would say that the test scores have nothing to do with how the children were schooled and simply show the results expected for children that come from this demographic group—households that are overwhelmingly white, well educated, two-parent, and middle class (see Coleman et al., 1966; Ogbu, 1987). This is not to say that these parents did not do a good job teaching their children, it is only to say that a comparable sample within the public or private schools may have scored just as well." (Welner and Welner)

Furthermore, this study does not take into account that testing is not required for a number of homeschoolers. And of those states that require it, most allow the parents to use a private professional to evaluate the child, the typical public school standardized testing being completely bypassed.

One obvious reason for this severely limiting aspect of the study is that it occurred amongst a culturally narrow group: those taking the tests at BJU. What about those families from less waspish environs?

To date, a Barna study strongly suggested what some thoughtful statisticians have pointed out elsewhere: homeschooling is not a middle-class, white, Evangelical monolithic entity (Welner and Welner).

Another problem not addressed by this study is the number of unschoolers and the percentage of mixed schooling (both homeschooling and private/public) that occurs among homeschoolers. As of this date, since investigating this issue, such numbers are greatly underreported in many homeschooling circles.

Another statistician's study highlights the limitations of the Rudner study as well:

"The recent achievement studies by Rudner (1999) and Ray (2000) are notable given their large sample sizes. However, Rudner’s study has been criticized (Welner andWelner, 1999) as having a biased sample. Given that his sampling frame originated from a conservative, religious institution, it is unlikely that the diversity of the homeschooling movement is represented in his data. The response rate was also low. Ray’s (2000) response rate was problematically low. Moreover, because he obtained student data from only 38% of the 29% of families that responded to the survey, the likelihood of bias increases. Although the sample underlying the research reported here also has limitations, the data and analysis make a unique contribution to this growing research literature. (p.314 Education and Urban Society / May 2005)

Note how neither critique ignores Rudner's study; they only point out its severe limitations. It helped prompt more studies and suggest areas of research. It did not demonstrate scientifically that homeschoolers outperformed their public or private school peers.

To date, I have not found any such study.

Playing the Telephone Game

When a child is excited about a new find--a toy--he tells all his friends how amazing the toy is. His friend then tells his other friend, ad infinitum. In the process, the toy turns from a mundane matchbox car to a remote-controlled tank. Each child heard elements of the original report he liked and did not hear other parts. Sit in a circle and play the telephone game by whispering into the next ear and similar results appear.

When an adult does the same thing, it is less appealing. And more dangerous. Homeschooling leaders that I am aware of have complained about burnout. And about low retention rates. This is what naturally follows when evidence is hyped out of proportion. The leaders certainly did not make up the numbers they heard; they probably heard it from a trusted friend, who heard it from another friend, ad infinitum.

With this posting, the telephone chain ends. There is no remote-controlled tank with a magical statistical bullet to silence the nay-sayers. Now the record is set straight. The numbers are not in and may never be discovered given the decentralized nature of homechooling.

Does this mean one should not homeschool? No, just realize that it takes work, hard organizational work. Then again, whatever schooling is used, proper parental nurture is hard work. Changing one type of schooling for another does not automatically make one's child an academic success--that it a combination of God-given talent, hard work and mercy from the Lord.


Hillary said...

Another factor (if you mentioned it, I overlooked it; and my apologies) is the issue of many families who fly so low under the radar that they are not even close to being "counted" or "studied" because they have an isolationist mentality. These families do not trust governments or even those who seem to have the best interests of homeschooling at heart, because 1) everyone has an agenda and 2) almost anyone, even in "Christianity", can be bought out.

I have ten siblings; if there were just ten other families like mine out there, 100 home-schooled kids is still a significant number worth contributing to a study. Unfortunately I can safely assume there are significantly more than that. . .yet we will never know for sure.

polymathis said...

You are correct, Hillary. I did not deal with the national numbers in this posting. I deal with it in an earlier posting, the Future of Homeschooling.

Out where I am there are many such families with isolationist mentalities. And I think when the false numbers I went over in this posting are pushed in those circles it just helps reinforce their mentality: "we are smart enough to be islands unto ourselves."

take care,

Bill said...

Hi Shawn,

I think you are correct about the looseness of some peoples view of history.

Here is a publication that you might want to has a little more detail to the studies you are critiquing.

Ray, Brian D. (2005). A homeschool research story. In Bruce S. Cooper (Ed.),
Home schooling in full view: A reader, p. 1-19. Greenwich, CT: Information Age Publishing.

Thanks, Bill Roach

polymathis said...

Greetings Bill,

Given my backlog of books to read about homeschooling, could you give me the name of the studies he depends upon? I noticed in that this book was partially scanned in but the academic section, pages 4-5, was not scanned in. He gives the study names in parenthesis with a last name and year.

Thank you for pointing out this unknown reference and thank you in advance for finding those references for me.

Bill said...


Backlog, huh?? I guess there is a quite a bit out there to know to be an expert on something and be able to write/blog about it!!

Here are some suggestions:

1. Read all of Cooper's book...especially the chapter written by Dr. Ray.
2. The 2000 double issue of the refereed scholarly Peabody Journal of Education on homeschooling
3. Rudner's 1999 study on homeschooling...make sure you read the whole thing.
4. Ray and Eagleson, 2008, Journal of Academic Leadership,

Happy Reading...research is tough, brother!! Because of time factors, you probably can see how easy it is for us homeschoolers to become lazy in our historical study and research and not be thorough...borrow from other people's work and become an expert on "what famous person was or was not homeschooled." I know I have been guilty of that myself.

That's what I have always appreciated about Dr Brian Ray. He is thorough, detailed and professional in his research, and he is a brother who interprets information from the perspective of the City of God. I thank God for him more and more.

May God give you wisdom, Shawn.

Your brother,


polymathis said...

Dear Bill,

Thank you for pointing out the work and professionalism of Dr. Ray. I do believe he is quite thorough and detailed. In his summary of the highlights of homeschooling in The Old Schoolhouse online article (The Evidence Is So Positive), he stated unequivocally:

"The design of most research to date does not allow for the conclusion that homeschooling necessarily causes higher academic achievement or better social and emotional development than does public (or private) institutional schooling."

This is precisely the conclusion of my posting on Rudner's study.

I am open to public correction and will publicly recant if it can be shown that I am in error. And I trust you share this attitude.

I strongly recommend my latest posting on statistics to understand where I am coming from and where I am going:

A Story About Scholarship

your humble researcher in Christ,