Today Philip Garland, Ph.D, a visiting scholar at the University of Washington (UW), published his findings that the online survey commissioned by the City of Seattle to determine if work schedules are a problem for shift workers, and if so, to what extent, is severely flawed.  Even though the researchers hired by the city are his colleagues from UW, Garland says the survey is poorly designed and unreliable, and should not be used as the basis for the city to craft regulations restricting how employers schedule and manage their workforce.

Why should one consider Dr. Garland’s analysis the authority in this matter?  In addition to his academic credentials, Garland is also the former VP of Methodology for SurveyMonkey, the world’s leading provider of web-based survey services. So Garland is uniquely qualified to analyze the online-only survey created and conducted by his UW colleagues for the city.

According to Garland, the methodology behind the survey is “substandard to accurately represent the Seattle worker population,” which he says makes it pretty easy to predict what the results will be.   Garland goes through a comprehensive explanation of the various factors that virtually ensure the survey will yield the result Mayor Murray, the City Council and organized labor are looking for.  As Garland puts it:

“Biased results are virtually inevitable.”

First are the “alarming” sampling issues that “all but guarantee that the survey findings will demonstrate an outsized proportion of unhappy workers and also overstate the extent of their unhappiness.” According to Garland, “accurate estimations of the prevalence of workplace abuse are simply unattainable from the sample that comprises the city’s survey.”

Then there concerns with the reward offered to participants that “exacerbates the unwanted effects” of the flawed sampling. 

Next, Garland explains the problems with how the questions are phrased and the order in which the responses are offered, “all but ensuring an over-reporting of respondents that agree with the scheduling practice abuses described in the questions.”

Garland’s conclusion:

“Taken together, these flaws suggest that the total survey error—the combined error produced from each component of survey design and analysis—of the city’s study is extraordinarily high.

Given research that shows purposeful and selective part time employment among freelancers has tremendous value, it is curious why the city’s survey excludes items that measure peoples’ availability, compensation, and benefits that would contribute to a robust understanding of the nature of part time work.

The city would be much better served by a large-scale, highly valid, probability study that measures scheduling practices of employers, the scheduling policy desired by workers, and general attitudes toward part time work.

Until then, the results of the city’s current survey are destined to point to a one-size-fits-all solution that ends up fitting nobody especially well and some disadvantaged groups rather poorly.”

In other words, when the survey results—which will undoubtedly show widespread scheduling abuse and worker discontent—are released to hype and fanfare next month, they should be taken with a huge boulder of salt.