The General Accounting Office (GAO) and Inspector General (IG) both released congressionally directed studies of Compliance Safety Accountability (CSA) and found numerous critical flaws with the system. Both studies can be accessed in their entirety from this post, but they are lengthy so I will highlight key conclusions.
The GAO report “Modifying the
Improve the Ability to
Identify High Risk Carriers” analyzes nearly 315,000 U.S.-based carriers that were under FMCSA’s jurisdiction and, with reasonable certainty, were active during the period from December 2007 through June 2011. GAO considered a carrier active during this period if it received a state or federal inspection, was involved in a crash, or reported the number of vehicles it operates to FMCSA. The GAO study corroborated many of the deficiencies we have reported over the years.
- The factors being measured cannot be correlated to safe driving: “Our analysis found that most of the regulations used in (Safety Measurement System) SMS were violated too infrequently over a 2-year period to reliably assess whether they were accurate predictors of an individual carrier’s likelihood to crash in the future. We found that 593 of the approximately 750 regulations we examined were violated by less than one percent of carriers. Of the remaining regulations with sufficient violation data, we found 13 regulations for which violations consistently had some association with crash risk in at least half the tests we performed, and only two violations had sufficient data to consistently establish a substantial and statistically reliable relationship with crash risk across all of our tests.”
- There is insufficient data on most carriers: “Most carriers lack sufficient safety performance information to ensure that FMCSA can reliably compare them with other carriers.” “About two-thirds of carriers we evaluated operate fewer than four vehicles and more than 93 percent operate fewer than 20 vehicles. Moreover, many of these carriers’ vehicles are inspected infrequently".” “Given that SMS calculates violation rates for carriers having a very low exposure to violations, such as operating one or two vehicles or subject to a few inspections, many of the SMS scores based on these violation rates are likely to be imprecise.”
- State-by-state variability means CSA is measuring where a company operated, not how safely it operates: “The frequency of an individual carrier’s inspections varies depending on where the carrier operates. States vary on inspection and enforcement practices. Some studies have shown that inspectors or law enforcement officers in some states cite vehicles for certain violations more frequently than in other states. Delays in reporting crash data to FMCSA, as well as missing or inaccurate data, can affect a carrier’s Crash Indicator SMS scores. These delays can vary by state.”
- Data sufficiency standards are inadequate: “For most BASICs, we found FMCSA’s data sufficiency standards too low to ensure reliable comparisons across carriers. In other words, many carriers’ violations rates are based on an insufficient number of observations to be comparable to other carriers in calculating an accurate safety score. Our analysis shows that rate estimates generally become more precise around 10 to 20 observations, higher than the numbers that FMCSA uses for data sufficiency standards.”
- CSA is ineffective at predicting individual carrier crash risk and is suspect even for its original purpose of allocating enforcement resources: “Overall, SMS is successful at identifying a group of high risk carriers that have a higher group crash rate than the average crash rate of all carriers that we evaluated. However, further analysis shows that a majority of these high risk carriers did not crash at all, meaning that a minority of carriers in this group were responsible for all the crashes. As a result, FMCSA may devote significant intervention resources to carriers that do not pose as great a safety risk as other carriers, to which FMCSA could direct these resources.”
GAO analyzed an alternative data sufficiency standard of at least 20 trucks or 20 inspections and eliminating safety event groups and found far more reliability of the scores in identifying at-risk carriers. Furthermore, GAO questioned the viability of FMCSA’s proposal to determine a carrier’s fitness to operate based on SMS scores.
“FMCSA officials told us the primary purpose of SMS is to serve as a general radar screen for prioritizing interventions. However, as discussed above, due to insufficient data, SMS is not as effective as it could be for this purpose. Further, if the same safety performance data used to inform SMS scores are intended to help determine a carrier’s fitness to operate, most of these same limitations will apply. According to FMCSA, the Safety Fitness Determination rulemaking would seek to allow FMCSA to determine if a motor carrier is not fit to operate based on a carrier’s performance in five of the BASICs, an investigation, or a combination of roadside and investigative information. However, basing a carrier’s safety fitness determination on limited performance data may misrepresent the safety status of carriers, particularly those without sufficient data from which to reliably draw such a conclusion.”
The IG study “Actions Are Needed To Strengthen FMCSA’s Compliance, Safety, Accountability Program” was far more limited in scope. The study was coordinated with the GAO study to avoid duplication of effort. The IG report found fault with FMCSA’s lack of discipline in requiring carriers to update census information which is essential to measuring crashes per mile or per power unit. FMCSA has begun efforts to deactivate US DOT numbers for carriers that do not update census data. The IG was also critical of the Agency’s failure to fully implement the enforcement intervention process. Only 10 states had fully implemented the process at the time of the report. The other 40 states and DC are waiting on software and training which is not expected until May 2015. Since the primary purpose of CSA is to prioritize enforcement resources, it is disconcerting to find that so few states are in compliance and that improvement is not expected until 2015.
In between the release of these two reports, FMCSA released a study by the Volpe Center titled, “The Carrier Safety Measurement System (CSMS) Effectiveness Test by Behavior Analysis and Safety Improvement Categories (BASICs)”. This was widely seen as a weak attempt by FMCSA to contradict some of the GAO findings. This report is basically a rehash of the Agency’s data showing that “on average” carriers with higher SMS scores have higher accident frequency, but does nothing to dispute the incontrovertible evidence that CSA is unable to predict anything about the safety of an individual carrier.
What was originally designed to be a “general radar screen” to help prioritize FMCSA enforcement resources has very inappropriately become a guilty verdict in the mind of some shippers, who refuse to do business with carriers that have even one BASIC above the intervention threshold. When half of all carriers with a percentile ranking have at least one BASIC above the intervention threshold and there is absolutely no correlation between SMS scores and individual carrier accident frequency, this is simply wrong. It is harming our nation’s supply chain and doing nothing to improve highway safety.
Following the GAO study release, the American Trucking Associations, the Owner Operator Independent Drivers Association (OOIDA) and the group I chair Alliance for Safe, Efficient and Competitive Truck Transportation (ASECTT) have all called for removal of SMS scores from the FMCSA website. “Since scores are so often unreliable, third parties are prone to making erroneous judgments based on inaccurate data, an inequity that can only be solved in the near term by removing the scores from public view,” said Dave Osiecki, ATA executive vice president and chief of national advocacy.