Yesterday, Transportation Secretary Anthony Foxx testified in Congress that it would take 2 years for FMCSA to complete its review of CSA as mandated by the FAST Act. The law required that FMCSA remove SMS scores from the Web until the review was completed. After immediately removing SMS percentile scores and alerts from the Web, FMCSA exploited a loophole in the law by posting raw SMS scores without percentiles or alerts. FMCSA then doubled down by communicating its proposed new Safety Fitness Determination rule that is based, in part, on the flawed CSA/SMS program.
So even though FMCSA admits it will take two years to remedy the flaws in the SMS system that have been pointed out by the General Accounting Office, Inspector General, and others, it still intends to utilize the system to put some unlucky carriers out of business with an “Unfit” safety rating.
The commentary period is now closed, but numerous industry participants strongly objected to the proposed rule and Congress is considering legislative action that will prevent the rule from being implemented.
I believe there is enough opposition to prevent this harmful rule from being implemented. Time will tell.
On March 5, the Senate Commerce Subcommittee on Surface Transportation and Merchant Marine Infrastructure, Safety and Security held a hearing that was harshly critical of FMCSA’s lack of action and follow up on the Compliance, Safety, Accountability (CSA) system and Hours of Service (HOS).
Director Physical Infrastructure Issues General Accounting Office (GAO), Susan Fleming’s testimony was critical of CSA, “First, SMS uses violations of safety-related regulations to calculate a score, but GAO found that most of these regulations were violated too infrequently to determine whether they were accurate predictors of crash risk. Second, most carriers lacked sufficient data from inspections and violations to ensure that a carrier’s SMS score could be reliably compared with scores for other carriers. GAO concluded that these challenges raise questions about whether FMCSA is able to identify and target the carriers at highest risk for crashing in the future.”
Fleming indicated that FMCSA has done nothing to address these issues, “FMCSA did not concur with GAO’s recommendation to revise the SMS methodology … Therefore, FMCSA has not taken any actions.”
The committee chairman Deb Fischer (R-Neb) criticized the FMCSA regarding HOS. “For example, FMCSA issued the final 34-hour restart rule in 2013 with complete disregard for congressionally-mandated requirements for an efficacy study on the rule’s impact. When the study was eventually issued several months late, the sample size was not representative of this diverse industry. In addition, serious concerns were raised about the rule’s perverse impact on safety because, in effect, it pushed drivers onto the roads during workers, students, and families’ morning commutes.”
Senator Fischer was also highly critical of FMCSA’s disregard of the GAO review of CSA. “In 2014, the GAO investigated the methodology behind FMCSA’s Compliance, Safety, and Accountability Program. Inaccurate CSA scores, publically available online, have cost companies contracts and raised insurance rates – all of this has occurred without a clear correlation to increasing highway safety. When confronted with these findings, FMCSA completely disregarded GAO’s recommendations. To address flaws in CSA implementation, major stakeholders, including law enforcement, requested that FMCSA remove CSA scores from public view.”
It is terrific to see congressional scrutiny of a federal agency that continues to disregard science and hard facts while promoting regulations that harm carriers and shippers while doing nothing to improve highway safety, and in the case of HOS rules, actually make the roads less safe.
The Commercial Vehicle Safety Alliance (CVSA) (a partnership between industry and enforcement groups) has requested in writing that the Federal Motor Carrier Safety Administration (FMCSA) remove from public view the scores from its Compliance, Safety, Accountability program’s Safety Measurement System.
The November 14th letter acknowledges the importance of safety and CSA’s success in leading carriers to increase their focus on safety initiatives. It also points out the shortcomings, including differences in enforcement practices from one jurisdiction to the next and the lack of relationship between some violations and crash risks. The letter sites the GAO study finding that CSA SMS scores are “unreliable predictors of individual fleet crash propensity.”
The letter sums up the essence of the problem very well. “Since the collective crash rates of fleets with SMS scores above thresholds are higher than those below, the SMS is useful as an enforcement prioritization tool. In short, enforcement agencies can focus on these fleets to conduct further investigations and determine which of them are truly risky. On the other hand, since the SMS scores are a poor indicator of an individual fleet’s propensity to be involved in a future crash, their utility in providing the public with information about fleets’ safety performance is limited.”
The letter concludes “… CVSA echoes stakeholders’ call to remove SMS scores from public view.”
Congressman Lou Barletta (R-Pa.) introduced legislation Sept. 18 to require the FMCSA to remove from public view the carrier rankings and scores. It was referred to the House’s Transportation and Infrastructure Committee.
Ten industry trade associates collaborated on a letter to DOT Secretary Foxx on August 22, 2104 also requesting the FMCSA remove SMS scores from their website. That letter concluded, “Given the many identified data sufficiency and reliability issues outlined by the Government Accountability Office, we urge you to direct FMCSA to remove carrier’s SMS scores from public view. Doing so will not only spare motor carriers harm from erroneous scores, but will also reduce the possibility that the marketplace will drive business to potentially risky carriers that are erroneously being painted as more safe.” The trade associations included the American Trucking Associations, the Truckload Carriers Association, The Owner Operator Independent Drivers Association, The National Private Truck Council, and the National Tank Truck Carriers.
The General Accounting Office (GAO) reviewed Defense Transportation’s handling of hazardous materials shipments. The GAO was critical of DOD’s reliance on CSA/SMS in carrier selection. DOD has strict requirements based on percentile scores on each BASIC in qualifying carriers for use on Hazmat shipments.
The report’s conclusions state that “DOD uses Safety Measurement System scores to determine safety performance of its Transportation Protective Services carriers. However, both our February 2014 report and the Federal Motor Carrier Safety Administration state that these scores should not be used to draw safety conclusions about a carrier’s safety condition. As a result, DOD may be determining which carriers should be eligible for the Transportation Protective Services program using the Compliance, Safety, Accountability’s Safety Measurement System that, for many carriers, lacks sufficient information to reliably assess carriers’ safety performance.” “To better ensure the safety and security of DOD’s shipments of sensitive arms, ammunition, and explosives, we recommend that the Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, direct TRANSCOM to examine the data limitations of the DOT Federal Motor Carrier Safety Administration’s Safety Measurement System raised in our February 2014 report on modifying DOT’s Compliance, Safety, and Accountability program…”
The report also recaps the flaws pointed out in the February GAO report. “In February 2014, we found that the Federal Motor Carrier Safety Administration faces challenges in reliably assessing safety risk for the majority of carriers. Among other things, most carriers lack sufficient safety performance data to ensure that Federal Motor Carrier Safety Administration can reliably compare them with other carriers using Safety Measurement System scores. Basing an assessment of a carrier’s safety performance on limited data may misrepresent the safety status of carriers, particularly those without sufficient data from which to reliably draw such a conclusion. In addition, previous evaluations of the Safety Measurement System have focused on estimating the correlations between crash risk and regulatory violation rates and Safety Measurement System scores. These evaluations have found mixed evidence that Safety Measurement System scores predict crash risk with a high degree of precision for specific carriers or groups of carriers. As we found, according to the Federal Motor Carrier Safety Administration’s own methodology, the Safety Measurement System is intended to prioritize intervention resources, identify and monitor carrier safety problems, and support the safety fitness determination process.
Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers, The Federal Motor Carrier Safety Administration also includes a disclaimer with the publicly released Safety Measurement System scores stating that the data are intended for agency and law-enforcement purposes, and readers should not draw safety conclusions about a carrier’s safety condition based on the Safety Measurement System score, but rather the carrier’s official safety rating. Due to ongoing litigation related to the Compliance, Safety, Accountability program and the publication of Safety Measurement System scores, we did not assess the potential effects or tradeoffs resulting from the display or any public use of these scores. We recommended that DOT improve the Compliance, Safety, Accountability program by revising the Safety Measurement System methodology to better account for limitations in drawing comparisons of safety performance information across carriers; and in doing so, conduct a formal analysis that specifically identifies, among other things, the limitations in the data used to calculate Safety Measurement System scores including variability in the carrier population and the quality and quantity of data available for carrier safety performance assessments. According to federal internal control standards, for an entity to run and control its operations, it must have relevant, reliable, and timely communications, including operational data.”
The General Accounting Office (GAO) and Inspector General (IG) both released congressionally directed studies of Compliance Safety Accountability (CSA) and found numerous critical flaws with the system. Both studies can be accessed in their entirety from this post, but they are lengthy so I will highlight key conclusions.
The GAO report “Modifying the
Improve the Ability to
Identify High Risk Carriers” analyzes nearly 315,000 U.S.-based carriers that were under FMCSA’s jurisdiction and, with reasonable certainty, were active during the period from December 2007 through June 2011. GAO considered a carrier active during this period if it received a state or federal inspection, was involved in a crash, or reported the number of vehicles it operates to FMCSA. The GAO study corroborated many of the deficiencies we have reported over the years.
- The factors being measured cannot be correlated to safe driving: “Our analysis found that most of the regulations used in (Safety Measurement System) SMS were violated too infrequently over a 2-year period to reliably assess whether they were accurate predictors of an individual carrier’s likelihood to crash in the future. We found that 593 of the approximately 750 regulations we examined were violated by less than one percent of carriers. Of the remaining regulations with sufficient violation data, we found 13 regulations for which violations consistently had some association with crash risk in at least half the tests we performed, and only two violations had sufficient data to consistently establish a substantial and statistically reliable relationship with crash risk across all of our tests.”
- There is insufficient data on most carriers: “Most carriers lack sufficient safety performance information to ensure that FMCSA can reliably compare them with other carriers.” “About two-thirds of carriers we evaluated operate fewer than four vehicles and more than 93 percent operate fewer than 20 vehicles. Moreover, many of these carriers’ vehicles are inspected infrequently".” “Given that SMS calculates violation rates for carriers having a very low exposure to violations, such as operating one or two vehicles or subject to a few inspections, many of the SMS scores based on these violation rates are likely to be imprecise.”
- State-by-state variability means CSA is measuring where a company operated, not how safely it operates: “The frequency of an individual carrier’s inspections varies depending on where the carrier operates. States vary on inspection and enforcement practices. Some studies have shown that inspectors or law enforcement officers in some states cite vehicles for certain violations more frequently than in other states. Delays in reporting crash data to FMCSA, as well as missing or inaccurate data, can affect a carrier’s Crash Indicator SMS scores. These delays can vary by state.”
- Data sufficiency standards are inadequate: “For most BASICs, we found FMCSA’s data sufficiency standards too low to ensure reliable comparisons across carriers. In other words, many carriers’ violations rates are based on an insufficient number of observations to be comparable to other carriers in calculating an accurate safety score. Our analysis shows that rate estimates generally become more precise around 10 to 20 observations, higher than the numbers that FMCSA uses for data sufficiency standards.”
- CSA is ineffective at predicting individual carrier crash risk and is suspect even for its original purpose of allocating enforcement resources: “Overall, SMS is successful at identifying a group of high risk carriers that have a higher group crash rate than the average crash rate of all carriers that we evaluated. However, further analysis shows that a majority of these high risk carriers did not crash at all, meaning that a minority of carriers in this group were responsible for all the crashes. As a result, FMCSA may devote significant intervention resources to carriers that do not pose as great a safety risk as other carriers, to which FMCSA could direct these resources.”
GAO analyzed an alternative data sufficiency standard of at least 20 trucks or 20 inspections and eliminating safety event groups and found far more reliability of the scores in identifying at-risk carriers. Furthermore, GAO questioned the viability of FMCSA’s proposal to determine a carrier’s fitness to operate based on SMS scores.
“FMCSA officials told us the primary purpose of SMS is to serve as a general radar screen for prioritizing interventions. However, as discussed above, due to insufficient data, SMS is not as effective as it could be for this purpose. Further, if the same safety performance data used to inform SMS scores are intended to help determine a carrier’s fitness to operate, most of these same limitations will apply. According to FMCSA, the Safety Fitness Determination rulemaking would seek to allow FMCSA to determine if a motor carrier is not fit to operate based on a carrier’s performance in five of the BASICs, an investigation, or a combination of roadside and investigative information. However, basing a carrier’s safety fitness determination on limited performance data may misrepresent the safety status of carriers, particularly those without sufficient data from which to reliably draw such a conclusion.”
The IG study “Actions Are Needed To Strengthen FMCSA’s Compliance, Safety, Accountability Program” was far more limited in scope. The study was coordinated with the GAO study to avoid duplication of effort. The IG report found fault with FMCSA’s lack of discipline in requiring carriers to update census information which is essential to measuring crashes per mile or per power unit. FMCSA has begun efforts to deactivate US DOT numbers for carriers that do not update census data. The IG was also critical of the Agency’s failure to fully implement the enforcement intervention process. Only 10 states had fully implemented the process at the time of the report. The other 40 states and DC are waiting on software and training which is not expected until May 2015. Since the primary purpose of CSA is to prioritize enforcement resources, it is disconcerting to find that so few states are in compliance and that improvement is not expected until 2015.
In between the release of these two reports, FMCSA released a study by the Volpe Center titled, “The Carrier Safety Measurement System (CSMS) Effectiveness Test by Behavior Analysis and Safety Improvement Categories (BASICs)”. This was widely seen as a weak attempt by FMCSA to contradict some of the GAO findings. This report is basically a rehash of the Agency’s data showing that “on average” carriers with higher SMS scores have higher accident frequency, but does nothing to dispute the incontrovertible evidence that CSA is unable to predict anything about the safety of an individual carrier.
What was originally designed to be a “general radar screen” to help prioritize FMCSA enforcement resources has very inappropriately become a guilty verdict in the mind of some shippers, who refuse to do business with carriers that have even one BASIC above the intervention threshold. When half of all carriers with a percentile ranking have at least one BASIC above the intervention threshold and there is absolutely no correlation between SMS scores and individual carrier accident frequency, this is simply wrong. It is harming our nation’s supply chain and doing nothing to improve highway safety.
Following the GAO study release, the American Trucking Associations, the Owner Operator Independent Drivers Association (OOIDA) and the group I chair Alliance for Safe, Efficient and Competitive Truck Transportation (ASECTT) have all called for removal of SMS scores from the FMCSA website. “Since scores are so often unreliable, third parties are prone to making erroneous judgments based on inaccurate data, an inequity that can only be solved in the near term by removing the scores from public view,” said Dave Osiecki, ATA executive vice president and chief of national advocacy.
CCJ Digital has published a number of very informative pieces on the problems of CSA. The latest post by Kevin Jones is terrific and highly recommended.
The American Trucking Associations issues a white paper questioning the reliability of CSA/SMS as a tool for shippers and brokers to use in selecting carriers. The paper concludes that “Though there are statistical correlations between SMS scores in certain categories and crash risk, as well as between the total number of alerts assigned and crash risk, individual carriers’ scores can be unreliable indicators of their safety performance. The identified correlations between scores and crash risk represent industry-wide trends that often don’t hold true for individual carriers. In most BASICs there are thousands of carriers (“exceptions”) whose scores contradict the trends (i.e. carriers with high scores but low crash rates and vice-versa).”
The paper is critical of the insufficient data collected by SMS stating that only 18 percent of active motor carriers have a score on even one of the five published BASICs. In an industry where 90% of the carriers have 6 or fewer trucks, only about 6% of those smaller fleets have enough data to generate a score in at least one BASIC.
Regional enforcement discrepancies are also highlighted indicating that scores are more a reflection of where a carrier operated than how safely it operates. For example, 10 states account for half of all speeding violations. Moving violations represent 29% of all violations in Indiana and Delaware but only 1.4% of all violations in Mississippi.
The system makes no attempt to determine who was at fault in crashes, so a carrier that is struck while legally parked is scored the same as a carrier that strikes a legally parked car. This is critical because it is estimated that 70-75% of all fatal truck-car crashes are caused by the automobile driver.
The Rube Goldberg system of severity weightings on every possible violation that are rolled up into a score on each BASIC is also criticized in the report. The statistical data was modified based on subjective input from “subject matter experts.” The report states that, “Doing so further blurred the statistical relationships between individual violations and crash risk.”
As we have stated all along, the ATA believs that SMS is an appropriate tool to help the FMCSA prioritize its enforcement resources and provides valuable feedback to carriers. The system is not appropriate for the shipping public to use for carrier selection.
The report concludes: “In almost all measurement categories there are thousands of fleets with high scores but low crash rates or vice-versa.
The relationship between scores and crash risk is impacted by a number of data and methodology problems that plague the system. A substantial lack of data, particularly on small carriers who comprise the bulk of the industry, hinders the system’s ability to render meaningful scores of comparative performance. Regional enforcement disparities likely cause fleets of all sizes operating in jurisdictions with robust enforcement practices to be perceived as less safe than those operating in other regions. Also, the questionable assignment of severity weights to individual violations can skew carriers’ scores. Finally, the underreporting of crashes by states, the use of crashes that were not caused by motor carriers, and the increased exposure to crashes experienced by carriers operating in urban environments, all affect the significance of Crash Indicator BASIC scores.”
In April, I had the privilege of sitting down with Russell Goodman, Editor in Chief of SupplyChainBrain, to discuss the regulatory challenges that will significantly impact the trucking industry. The “Executive Briefing” can only be viewed at SupplyChainBrain’s website with a valid login, so we have provided a short synopsis below.
The sweeping regulatory mandates that will take effect over the next few years will compound the preexisting capacity challenges due to the shortage in OTR drivers. Shippers must change their logistics strategies to avoid cost increases due to the reduction in effective capacity.
The new Hours of Service (HOS) rules that are scheduled to take effect July 1, 2013 are estimated to reduce effective capacity between 2% – 5%. Some carriers believe this estimate is too conservative, but we believe this estimate accurately accounts for the new 34-hour restart and rest-break provisions.
The ongoing controversy surrounding CSA scores is wholly deserved. FMCSA is punishing carriers despite the lack of correlation between CSA scores and accident frequency. More than 50% of all measured carriers have at least one BASIC score over the intervention threshold, rendering the scores useless as a means of evaluating carrier safety. There is a clear disconnect between safe operations and CSA scores. The motor carrier industry’s relentless pursuit of highway safety is demonstrated by the industry’s best safety record ever. The danger of shippers establishing an arbitrary threshold for selecting carriers based on CSA scores is a costly proposition that denies safe carriers access to the freight they need to survive in the highly competitive trucking market.
The broad effects of HOS changes and CSA scores are harmful to the industry, but electronic on-board recorders (EOBRs) make a lot of sense. Paper logs are prone to input and analytical errors that decrease reporting accuracy. Conversion to the technology has proven to be affordable, and some larger carriers have even achieved higher utilization with EOBRs. Unfortunately, full implementation of EOBRs will not occur until 2016 or later.
Shippers must embrace innovative logistics strategies. The impending capacity shortage greatly increases the value of converting truck freight to intermodal. The infrastructure improvements made by Norfolk Southern and CSX east of the Rockies promise a service level equivalent to single driver trucking in many lanes.
Despite having several years to fine-tune CSA, the fact is, the system still fails to provide any measurement for most carriers on the highways. For most carriers that CSA does claim to measure, small sample sizes of roadside inspections undermine the validity of the scores. Finally, the system still brands over half of the carriers it does measure as being deficient on at least one BASIC.
Only 89,134 out of 786 thousand carriers in FMCSA’s dataset have a score on at least on BASIC. To be fair, only about 500 thousand carriers are active on the highways but that would still mean that less than 20% of carriers are measured. Only 330k carriers have had at least one roadside inspection in the last 24 months. Of the 89k carriers that are measured, 49k have a score on at least one BASIC that places them over the intervention threshold (an Alert or “Golden Triangle). That means 55% of all measured carriers are branded as questionable by the FMCSA, which is ludicrous when one considers the exceptional record of highway safety in the trucking industry.
Fleets with less than 50 trucks make up 99% of all trucking firms and account for 49% of all trucks on the highway according to data downloaded from the FMCSA website. For single truck carriers, two thirds have not had a single roadside inspection in the last two years and only 4% have had enough inspections to receive a score on even one BASIC. For fleets of 2-10 trucks about half have no inspection data at all and only 14.5% have a score.
Of those small carriers unlucky enough to be trapped in the CSA web, 67% of the single truck fleets and 59% of 2-10 truck fleets have at least one Alert. The reason for this is the faulty math of the FMCSA, not the actual safety performance of small fleets. Dr. James Gimpel of the University of Maryland published a study critical of the CSA methodology. One of his points was that most studies with sample sizes of less than 20 would not be published because the sample is too small to accurately represent true performance. Less than half of all carriers with a CSA score have had 20 or more roadside inspections entered into the system in the last two years. Notice I wrote “entered into the system”. Many clean inspections are not uploaded into the system meaning that small carriers, who are seldom inspected anyway, find it almost impossible to offset one or two negative inspections and as a result have poor CSA scores. As shown in the chart below, 18 thousand carriers, which is 28% of all carriers with only 5-9 inspections, have a score on at least one BASIC. Of those, 64% have an Alert. None of those carriers should even have a score.
CSA was originally designed to help the FMCSA prioritize its enforcement resources. It is a perfectly good system to be used in that way. In addition, many carriers tell me they benefit from the data they are receiving from roadside inspections that allow the carrier to enhance their safety programs and better manage drivers. However, CSA was not designed to be used by the shipping public as a mechanism to screen and credential carriers and it is totally inappropriate to utilize the system to aid in carrier selection. There are many exceptionally safe carriers that have high CSA scores and many other carriers with high accident frequency that have very good CSA scores.
I spoke at the Warehouse Education and Research Council (WERC) meeting in Dallas yesterday on regulatory issues. In advance of that, we decided to update our analysis of the relationship between Compliance Safety Accountability (CSA) scores and carrier accident frequency and found there is no relationship between a carrier’s percentile scores on Unsafe Driving and Hours of Service and their accidents per million miles. These two charts show data for 28,000 carriers for Unsafe Driving and 48,000 carriers for Hours of Service from the March 2013 FMCSA data available on their website.
There have been numerous changes to the CSA program including a redefinition of BASICs, but the outcome remains the same. A carrier’s CSA scores are not a valid predictor of the likelihood of that carrier having an accident and should not be used by shippers and brokers in the carrier selection process.
We also reran the numbers removing all carriers with less than 25 trucks since those carriers have very few inspections and their scores are particularly unreliable. Even excluding the smallest carriers, it is still clear that there is no relationship between CSA scores and accident frequency. There are 9,905 carriers represented on the Unsafe Driving graph and 8,470 carriers on the Hours of Service graph.
The CSA data remains incomplete as well. As of March 2013, there are still only 89,134 carriers with a score on at least one of the five published BASICS out of a universe of at least 500,000 carriers, so less than 20% of carriers are measured. Of those carriers that are measured 55% have at least one BASIC over the intervention threshold. Given the phenomenal track record of the trucking industry in reducing highway accidents and fatalities, it is incredulous to think that 55% of carriers could have some deficiency in their safety programs.
Inspector General and General Accounting Office studies of the CSA program are underway and we are hopeful that these efforts will convince (or force) the FMCSA to use the CSA program as it was originally intended – that is as a mechanism to help the agency allocate its scarce enforcement resources. Many carriers also benefit from the feedback they receive from the inspection process. A federal lawsuit by ASECTT challenging the publication of scores is also heading towards oral arguments early this summer. The fundamental problem with this program is that the data is not valid for credentialing carriers, is not a suitable substitute for the Safety Fitness Determination that the FMCSA is federally obligated to provide, and should not be made available to the shipping public. Small businesses (carriers) are being harmed because they are denied freight that they should rightfully be allowed to haul. Shippers and brokers are needlessly being exposed to meritless liability claims.
Compliance, Safety, Accountability
> Regulatory Legislative Issues > Compliance, Safety, Accountability