07 - Garbage In - Garbage Out.

Back to Top

Bad Intel is Worse Than No Intel at All:

The primary task of the Salt Working Group was solve this nation's interoperability crisis. The data necessary to form an understanding of the nation’s degree of fragmentation or the number of triage systems in use does not exits. The only data that does exist is inaccurate and unverified. Tangible data suggest there is no interoperability or lack of standardization issue that needs to be addressed.

U.S. S.T.A.R.T. triage Adoption by State.

Alabama

Hawaii

Massachusetts

New Mexico

South Dakota

Alaska

Idaho

Michigan

New York

Tennessee

Arizona

Illinois

Minnesota

North Carolina

Texas

Arkansas

Indiana

Mississippi

North Dakota

Utah

California

Iowa

Missouri

Ohio

Vermont

Colorado

Kansas

Montana

Oklahoma

Virginia

Connecticut

Kentucky

Nebraska

Oregon

Washington

Delaware

Louisiana

Nevada

Pennsylvania

West Virginia

Florida

Maine

New Hampshire

Rhode Island

Wisconsin*

Georgia

Maryland

New Jersey

South Carolina

Wyoming

A Nation of S.T.A.R.T.

Ground Truthing 101:

We thought it strange - all these reports of multiple triage systems. So, in our off days, we did some research of our own. We made contact with hundreds EMS agencies across the country and obtained their MCI protocols. All had START, save for those few cities that have adopted SALT. We spoke to dozens of members of NAEMSE (National Association of EMS Educators) - All teach START and most have never heard of SALT and none, save for one, has ever trained on it. We contacted CERT and Civil Support Teams representing almost every state and again found only START being the standard. We used data available from the triage tag manufacturers and conducted research looking for agencies that use “other” systems. We aggregated all that raw information and constructed a population-based research approach to see what triage system covers what portion of the U.S. population.

It could, and should, be argued our approach was unscientific and skeptics of our findings may suggest we were biased and looking for a given outcome. Those are healthy opinions to hold and we and look forward to third-party debunking. In the meantime we stand by our preliminary finding that 98% of the U.S. population falls within the boundaries of a START-oriented response agency. This does not include the city fragmentation recently caused by SALT because we know the responders in those agencies will still be able recall their START training, for now.

In the interest of third-party verification of our information we have supplied representative links to get you started. All 50 states use START save for parts of Wisconsin, Georgia and Ohio that have begun transitioning to SALT. Hire an intern.

Perception is Not Reality:

Sometimes we see what we want to see and sometimes we don’t know what to look for. When doing data research upon which any critical decision is to be made we owe it to ourselves at a minimum, and to our public as a standard, to validate our information set we base those decision on. For some reason the allure of SALT seems to obscure this basic principal and drives entities to see a fragmentation ghoul where one does not exist. Take for instance Georgia's Public Health Department’s interpretation and lack of validation of data they collected while surveying their EMS agencies. Their interpretation led to their adoption of SALT.

The Questions They Asked

The Assessment They Inferred

The first issue with the validity of the data was the framing of the question and the Health Department’s confusion as to what a triage system is vs. a triage tool. (A common mistake for people with little field experience) START is a system. MetTag and Disaster Management Systems are tools (tags) for START triage. So the real data from their survey is indicated in Chart 2 below. The second issue is referred to as “probable outcome” in the survey world. (If you are against smoking, chances are you would not buy yourself an ash tray) So looking at the responses of “Other” we can infer that it is not SALT because the requesting agency is the one that has introduce SALT, ergo SALT is so new any responding agency would know it’s name. Likewise, since the agency is reporting they do have a triage system but do not know what it is, we have to look at the alternatives. There are none. All systems need a tool and all the tools available today are START-based. Chart 3 reflects the outcome of this data properly adjusted.

Finally, and most inexplicably, is the “No answer” quotient. No answer from your EMS Agencies? Is that not an immediate call to action to pick up the phone and get one? Especially when the data is part of an information set that creates a fundamental change? Regardless of the politics, the math is simple. When they do get the answer it will not be SALT. The outcome will most likely look like Chart 4.

The only interoperability issue in the state of Georgia was the one created by SALT. In fact Georgia is looking the same as our national SALT fragmentation pie chart if one more state adopts it. The objective here is not to bag on our Brothers in Georgia, it was a necessary evil to demonstrate that SALT has an influence of attraction we are baffled by and unable to comprehend. We know Georgia was a strategic state in SALT’s objective for national adoption. We also know the Salt Working Group knew the Georgia survey data was flawed. (They know better than any that DMS, Met tag and SMART are START tools) yet they let Georgia run information they knew was bad regardless of the fragmentation it would cause. What could possibly motivate them to to that?

Did Georgia get coached into looking for fragmentation where it did not exist or did they simply make a data-analysis error. Either way, they never validated their data and made a decision based off an inaccurate assumption. Disclosure: The Working Group had multiple members from Georgia.

Chart 3

Chart 2

Chart 4

That’s the Signpost Up Ahead…:
In another example of how bad information plays a key role in the the argument to adopt SALT is this slide from the Dayton Ohio deck:
 

1. START requires 60 seconds to assess a patient: In what world is that possible?
The author is a senior paramedic and headed up an EMS training bureau. Maybe the POI is to count respirations for a full minute.  

2. Psychological criteria never validated: Does the letter “M” come to mind? We must be missing something here. Nobody in our industry could be that naive.

3. Real world use limited…: Maybe true if your “real world” is Dayton and take 60 seconds to count respirations. Any competent operator can apply START, or any system, in well under 30 seconds. Ironically it is SALT that extends the assessment time well past 60 seconds by allowing up to 59 seconds to apply LSI prior to assessment.
“…the intervention can be performed quickly (less than 1 min)”
                                                    - Salt Core Criteria 3.1 - Lifesaving Intervention.

Dayton’s other mistake. Using SALT for both initial and secondary triage. (They use ribbons for initial and tags for secondary). Maybe they realized SALT’s limitation of being only a initial triage system was counter-intuitive so they made SALT more START-like.

SALT Acceptance Syndrome: (SAS) Symptoms: Data-blindness, delusional convictions and a strong willingness to get federal funding.

Introduction:

Two types of data went into the development of SALT. Quantitative data related to determining the triage systems being used today in the U.S. and science/evidence-based data to develop the algorithm itself. Both data sets were flawed.

A Standard by Any Other Name:

In all of our travels, from the National Fire Academy to Anniston to conventions, CERT jamborees and association meetings nationwide, the only triage process being taught, used and recommended was START.  It may have come in different forms: A SMART tag, a DMS tag, a Met tag, custom tags like Virginia’s or New Jersey’s or just a simple piece of surveyor’s tape but it was all START. Other systems came, such a STM in a few counties, SIEVE and the NDLSF pushing MASS as a bolt-on to START (unnecessary) but at the end of the day our EMS agencies stayed with START in the interest of patient care, simplicity and interoperability.    
To this day some agencies are trying new processes, as they should. New York City is trying out a fifth START category of orange that falls in-between red and yellow while a county in Washington state is trying out ‘sick-not-sick’. Maybe we’ll learn something from them. In the meantime they know START and are ready to inter-operate as needed.

The warning next to the map that the data was prone to error and not validated.

The infamous map used by the Working Group validate a fragmentation crisis exists.

Chicken Little this…:
All parties involved in the creation of SALT knew of, but ignored, these issues and presented the bad data as gospel and built a citadel of crisis upon it. Maybe they thought it would slip under the radar and go unquestioned. Consider it questioned. These are our academic best and industry leaders that we intrust and empower to make accurate directional decisions based off facts, not survey-generated unchecked assumptions. The internal system of reasonable checks-and balances we rightly expect to be part of our nation's decision making process for EMS matters is apparently flawed and needs to be corrected. Correction does not come in the form of silent omission, but in the form of formal accountability. From the Working Group to the FICEMS, we deserve to either see the verified facts upon which they based their conclusion that an interoperability crisis exists or a acknowledgment that the process was flawed and a reset should put in place until the facts are in and the directional decision to move forward is grounded in fact. Until either of those events occur it is reasonable to be suspect.

NASEMSO 2010 Survey

The  2010 NASEMSO survey also warns they asked the wrong people the right question with regard to triage systems. The Working Group included a senior member of NASEMO. It would be implausible to suggest they were not aware of their own SALT-related survey.

“The Model Uniform Core Criteria for Mass Casualty Triage were designed to increase interoperability among various mass casualty triage protocols.” - SALT Working Group

“The intent of the Core Criteria is to ensure that providers at a mass casualty incident use triage methodologies that incorporate core principles in an effort to promote interoperability and standardization.” - SALT Working Group

See the pattern here? In medical terms we refer to this as -

All existing surveys of our triage structure are unscientific, have no data validation component and were not constructed in a manner that their resulting data could be used for any reliable measure of the condition.
The creators of the 2011 National EMS Assessment Survey appropriately included a repetitive disclosure making it abundantly clear to the reader the data was prone to error and objective. And for good cause, the data was radically incorrect and in no way represented anything remotely close to the actual condition.

Science is in the Eye of the Beholder:

It used to be that science was fairly easy to define. A hypothesis is made and an outcome predicted. Tests are performed to validate the hypothesis and after the results are validated by repetitive blind tests, science is born. That’s now how it works anymore.

Here is how the Working Group defined “science” as it pertains to SALT.

Science” was defined as the existence of at least 1 peer-reviewed publication that explicitly tested the criterion in regard to mass casualty triage. The data within those articles were found to support the concept identified in the criteria, but the workgroup did not score the articles based on the quality of the study. Therefore, this designation indicates that data exists, but the strength of its support is not represented.”

In layman terms: Somebody wrote an opinion paper on the MCI triage, threw in some untested data, someone else read it, and science was born. It did not matter if the supporting data was accurate or embellished because it was never tested. As long as it was from a ‘peer’ and it supported the Working Group’s position, it was science.

The only ‘science-based’ evidence in SALT is the use of tourniquet, airway management, peripheral pulse and chest decompressions.

Arguably the antidote science conclusion is flawed. One of two article cited makes no reference to antidote usage during triage and the second uses the word ‘triage’ in a different context than that of obtaining an acuity class based on an algorithm. If you can’t get a pulse you can’t perform triage in the context of SALT. We can conduct LSI, as the author was implying. To what beneficial extent immediate application of that LSI has on a patient who has been unattended for 40-plus minutes is a good discussion to have but it is not this discussion. Because antidote usage was not used in conjunction with mass casualty triage as it relates to obtaining an acuity class in either article it does not meet the Working Groups own criteria of being relative to the subject. So antidote usage must be pulled from the science category.

The chest decompression citations are compelling and it would be hard to question this treatment's value in pre-hospital care settings. Neither of the articles, however, studied needle use in an MCI environment where patients will be left unattended for prolonged periods. In fact both the articles highly recommend continuous monitoring of the patient. It is likely true that any patient needing this procedure is better off with it than without it, but again this is not something that should be considered scientific fact in an mass casualty incident setting. Because neither paper supports the use of needles in such a setting it must be removed from the science category per the Working Group's criteria.


What we are left with is exactly what START began with: Tourniquet, pulse and airway management. Take out the items of clear conjecture above and START is as ‘science-based’ as SALT.


The Second Science:
There is even a new kind of science, something called “indirect science”. The Working Group classifies it as such:


“Indirect science” was defined as the existence of at least 1 peer-reviewed publication that studied the criterion under different circumstances or in a different patient population. Again, this term was not intended to reflect the strength of the evidence within the cited publications, but instead to reflect the population that was studied.”


In laymen terms: Read one peer-reviewed document on the subject of patient care that has unverified data within it and it’s indirect science.

 

The subject here is SALT being based on science. Indirect science is not that so we won’t delve into the issues related to it. However, one reference we are not clear on is the indirect science for not counting vitals. We may be wrong on this, but search as we may we cannot find a clear answer. We believe this is in reference to respirations since it is already agreed science that pulse-rate is ruled out in favor of palpable pulse. We are looking for the reasoning that “respiratory distress’” is used in SALT as opposed to START’s >30. We know that START used >30 because inexperienced responders, such as CERT, may not diagnose respiratory distress properly. In their life experience, even being startled can cause someone to breathe rapidly so a respiratory rate of even 50 could seem normal to them. They don’t have the clinical wherewithal to understand that a patient with a slightly elevated rate of one breath every two seconds has been breathing as such for an extended period of time - indicating a poor physiology. We realize untrained laymen assume this means we count the breaths for extend times but in reality it cannot take more than 10 seconds in any condition and usually less than 5. We will keep searching but it would be nice to know where this indirect science came from. It would be a shame to deprive our largest response force such a vital non-subjective tool.
On this subject, START does not include shallow breathing for a reason. It’s covered by checking Mental. There is no need for the <10 we’ve seen in some department guidelines for adults.

Searching for the Science:
Every proponent of SALT we have spoken to knows very little about it. They seem content with trusting that since its ‘science based’ it must be better. After all it is comprised of the ‘best available evidence’ derived from subject matter experts. What could possibly be wrong with that? Glad you asked.

“Qualitative and quantitative data sources reviewed by the Preparedness Committee indicate variability across the nation in the MCI triage systems used by state and local EMS systems.”

Stop the Madness:
In what could be the worst exaggeration of data is the citation inside the FICEMS report in reference to a 2013 publication from the Institute of Medicine’s Crisis Standards of Care: A systems Framework for Catastrophic Disaster Response. Upon analysis by the Working Group they claim the Preparedness Committee found qualitative and quantitative data showing variability across the nation with respect to MCI triage systems being used.

This is blatantly false. In reviewing the 520 page document there is only mention of two systems being used. START and SALT and no specificity is given to the quantitative coverage area of each. (Close to 98% to 2%).

Further, a member of the working group is quoted in the publication states:


“EMS personnel utilizing disaster triage systems (sort, assess, life-saving interventions, treatment/ transport; simple triage and rapid treatment [START]; and JumpSTART triage methods) so they can assess patients within 60 seconds and categorize them for immediate or delayed care.”
-(HHS,2011; Lerner et al., 2011; Romig, 2011)      

However in the publication the Preparedness Committee does make this recommendation with regard to how new crisis management concepts should be considered and adopted:


The state EMS office, in cooperation with the state health department, should ensure that EMS agencies have an opportunity to review and discuss the CSC plan at the state, regional, and local levels. EMS agencies should be engaged in the planning process from the beginning, and this can easily be accomplished through regional advisory councils or committees.”


We don’t recall that process occurring.

Best Available Evidence?…Not So Much:
This is the last information type the working Group used to draw its conclusions.


“Consensus” was defined as criteria that were unanimously agreed to by the workgroup; there also may be existing nonresearch publications that support the concept. A criterion was considered to be part of an existing triage system if it was included in 1 or more of the triage systems that are in use today, excluding SALT Triage.”


In laymen terms: A show of hands.

 

No need to discuss this process save to say these are highly skilled medical professionals that did their best due diligence to arrive at what they felt were qualified sound decisions based on medical knowledge and extensive research on a complex subject where little data exists. Our only issue is that they apparently limited their evidence gathering to the realm of academia and selected subject-matter-experts. We believe the best available evidence lies outside that realm and is found in even greater abundance in the real world. We believe valuable information could have been obtained by interviewing ICs and Triage Unit Leaders that have managed MCIs and reviewed their After Action Reports and recommendations. They have been working on this project for almost a decade yet not one AAR or hotwash is cited in any of their publications. A lot of MCIs have occurred in that timeframe and each one should be looked at as an opportunity to learn to do better. One good example of this is the following statement by the Working Group with respect to assessing the ‘walking wounded’ last:

“…Alternatively, it is possible that minimally injured or noninjured victims may carry injured children or smaller adults to the designated area when the “walk” command is given. Although there is no scientific evidence to support these possibilities, it was the consensus of the panel, with support of the published data described above, that every victim must be assessed individually.”


No scientific evidence? Tell that to the Triage Unit Leader of Century Division in Aurora. An interview with him certainly would have removed the necessity of a consensus opinion and quite possibly would have given the Working Group the insight to see just how important it is to assess the walking early on instead of requiring it to be dead last.     


Expert from an article co-authored by two of the Working Group’s leads where discuss SALT and other triage systems. This is after all the SALT research had been completed.
Question: What changed?

…Accordingly, this chapter will describe the existing triage systems but will be unable to recommend one system be used over the other due to lack of scientific data.”
-
Brook Lerner, David c. Cone and Christopher A. Kahn

“At this time, no specific recommendation based on strong evidence can be made to support any one triage system over another.”
-
Brook Lerner, David c. Cone and Christopher A. Kahn

September 2009

Conclusion:

Review this document in our research section.

District of Columbia

In Closing:

Not long ago a well-known group made a very short video that best sums up what we are experiencing. If you have a few moments its worth the click.

Is There a Doctor in the House?:

It’s not just us. Several predominate members in the medical community concur with our findings that the process used by the Working Group is flawed, or at a minimum premature. As one example is this interview with the professor of emergency medicine at the University of California at Irvine and codirector of the school's EMS and disaster medicine sciences fellowship.