Why Compliance Isn’t Security: And How to Fix It.
If I told you that behind this garage door are four tires, an engine, windows, a radio, a transmission, a gas tank and an exhaust pipe, you would likely conclude that I had an automobile in my garage. It’s not a bad assumption of course as most people do keep cars, trucks, or vans in their garage but how would you prove that I did in fact have an automobile in my garage? You could take my word for it, since I brag about my new car at work and seem to be a trustworthy guy or you could simply open the door and assess the situation.
Unfortunately, I wasn’t being honest, and I have many parts to an automobile, but I have not yet assembled them. How many times have we seen this in the cybersecurity world where organizations have purchased plenty of hardware and software but have not architected their networks correctly or built a secure environment yet. Imagine someone you know building a car the way most enterprises build a Zero Trust architecture. They buy a chassis, an engine, a transmission, a battery, a set of brakes, an electronic control unit, four wheels, and a body. They lay the parts out neatly on the floor of a clean, well-lit garage. They run a checklist: chassis present, engine present, transmission present, brakes present, electronics present. Every box gets a tick. They post on social media announcing that they have a new car.
Nobody, of course, would actually call that a car. A car is not the sum of its parts. A car is what happens when those parts are assembled correctly, calibrated to one another, integrated into a working whole, and then driven (accelerated, braked, steered, parked, crashed, and serviced) under conditions that resemble the road. The car is the emergent function of the parts working together.
To take this point one step further, you could open up the door to my garage and find something else. You could find that I am building a vehicle, but it does not work because it is either unfinished, has incorrect parts, or has been assembled poorly. In other words, am I missing a windshield? Are all four tires the same size? Do I have a motorcycle engine inside of my pickup truck frame? This also happens every day in IT shops around the country. Network architectures are constructed that barely function, function incorrectly, or do not function at all. Until the car works, it is hard to call it a car. It’s essentially still an inventory.
Most cybersecurity assessments (let’s take Zero Trust for example) grade the inventory. They ask whether you have an identity provider, MFA, conditional access, EDR, micro segmentation, and a SIEM. They do not ask whether those systems integrate cleanly, whether their policies agree, whether their telemetry actually flows, whether the policy engine fires when it should, whether the enforcement points actually block what that engine decides. This “car” could not pass a road test.
In both the first scenario and the second scenario above I could have taken an industry checklist and passed a self-attested inspection because all parts are accounted for. These self-attestations are an easy button, and a dangerous one at that. They are clearly meant to be a scalable effort to force organizations to do things the right way but there is, as we have pointed out, a glaring loophole. No objective validation of functionality or security exists in this model of compliance. They are dangerous as assessment instruments, for three reasons.
Number one, as we mentioned they are oftentimes self-graded. The team that owns the controls answers the questionnaire that grades the controls. Even with the best intentions, this produces optimism bias.
Secondly, they measure presence, not effectiveness. “Do you have phishing-resistant MFA for administrators?” is a different question from “Do all administrative paths into your tier-zero systems require phishing-resistant MFA, all the time, with no exceptions and no break-glass shortcuts?” Almost every organization will answer yes to the first. Very few would survive the second.
Lastly, they do not ask the integration questions. Each Zero Trust pillar is graded on its own. User. Devices. Networks. Applications and Workloads. Data. Visibility and Analytics. Automation and Orchestration. Whether the user/identity actually informs network policy, whether device posture actually blocks unhealthy hosts from reaching sensitive applications, whether data classification actually changes what conditional access does, those are integration questions. Cyber scorecards cannot capture these.
There is another scenario that exists. In this scenario, you open my garage door, find an automobile that seems well put together, and ask for my keys. You open the hood, check the fluids, get into the vehicle, test the wipers and turn signals, then start the engine and go for a test drive. What are you measuring? 0-60 mph time? Breaking distance? Actual miles per gallon of gasoline? Similarly, true compliance with government cybersecurity regulations should require Outcome-Based Assessments that provide objective and inarguable results. Outcome-based assessments get to the heart of a very important question: What can a realistic adversary actually do within this environment right now, and how does your environment respond? The answer is produced by testing, not by checklist. Environments need to be stress tested for real life scenarios that will eventually happen thereby helping organizations be prepared for breaches BEFORE they occur. We must begin to examine outcome metrics that actually matter. Most of the metrics on a typical cybersecurity dashboard (number of policies, percentage of users on MFA, number of agents deployed) are inputs. Inputs cannot tell us whether we are winning. The outputs are what truly matter and here are a few important ones:
Mean Time to Detect (MTTD). From the moment an attacker action occurs to the moment your team has a credible signal of it. This should be measured per attack stage, not as a single number, because detection in the initial-access stage is worth dramatically more than detection in the exfiltration stage.
Mean Time to Contain (MTTC). From detection to the attacker losing the ability to do further damage. Containment, not investigation, is the metric as investigations can run for weeks; containment must not.
Attack-Path Coverage. Take a representative starting point (a phished workforce credential, a compromised contractor laptop, a leaked SaaS API token) and enumerate the paths to each tier-zero asset. What percentage of those paths are blocked at the first hop? The second? Pillared maturity models cannot answer this. Attack-path graphs can.
Conclusion
I know what you are thinking. Hands-on assessments of functionality and security take a long time, not to mention they pull talented resources away from their current work to perform a task that doesn’t help the immediate bottom line. The good news for you is that this is a thing of the past. Automated Outcome-Based Assessment Software is now a reality, and these tools save significant time and money while providing government accepted reports and recommendations with the click of a button. The evaluations measure outcomes only, so you leave knowing whether your architecture and tools responded to threats as expected. It is imperative that we encourage system owners, administrators, network and cybersecurity engineers, CISOs, CTOs, and CIOs to abandon the checklist model of self-attestation and embrace the future of compliance assessments that truly guarantee functionality and security.
The shift from checklist-based self-attestation to automated, outcome-based assessment isn’t a far-off ideal, it’s already operational. At Tensley Consulting, we built ZT ROAM (Risk-Based Objective Assessment Methodology) to deliver exactly what modern compliance demands: empirical evidence that your architecture, security tools, and defensive controls perform as intended against real threat behaviors. Rather than asking your team to certify what should work on paper, ZT ROAM measures what actually works in practice, then produces the government-accepted reports and recommendations to prove it (in a fraction of the time and cost of traditional hands-on assessments).
For system owners, administrators, engineers, and executives carrying the weight of accreditation and risk acceptance this means faster authorizations, defensible findings, and engineering hours returned to mission. It means trading the false confidence of a completed checklist for the real confidence of a validated outcome.
If you’re ready to see what an objective, outcome-based assessment looks like for your environment, request a demo of ZT ROAM and let our team show you how Tensley Consulting is redefining what “compliant” really means.
Written by:
Ryan Rosencranz, CXO ZT ROAM
CPT William Johnson, CISSP/PMP U.S. Army Ret.
Dr. David Moured, AI Security Architect | Red Team Lead | Zero Trust SME