The Question We Didn't Ask

In 2020 I co-wrote a white paper on securing video surveillance systems. Reading it now, the omission matters more than anything in it — and software supply chain tooling is making the same one.

In short

Six years ago I helped write a vendor white paper about the cyber posture of networked cameras. It got the hard part right: it identified that the people watching a security system were usually not the people responsible for fixing it, and that nobody clearly owned the device. It got one thing wrong by omission — every posture question it asked was about how a device had been configured, and none asked what the device was or who wrote its firmware. That omission has a cause, and the cause is repeating right now in software supply chain tooling: configuration is visible from the network, provenance is not, and products measure what they can see.

What we wrote

In July 2020, at Razberi, I co-wrote a white paper with a sales engineer colleague on surveillance system resilience and health monitoring. It was a marketing document. It was also, within its scope, accurate.

It covered what could take a camera system down — drive failures, silent Windows service stops, thermal shutdowns — and what was actively hunting it: Mirai, still crawling the internet four years after it conscripted CCTV cameras; Dark Nexus, going after internet-exposed cameras and routers; Ripple20, nineteen vulnerabilities in an embedded TCP/IP stack from Treck, four of them critical, disclosed by JSOF on June 16, 2020, a month before we published. Estimates of the installed base ran from tens of millions of devices to hundreds of millions, across a vendor count nobody could pin down. Our paper said billions, which was wrong by at least an order of magnitude and is a decent illustration of how carefully we were reading our own citations.

And it argued that a monitoring platform should assess the system's cybersecurity posture continuously, answering four questions: is antivirus installed on servers and clients, are devices following hardening best practices, are any of them using guessable credentials, and is anything communicating out to unknown destinations.

What we got right

Buried in the middle of that paper is a sentence I'd now put in the title. Security operators monitor the system's outputs, but they are frequently not the team responsible for fixing the technical problems.

That's the ownership gap, stated plainly, in 2020, by the vendor.

It wasn't an isolated observation. Razberi's Chief Product Officer said in the Razberi Monitor launch release that the platform existed to align physical security and IT departments — a product company stating that the org chart was the problem it was solving. The customer in that release, Chad Nelson at Tropical Shipping, independently described the value the same way: aligning his network and surveillance departments.

And in a 2020 survey I produced of more than 80 physical and IT security professionals, respondents ranked not aligned with network IT policy among the top five cyber attack vulnerabilities in their own deployments, sitting in the same list as weak passwords and unpatched firmware. They classified an organizational failure as an attack surface, which is the correct classification. In the same survey, 51% named cybersecurity a top concern and 85% planned to install more networked cameras that year. Worried, and deploying anyway.

The industry knew. It said so, repeatedly, in public.

What we missed

Every posture question in that white paper is a configuration question. Is antivirus present? Are credentials weak? Is hardening applied? Is the device talking to somewhere it shouldn't?

Not one of them asks what the device is, or who built it, or what firmware is actually running on it.

The uncomfortable part is that we cited the answer and walked past it.

The hardest thing about Ripple20 was never the nineteen vulnerabilities. It was that nobody could determine which devices contained the affected component. JSOF spent much of the disclosure tracing where a twenty-year-old software library had ended up, and reported that identifying the full set of affected vendors and devices was obstructed by both logistics and licensing. Forescout, working the same problem alongside them, named the root cause without hedging: in the IoT world there is no public bill of materials that lets users know what components are inside the devices they use.

That was published in June 2020. We listed Ripple20 as a threat in July and then proposed four posture questions, not one of which would have told a customer whether they were exposed to it.

We published in July 2020, eleven months after Section 889(a)(1)(A) took effect, barring federal agencies from procuring, and contractors from delivering, any system that uses covered video surveillance equipment as a substantial or essential component.

Part (B) went further, barring agencies from entering into, extending, or renewing a contract with any entity that merely uses covered equipment anywhere in its own operations — including on work with nothing to do with the government. Both parts were already law or law on a published schedule before we wrote: (A) had been in force for eleven months, and (B)'s effective date had been set since the 2018 statute. For anyone selling surveillance into federal facilities or critical infrastructure, the provenance question was already procurement law. It does not appear in our paper.

I don't think that was negligence, and I don't think we were unusual. Configuration is observable from the network. A monitoring platform can see an open port, a default credential, an outbound connection. It cannot see who manufactured a board or who compiled an image. We wrote about what the product could measure, and what the product could measure quietly became the definition of the problem.

What the market did instead

The gap got filled from outside the tooling, and badly.

Section 889 names five companies: Huawei and ZTE for telecommunications equipment, and Hytera, Hikvision, and Dahua for video surveillance and telecommunications equipment used for public safety, security of government facilities, physical security surveillance of critical infrastructure, or other national security purposes. The covered foreign country, per the clause, is the People's Republic of China.

But the list is not closed. A further category sweeps in equipment or services from any entity the Secretary of Defense, in consultation with the Director of National Intelligence or the FBI, reasonably believes to be owned by, controlled by, or otherwise connected to that government.

Read that as an admission. Congress could not write down a finite list of manufacturers, so it wrote a clause allowing the list to grow by later determination. The entity-ban approach concedes its own limit inside its own text.

Even the named part was hard to apply. The camera makers are among the largest on earth, and their hardware ships under a long list of other labels. The statute reaches those relabels — a covered product stays covered regardless of the logo — but legal reach is not detection. Buyers got an obligation with no instrument.

So the market produced paper: vendor compliance statements, attached to bills of materials. And then field technicians started checking MAC address OUI prefixes and reading firmware fingerprints off device interfaces, because the hardware maker registers those and the enclosure label doesn't change them. Guidance moved from which brands are safe to per-SKU verification.

That's forensic provenance, and it exists because attestation didn't. Note who ended up carrying it: integrators, facilities managers, contracting officers. The exact same people who never owned the device to begin with. The signature line and the technical capability were in different departments.

The same omission, in software

Ask what tooling measures about AI-generated code today and you get the 2020 list in a new costume: does it have known vulnerable dependencies, does it match insecure patterns, does it pass policy. All observable from the artifact. All configuration questions.

Ask who or what wrote a given function and whether anyone competent reviewed it before it entered the build, and the tooling is mostly silent — not because vendors are lazy, but because that evidence has to come from the producer and no scanner can synthesize it.

And the code sits in the same organizational blind spot the camera did. The developer accepted the completion. The security team inherited the result. Ask who vouched for it and the honest answer is often nobody, because no one's job description contains the question.

With one difference that makes it harder. Section 889 had a factory — an entity, in a place, with a name. A model has none. There is no logo on the enclosure and no MAC prefix to look up, which means the instrument federal procurement reaches for first has nothing to attach to.

What I'd write differently

If I wrote that white paper again, I'd put ownership before instrumentation. Not "here is what a monitoring platform should watch," but: name the person accountable for what runs on this device, then buy the tool that gives them evidence.

The industry bought the tool first for a decade. It got very good dashboards and never answered the question.

Software is about to make the same trade. Most of the current energy is going into generating SBOMs — a document that says what's in a build — rather than into evidence a third party can verify, bound to the build, that fails loudly when false. Generating the document is the easy half. It's also the half that produces paper.

I know what the paper phase looks like. I wrote some of it.

Common questions

What is the difference between configuration and provenance in security tooling?

Configuration is how a device or artifact is set up: open ports, credentials, hardening, outbound connections. Provenance is what the thing is and who made it — which manufacturer, whose firmware, which model wrote a function. Configuration is observable from the network, so tooling measures it. Provenance has to come from the producer, so tooling is usually silent on it, and the silence quietly becomes the definition of the problem.

What does Section 889 of the FY2019 NDAA prohibit?

Section 889 bars the federal government from procuring covered telecommunications and video surveillance equipment from five named Chinese companies and their affiliates (part A, effective August 13, 2019), and from contracting with any entity that merely uses such equipment anywhere in its operations (part B, effective August 13, 2020). A further clause lets the list grow by later determination of the Secretary of Defense.

Can a Software Bill of Materials establish who wrote AI-generated code?

No. An SBOM records what components are in a build. It does not record who or what authored a given function, or whether anyone competent reviewed it before it entered the build. That evidence has to come from the producer and no scanner can synthesize it from the artifact — the same provenance gap that networked-device security left unanswered a decade earlier.

Why is AI-generated code provenance harder to verify than hardware provenance?

Hardware has a factory: an entity, in a place, with a name, and identifiers like MAC OUI prefixes that a technician can look up. A model has none of these. There is no logo on the enclosure and no prefix to check, so the instrument federal procurement reaches for first — an entity ban — has nothing to attach to.

This piece revisits a marketing white paper I co-wrote in 2020. Survey figures are from a reader survey I produced that year; policy details are cited to the FY2019 NDAA Section 889 and its implementing FAR rules. Corrections welcome, particularly from people who work procurement or authorization for a living.