The perimeter keeps moving: why an annual pentest starts ageing the day it is delivered

When a penetration test report is delivered, the company receives a snapshot of its digital exterior. The snapshot records the doors, windows and service entrances that were visible on that particular day. The difficulty is that the real network architecture changes faster than the audit planning cycle. A new version of an application ships on Friday evening. On Monday morning a DevOps engineer spins up a container to test an integration with a new payment gateway. On Tuesday a port is left open, or a test DNS record is never removed.

By Wednesday the new service is reachable from the internet. By Thursday it can be picked up by an automated bot scanner. And the last penetration test, carried out three months ago, states that no such service exists. This is not a security defect in the classic sense; it is a blind spot created by the time between checks. The one-off check is treated as sufficient largely for historical reasons, and today that model runs into a plain contradiction: infrastructure changes faster than manual audits can be scheduled and completed.

Where the belief in the annual cycle came from

The idea that security can be verified once a year came from physical protection and financial audit. In those fields the object of the check is relatively stable. A bank does not relocate every quarter, and its core financial processes stay the same across a year. The analogy was convenient: the notion of an audit was taken and applied to code and networks.

The approach fitted the era of monolithic applications. If a company ran one large system deployed on a handful of servers in a data centre, its external interface, the perimeter, genuinely changed rarely. Adding functionality usually meant updating internal logic rather than creating a new entry point from outside. A penetration test in that context worked like a building inspection: you check the locks, the walls and the roof. If the building is not rebuilt, the result of the inspection stays valid for a long time.

Infrastructure then moved to microservices and cloud environments. The building now consists of hundreds of small modules that are connected and disconnected constantly. The external surface of a company, its set of domains, IP addresses and open ports, has become dynamic. It expands and contracts hour by hour, following the development lifecycle. The confidence that nothing has changed since the last test is usually mistaken, because the changes happen at the level of configurations and temporary resources that never appear in the release checklist.

What changed in attacker behaviour

The attacking side adapted to this pace faster than the security industry did. The reconnaissance techniques used to collect information about a victim's network are catalogued in MITRE ATT&CK as T1590, Gather Victim Network Information. The stage covers determining network structure, mapping hosts and collecting data about devices. It used to be done by hand, selectively and at cost. Today much of it is automated.

Bots scan the internet continuously, looking for new targets: recently registered domains, new DNS records, IP addresses that have just started answering requests. For an attacker, the appearance of a new public service is a signal. A fresh configuration, possibly not yet fully patched, possibly still carrying default credentials or a debug interface.

Attacks rarely start at the hardened core of a system. A forgotten admin panel, a test API or a database backup left in a publicly readable storage bucket is an easier starting point. Assets like these appear by accident, live briefly and disappear a few days later, which is still long enough for them to end up in an attacker's inventory.

It is worth separating what is known from what is an estimate. What is known is that automation has driven the cost of mass scanning down far enough to run it against the whole address space. What is an estimate is that recently added assets account for a disproportionate share of successful compromises, simply because older assets have been through several rounds of fixes. The precise picture for any specific company comes only from analysing a year of access logs against the creation date of each resource.

The gap between a snapshot and a stream

Picture an office photo shoot arranged before an important investor meeting. The photographer takes an excellent frame: everything is tidy, the documents are in order, the staff are smiling. While the photograph was being processed and printed, someone left the safe open and the bins were not taken out. The frame is still excellent. It simply no longer reflects the current state of things.

A penetration test works the same way. A team of experts spends weeks studying the architecture, looking for logic flaws and testing authorisation. The result is a list of problems found at the time of the test. As soon as the report is signed, the information starts to age. Every deployment, every new DNS record, every new IoT device on the office network is a change to the perimeter that the last audit did not cover.

The risk is not that old code becomes vulnerable. The risk is that a new environment grows around it that has never been checked. A new frontend may have a misconfigured CORS policy. A new backend may use an outdated version of a library that is not on the old dependency list. A new IP address may belong to a monitoring service running with weak default credentials.

In the traditional model, changes like these are discovered either by accident, when a developer realises something was left behind, or at the next annual penetration test. The second case means a potential vulnerability has been living in production for months. In business terms that is a window of exposure, during which anyone looking could have found the new door.

Why manual tracking does not scale

The obvious question: can a company not simply keep a spreadsheet of all external services and update it on every release? It can, and many do. The practice tends to break down for two reasons.

• Fragmented ownership. Domains are handled by marketing, IP addresses by network engineers, APIs by developers, public storage by DevOps. No single person holds the full picture of the external estate. The spreadsheet stays incomplete, because somebody somewhere created a resource without telling the other teams.
• The human factor. Keeping the spreadsheet current takes discipline and attention. Under deadline pressure, registering a new asset in the security inventory slips down the list. The asset goes to production and the paperwork follows later. By the time the spreadsheet is updated, the asset may have been publicly reachable for a month.

Relying on a manually maintained asset inventory means tracking on foot something that moves at the speed of deployment. The tooling has to match the pace of change in the environment.

From event to state

Closing this gap requires a change of framing: to stop treating security as an event, the penetration test, and start treating it as a state, monitoring. That does not mean penetration tests are no longer needed. They remain essential for deep analysis of business logic, for complex exploit chains, and for internal networks that a scanner cannot reach.

What has to become continuous is the base layer of control: a mechanism that looks at the external boundary around the clock and compares the current picture with a reference one. Regulatory practice points the same way. The US directive CISA BOD 23-01 for federal networks is built around improving asset visibility and vulnerability detection. Its premise is that static asset lists go out of date too quickly, and that organisations need the ability to see changes in their digital environment close to real time. The underlying requirement is simple: what you do not know is on your perimeter right now, you cannot protect.

What continuous control looks like in practice

Picture a system that takes a photograph of the external estate every few hours, or every few minutes. It asks DNS servers which names are attached to the company's zone. It scans IP ranges to see which ports answer. It checks TLS certificates to see which sites operate under the company's brand.

If the system sees a new name, staging-api.company.com, that was not there before, it does not try to break into it. It marks it as new. That is the point where two processes meet.

• Perimeter change monitoring answers the question "What has appeared?" Its job is detection: new domains, new IP addresses, new types of device, a router where a web server used to be.
• Continuous scanning answers the question "Is this normal?" When monitoring reports a new asset, the scanner automatically runs a basic check against it. Not the depth of a penetration test, but a fast pass: open ports, known CVEs, HTTP settings, exposed sensitive files.

The output is not a static report but a live list of hypotheses. Every change on the perimeter creates a task in a verification queue, and security engineers receive an alert of a familiar shape.

This changes the character of the team's work. Instead of waiting six months to check the whole network at once, specialists react to changes. Triage, the sorting of findings and the assessment of their severity, becomes a continuous process, and new exposures are assessed in hours rather than months.

What this means for the business

Translated into the language of risk: if a new service appears on a Tuesday and the next penetration test is eight months away, that service sits on the perimeter unexamined for those eight months. Nothing about that makes an incident inevitable. It means that nobody on the defending side is in a position to say what condition the service is in, or whether anyone else has already looked at it.

With continuous monitoring the reaction time shrinks to days, sometimes hours. A new asset appears, the scan points to a problem, missing HTTPS or a known vulnerability in the software version, and the team can close it in the same sprint. You do not wait for the next audit to learn that a door was left open.

There is a caveat that should not be skipped. Automation is cheaper than manual work, but it is not free. Deploying monitoring tools and wiring them into the CI/CD pipeline takes effort: rules have to be tuned, false positives filtered out, and the team has to learn to work with a new stream of alerts. It is an investment in operational capability and should be planned as one.

Where automation ends

It is important to understand the limits of the approach. Perimeter monitoring and continuous scanning address visibility and simple configuration errors. They are good at finding open ports, outdated software and mistakes in web server settings. They do not replace a penetration test.

Automated tools have a poor grasp of business logic. They will not see that user A can open user B's profile when the first letters of their surnames match. They will not work out that an upload feature accepts executable files, because that behaviour breaks the logic of the application rather than a security standard. Finding complex, multi-step vulnerabilities of this kind takes people, creative thinking and an understanding of how the application is supposed to work.

So the workable security architecture is a hybrid. Continuous monitoring keeps awareness of the perimeter current and clears low-level problems quickly. Periodic penetration tests go deep into logic and search for complex attack paths. The results of a test then feed back into the monitoring rules: if testers found a way around an authorisation check through a specific API field, that check can be added to the regular scans, or at minimum the component can be flagged as requiring closer attention whenever it changes.

A question for your IT leader

At the next meeting with a CISO or CTO, one direct question is worth asking.

How do we find out that a new public service or domain has appeared, if it was not part of the current quarter's plan?

A question to put to your security lead

An answer of "we run a penetration test once a year" means the organisation is accepting the risk of not knowing about part of its own digital estate. An answer of "we keep an asset inventory" is worth following with a second question: how current is that inventory, and is it verified automatically, or does it depend on developers remembering?

A concrete next step is to audit how the teams register new external dependencies, and to measure the time between the deployment of a new service and the moment it becomes visible to the security tooling. If that interval is measured in weeks, that is where the response gap is.

Verdict

The belief that an annual penetration test is enough on its own does not fall apart because penetration tests got worse. They became insufficient as the only instrument of control. Infrastructure became dynamic, and reconnaissance became automated.

The correction is simple: the one-off check becomes part of a continuous cycle. The proposal is not to give up penetration testing, but to add a layer of ongoing observation of the external perimeter alongside it. Change monitoring makes a new element in the architecture visible shortly after it appears. Continuous scanning turns that observation into action by returning the new asset to the verification queue.

The result is that penetration test findings stop being an archive document. They become a live list of hypotheses that is updated alongside the infrastructure. The organisation spends less time being surprised by its own exposure and more time managing it. That is not magic; it is discipline supported by the right tooling. It starts with accepting that the external wall keeps moving, and that the defence has to move with it.