Abstract
Security guidance is commonly presented as a checklist: inspect a setting, review a permission, confirm that a control is enabled, and repeat the exercise at a later date. A checklist can be useful, but it describes work to be completed rather than conditions a system must continue to satisfy. This distinction matters in long-lived software. Code is added, modules are installed, production settings are modified, and temporary exceptions are introduced under operational pressure. A system that passed a security review once may no longer satisfy the same requirements.
This paper presents the security baseline as a practical response to that problem. A baseline is a selected set of explicit rules against which a system is assessed repeatedly. Initial failures identify the gap between the current system and the required state. After review and remediation, the baseline reaches a passing state. Later failures show that a previously satisfied condition has been broken and needs attention. The baseline may also be revised when legitimate business requirements change, but it should not be weakened merely to make failed checks disappear.
The purpose of a baseline is not to prove that a system is secure. Its purpose is to make a defined set of security expectations visible, testable, and maintainable over time.
1. Security Is Often Lost Gradually
Security failures are often discussed as discrete events: a vulnerability is exploited, an account is compromised, or malicious code is introduced. These events matter, but they do not describe the full path by which an application becomes vulnerable. In many systems, security is weakened gradually through ordinary work.
Magento provides a useful example. A new installation that remains close to the supported core can begin from a relatively controlled state. Over time, however, a production store accumulates custom modules, integrations, rewrites, operational exceptions, and changes made directly under commercial pressure. The system becomes more valuable to the business at the same time that it becomes harder to understand as a whole.
Sales campaigns make this tension easy to see. A release may need to go live on a fixed date. Developers are asked to make the campaign work, sometimes with little room to reconsider the surrounding architecture. A setting may be changed temporarily, a module may be installed quickly, debugging may remain enabled, or a permission may be relaxed to resolve an immediate problem. The campaign runs, the commercial deadline is met, and the team moves on.
The temporary state often remains.
This is not necessarily evidence of careless people. It is a predictable result of how success is defined. The immediate objective is visible: the campaign must launch. Restoration work is less visible because the harm caused by a weakened security condition may not appear at once. Once the urgent event has passed, the team has little incentive to revisit changes that appear to be working.
A useful security method must therefore account for more than known attacks. It must also detect when normal development and operational work move a system away from conditions that were previously considered acceptable.
2. The Limits of the Security Checklist
Security checklists are a reasonable starting point. They turn broad advice into a set of tasks and give reviewers a way to avoid relying entirely on memory. For Magento, available guidance commonly includes checks for configuration, permissions, administrative access, deployment hygiene, dependencies, and related concerns.
The limitation is not that checklists contain the wrong work. The limitation is the way that work is represented and repeated.
A manual checklist may say:
- Check whether Magento is running in production mode.
- Review sensitive file permissions.
- Inspect administrative accounts.
- Verify that required security controls are enabled.
- Review installed modules and dependencies.
Each item asks a person to perform an activity. Once the activity is complete, the item can be marked as done. The checklist itself does not necessarily preserve the result as an expectation that remains applicable after the review.
Manual execution also has a cost. In the experience that led to this methodology, a meaningful Magento security checklist could consume roughly a working day. When a review competes with releases, client work, and production incidents, it is unlikely to become a frequent routine. It may be performed before a launch, after an incident, or whenever someone remembers that enough time has passed.
The system continues to change between those reviews.
This creates an awkward gap. The organization may know that a security review was completed in the past, but it cannot easily say whether the conditions verified during that review still hold today. A completed checklist records an activity. It does not, by itself, maintain a state.
3. From Tasks to Rules
Many checklist items can be expressed more precisely as rules.
Instead of:
Check the Magento application mode.
A rule states:
A production Magento environment must run in production mode.
Instead of:
Review the permissions of sensitive configuration files.
A rule states:
Sensitive configuration files must not be readable or writable beyond the access required by the application.
This change in wording is more than an editorial improvement. It separates three elements that a checklist often leaves implicit:
- the condition that is expected;
- the condition that is observed; and
- the difference between them.
Once an expectation can be evaluated consistently, it becomes repeatable. The same rule can be run after a deployment, after a module installation, during a scheduled review, or as part of a delivery pipeline. Automation reduces the cost of performing the check, but the more important change is conceptual: security guidance has become a condition the system is expected to preserve.
This leads to the central distinction of this paper:
A checklist is completed. A baseline is maintained.
A checklist describes work to be performed. A baseline describes rules that must continue to pass.
Not every security judgment can or should be automated. Some rules can be evaluated directly from configuration or filesystem state. Others require source-code analysis, external information, or human interpretation. A machine can identify that an expected condition does not hold; it may not be able to determine why the condition was broken, whether the surrounding design is acceptable, or which remediation is appropriate.
Automation makes repeated evaluation practical. It does not remove the need for security judgment.
4. Ruleset, Profile, and Project Baseline
A baseline needs rules that are sufficiently stable to be evaluated consistently. In the model developed for Magento, the starting point is a ruleset of 99 purpose-built rules grouped into 12 security controls. The rules draw from Magento security guidance and are informed by broader sources such as OWASP and PCI DSS requirements, then tailored to conditions that can be examined meaningfully in a Magento environment.
The ruleset defines what the assessment method knows how to examine. It does not require every project to use every rule.
Different Magento projects have different architectures, payment flows, integrations, data exposure, operational practices, and business obligations. Applying the same selection to every project would ignore those differences. Allowing each project to redefine the meaning of every rule, however, would make results inconsistent and difficult to compare.
The model addresses this by separating rules, profiles, and project baselines.
Rule
A rule expresses one specific security expectation and defines how that expectation is checked. Its identity and intent remain stable across projects.
Profile
A profile is a pre-built group of rules selected for a common assessment need. Profiles such as Basic, Standard, OWASP-oriented, or PCI-oriented provide practical starting points. A Full profile can select all 99 rules.
A profile is not a complete judgment about a specific system. Passing an OWASP-oriented or PCI-oriented profile does not prove that the application is secure or that the organization is compliant with an external standard. The profiles select relevant technical checks; the broader assessment still depends on scope, architecture, process, people, and evidence outside the application.
Project baseline
A project baseline is the set of rules that a particular Magento project is expected to satisfy after its requirements have been reviewed. It may begin with a pre-built profile and then be adapted to the system's actual security level and business needs.
The distinction can be summarized simply:
The ruleset defines the available checks. A profile provides a starting point. A project baseline records the decision.
The flexibility belongs to the composition of the baseline, not to the meaning of the underlying rules. A rule should not mean one thing for one merchant and the opposite for another. A project may include or exclude a rule after review, but inclusion means accepting the condition that the rule expresses.
5. Establishing Baseline v1
The first version of a project baseline does not emerge by recording everything that happens to exist in the system. A snapshot describes the current state, including its mistakes. A baseline describes the state the project is required to reach and maintain.
Establishing Baseline v1 begins by selecting a suitable profile or, when appropriate, the full ruleset. The selection is then reviewed against the project's architecture, business requirements, operational environment, and security needs. A generic profile may be sufficient for an initial assessment, while a project-specific application security review may produce a more deliberate selection.
The selected baseline is run against the current system. Failed rules become findings.
These initial findings represent the gap between the observed system and the selected security requirements. They do not become accepted merely because they were present during the first scan. A person must review the findings, understand their causes, and remediate the system. The process continues until every rule selected for the baseline passes.
The sequence is:
Select a profile
↓
Review the project and select appropriate rules
↓
Run the baseline assessment
↓
Review failed rules
↓
Remediate the system
↓
Run the assessment again
↓
All selected rules pass
↓
Baseline v1 is established
Baseline v1 is therefore not a version of an assessment tool or of the ruleset. The version belongs to the Magento project. It identifies the project's first reviewed and satisfied set of security rules.
This is an important difference from a one-time audit report. The report describes what was found during an assessment. The baseline remains after the report and provides the reference for the next assessment.
6. Findings, Remediation, and the Passing State
A failed rule identifies a broken expectation. It does not automatically explain the cause.
The cause may be a configuration mistake, a direct modification in production, a newly installed module, a custom implementation, a vulnerable dependency, or an operational workaround that was never restored. The same failed rule may require different remediation in different systems.
For that reason, the output of automation is a finding, not a final decision. Human review remains necessary to determine:
- why the rule failed;
- what risk the failure creates in context;
- which code, configuration, dependency, or process must change; and
- how to verify that the remediation is complete.
After remediation, the baseline is run again. A passing result means that the observed system satisfies the selected rules at that time.
It does not mean that no vulnerability exists. It does not guarantee that the system cannot be compromised. It means something narrower and more defensible: every security condition represented by the selected baseline was satisfied by the evidence available to its checks.
That narrower claim is useful because it can be repeated.
7. Maintaining the Baseline Over Time
The work does not end when Baseline v1 passes. Passing establishes the state that subsequent checks are intended to preserve.
Software continues to change. Developers add code, modules are installed or upgraded, infrastructure is modified, and production settings are adjusted. Some changes are planned and reviewed. Others are made under time pressure or directly in response to an incident. Any of them can break a condition that previously passed.
Running the same baseline again makes the change visible. If a rule that previously passed now fails, the project has moved away from an accepted security condition. The failure is a signal of security drift or regression.
There is a useful distinction between an initial finding and a later failure:
- During the first assessment, a failed rule reveals an existing gap between the system and its intended baseline.
- During a later assessment, a failed rule shows that a previously satisfied expectation no longer holds.
The technical result may look the same, but the history is different. In the second case, the team knows that the project had reached a passing state and that something subsequently broke it.
Repeated evaluation can occur on a schedule, after deployment, after a module change, before and after a major sales event, or within CI/CD. The method does not require every rule to run every second. Continuity comes from preserving the same expectations and returning to them whenever the system changes.
The cycle is:
Assess
→ Review
→ Remediate
→ Pass
→ Change
→ Reassess
→ Restore
Security is treated here not as a certificate earned once, but as a condition that requires maintenance.
8. Revising the Baseline
Not every change is unwanted. A project may introduce a new payment flow, integration, module, deployment model, or business capability. When legitimate requirements change, the baseline itself must be reviewed.
This creates two different responses to change:
The system changes unexpectedly
→ Keep the current baseline
→ Remediate the system
Business requirements change deliberately
→ Review the baseline
→ Add or remove rules as appropriate
→ Establish a new baseline version
A failed rule is not, by itself, a reason to remove that rule. Otherwise, the baseline could always be made to pass by redefining security around the system's current weaknesses. The baseline should be revised when the project's accepted requirements or risk context change, not merely when remediation is inconvenient.
After review, a project may move from Baseline v1 to v1.1 or v1.2. These versions belong to that project. They record the evolution of the rules the project is expected to satisfy.
Routine scans do not create new versions. Fixing a regression does not create a new version when the underlying requirements remain the same. A new version records a reviewed change to those requirements.
This gives the baseline two complementary properties:
- It is stable enough to detect unwanted change.
- It is revisable enough to remain relevant when the business changes legitimately.
Without stability, the baseline cannot function as a reference. Without revision, it eventually stops representing the system it is intended to protect.
9. The Roles of Automation and Human Judgment
The baseline method divides work according to what machines and people do well.
Automation can execute defined rules consistently, collect evidence, report failures, and repeat the same assessment at relatively low cost. It reduces dependence on memory and makes it practical to run checks more often than a manual review.
Human judgment is needed to select the baseline, understand the project, interpret findings, choose remediation, and review changes in business requirements. It is also needed where the available rules do not capture the full security question.
The relationship is:
Automation identifies the broken expectation. Human review determines why it broke and how it should be restored.
This is especially important in application security. Two Magento stores may use the same platform while having very different payment scope, custom code, integrations, administrative workflows, and exposure. A generic profile can provide coverage and a useful starting point. It cannot determine the full risk context of each business.
The baseline therefore supports judgment rather than attempting to replace it.
10. What a Baseline Cannot Prove
A baseline makes a bounded claim. Its value depends on keeping that boundary explicit.
A passing baseline does not prove:
- that the application has no unknown vulnerabilities;
- that every relevant security condition has been included;
- that a system cannot be compromised;
- that all organizational processes are effective;
- or that an external compliance standard has been satisfied in full.
The baseline can only evaluate the expectations represented by its selected rules and the evidence available to those rules. Some security properties cannot be determined through static or automated checks alone. Others exist outside the application, including organizational process, personnel, network architecture, physical controls, and third-party operations.
This limitation does not make the baseline weak. It makes its result interpretable.
The baseline replaces a vague statement such as "the site is secure" with a narrower one:
At the time of assessment, the system satisfied the selected and reviewed security rules defined by its current project baseline.
Because that statement is specific, it can be tested again after the system changes.
Conclusion
Long-lived applications rarely remain in their initial state. They accumulate features, modules, integrations, exceptions, and operational compromises. Security guidance delivered only as a periodic checklist struggles to keep pace with that change.
A security baseline raises the checklist from a list of activities to a set of conditions that must continue to hold. Rules make those conditions explicit. Profiles provide practical starting points. A project baseline records the rules selected for a particular system. Initial assessment reveals the gap between the current and required states; remediation establishes a passing Baseline v1; repeated assessment identifies when later changes break previously satisfied rules.
The baseline is not frozen. When legitimate business requirements change, it is reviewed and revised as a new project version. But it is not rewritten merely to excuse failures. Its purpose is to remain a stable reference for security while the system around it continues to evolve.
The central idea is simple:
Security is not maintained because a checklist was completed once. It is maintained because important conditions are defined, checked again, and restored when they are broken.