A sample is defensible only when it answers a defined audit question from a validated population. This practical guide helps internal auditors, CISOs, IT managers, and control owners distinguish representative sampling from targeted testing, choose a repeatable selection method, and avoid conclusions that exceed the work performed.
Start with the conclusion you need to support—then build the population and selection design that can support it.
Sampling reduces the number of items tested; it does not reduce the need to define the audit objective, applicable criteria, expected condition, period, sampling unit, source system, and deviation. If the source listing is incomplete, duplicated, filtered incorrectly, missing deleted records, or limited to one identity store, the selection can be perfectly random and still answer the wrong question.
Before calculating a size or choosing items, state whether the procedure is intended to support a conclusion about control design, implementation, operation over time, a configuration snapshot, a defined set of high-risk items, or only the particular records examined.
These approaches can be combined, but they do not support identical conclusions. Label each selection honestly so reviewers can understand what was covered and what remains uncertain.
Use complete testing when the population is small, every item is material, automated analysis can reliably evaluate all items, or missing one exception would create unacceptable risk. A full-population query still requires validation of source completeness, query logic, exclusions, time coverage, and false positives.
Conclusion boundary: may address the entire validated population for the attributes actually tested.
Use a statistical or appropriately designed nonstatistical sample when testing every eligible item is impractical and the objective requires evidence about the broader population. Each eligible unit needs an opportunity for selection consistent with the design, and the evaluation method must address sampling risk.
Conclusion boundary: may extend beyond tested items only as supported by the sampling design and evaluation.
Select privileged accounts, internet-facing assets, emergency changes, failed jobs, unsupported systems, large transactions, unusual exceptions, or specific incidents when those characteristics matter. Targeted work is valuable, but the results generally apply to the selected items—not automatically to untested items.
Conclusion boundary: describe the targeted items and avoid presenting them as a representative sample.
Do not combine a targeted selection and a random sample into one undifferentiated pass rate. Preserve separate purposes, populations, results, and conclusion logic. A high-risk exception can be reportable even when it is not projected statistically, while a representative sample may require a defined evaluation of deviation rate and sampling risk.
The sampling frame is the actual set of eligible units available for selection. Population reconciliation often provides more audit value than the sample itself because missing classes of devices, accounts, changes, tickets, or events can conceal systematic failures.
State the inclusion and exclusion rules, audit period, sites, tenants, systems, control owners, and one unit of analysis. A unit might be a user termination, privileged account, firewall rule change, monthly access review, patch deployment, backup job, incident, vendor, device, vulnerability, or restore test.
Determine whether the population comes from a system of record, configuration export, identity platform, ticketing system, HR system, SIEM, vulnerability platform, backup console, asset repository, cloud API, or a reconciliation of several sources. Record permissions, query, filters, time zone, pagination, and extraction date.
Reconcile totals to independent sources, investigate gaps in sequence, compare beginning and ending inventories, account for deleted and disabled objects, validate API limits, inspect null identifiers, and determine whether local systems, shadow IT, acquisitions, or disconnected devices are absent.
Confirm that material fields reflect the source, timestamps use the intended time zone, status values are interpreted correctly, duplicates are resolved, composite records are not counted twice, and each selected identifier can be traced back to an authoritative object.
Preserve a read-only population file or reproducible query result with an identifier, extraction time, record count, field list, filters, hash or integrity record when appropriate, repository location, sensitivity marking, and retention rule. Later changes should create a new version.
Determine whether one population actually contains materially different control processes. Separate locations, platforms, administrators, business units, risk tiers, manual versus automated workflows, or time periods when control execution and expected deviation may differ.
No single method is best for every cybersecurity test. Document the method before examining results, preserve enough information to reproduce the selection, and obtain statistical assistance when the design or evaluation requires specialized expertise.
| Method | Useful when | How to preserve reproducibility | Strength | Material limitation | Cybersecurity example | Reviewer challenge |
|---|---|---|---|---|---|---|
| Simple random | Eligible units are sufficiently comparable and each unit should have an equal selection opportunity. | Freeze the ordered frame, preserve the tool and version, random seed, requested size, generated positions, selected IDs, and date. | Reduces conscious selection bias and supports probability-based designs when properly evaluated. | May underrepresent small high-risk subgroups unless they are separately addressed. | Randomly select user-access recertifications from a validated annual population. | Could every eligible unit be selected, and can the identical selection be regenerated? |
| Systematic random-start | A stable, complete ordered population is available and a periodic interval is operationally efficient. | Record the frame order, population size, interval calculation, random start, wrap or end rule, and selected IDs. | Simple to execute and spreads selections across the frame. | Hidden periodicity or a meaningful sort order can bias coverage. | Select every nth approved firewall change after a documented random start. | Does the source ordering correlate with administrator, site, time, or control outcome? |
| Stratified random | Risk, platform, location, value, privilege, or process differences could be masked in one pooled population. | Define strata before selection; preserve membership rules, counts, allocation, seeds, and separate evaluation logic. | Ensures coverage of material subgroups and may improve efficiency. | Incorrect strata or unsupported aggregation can distort a combined conclusion. | Sample privileged, standard, service, and guest accounts separately. | Are strata mutually exclusive, collectively appropriate, and evaluated at the right level? |
| Cluster or multistage | Populations are distributed across many sites, business units, systems, or providers and travel/access cost is material. | Document each selection stage, probability or judgment at every stage, cluster comparability, and weighting/evaluation requirements. | Can reduce collection cost and coordinate fieldwork. | Units within a cluster may be correlated; a few selected sites may not represent all sites. | Select locations, then randomly select endpoint hardening records within each chosen location. | Why do selected clusters support the stated enterprise conclusion? |
| Risk-based targeted | The audit needs direct coverage of critical, unusual, failed, or suspected items. | Record the risk rule before testing, the full set meeting the rule, selected identifiers, exclusions, and why remaining items were not examined. | Concentrates effort where business impact or likelihood is greatest. | Not a representative sample unless combined with a separately designed representative selection. | Test all domain administrators, emergency changes, and internet-facing critical vulnerabilities. | Does the report clearly limit results to targeted items? |
| Haphazard nonstatistical | A nonstatistical design is approved and the auditor selects without a conscious pattern or deliberate exclusion. | Preserve the frame, instructions, selected IDs, timing, preparer, and evidence that convenience or result knowledge did not drive selection. | May be practical for some low-complexity tests. | Selection probabilities are not measurable; subconscious bias and convenience selection remain risks. | Select change tickets across the audit period without favoring easy-to-retrieve records. | Why is this approach sufficient, and how was bias controlled? |
| Full-population analytics | A reliable rule, query, or analytic can evaluate every unit for a defined attribute. | Preserve source validation, code/query, tool version, parameters, exclusions, error handling, results, and manual validation of the analytic. | Can detect all rule-defined exceptions in the validated frame. | It tests only what the logic can observe; data defects and false positives can affect every result. | Evaluate every terminated user against directory disable timestamps. | Was the analytic independently validated, and what conditions could it miss? |
| Hybrid coverage | Critical items need complete or targeted testing while the remaining population needs representative coverage. | Separate the census/targeted segment from the residual sample; preserve distinct populations, methods, results, and conclusions. | Balances high-risk certainty with efficient broader assurance. | Blended percentages can mislead if segments are aggregated without a valid basis. | Test all privileged accounts plus a random sample of standard accounts. | Can reviewers trace each result to the correct segment and conclusion? |
A number that looks precise can still be indefensible when its assumptions are missing. Statistical and nonstatistical approaches require professional judgment about the audit objective, desired assurance, population characteristics, expected exceptions, acceptable error, and how results will be evaluated.
Design, implementation, and operating-effectiveness questions require different evidence. A current screenshot cannot establish operation throughout a year, and a transaction sample may not address whether the control was suitably designed.
Greater assurance generally requires stronger evidence and may require more selections. Define the acceptable sampling risk and whether statistical evaluation will quantify it.
State the maximum error compatible with the control conclusion. In cybersecurity, one critical deviation may be reportable even when an overall rate appears low.
Prior findings, process maturity, monitoring, system changes, manual steps, staff turnover, incidents, and preliminary analytics can influence the anticipated exception rate and design.
Population size may matter, but heterogeneity, small critical subgroups, multiple process owners, and site-specific execution can matter more than the raw record count.
More weak evidence does not automatically create strong assurance. Consider source reliability, corroboration, reperformance, observation timing, automation, inquiry limitations, and false-positive analysis.
A sample size should not be copied from another audit without validating its assumptions. When the engagement requires probability-based confidence, projection, complex strata, cluster designs, or specialized estimation, involve a qualified audit-sampling or statistical specialist.
Freeze important decisions before results are visible. That separation helps prevent replacing inconvenient items, expanding only after favorable results, or quietly narrowing the population when evidence is difficult to obtain.
Identify the risk, control, criteria, expected condition, period, and exact conclusion the test is intended to support.
Describe one eligible item and the specific condition that counts as a failure, including partial, late, overridden, or missing evidence.
Obtain the complete eligible set from authoritative sources, reconcile systems, and record exclusions and coverage limitations.
Test counts, uniqueness, required fields, dates, filters, pagination, status interpretation, deleted items, and source-to-record accuracy.
Separate critical items or strata when privilege, exposure, location, platform, administrator, process, or time period affects the risk.
Document complete, representative, targeted, or hybrid coverage; size assumptions; method; specialist input; and planned evaluation.
Preserve the selection tool, version, seed, random start, interval, stratum rules, selected IDs, order, date, and preparer before testing.
Apply the planned procedure consistently. Record evidence, result, deviation, limitation, and any approved procedure change by item.
Determine cause, extent, systemic implications, compensating controls, contradictory evidence, and whether expansion or redesign is required.
Reconcile selected items, evaluate sampling risk, limit the conclusion, resolve review notes, and link findings, remediation, and retesting.
A difficult item is not a reason to substitute an easier one. The workpaper should make it impossible to hide missing evidence, unavailable systems, contradictory records, control overrides, or deviations discovered after selection.
Determine whether absent records indicate a population defect, control deviation, retention failure, access limitation, or an authorized scope restriction. Do not mark an item not applicable merely because documentation cannot be produced.
Replace an item only under a defined and approved condition such as verified ineligibility or duplicate sampling unit. Preserve the original selection, reason, evidence, approver, replacement method, and effect on evaluation.
Additional testing may assess whether a failure is isolated, validate a suspected root cause, cover a newly identified subgroup, or obtain sufficient evidence. Predefine decision rules where practical and do not expand only until a preferred result appears.
A single active former employee account, unprotected privileged identity, failed restore, exposed administrative interface, or unapproved firewall rule can warrant action independent of an estimated population rate.
A clean sample means no deviations were found in the tested units under the performed procedure. It does not prove that no deviations exist, that the frame was complete, or that another attribute would also pass.
When tickets, logs, screenshots, interviews, and system states disagree, preserve the conflict, investigate source reliability and timing, and explain why the final conclusion accepts or rejects each source.
These examples illustrate sampling logic, not prescribed sample sizes. Engagement authorization, platform behavior, contractual requirements, regulatory expectations, privacy, risk, and available evidence determine the final procedure.
Population: all HR-recorded terminations during the period reconciled to every relevant identity store. Coverage: consider full-population analytics for disable timing plus targeted review of privileged or delayed cases. Deviation: disablement exceeds the approved threshold without an authorized exception. Watch: contractors, shared accounts, local accounts, application identities, and time-zone differences.
Population: completed review decisions and associated privileged identities for each required cycle. Coverage: stratify by platform, privilege class, reviewer, and risk; test all break-glass and domain-level access where appropriate. Deviation: missing, late, incomplete, conflicted, or unsupported approval. Watch: reviewers approving their own access and groups expanded only at review time.
Population: production configuration changes reconciled between the ticketing system and device logs. Coverage: test emergency and internet-facing changes completely or as targeted items, then sample the residual standard population. Deviation: absent authorization, testing, rollback, peer review, or accurate implementation record. Watch: out-of-band administrator changes and shared credentials.
Population: findings due during the period, linked to affected assets and validated scanner coverage. Coverage: test critical and externally exposed items directly; stratify remaining findings by severity, asset class, business owner, and closure reason. Deviation: unsupported closure, failed validation, overdue remediation, or unapproved risk acceptance. Watch: rescans that no longer include the original asset.
Population: scheduled jobs, protected systems, restore tests, failures, and exceptions across the period. Coverage: target critical systems and failed jobs; sample successful jobs across platforms, sites, and time periods; inspect restore evidence separately. Deviation: missed coverage, unresolved failure, unencrypted or inaccessible copy, or restore result that does not meet approved objectives. Watch: backup success being treated as recovery proof.
Population: reconciled in-scope endpoints with deployment status, exceptions, and device activity. Coverage: use full-population analytics when data is reliable; target internet-facing, privileged, unsupported, and repeatedly noncompliant devices; validate a sample directly against endpoints. Deviation: missing control, overdue update, stale agent, false inventory status, or unsupported exception. Watch: inactive devices disappearing from compliance reports.
The reviewer should be able to reconstruct the eligible population, regenerate the selection, trace every tested item, understand each exception, and determine why the conclusion does or does not extend beyond the examined records.
Use the cybersecurity audit workpapers guide to connect the population, sample, procedure, evidence, exception analysis, reviewer notes, and conclusion in one controlled record.
A credible sample depends on decisions made earlier in the engagement. Scope determines the eligible universe; criteria define the expected condition; evidence requests determine whether fields and source records are sufficient; workpapers preserve the selection and evaluation.
When sampling exposes unreliable inventories, fragmented identity sources, inconsistent change workflows, missing monitoring, or weak operational records, co-managed IT implementation support can help improve the approved technical and operational controls.
Set authorized boundaries, systems, periods, owners, and stop conditions with the internal security audit planning and scope guide.
Define the tested requirement and expected condition with the cybersecurity audit criteria and control-mapping guide.
Protect source exports, derived frames, selections, and item-level results using the audit evidence collection and protection guide.
Specify required source fields, formats, periods, and validation checks with the technical audit evidence request guide.
Follow the full sequence from authorization through remediation with the internal security audit process guide.
Ali Hassani brings 25+ years of IT, cybersecurity, compliance, Microsoft infrastructure, cloud, firewall, network, vulnerability-management, and operations experience to internal security audits. That technical context helps distinguish a clean report from a complete population, a configuration snapshot from sustained operation, and a statistically neat selection from one that misses the systems where risk is concentrated.
Review Ali Hassani’s professional background or contact OC Security Audit to discuss an authorized internal audit, independent sampling review, population validation, or control-testing engagement.
The resources below provide recognized audit and assessment foundations. Determine which standards apply to the organization, engagement, sector, contract, and assurance objective; specialized statistical designs may require expert assistance.
The current FAM provides detailed audit methodology and sampling documentation resources. Its financial-audit context should be adapted carefully to the cybersecurity objective.
PCAOB’s audit sampling standard discusses sampling uncertainty, tests of controls, substantive details, and selection approaches for applicable financial-statement audits.
NIST assessment procedures help structure assessment objectives, methods, objects, and evidence for security and privacy controls.
The IIA standards address effective engagement planning, conducting engagement work, evidence, professional judgment, communication, and quality.
No universal number is appropriate for every test. The objective, desired assurance, tolerable and expected deviation, population characteristics, selection method, evidence quality, criticality, and planned evaluation all affect the design. Apply the standard relevant to the engagement and obtain specialist assistance when needed.
It may be described as a targeted or judgmental selection, but the report should not imply that it is representative of the untested population. Preserve its risk criteria, selected items, and conclusion boundary separately from any representative sample.
Critical items may warrant complete or targeted testing independent of the random sample. A hybrid design can test all critical items and separately sample the residual population, with distinct results and conclusions.
Investigate why. The unavailability may be a deviation, population defect, retention failure, access limitation, or verified ineligibility. Preserve the original item and use an approved replacement rule only when replacement is justified.
It means no exceptions were identified in the tested units under the performed procedure. The conclusion still depends on population quality, sampling risk, evidence reliability, time coverage, control objective, and whether the procedure could detect the relevant deviation.
Full-population analytics can be stronger for rule-defined attributes when the population and logic are validated. Manual verification may still be needed to test source reliability, context, false positives, evidence quality, or conditions the analytic cannot observe.
Expansion may be appropriate when exceptions exceed the planned decision threshold, indicate a systemic cause, reveal a missing subgroup, undermine population reliability, or leave insufficient evidence. The expansion purpose and rule should be documented, and it should not continue only until a preferred outcome appears.
State the population, period, method, targeted or representative nature, material exclusions, evidence constraints, exceptions, and conclusion boundary. Avoid language that implies complete coverage when only selected items were tested.
OC Security Audit can help validate populations, design risk-aware selection approaches, review sample documentation, test cybersecurity controls, and connect supportable results to practical remediation and retesting.
Prepared by Ali Hassani, CISO, drawing on 25+ years of IT, cybersecurity, compliance, and infrastructure experience. This sampling guide is for initial guidance only and does not replace a professional cybersecurity audit, compliance assessment, penetration test, legal/compliance review, privacy review, statistical consultation, or engagement-specific audit methodology.