Checklist · free
Security testing buyer’s checklist
For security, IT and engineering leaders buying a penetration test: what to settle before you ask for quotes, what to ask any testing firm, and what a credible report and attestation letter should contain.
Before you ask for quotes
| # | Question | Why it matters |
|---|
| 1 | Do you know who has to accept the result (an auditor, underwriter, customer, regulator or the board) and what exactly they need to see? | The audience decides the scope, the depth of the report and whether you need an attestation letter. |
| 2 | Do you have a firm deadline, and know what happens if you miss it? | A date tied to a renewal, audit or contract shapes the timeline you can accept. |
| 3 | Do you know which framework you're measured against, such as SOC 2, ISO 27001, HIPAA, PCI DSS 4.0, CMMC or NIST CSF? | The framework tells you what the test and its evidence need to support. |
| 4 | Have you decided whether you need a test only, or also a readiness review of policies, controls and evidence? | An auditor or customer may ask for approved policies and evidence, not only a test report. |
| 5 | Has someone been named to own remediation, with time to act on the findings? | Findings with no owner become a filed report, not a fix. |
Scope and method
| # | Question | Why it matters |
|---|
| 6 | Will the firm test manually, supported by tools, or only run an automated scan? | Scan output and manual testing support very different claims in a report. |
| 7 | Will the firm put targets, exclusions and off-limits systems in writing before testing starts? | A vague scope makes the report and any letter hard to rely on. |
| 8 | Will there be a written rules-of-engagement document and agreed testing windows that fit your release schedule? | Explicit rules and windows are how testing avoids disrupting production. |
| 9 | Does the firm require written authorization from you and from any third-party hosting provider? | Testing hosted assets without the provider's authorization creates its own risk. |
| 10 | Who will actually do the work, and can you reach them during the engagement? | You are buying the people, not the brand on the proposal. |
| 11 | Is the tester independent of the team that runs or manages your systems? | An auditor or customer may ask who tested the environment and who manages it. |
| 12 | Is a retest of fixed findings included, and on what terms? | A retest is how you show that findings were closed, not just reported. |
| 13 | Are the scope, fee and end date all defined up front? | Open-ended time and materials makes the cost and the finish date hard to plan for. |
| 14 | How will the firm handle your sensitive information, and can the work be scoped with your counsel involved if needed? | You will share details about your weaknesses; agree how they are protected before you do. |
What a credible report contains
| # | Question | Why it matters |
|---|
| 15 | Does it state what was tested, when, and what was excluded? | A reader can only rely on a report whose scope is clear. |
| 16 | Is there a plain-language executive summary for leadership? | Leaders and boards need the conclusion without the technical detail. |
| 17 | Are findings prioritized by risk, rather than a raw dump of tool output? | Prioritized findings tell a small team what to fix first. |
| 18 | Does each finding include steps a developer can use to reproduce it? | If engineers can't reproduce a finding, they can't fix or verify it. |
| 19 | Does each finding include specific remediation guidance? | Guidance turns a list of problems into a plan your team can act on. |
| 20 | Is there a readout session that walks engineering and leadership through the results? | A walkthrough settles questions faster than a report sent by email. |
| 21 | Are retest results reported, showing which findings were closed? | This is the evidence most audiences want after remediation. |
What an attestation letter should contain
| # | Question | Why it matters |
|---|
| 22 | Does the letter state only what was actually tested or reviewed? | A letter that claims more than the work covered won't survive a customer's security review. |
| 23 | Does it state when the testing and any retest took place? | Customers and auditors look for a current test, not an old one. |
| 24 | Does it clearly name the firm that performed the work? | The reader needs to know who stands behind the statement. |
| 25 | Does it avoid any claim of certification, compliance or a "clean" or "passing" result? | Only auditors and certification bodies issue opinions and certificates; a test letter can't. |
How to read your result
- Any unchecked item in "Before you ask for quotes": settle it first. Quotes will be hard to compare until you know the audience, deadline and framework.
- Two or more "no" answers from a firm in "Scope and method": treat the quote with caution, whatever the price.
- Use the last two sections on a sample or final report and letter. Each unchecked item is a question to put back to the firm before you send the result to your auditor or customer.
If you’d like help scoping a test or making sense of a quote, contact Kalosys today. Your note goes straight to our cyber security practice.
