A CISO evaluates an AI vendor by defining its permitted work, testing its permissions and retrieving evidence independently. Use ten working days to decide on one workflow, with legal and operational owners involved. Stop or narrow the evaluation when essential proof is missing. A polished demonstration should not decide whether customer data enters the system.
How a CISO evaluates an AI workspace in ten days.
An evaluation plan for AI workspace and agent governance. Define permissions, test the workflow and give the risk committee a decision about that specific use.
What must be agreed before day one?
Choose one funded job and test data with no real customer information. Name the business owner, security reviewer and person who accepts the remaining risk. Give the supplier the allowed actions and explicit rejection conditions in advance.
Compare the same task across suppliers. A hosted writing service, an agent framework and a customer-operated workspace have different responsibilities. State which operating model the organisation needs, then test the proposal against that requirement.
| Window | Work with the supplier | Required review record |
|---|---|---|
| Days 1-2 | Map purpose, types of data, identities, source access, external connections and support staff's administrative access. | A scoped workflow and responsibility map, with unresolved questions. |
| Days 3-4 | Inspect the proposed settings. Run the ordinary task using test data. | Software version and settings, expected result and observed state of the receiving system. |
| Days 5-6 | Test unauthorised access, rejected approval, changed request contents and removed permissions. | Failure evidence and checks of the receiving system, linked to the original request. |
| Days 7-8 | Retrieve evidence independently. Rehearse an interruption and an agreed export. | Reconstruction notes, recovery limitations and a record of who holds the exported data. |
| Day 9 | Review the contract, retention and suppliers that process data on the provider's behalf. Review incident duties and open findings with the responsible teams. | A decision register with risk owner, due date and the requirement that must be met. |
| Day 10 | Choose a bounded pilot, further investigation or rejection. | A signed internal decision covering only the tested configuration and use. |
Which vendor answers need a live demonstration?
Ask the supplier to show an operation performed by the intended identity. Repeat it after removing permission. Inspect both the request decision and the receiving system. A denial banner alone leaves an important question unanswered: did anything change in that system?
Ask a reviewer who did not run the demonstration to find the corresponding evidence. Check whether the records distinguish the requesting human, acting worker, approval and execution result. Record missing events and ambiguous attribution.
Challenge update and recovery assumptions. An export that downloads successfully is not yet a restore test. Agree what continuity evidence can be produced inside the review window and what remains a condition of a later pilot.
How do regulatory and management-system reviews fit?
Use NIST AI RMF 1.0 to organise risk ownership, context and measurement. It is voluntary guidance, not a product certification. Keep the practical tests tied to the organisation's chosen controls and intended use.
For financial services, involve the firm's regulatory owner early. The FCA AI Lab is a source of current UK regulatory engagement. It is not an endorsement of a vendor. For EU exposure, use the linked review to check which AI Act rules apply. Do not classify every internal AI task as high-risk.
ISO/IEC 42001 concerns an organisation's AI management system. Ask which entity, scope and activities any certificate covers. Do not infer that a supplier certificate proves your own operating process or the performance of a proposed integration.
What should the risk committee receive?
Deliver a short decision brief with the exact workflow, tested settings and supporting records. Separate observed behaviour from supplier statements. Mark each requirement as met, failed, not tested or outside scope. Name the person who can close it.
Include commercial and operating responsibilities. Specify support access, who approves updates, who holds the evidence, retention and how the customer can leave. A technical pass does not settle those terms. The committee should be able to reject the proposed scope without losing the work already done on the evaluation.
How can we apply this to a Vantage evaluation?
Use the same scorecard and rejection conditions for a Vantage proposal. Request current evidence for the intended deployment and integration. Do not infer availability from a category description or a different demonstration.
Bring one workflow to the evaluation discussion. Agree which tests can run, who observes them and where the records will be held. If a required capability is not demonstrated, keep it as an open purchase condition.
Which sources inform this review?
TAKE IT INTO THE VENDOR DEMO
Use the questions. Make the evidence reproducible.
The guide establishes what to ask. These three resources turn the same evaluation into a scorecard, an interactive committee record, and a guide to checking the product's evidence.
Frequently asked questions.
Can ten days establish full production readiness?
No. This is a bounded review window for one workflow. Broader deployment, assurance, performance and recovery testing may need additional work.
What if the supplier cannot provide a required record?
Record the missing evidence, its consequence and the owner of the next check. Do not score an undocumented claim as a demonstrated control.
Should the pilot use customer data?
Begin with test data containing no real customer information. Any later use of real information needs its own authorisation, rules on where data may go and safeguards.
Does a framework mapping prove compliance?
No. A mapping organises questions and evidence. Certification scope, legal obligations and operating effectiveness require separate assessment.
Further reading
RUN THE EVALUATION
Bring the questions. Make the vendor produce the evidence.
Use the Procurement Workbench to compare vendors on the same tests, record the decision rights, and export the committee brief. The evaluation stays in your browser until you choose to export it.
Open the Procurement Workbench →Or write to hello@handvantage.com directly.