Spec sheets are easy to compare; services are harder. When a small business or a growing online shop decides to have somebody watch its systems around the clock, the brochures all say the same things: “24×7”, “rapid response”, “advanced analytics”. This guide gives you a no-nonsense method for testing those claims in the way a reviewer tests a gadget, by running checks that produce evidence. The subject is SOC Ops, meaning the practical running of a security operations centre, whether you build one or buy the service from a provider.
The method draws on the Ops Atlas, a reference that summarises how these centres are meant to operate. Its entry, the xOps School reference on SOC Ops, takes only minutes to read, and several of the checks below come from it.
Step 1: Decide what you are actually reviewing
Write down, in one sentence, what the service is supposed to do. A workable version: “Turn warning signals from our systems into timely, recorded decisions that a named person can answer for.” If the provider’s description contains nothing about decisions and records, you are looking at a feature list, not a service.
Also check the boundaries. The Atlas treats detection writing, full incident handling, threat intelligence and data pipelines as neighbouring areas rather than part of the core. A provider may offer them, but they should be itemised and priced separately so you know what the monitoring fee covers.
Step 2: Ask how each alert ends
This is the equivalent of testing a battery under load. Ask the provider to show you the closing categories on a sample day. The Atlas recommends four: a confirmed threat, a benign trigger, a false alarm and an undetermined case. If the answer is only “resolved” or “closed”, you cannot tell whether anyone investigated.
Look for a written reason beside each closure. The Atlas lists “false alarm” closures with no reason as a classic weakness, because they teach the team nothing and hide recurring noise.
Step 3: Demand percentiles, not a single average
Any single figure for response time should make you sceptical. A mean can be dragged by one outlier in either direction, and the Atlas quotes MITRE’s remark that one old email can wreck it. Ask instead for the median and the 90th percentile, split by severity and by shift. Then ask specifically about nights and weekends. The Atlas includes a lab example where the daytime-heavy overall median was a handful of minutes while the after-hours median stretched beyond eight hours. If the vendor only offers a daytime sample, note it in your review.
Step 4: Check the staffing arithmetic
Round-the-clock claims are testable. A single seat covered every hour of every week needs about five people once rest, leave and training are included, a figure the Atlas takes from MITRE. MITRE additionally sets a floor of two analysts present at any moment. Ask how many names sit on the rota behind that promise, and who covers a festival week. Vague answers are a red flag.
Step 5: Read the contract for perverse incentives
Here the Atlas is blunt. Contracts that grade a provider on how quickly tickets are closed encourage rushed closures, and the reopen rate climbs as the target tightens. Look for measures that reward quality of the decision: sampled reviews, reopen rates, and the share of escalations the customer accepts. If you must have a time target, make it about when a human first acts, not when the ticket vanishes. Automated touches should not count as the first look.
- Time-to-first-human-action, reported as median and 90th percentile.
- Escalation quality: does each escalation arrive with context?
- Reopen rate, tracked monthly.
- Number of items left undetermined, and what happens to them.
Step 6: Confirm who stays accountable
Buying a service is like buying a warranty: it helps, but it does not transfer ownership. The Atlas, relaying MITRE, says system owners and executives stay accountable even when an outside team does the watching. Check the contract for named escalation contacts on your side, written criteria for declaring an incident and a clear list of actions the provider may take without asking you. The Atlas would confine automated containment to rules with a track record, applied to staff laptops and phones at a throttled pace, and keep every critical system under a human’s call.
Step 7: Look at handover and playbooks
Ask to see a redacted shift-handover log. The Atlas describes a pass-down record countersigned by the people leaving and the lead arriving; whatever the newcomer has not accepted stays with its previous owner. Then ask about written procedures: the Atlas’s rule of thumb is that anything occurring about once a month or more often should have a signed-off procedure ready. If the provider cannot produce one for a common case, such as a suspicious login, the service is more improvised than it sounds.
Step 8: Score maturity with an open yardstick
You do not need to invent your own scorecard. The Atlas points to SOC-CMM, an openly available self-assessment that covers five areas: business, people, process, technology and services. Ask a provider whether they have used it and when they last reassessed. A yearly review is the cadence the Atlas mentions. The score itself matters less than whether the team can talk about its weakest area without flinching.
Step 9: Write the review
Finish the way you would any product test: findings, caveats, verdict. Record which checks the provider passed, which they dodged and which you could not verify. A verdict like “documented, honest about nights, vague on rota size” tells a reader more than any star count could.
If you are used to judging devices, our earlier notes on tracking down dependable, impartial product reviews cover the mindset. The same scepticism helps with a service pitch. For a related exercise in comparing offers on specification and fine print, see the walkthrough of locating and reserving a hire vehicle without surprises.
Quick checklist
- One-sentence definition of the service, with decisions and records in it.
- Four closing categories, with reasons.
- Median and 90th percentile, including after-hours.
- Rota size that matches the 24×7 claim.
- Incentives that reward good decisions, not fast closure.
- Named accountability on your side.
- Handover log and procedures on file.
- A maturity self-assessment, repeated yearly.
None of this needs technical wizardry. It needs the same stubbornness you bring to a product test: ask for the evidence, and write down what you were shown.