If you review gadgets or write about consumer technology for a living, you already own the most transferable skill in the AI-security job market: the ability to test a claim instead of repeating it. AIOps for Security is the working shorthand for security operations assisted by machine learning and LLMs — a label that still lacks any formal standard — and it has become a hiring category faster than it has become a discipline. Teams have discovered that their scarcest people are not model builders but evaluators: staff who can label data honestly, compute what a result actually proves, and say no to a demo. This piece is a practical map of that skill set: what it consists of, which published numbers every evaluator should know cold, and how to assemble proof of it that a hiring manager can check. No certifications required; the evidence is the credential.
Why Reviewers Translate So Well Into AIOps for Security Work
Product reviewers live by a discipline that security teams are only now institutionalising: controlled comparison, disclosed methodology, and suspicion of any number that arrives without its measurement conditions. The security world calls this evaluation; the trade calls it a rubric. The parallel runs deep because both fields share the same failure mode — a plausible-looking score produced by an unexamined setup. Academic work has documented this brutally. The TESSERACT programme (2019–2022) showed that headline F1 scores around 0.99 could be manufactured by spatial and temporal leakage hiding in the test data. And when Arp and colleagues audited thirty leading security-ML papers, sampling bias turned up in nine of every ten, with data snooping in close to three-quarters. In other words, the most-cited results in the field arrived pre-inflated. Anyone who has ever caught a review website scoring products it never tested will recognise the pattern immediately. The habits that transfer are concrete: write the test before you see the product, keep the labelled truth in a file the vendor cannot edit, and report misses with the same prominence as hits. The same scepticism, applied to an alert-scoring vendor, is a billable skill.
The Numbers an Evaluator Knows Before the First Meeting
Think of benchmarks as this field’s test bench; their results set expectations that marketing cannot. ITBench, published at ICML 2025, watched autonomous agents crack 25.2% of its compliance-and-security set. CyberSOCEval, released in 2025 by Meta and CrowdStrike, landed agents between roughly 23% and 34% on malware analysis and between about 43% and 53% on threat-intel reasoning — against guesswork baselines of 0.63% and 1.7%, so the agents clearly beat coin-flipping and clearly cannot run a programme unsupervised. Gartner, for its part, parked AI SOC agents on its 2026 hype-cycle peak of inflated expectations and flagged “AI washing”: accuracy claims dressed well past their evidence. An evaluator walks into vendor conversations carrying these figures the way a reviewer carries spec sheets — not as verdicts, but as the floor any specific product must argue past. When a sales deck quotes numbers far above the published ranges, the honest question is not “how” but “measured where, on whose data, and with what held out”.
The Skill in Practice: One Month of Blind Labels
Evaluation stops being abstract the moment you label data. Consider the shape of a real triage assessment, as documented in the field’s teaching material: 1,240 emails were blind-labelled — the truth recorded before any model saw a verdict — and 71 of them were genuine threats. The scoring agent missed 2 of the 71, an observed miss rate of 2.8%. The statistical ceiling on that small sample sat near 8.6% at 95% confidence, so the trial failed a gate requiring 2%. Alongside the misses sat genuinely impressive figures: 99.8% precision on its closings, an 86% precision when it asserted something was a true positive, and a finding that 93.5% of the triage workload was removable without any autonomy at all. The lesson an evaluator takes from this: impressive and safe are different axes, and only blind labels reveal which one you measured. A candidate who can narrate this distinction in an interview — sample size, bound, gate, verdict — sounds like staff, not staff-audience. The labelling itself is unglamorous and cheap: an afternoon with a spreadsheet, a fixed rubric agreed before you start, and a second labeller for the disputed rows. That is the entire secret of why such small exercises carry weight — the truth was fixed before the machine spoke, and nobody could adjust it afterwards.
The AIOps for Security Fine Print Nobody Quotes
Claim-testing extends beyond accuracy into the quiet defaults. Security Copilot turns its customer-data-sharing settings on by default; only M365 E5 or E7 licensing changes that posture. Gemini in Security Ops processes on global endpoints, though it keeps customer prompts out of training. Microsoft’s phishing agent closes mail by itself from its very first run — nobody has documented an observe-only switch for it (as of October 2026). Google’s triage-and-investigation agent sits one rung lower, in recommend-only territory, at roughly a minute a case. Sentinel’s Fusion engine is on its way out as Defender XDR correlation takes over; workspaces must finish that move before 31 March 2027. None of these facts appears in an accuracy table, and all of them change the risk picture. Reviewers know this feeling well: the spec sheet is never where the problems live; the defaults are. And any agent that reads attacker-controlled content — forwarded mail, uploaded documents, pasted logs — carries prompt-injection risk that no filter reliably removes, which is why the surrounding permissions deserve as much review as the model’s scores.
Building the Portfolio: What to Show Instead of a Certificate
The fastest route into this work is a small, honest, self-generated dataset and the write-up of what it proved. A ninety-day arc borrowed from the field’s maturity guidance fits an evening-a-week schedule:
- Days 1–30: export your own dispositions (or a public dataset), pick two alert categories, run the base-rate arithmetic, and shadow-score with blind labels.
- Days 31–60: evaluate per category monthly, and set explicit gate values for what you would accept before any autonomous action.
- Days 61–90: plant canary cases, add drift monitors, and write down the cost baseline — what an analyst hour actually spends.
The write-up becomes the portfolio piece. Pair it with a one-page claim audit of a real vendor — claims in one column, evidence in the other — laid out with the same rigour reviewers bring to finding genuinely reliable, unbiased product reviews. Hiring managers for security programmes report that this kind of artefact outperforms any certificate, because it demonstrates the exact act the job requires: measuring something and publishing the conditions of the measurement.
Where the Skill Rates an Offer
Teams adopting this technology increasingly use a simple scored rubric before any deployment: a decision score out of a possible 27, where results above 14 buy shadow mode, 9 to 14 buy a time-boxed pilot, and anything below 9 gets declined. Knowing that rubric exists — and being the person who can fill it in honestly — is the career move. Titles in the field are still settling, but the shape of the work is stable: evaluation engineer, detection quality analyst, or the reviewer-adjacent role of verifying AI claims before customers do. Titles matter less than the artefacts, and the artefacts are exactly what the ninety-day plan produces. The work rewards the temperament reviewers already have: patience with labelling, pleasure in a clean table, and no interest in being fluent about things nobody measured. Start from the material aimed at practitioners, the AIOps for Security entry published by the Ops Atlas, then practise on whatever queue you can ethically label. The judgment to book a test before a purchase applies everywhere; the same readers who trust a straightforward walkthrough for booking a rental vehicle will recognise the shape of this discipline at once. Evaluation is the rare skill that gets more valuable as the products get smoother — because smooth is exactly what hides the untested part.
