..

Lessons from passing the OSAI

Hello reader, I recently landed a passing grade on the OSAI!

Mario jumping into the OSAI+ badge

The exam was challenging, and surprising - though not in the way I thought it might be. I expected to spend most of my time stumped by clever variations on course materials, but mostly I found myself struggling with AI automation and scale.

I’m sharing some exam preparation tips here that I think anyone going into the exam would find helpful to consider. This article is a brain-dump in no particular order, not a structured guide.

(1) This exam tests your ability to orchestrate AI as much as it tests attacking AI

From the exam guide:

Above all, this exam is designed with the active use of AI in mind, and we strongly encourage you to leverage it throughout every stage of your attack…

Take this section very seriously and think about how you want to structure this ahead of your exam. 24 hours with a formidably sized lab is a very compressed timeline. Parallelism and efficient collaboration between agents and operator help this go a lot more smoothly.

As I built knowledge, workflows, and skills I wanted to use in my upcoming exam I dogfooded them in HTB Pro Labs. There were countless rough edges that would have cost me time had I not tried my orchestration on other lab-style targets.

Note: I included report writing as part of my dogfooding, which brings me to my next thought.

(2) I found creating a professional quality report one of the hardest parts of this exam

I didn’t have trouble reaching a passing number of points when I attempted this exam, but what was hard was creating a report that I considered professional grade. OffSec’s official suggested documentation gives you an idea of how many machines with interconnected steps you’ll approach. Trying to capture everything you and your agent did, while still creating a concise usable walkthrough during the exam is a balancing act. In full transparency I don’t think I nailed it, and given another try I’d focus more on honing my report strategy earlier.

On one side you need to be careful to include enough detail to show everything done, but also your agent’s approaches will likely be winding, overcomplicated, and chain-command-dense. Trying to capture every bit of every conversation can leave you with an unwieldy pile of slop that is daunting to carve into something appropriate for a human audience.

What helped when it came to report time:

Record everything and structure it meticulously

While I was taking the exam I made sure every agent was saving chat transcripts that could be referred to later. All commands they ran were instructed to take place in named tmux sessions with filesystem backed history. Agents used a virtual framebuffer to take terminal screenshots at key discoveries / achievements. Screenshots had detailed descriptions saved along side them. All transcripts from chats and tmux, as well as screenshots were stored on disk in a heirarchy that made it easy to reference later:

<machine>
  -> screenshots/ # (saved as short-description slugs, and long description txts)
  -> tmux_transcripts/
  -> chat_transcripts/
  -> notes.md
index.md  # top level documentation of machine approaches and interdependency
loot.md   # looted credentials and xrefs
readme.md # structure conventions, general instructions
          # (my preferences, tool choices, broad strokes approach to labs)

You’re going to have to cover an immense amount of material in the 24 hours after your lab closes, missing evidence is a major risk without a system in place.

Create tools to help with ergonomics / building reports

I had Claude take the official exam documentation template and use it as a guide for an automated PDF generation pipeline. This pipeline was markdown based, and made it easy for me to collaborate on the report along side agents. Trying to work in Word is a huge pain. Taking the time to format code blocks, narrative, diagrams, and evidence screenshots is all time you could be using to work on documentation and landing more points.

I built hotkeys that wrapped screenshot utilities so my own manual evidence documentation landed with the right metadata in the right folders. Think about what things take the most time while you’re working on documentation, dogfood building a report. Use your findings to inform your own set of tools that speed up how you approach it.

Integrating things like the following are a major boon on any engagment, not just this one:

  • Context7 for software reference
  • Exa search for research
  • Playwright / CDP for headless browsing

Lean on Appendices to keep the main attack chain documentation clean, while still fully documenting the approach

Chat/command transcripts will sprawl and contain lots of back and forth. I personally opted to include these near-fully in my report to avoid leaving gaps in transparency. Though, I did find trying to inline too much information into the attack chain was an easy way to accidentally create a sloppy report. I eventually landed on creating a short snippet that explained how AI contributed to each machine walkthrough as footnotes - linking directly to the detailed appendix with the full context.

(3) Know your agent’s footguns (refusals, papercuts)

By the time you take this exam you should know what models, workflows, and harnesses you’re bringing. You should have tried them out extensively and know how you deal with common blockers in other real-world lab environments. Think about how you are planning on working around things like refusals (falling back to older models, using other providers, etc…)? If you want to try working with abliterated models make sure you have a gut feel for how they operate in comparison to non-abliterated models.

Most importantly - have backup plans: providers can 5xx, your self-hosted models can crash, provider cyber-verification requirements can change mid-exam. Don’t put all your chips in one basket.

Also think about papercuts, what do your agents do that translates to lost time on exams? Do they frequently send signals to basic reverse shells and blow them up? Do they rabbit hole on a particular vulnerability time with low yield? Providing up-front agent guidance is all time better spent in the exam.

(4) Think about your approach to agent orchestration

I’ll reiterate you have 24 hours to land your points and collect your evidence. Whenever you have a single agent pinwheeling on one area alone you’re creating a single point of failure for the exam. Parallelize your enumeration, idea generation, and execution to give yourself breathing room and avoid losing all your exam time to a single incorrect assumption.

I recommend experimenting with ways to coordinate these agents. I tried to use common markdown ledgers with agent instructions that work done on enumeration or testing should be documented and prioritized with evidence. Work needs to be assigned to an agent by id (self-generated / reported) before it’s started. With a formidably sized exam you will almost certainly hit situations where an agent can’t keep all lab context in its window, along with all its theories. Each compaction reduces its fidelity and creates opportunity to retread old ground. By documenting work in ledgers and explicitly avoiding retreading ideas without additional evidence that an approach can work, I avoided many dreaded rabbit holes / agent loops.

This is one very basic approach of many, honed in real-world labs ahead of the exam. Figure out what works for you.

(5) Consider how your judgement and expertise fit in

One reason I used tmux heavily in my automation was because it allowed me to observe the command sessions my agents drove in detail. I caught many rabbit holes, poor uses of time, and dangerous behaviors simply by participating interactively in terminal sessions with my agents. I also iterated on my documentation scaffolding ad nauseum to make sure the skeleton I eventually refined into my report reflected my vision. I worked with my agents interactively to help prioritize enumeration outputs.

Think about how and when your agents should report information back to you for guidance. The amount of autonomy vs human judgement you approach the exam with is your personal choice.

How do you see yourself guiding the ship as the exam takes place?

Closing

Good luck, you got this! Remember to also consider this a test of how you use AI, not just a test of your ability to hack AI. 🚀