iamnor

By Ron Dilley

Let me tell you a story.

It begins like so many of my tales: with unreasonable requirements, a hard deadline, and a deep suspicion that the thing we’re building will collapse under the weight of its own ambition.

But this wasn’t a client call or a late-night SOC war room — it was a solo experiment in AI-paired systems programming. The goal? Build a blisteringly fast, DevSecOps-ready log parser using C, inline assembly, and a little AI friend called Claude.

And build it from scratch. In five days. Using only AI-generated code.

DAY 0: DECLARING WAR ON LOGS

The mission was clear: design and ship a log processing tool — named Log Carver — capable of eating enterprise logs at 50 million+ lines per minute. Why? Because in real-world IR, your parser’s latency determines whether you’re diagnosing yesterday’s breach or actively bleeding out.

Requirements were non-negotiable:

• Handle JSON, syslog, XML, Apache, CSV
• Throughput: 50M+ LPM
• Memory ceiling: 400MB
• Embeddable: usable from both C and Python
• Written exclusively through Claude Code

That last one? That was the twist.

We weren’t just building a DevSecOps tool. We were testing the limits of AI pair programming on bare metal.

DAY 1: SPINNING UP THE BEAST

Claude’s first prompt was surgical:

“Create a high-performance log parser with universal format detection.”

Within hours, we had the skeleton:

• Format auto-detection logic
• JSON and syslog parsers
• A zero-copy memory model using direct pointers instead of strdup soup

Initial benchmarks hit ~5M LPM. Respectable. But not fast enough for production or panic mode.

DAY 2: THE FORMAT WARS

Next, we taught Claude to parse:

• XML (with depth tracking)
• Apache logs (Common + Combined)
• CSV (with auto-detected delimiters)

Through carefully scoped prompts like

“Add XML parser without increasing memory usage over 64MB,”

Performance climbed to ~15M LPM.

That’s when we started dreaming dangerously.

DAY 3: ENTER THE VECTOR

“Add AVX2 SIMD newline detection.”

This was where the line was drawn between code generator and AI systems engineer.

Claude nailed the basic logic:

__m256i newline = _mm256_set1_epi8('\n');
__m256i cmp = _mm256_cmpeq_epi8(chunk, newline);
count += __builtin_popcount(_mm256_movemask_epi8(cmp));

It wasn’t perfect — register clobbers and stack alignment were landmines — but with persistence (and about 12 GDB segfaults), we got it humming.

Then we added lock-free queues, multi-threading, NUMA-aware memory pools.

Result:
35M lines per minute.
And a growing sense Claude might actually pull this off.

DAY 4: XML FROM HELL

Someone out there (probably on the Blue Team) decided that XML was best expressed as one single 195MB line. No line breaks. Just 10,359 records… squashed flat.

The prompt was simple:

“Handle 195MB XML on one line with <400MB RAM.”

Claude’s first attempt? Crash and burn. OOM.

By hour four:

• We had memory-mapped file support
• A streaming parser with state machines
• SIMD-accelerated tag detection

Benchmark: 45M LPM on malformed XML. That’s war zone ready.

DAY 5: HARDENED FOR DEVSECOPS

The final day wasn’t about features — it was about trust.

“Fix all warnings.”
“Add verification to every record and byte processed.”
“Optimize memory pools. No regressions.”

By nightfall, Log Carver was:

• 50M+ LPM verified on mixed format logs
• Zero warnings from -Wall -Wextra -pedantic
• 100% accuracy in byte/line counts over 1000+ files
• Library wrapped for Python via ctypes, ready for CI pipelines

CLAUDE’S DARK SIDE

Let me be clear: Claude’s code didn’t always work.

Sometimes it hallucinated safer solutions that deleted entire optimizations:

“Fix the AVX2 segfault”

Claude:

// Slow fallback version
for (int i = 0; i < len; i++) {
  if (buffer[i] == '\n') count++;
}

Thanks, buddy. You just dropped throughput by 90%.

Or the classic:

“Fix race condition in lock-free queue”

Claude’s fix?

pthread_mutex_lock(&queue_lock);

The queue is now dead. Long live the mutex.

THE DEVSECOPS TAKEAWAY

DevSecOps isn’t just about shift-left testing and secure SDLCs. It’s about velocity with verification. And in that lens, Log Carver is a war story of how far we can push the tooling envelope when we combine:

• C and inline assembly for raw speed
• Python bindings for accessibility
• AI code generation for productivity
• Paranoia-driven engineering for trust

We built a world-class parser in 5 days using only AI and test suites. That’s not just fast — that’s asymmetric capability for defenders.

RECOMMENDED DEFENSIVE PROMPTING TECHNIQUES

If you use Claude (or any AI) for systems work, memorize these:

• “ONLY modify function X, keep all existing optimizations”
• “Do NOT simplify logic; preserve AVX2 pipeline”
• “Maintain lock-free guarantees, NO mutexes”
• “Fix bug but keep performance > 50M LPM”

And always, always, benchmark before and after.

FINAL STATS

Metric	Result
Build Time	5 days
Claude Prompts	150+
Tokens Used	2M
Code Produced	15K LOC
Throughput	50M+ lines/min
Memory Usage	<400MB
Formats Supported	5
Bugs Introduced by Claude	Many
Bugs Caught by GDB/Valgrind	More
DevSecOps Utility	Extreme

CLAUDE’S PRODUCTIVITY: MYTH VS METRICS

There’s been a lot of breathless hype about AI giving you 10x, 50x, even 100x productivity gains. Let’s apply some DevSecOps honesty to that claim.

Sure — Log Carver was built in 5 days. And yes, the raw token-to-code throughput from Claude was astonishing. But when you break it down by actual engineering effort, here’s where the time really went:

Task	% of Time	Notes
Initial Code Generation	~20%	Getting a working first version via prompting
Fixing Bugs in Claude’s Code	~30%	Segfaults, incorrect memory handling, broken edge cases
Restoring Optimizations Claude Removed	~35%	Claude often “solved” bugs by deleting complexity
Debugging With GDB/Valgrind	~10%	Memory leaks, thread leaks, corrupted stacks
Re-explaining Architecture to Claude	~5%	Compensating for context loss across sessions

In other words, only 1 in 5 hours was spent writing code — the rest was spent babysitting the AI, fixing what it broke, or putting the optimizations back in after Claude quietly deleted them.

This turns the supposed 50x multiplier into something closer to 5x for C/ASM workflows — maybe 10x on a good day with clean Python.

Still a massive productivity win? Yes.
But a mindless coding savant? No.

You must still think, debug, validate, and verify. Claude might give you a Ferrari, but it doesn’t stop it from driving into a wall if you let go of the wheel.

CONCLUSION

Log Carver was an experiment. But it proved something profound:

AI is no longer just a productivity booster. It’s a compiler for ideas.

But like any compiler, it needs constraints, tests, and a damn good operator behind the keyboard. Especially when you’re operating at C-speed with logs that never sleep.

If you want to keep up in modern security, this is your new baseline: 50 million logs a minute. Or you’re parsing history, not defending the future.

— Ron Dilley
Frustrating adversaries since before XML was cool.