I tried to replace myself with AI and automate my code reviews

I’ve always struggled to find a use case where a language model could completely replace me, even for singular tasks. It’s an aid, a pair programmer, a proofreader, a code generator. But it’s never reliable or smart enough for me to trust it to act or write in my name. And honestly, I wouldn’t want that anyway. Even mundane tasks require context that’s hard to convey to a model.

But that’s no reason not to experiment!

The Role of Code Reviews

In my engineering work, I spend a significant amount of time reviewing others' code. In fact, in most companies, developers spend 2-5 hours per week on code reviews. Some find it boring or see it as an extra burden. However, there are three key goals:

Ensure quality and catch mistakes
Stay up to date with changes
Help others improve their programming skills

All of these are critical, but there’s also an indirect consequence of code reviews in modern development practices. Usually, a code review is a step taken before deployment. Performing a swift code review becomes the single most important thing you can do to help the team ship the next feature. I prioritize reviews ahead of my own work.

At most of the places I’ve worked, there’s been discussion about codifying standards for great code reviews to help everyone improve. My current workplace was no exception.

Experimenting with AI

I drafted a set of guidelines for effective code reviews. As I looked at them through the lens of my AI engineering experience, I realized they could make a perfect prompt for a language model. This inspired me to try making AI assist with reviews. My goal was to spot more issues and perhaps even provide automated reviews for the team, even if they weren’t perfect.

The Plan

I devised a plan in minutes. There’s a tool I often use for low-code automations, n8n. I discovered that I could easily connect to GitHub via its API to read code and post review comments.

Prototype Phase

I built a prototype using GPT-4o and tested it with my personal projects. The outputs were, frankly, terrible. The comments were overly polite, and their usefulness got buried under layers of fluff. I shortened the prompt, made it more specific, and explicitly asked the model to be critical in all comments except the summary. I also provided examples of effective feedback.

Version 2 was far more useful but still included some unnecessary and impractical remarks. I encountered several limitations:

Tool constraints: n8n, the tool I used to prototype the workflow, didn’t support the latest OpenAI API functions. This resulted in subpar outputs compared to what was possible.
Contextual limitations: The diff provided limited context about the project and its tasks. Providing a great review requires understanding the business context.

That said, the AI managed to spot mistakes in the code that I had missed. Some of its outputs were genuinely helpful. The process involved reviewing all generated comments, removing 75% of them, adding my own insights, and then submitting the review. Despite the overhead, this workflow was actually useful enough.

Next Steps

I haven’t gone further yet. To improve the system, I’d need to implement new API functionalities (like JSON outputs), pull extra code and project context from the codebase, and possibly integrate task management software to fetch relevant business information. The approach shows promise, and I’m aware of several companies exploring similar solutions. I want to refine it further and potentially turn it into a small product.

I’m sharing this post as a checkpoint and a possible discussion starter. If you think this is interesting and would like to test it, let me know. I’ve been considering building a version tailored for companies to automate code reviews, as well as one for individuals to help them become better reviewers and save time.

What do you think is the best route?

If you liked this content, please support me by sharing this post and subscribing to my Newsletter.

You can also find me here: