The AI Gateway: Building the Same Idea Twice

Over the last couple of weeks I have had the opportunity to build an AI gateway for a private-markets syndication platform, and in doing so I found myself going back to the first one I built earlier this year, for a platform where a group of family offices collaborate on investment opportunities. Comparing the two has been an interesting exercise, mainly because the second build finally gave me a clear way of explaining why the approach matters, something I had struggled to do the first time round.

To explain the idea properly it is necessary to understand:

• What is an AI gateway?
• Why build it as a separate service?
• Why should anyone (other than the architect) care?

What is an AI gateway?

When I started building the family-office platform, I made a decision very early on that the application itself would never talk to an AI model directly. Instead every AI call would go through a dedicated service, the "gateway", and the application would simply ask it to perform a named job ("summarise this document", "classify this file", "research this company") and receive an answer back. Which model to use, which prompt, and how the job is actually run all sit behind the gateway.

I must give credit where it is due, as I did not invent this idea. I borrowed it from Peter Steinberger's personal AI assistant at a time when it was still called Clawdbot, before it became OpenClaw and gathered the enormous following it has today. His design had a gateway at the centre which every channel spoke to, keeping the AI separate from everything around it.

Why build it as a separate service?

Those who know me will know that I have long been a fan of building systems in a modular way, often referred to as "the Unix way": small components that each do one thing well, each with a clearly defined interface, talking to one another through APIs. It is the same thinking behind Jeff Bezos's famous "API mandate" at Amazon, where every team had to expose its data and functionality through service interfaces and nothing else. The AI gateway is simply that principle applied to AI. The AI becomes its own component which "speaks to" the rest of the application through a defined interface, so you can change the model, the prompts or the provider without touching anything else.

I have to confess that while the technical reasons were obvious to me, I was never really able to explain them clearly to others. People could see it was a good idea, but every time I started talking about services and interfaces I could see I was losing them. Nevertheless, when I later joined the team building the syndication platform, building a gateway into that application too was one of the first things we agreed.

Why should anyone care?

This is where the second build answered the question for me, for two reasons.

The first is that we were able to take the modularity a step further. The family-office platform already used "skills" for some of its AI jobs, but on the syndication platform every AI job became one, and each skill became a more complete package: the instructions in plain English, a header stating who owns it, which kind of model it needs and which tools it may use, a definition of what a valid answer looks like, and a set of test cases. Skills started life as an Anthropic concept, but have since been handed over to the Linux Foundation as an open standard, which matters because a skill is no longer tied to one vendor or one model.

skills/company-report/
  SKILL.md        what it's for, then the instructions;
                  owner, risk tier, status, model, tools
  schemas/        what a valid answer looks like
  evals/          test cases, run before approval

The second reason is regulation. The syndication platform is used by firms regulated by the FCA, whose recent good practice on AI in financial services asks the questions you would hope a regulator would ask: what AI are you using and for what, who is accountable for it, what changed and who approved it, and how do you know it works? Where AI calls are scattered throughout an application, each of those questions becomes a project in its own right. With a gateway most of the answers already exist, because the list of skills is in effect the register of AI use cases (each with a named owner), and every call is logged.

The question of what changed and who approved it is answered by giving every skill an approval lifecycle. Each change to a skill creates a new version, which starts as a draft, is run against its test cases, and can only be approved by someone other than its author once they have seen the results. In production the gateway will only run an approved version, and when a skill is replaced the old version is retired rather than deleted, so there is always a record of exactly which version produced which output (and rolling back is simply a case of switching to the previous approved version).

I had always argued for the gateway as good architecture, which I still believe it is, but good architecture is rarely a reason anyone outside the technology team cares about. Being able to answer the regulator's questions is.

Bringing it back to the first platform

The family-office platform was built for a small private group, with different needs to a regulated platform, and its gateway served it well. But having raised the bar on the second build it made sense to bring the improvements back, and because every AI call already went through the gateway this took a matter of days rather than months. The learnings from doing so are worth sharing:

(1) Everything through the gateway is a skill. Rather than some AI jobs using skills and others carrying their prompts in the application code, every call to the gateway is now a call to a named, versioned skill. That one rule is what makes all of the following possible.

(2) Each job gets only the tools it needs. Every skill names the tools it requires and is given those and nothing more, which for most jobs (summarising, classifying) means none at all. The lesson here was to test what the AI can actually do rather than trust what the configuration says: when I asked a job configured with no tools what it could do, it listed several, inherited from the account it was signed in with. Only a live test would have caught this, and in fairness to myself I found exactly the same thing in my own personal AI system the same week!

(3) Check every answer. Each skill defines what a valid answer looks like, and the gateway checks every response against it, asking the model to try again (with the specific problem) if it does not match. This replaced a variety of different ways the application had been interpreting the AI's answers.

(4) Log every call. A record of every AI call (which skill and version, which model, how long it took, what it cost and whether it succeeded) makes errors visible straight away and shows where the money goes. Web research, for example, turned out to cost many times more per run than any other job, which changes how you think about when to run it.

(5) Build up the evaluations. A good set of test cases for each skill is what makes it safe to change one, since every new version can be checked against the same cases before it goes live. This is the one we are still building out on the first platform, and I am not convinced either platform has fully solved it yet. Test cases are a good start, but the real measure is what people do with the AI's output (do they accept it, edit it or throw it away?), and capturing that properly is the next piece of work on both.

Looking back, the decision to build a gateway early this year was the right one, though not quite for the reasons I gave at the time. Its real value has been that, because every AI call passes through one place, the platform can keep improving without having to unpick the whole application. Hopefully by sharing these experiences others can benefit before making the same mistakes.