Mr. Elite & agents
AI employee, Admin AI employee and chief of staff, on a multi-agent orchestrator I designed.
Before GeniOS became what it is today, we built AI employees: an Admin AI Employee to automate email, calendar, documentation and writing, on a multi-agent orchestrator I designed. I did the system design, the high-level design and the architecture myself, before the recent LLM advances made any of this simple.
What we were building
An employee, not a chatbot.
A chatbot answers. An employee takes the work off your plate. We wanted the second one, and the hard part was never the model. Agents could act, but nobody had taught them to think in order. So the first thing I designed was not an agent. It was the thing that sits above them.
- 1AI employee
Email and calendar handled by agents. Piloted with 5 startups.
- 2Admin AI employee
Email, calendar, writing and documentation in one place. I ran it myself first, then piloted it with 1 company.
- 3Chief of staff
An AI chief of staff on LangGraph, following the work across agents.
The high-level design
Mr. Elite, layer by layer.
I started with the HLD on paper: what a request touches on its way in, what has to be true before anything runs, and where memory lives. Mr. Elite is built with Python and FastAPI, routes through LangGraph, and thinks with Gemini. It has six layers, and each one has one job.
- L1Interface
Python and FastAPI. Takes a request in plain language.
- L2Pipeline
Seven stages that turn the request into an ordered plan.
- L3Reasoning
Runs in parallel: policy enforcement, memory retrieval, resource validation.
- L4Control
Prompt injection protection and rate limiting, before anything executes.
- L5Agents
Email, Calendar, Writing, Documentation and Mr. Happy, routed through LangGraph.
- L6Memory
One shared brain on Supabase, so no agent starts from zero.
The system design
Seven stages, in order, every time.
Every request goes through the same pipeline. The order is the point: an agent should never act on something that has not been understood, broken down, given context and checked.
- 01Intent classification
Work out what the person actually asked for.
- 02Task atomization
Break it into the smallest pieces that can each be done and checked.
- 03Context retrieval
Pull what each piece needs to know from shared memory.
- 04Policy validation
Check every step against the rules before it runs.
- 05Agent execution
Route each piece to the right agent through LangGraph.
- 06Result aggregation
Bring the results back together into one answer.
- 07Learning updates
Write what happened back to memory for next time.
The architecture decisions
Why it is built this way.
| Decision | Why |
|---|---|
| One central orchestrator | Agents that call each other directly lose track of why they are running. |
| Policy before execution | A wrong email cannot be unsent, so checks come first, not after. |
| Reasoning in parallel | Policy, memory and resources can be checked together without slowing the plan. |
| A separate control layer | Prompt injection and rate limits should not depend on any one agent behaving. |
| One shared memory | Context stays with the company, not inside one conversation. |
Five agents, one brain
Coordinated through one orchestrator, on one shared memory.
| Agent | What it does |
|---|---|
| Summarises threads, drafts replies, extracts actions and prioritises. | |
| Calendar | Schedules, detects conflicts, slots by priority and follows up. |
| Writing | Drafts the work around the inbox. |
| Documentation | Keeps the documents up to date. |
| Mr. Happy | A personal assistant on OpenClaw for reminders, suggestions and action tracking. |
We also built Data 2 JSON along the way, which turns multi-source inputs into structured datasets for RAG pipelines, using Gemini, Supabase and prompt-based validation.
Who looked at it
Senior AI engineers at IBM, Adobe and Google reviewed the work and validated it, and several stayed on as advisors. Then it went to real users: 5 startups on the AI employee, and 1 company on the Admin AI employee.
What it taught me
The missing piece was never the agent.
The agents worked. What they lacked was an understanding of the company they served. They did what you said, not what the company needed. That lesson led to the context brain, and then to GeniOS.
Read the GeniOS storyWhat shipped
- 01. Piloted the AI employee with 5 startups, and the admin AI employee with 1 company.
- 02. Validated by senior AI engineers at IBM, Adobe and Google.
- 03. Led directly to GeniOS.
