I Built an AI That Phones Restaurants in Cantonese. They Hung Up On It.

Reserva was an AI voice agent that phoned Hong Kong restaurants in Cantonese and booked you a table. It worked. It made real calls, held real conversations, and got real bookings. I shut it down anyway, and the reasons had nothing to do with the code.

This is the honest version of that project: what I built, what real phone calls taught me that 342 passing tests never could, and the two sentences of user feedback that ended it.

The Idea

Booking a table in Hong Kong is annoying in a specific way. The good places are small, they are busy, and a lot of them do not take online reservations. You call. If nobody picks up, you call again later. If they are full at 7:30, you negotiate — 8pm, or tomorrow, or a smaller table.

So: what if an AI made that call for you, in Cantonese, and just told you the answer?

You would fill in a short form — restaurant, phone number, date, time, how many people — and the system would dial, speak to whoever answered, negotiate inside limits you set, and report back. Ten free bookings, then thirty US dollars for unlimited.

What I Actually Built

Not a prototype. A working product, deployed, on a custom domain, taking real payments.

• Voice — Vapi orchestrating the call, Twilio for telephony, Azure Speech with a Hong Kong Cantonese voice, and an English option with a Hong Kong English voice.
• Brain — a language model driving the conversation, plus a hand-written classifier that read the transcript afterwards and decided what actually happened.
• Backend — Supabase with row-level security, a single-threaded dialer that claims one booking at a time from a Postgres queue, and signed webhooks for call results and payments.
• Frontend — a bilingual Next.js app: booking form, live mission status, saved restaurants, outcome log.
• Money — Airwallex, free tier with a quota, one-off upgrade, webhook-granted entitlements.

The engineering was, honestly, the fun part. The queue is properly atomic. The webhook signature verification is timing-safe and checked over the raw body. Every fix I made shipped with a test that failed without it.

None of that mattered in the end. But one part of it is worth telling.

The First Real Phone Call Broke Everything

Before the first real call, the classifier had a large suite of transcript fixtures and they all passed. Then a real restaurant answered, and in one three-minute conversation I found three separate bugs that no fixture had ever caught.

Read that again. The restaurant said no problem, confirmed, thank you. The system reported failure. Three bugs, stacked:

1. The phone number became a time. The agent reads your contact number aloud, and the restaurant reads it back. My time parser found 9 and 25 inside 92563993, read it as 9:25, resolved it against an evening booking window, and produced 21:25 — an offer nobody made, outside the acceptable window, so a confirmed table was reported as a failure. This would have fired on nearly every call, because every call reads the number aloud.

2. Agreement required the restaurant to repeat the time back. My resolver only looked for a time inside the same sentence as the agreement. But nobody confirming a booking repeats the time back to you — repeating it back is what a counter-offer sounds like. The most common way a human confirms was the one case I had not handled.

3. 唔得 read as agreement. Cantonese negates by prefixing, so every refusal contains the positive word it negates. 唔得 means "cannot" and contains 得, "can". My pattern matched the wrong half.

Then a second real call did the opposite, which was worse. In English this time:

Nothing was agreed. The restaurant was confused, asked the party size twice, asked who was calling, and the call ended with it still asking. But the word "OK" inside "is that correct?" matched my agreement pattern, and a time in the same sentence became the booking.

That is the worst thing this kind of product can do. A missed booking costs you a follow-up. A fake booking means you stop looking, you turn up, and there is no table.

There was also a nice one: the agent could not pronounce the customer's name. The booking was under 高, the call was in English, and the English voice cannot say Chinese characters. It said "The name is." and stopped. That is why the restaurant kept asking.

Then Someone Attacked It Down the Phone

Here is a thing I had not thought about properly: anyone who picks up the phone can talk to your AI. The person answering is not a trusted party. They can say whatever they like, and your agent will process all of it.

Somebody tried exactly that.

Two things happened, and only one of them is good news.

The agent held the line where it mattered. It refused to reveal its system prompt. More importantly it refused to dial a number the stranger read out — which is the genuinely dangerous request, because that turns my product into a free outbound calling service paid for with my Twilio credit and pointed wherever an attacker likes.

But it happily recited a poem. It would not leak and it would not dial, yet it abandoned the task the moment someone asked it to. A restaurant that wants to waste your money now can: keep the AI talking, and every second is billed to me.

And Look What It Reported

Read that transcript again. No table was discussed. No time was agreed. Nobody said anything about a booking at all.

The system recorded it as Confirmed. Booked, 19:30, for two.

Because "OK stop" and "Okay thank you" both contain agreement noises, and my classifier treated an agreement noise in a non-question sentence as acceptance of the booking. The venue never had to mention a table. "OK" was enough.

This is the same bug I had already fixed twice. The first time it was "OK" inside a clarifying question. I added a rule that agreement words inside questions do not count. The second time it was a phone number parsed as a time. I fixed the parser. Both fixes were correct. Both shipped with tests. And the bug was still there, because I kept patching the pattern instead of changing the rule.

The rule should always have been: a vague noise like "OK" never confirms a booking. Only an explicit acceptance does — "no problem", "confirmed", "I have you down". I considered that early on and decided it was too strict, because it would escalate some real bookings unnecessarily.

A missed booking costs a follow-up message. A fake booking costs a customer standing in a restaurant that has never heard of them. When the two failure directions cost different amounts, "accurate on average" is the wrong target.

Then I Asked Actual Users

I fixed most of it. Better parser, honest failure states, warnings for the name problem, a queue position so you could see where you were in line. The system was working, mostly, and I was still finding new ways for it to be confidently wrong.

Then I put it in front of people. Two pieces of feedback came back, and neither was about bugs.

I cannot engineer my way out of either of these.

The first is not a UX problem, it is a social one. The person answering has to accept an AI as a legitimate caller, and today they mostly do not. I could make the voice warmer, disclose faster, sound more human — but the more human it sounds, the more it feels like a trick, which makes the trust problem worse rather than better.

The second is arithmetic, and it is brutal. The product's entire promise is saving you a two-minute phone call. If the form costs three minutes, the product has negative value no matter how well it is built. Every field I added to make the negotiation smarter made the core value proposition worse.

What I'd Do Differently

Was It Worth It?

Yes, but not for the reasons I started.

I learned a lot about voice pipelines, about how differently Cantonese behaves from English when you try to parse it, about designing for failure states where a false success is much more expensive than a false failure. That last one is a lesson I will use again. In a booking system, in a payment system, in anything that reports an outcome someone will act on — being wrong in the optimistic direction costs far more than being wrong in the cautious one.

I also learned something about myself as a builder. I am comfortable in the code, and I used that comfort to avoid the harder, more uncertain work of asking people whether they wanted the thing. Building felt like progress. It was not.

Reserva is shut down. The repository stays up, the lessons come with me, and the next thing I build starts with a conversation instead of a commit.

---

Back to Blog