A Small Flood That Felt Like a Tsunami

One of my clients, Tessa, organises a one day music and food festival by the river. It’s the highlight of her year, and honestly of mine, because I look after the little server that sells her tickets. They go on sale at noon on a set date. Every year a few hundred people are waiting, and they sell out in an hour.

This year she’d promoted it harder than ever, and she was nervous and excited in equal measure. She sat at her folding table, surrounded by posters and wristbands, with her laptop open on the ticket page. I sat at my own desk with the server logs scrolling on one screen and my phone beside the keyboard. At 11:59 she texted me a single row of exclamation marks.

Noon

At exactly noon the first sales began to tick in. One, three, eight, twelve. Tessa messaged me a grinning face. Then the numbers stopped. The page slowed to a crawl, and then it gave her a timeout.

She rang me, and I could hear half a dozen people already posting on the festival’s social page that the site was down. I told her to stay calm, and I opened the logs.

She assumed the site had collapsed under demand. It was the flattering and obvious explanation, since she had promoted it well. I’d have guessed the same, if I hadn’t been looking at the traffic.

What was really happening

Here’s what was actually happening on the server. A large burst of requests had arrived in the same second from thousands of different addresses, nearly all of them doing something pointless, like hammering the same page over and over. That’s junk traffic. A distributed flood, in the technical term. Somebody or something had spotted the sale time and decided to exploit it.

Real buyers were a thin trickle inside that wave. The server was spending nearly all its effort answering requests that would never turn into a ticket, and my small plan had nothing spare.

I opened a ticket with the host while Tessa was still on the line. Their automatic protection is designed to spot floods like this and filter them away, and it did work. But it takes a few minutes to recognise a pattern, and the first few minutes were exactly the ones that mattered to her. During them, the junk was crowding out her real buyers.

It took about nine minutes for things to settle. To me, watching the request count, it was a long nine minutes. To Tessa, standing at that table with her heart in her throat, it felt like an hour.

The scramble

For those nine minutes we each did what we could. Tessa posted on social media telling people to hold on and keep trying, and answered messages. I tightened what I could on my side, then watched the sales count sit frozen at twenty six.

When the host’s protection kicked in, sales rushed through, and the festival still sold out that evening. We were lucky. Some customers had given up and gone elsewhere, but the damage was small. The size of it wasn’t the point. What bothered me was that I’d known this could happen to a small plan, and I hadn’t put a plan in place before the big day. That one was on me, and I’d been doing this for years.

What we changed

Before the next sale, I asked the host the questions I should have asked in the beginning. How fast does your flood protection react? Can you turn it up to its most sensitive setting before a launch? They were very helpful, and offered to watch the site at launch time.

I also moved the ticket landing page onto a cached version, so that fewer requests needed the ticketing software behind it, and added a simple waiting room that lets people in a few at a time. Cached pages cost the server almost nothing to hand out, which is the whole trick. A flood can hit a static page all day and the heavy parts never wake up.

Looking back on the day

What I remember most is how little the festival’s real work mattered for nine minutes. Tessa had planned the stalls, the stage, the food trucks and the toilets, and none of it counted because a web page wasn’t answering. The thing everyone pays least attention to turns out to be the front door of the whole event.

Her volunteers were wonderful about it. One set up a phone line for anyone who couldn’t buy online, and they sold about thirty tickets that way. It was old fashioned, and it worked.

For the next event, we wrote a one page launch plan together. It lists who is watching the site, who is answering messages, who rings the host if needed, and what we say to the public. I hope we never need it. But I’ve noticed we both sleep better before a sale with it in the folder.

A few things I’d tell you

• Launch moments attract junk. Assume a crowd will include bots.
• Ask your host about their protection before the big day, including how quickly it responds.
• Cache your landing page so most visitors don’t touch the heavy parts.
• Use a waiting room or queue for limited tickets.
• Have a plan for communicating, like a pinned social post, so people know what’s happening.

I’ll always remember those nine minutes. It was a small flood, in the scheme of things. It just happened to land at the most important moment Tessa had.