The Green Cubes

Oz Giat has a meeting to get to. Before he goes, he wants to show me the cubes.

He turns his laptop around. The screen is covered in small green squares, one for every flow a player can take through an Appcharge web store or checkout.

Every few seconds, a synthetic player runs each of those flows again, from regions around the world, and the system measures how long each one took.

Oz, Appcharge’s Director of Infrastructure, is obsessed with how that experience feels to the player. If one of these flows slows down past its threshold, even for a second, its cube turns red and someone on Appcharge’s operations and engineering teams gets paged.

* * *

On Black Friday, it’s money time for those cubes.

For the mobile games that sell through Appcharge, the stretch from Halloween to Cyber Monday is the most valuable part of the year, and Black Friday is its peak. Hundreds of games run that weekend through the Appcharge platform, at the same time, all with traffic and sales spikes.

Gaming traffic is hard to plan for in the first place. Oz came to Appcharge from Moon Active, so he’s seen it from the studio side as well.

“Their traffic charts are very unique," he says. “It has these sharp, short peaks. And infrastructure and services usually suffer a lot when it comes to sharp peaks."

Staying up is only the starting point. “This is binary, like on and off," Oz says of uptime. “If you’re off, you’re off." No studio would work with a payments company without it. What players feel is speed, which is why the cubes measure latency, flow by flow.

With payments, the stakes are high. Oran Stern, Appcharge’s Director of Engineering, explains it the way a player would feel it.

“If you try to pay and something goes wrong, you usually won’t try again, because you’re scared you’ve already paid."

That’s why a flawless checkout UX is mission critical. 

The night that rewrote the plan

Last year, a major publisher, who was working with another DTC payment platform, turned to Appcharge with an urgent request: their payments system had crashed, on Black Friday of all days, and could the team jump in to save the day?

The answer, of course, was yes.

Oran and the team monitored the operation 24/7, testing every edge case they could find, across devices, countries, and localization, and ensuring the publisher’s user experience was perfect throughout their skyrocketing traffic. 

But despite this extra attention, the work Appcharge’s engineering team did in the preceding months meant the platform’s capacity could handle this traffic without much stress. “It was another day at the office for the Appcharge infrastructure team", Oz recalls.

Together, that’s the whole weekend: a platform with room to spare, and a team that spent the night on the details, making sure every player, in every market, got the experience they’d get on any other day.

Oran is open about the parts that weren’t perfect. Some non-critical paths, internal analytics and data events Appcharge uses for its own calculations, ran with delays. None of it touched players’ payments. It did show the team where its weaker points were, and it shaped how they test now, layer by layer: the money path first, then everything that supports it.

The bigger lesson was about planning. The team used to plan for the additional traffic that seemed reasonable. Now, Oran says, they plan for the physical maximum: every player Appcharge’s partners have, arriving at once.

Hunting for bottlenecks, every week

The tool behind that planning was built by Daniel Nagar, one of Appcharge’s longest-serving software architects. It uses agents running simultaneously, placing stress on the platform and producing an agentic report of all bottlenecks. 

Daniel describes the method. “Regular traffic, 2x traffic, 5x traffic, 10x traffic, 20x traffic, 200x traffic," he says. “And each time we try to see where the bottlenecks are, what’s starting to slow down, what starts to break".

Find the bottleneck, fix it, run it again. Oz adds that the bottleneck can sit somewhere unexpected: in a service upstream, in one downstream, or inside the service being tested.

The easy assumption is that all of this is seasonal work, something they just do when Black Friday approaches. Oz pushes back on it.

“For me to say that we are doing it just for Black Friday would be a lie" he says. “We do it every week". On every product change, every new feature, every configuration change, and every feature flag switched on or off.

“You don’t prepare for Black Friday one week before," he adds. “You just can’t. You’re going to fail. You prepare for Black Friday every day".

The overseer

One key part of Oz’s team’s preparedness is an AI agent they call “the overseer". It’s a game-changer, he tells me. 

When a new version of a service is gradually promoted to production, the AI agent monitors and compares the service's behavior with how it ran in its previous version. It monitors  the service during all stages of the promotion process, and hours later.

Oz calls it a compensating control: a check for the areas where writing a test for everything isn’t realistic. It’s also, he says, something “you couldn’t do before the AI era."

The team is careful about how far that goes. Agents suggest fixes when alerts fire, but people still have the final decision.

Everyone in one room

All of that groundwork makes Black Friday, in many ways, business as usual for Appcharge’s engineers - but the season still gets its own extra measures.

The first thing that changes is what ships. 

With AI writing around 95% of Appcharge’s code, developers push out far more changes than they used to, and in the weeks before Black Friday, every change is a risk. Two or three weeks out, Rony’s team starts asking the same question about each one: is it urgent, or can it wait until after Black Friday?

“It’s better to have what we have right now," she says, “than adding something small and risking it breaking even the smallest part of the system. It’s not worth the risk."

Oz’s team adds even more player flows to the checks.

Then, from three days before Black Friday until two days after, the team works around the clock from the office in shifts. Being on call around the clock isn’t new for them; what changes is having everyone in one room.

“If there is an issue, we can communicate faster together with other teams and departments," Oz says.

* * *

Oz has a line he uses to explain his work to people outside it.

“It’s like electricity," he says. “Nobody cares about it. You don’t think about it." That, he adds, is “the highest compliment that our infrastructure can get." 

Then he closes his laptop and heads to his meeting.

Out of sight, the cubes turn green again.

* * *

Oz Giat is Head of DevOps at Appcharge. Rony Sason is an Engineering Manager in Appcharge’s monetization R&D group. Oran Stern is Appcharge’s Director of Engineering. Daniel Nagar is Appcharge’s Software Architect.

Charged is Appcharge’s editorial publication, covering the people and craft inside the company and beyond.