Vibe Coding an Offer Weighting System: No Midnight Swaps

Today I’m sharing an automated offer weighting system I built with AI tools. It solved a pain point I’d had for a long time.

We all know the iron rule: don’t launch new campaigns before bed.

The problem is, while you’re sound asleep, some offer suddenly goes bad in the middle of the night, and the traffic is still running on the weights you set before bed.

You wake up the next day, open the tracker, and the offer that’s doing great only has 10% of the weight, while the one that collapsed has 90%.

Overall ROI, which had a shot at +50%, ends up forced down to -30%.

To stop getting burned by this, I built an automation system dedicated to watching it. It’s been running for a while and feels pretty good.

1. The off-the-shelf options all fall a little short

My first instinct, of course, wasn’t to build it myself but to let the tracker’s built-in Random Rotation watch it for me. For example, BeMob’s Random Rotation or Voluum’s Traffic Distribution. I went through the options on the market, and the conclusion was:

Basic subscriptions: most trackers’ basic plans don’t have this feature; you can only change weights by hand.
Premium plans: a few trackers have built an AI Optimizer, but it starts at $200 a month.
Doesn’t get there: they can only run fixed rules on CR / ROI / CV. The logic I wanted to add (bailout verification, score linkage, automatic alerts) isn’t there.

After looking around: either too expensive, or doesn’t exist, or can’t be modified. So, build my own.

Tracker Traffic Distribution dropdown: Random Rotation, Paths Rotation, Paths and Landings Rotation, Fix on Paths and more

2. What this system actually solves

My mechanism sits on top of a serverless backend and solves 4 core things:

No watching 24 hours a day: the system checks automatically every hour and adjusts weights dynamically.
Fast reaction: when a trend changes, it follows within 30 minutes (the old algorithm dragged on for 2-3 hours).
Not fooled by historical data: a 1-2 hour bailout verification, where the AI goes back and checks whether it misjudged.
It calls me when something big happens: an anomaly triggers a report straight to WeCom.

One key point here: this isn’t the AI making decisions for me unilaterally. It’s the result of the two of us working together.

An example: I change the weights by hand → the system steps aside for 1 hour and feeds my judgment into the algorithm as a prior → after the AI takes over, it uses real data to test me in turn: if I was right it keeps my call, if I was wrong it overturns it.

function seedFromWeights(weights: Record<string, number>, strength = 40) {
  const max = Math.max(...Object.values(weights));
  const out: Record<string, Beta> = {};
  for (const [offer, w] of Object.entries(weights)) {
    const r = w / max;                    // relative preference 0~1
    out[offer] = {
      alpha: 1 + strength * r,            // higher preference → larger α → higher posterior mean
      beta:  1 + strength * (1 - r),
    };
  }
  return out;
}
// you set A:70 B:20 C:10 → A's posterior mean = 41/42 ≈ 0.976 ("I believe A is good")
// the AI needs a lot of contrary data before it can shake you → respects the human but keeps the ability to push back with data

3. Why Bayesian, and not sorting by EPC

Most people’s first reaction is “just sort by EPC.” That’s a trap.

Some offer gets 1 conversion at $5 from 10 clicks, and the EPC looks nice and high at $0.5.

But the next 100 clicks might not produce a single conversion.

Sort directly by EPC → the weights keep flip-flopping between two offers, and the flow never settles down.

The Bayesian approach is different: for each offer, maintain a belief distribution of its true performance (something like a normal distribution).

New data arrives → belief updates → weights get computed.

Essentially it’s automatically balancing two things: exploring new offers and exploiting known winners. With few samples it explores more; with many samples it exploits more.

Judging this by hand relies on intuition; the algorithm relies on math, plus continuously accumulating data.

Dashboard table of Bayesian posteriors per offer: alpha, beta, average CVR, 95% confidence interval, clicks and conversions

4. Before going live, I ran a week of shadow mode first

Code that’s written isn’t code that’s ready to go live.

I first had it do a week of shadow running: the system made its decisions as usual, but I didn’t write the real weights back to the tracker.

Then compare: the system’s decisions vs my manual decisions. Only an agreement rate > 90% was allowed to switch to live.

That week is the system’s preparation period. Any algorithm that looks pretty, if it hasn’t passed the shadow run gate, is easy to crash when it goes straight to live.

Decision stability panel: current winner at 97%, 3 winner switches in 24h, and winner verification with measured CR 23.80%

5. The core ideas behind the 5 mechanisms

Each mechanism is a pure function of a few dozen lines. Here I’ll only go over the ideas.

1. The 1-hour bailout re-evaluation

Trust the AI’s current judgment, but it must go back and check after 1 hour.

If actual performance is too far off the prediction (say, below 50% of predicted), consider a reset and recompute.

This move compresses an AI misjudgment from taking hours to correct down to auto-correcting within 1 hour.

def check_bailout(predicted_cr, actual_cr_past_1h):
    # actual CR < 50% × predicted → trigger reset
    if actual_cr_past_1h < 0.5 * predicted_cr:
        return "BAILOUT"  # recompute with fresh data from the past 2h
    return "OK"

2. Half-life + scheduled runs

Align the algorithm’s reaction speed with a human’s.

A slow algorithm is a deaf algorithm. Something I can judge by hand in 10 minutes, if the AI can’t keep up, there’s no point.

So from the original 24h half-life + 30-minute cron, I changed to a 1h half-life, and the reaction speed is basically aligned with manual judgment now.

Note: the cron frequency was originally set to 15 minutes, then I found the tracker API was being rate-limited, so the load was cut straight to 1/4. Reactions are slightly slower, but it bought back stability. Worth it.

3. Softmax automatic weight allocation

Don’t go 100% all-in with a hard switch, and don’t split evenly for the sake of fairness.

Use softmax so the weights change automatically with the CV/ROI gap: a big gap concentrates (a 2pp gap → 88/12), a small gap spreads out (a tie → 50/50). One α parameter controls how aggressive it is.

4. Score linkage

The score is the intermediary platform’s rating of your traffic quality.

The lower the score, the more exploration is forbidden, and the more it concentrates on the winner to stay alive.

A high score (≥60) allows balanced exploration; dropping to 40-60 doubles the concentration; below 40 it goes straight to 100% all-in. When you’re about to die, what’s there to explore.