Teaching a Computer to Find Made Baskets

How do you teach a computer to find a made basket?

A person watching the game just knows when a shot drops. A computer doesn't. It sees video as millions of colored dots changing 60 times a second. So you have to break “a basket was made” into things it can actually look for: where's the ball, where's the rim, and did the ball pass down through the rim?

Two things make that harder than it sounds. First, the footage is huge (4K, gigabytes per game), so the first step is shrinking each video down to a small, low-res copy the computer can scan quickly. The timestamps still line up with the original, so a hit at 9:42 in the small copy is 9:42 in the real game. Second, and this is the one that got me: on a clean make, the ball drops through the net and basically vanishes for a moment. A perfect swish, all net and no rim, is the worst case. The better the shot, the less there is to see. Early on, that meant the computer was great at spotting misses and blind to the makes.

The journey: dead ends, one big surprise, and a fix

What followed was a Sunday of dead ends, one big surprise, and a handful of fixes, every wrong turn included. Here's the short version.

It started at zero. On the first real test game, the system found 0 of 7 made baskets. Not a great start.

The surprise: it wasn't failing at random. It was doing exactly the wrong thing. It kept flagging near-misses and ignoring clean makes. Watch these two moments back to back. In the first, a shot clanks off the rim and the ball stays in view, bouncing around, and the system fires. In the second, a clean make drops through the net, the ball disappears, and the system sees nothing.

That's the “ball disappears on a good shot” problem from above, in action. The cleaner the make, the more invisible it was.

The fix wasn't to be cleverer. It was to be simpler. Instead of trying to judge make-vs-miss in the moment (which is genuinely hard, even the disappearing ball trips it up), I flipped the goal: just flag every time the ball reaches a rim, and let me skim the handful of clips. I'd rather glance at a few extra moments than miss a highlight.

The other half was a smarter “rim detector”: early on it couldn't reliably tell the actual rim from the wood floor or a player's jersey, and getting that right made all the difference. Here's the same made basket: first the old approach flailing, then the new one locking onto the rim and flagging the make.

The result

By the end of the morning: 7 of 7 made baskets on the test game. Then the real test. I ran it, untouched, on a completely different 15-minute game it had never seen, and it found every made shot that had a visible rim: 14 of 14.

This changes my editing drastically. Instead of scrubbing two hours of footage to find the makes, I get a short list of moments to review. Every real make is in there, plus a few false alarms I dismiss in a couple seconds each. The tedious part of editing highlights basically went away.

What was different this time

Two years ago this was weeks of work. This time it was a Sunday morning. Part of that is just how far the tools have come, but a few things stood out about how the work went:

• Looking beat theorizing. The biggest jumps didn't come from clever ideas. They came from building a quick “debug view” that drew what the computer was seeing on each frame, then actually watching it. Every time I guessed at why something failed, I was wrong. Every time I looked, the answer was obvious.
• “Catch everything” beat “be perfect.” Chasing a system that only ever flags real makes was a trap. Accepting a few false alarms, and trusting myself to dismiss them quickly, is what made it actually useful.
• The hard part was the basketball, not the code. Knowing that a made shot ends with the ball vanishing into the net, that the other team inbounds right after, that the camera pans to follow the ball: that domain knowledge is what unlocked each fix. I expect the next features to work the same way. Figuring out who made a shot, for instance, is really about following a player through a possession, which is a basketball problem long before it's a coding one.

What's next

A few years ago, this project felt like I was just dabbling in computer vision. I still don't have much depth in the field, but what I can produce with the right prompts is surprising. For me personally, I want to keep learning how computer vision detects objects, both for the fulfillment and a better understanding. For Saturday Hoops NYC, I want to take the automation further: cut the editing time down even more, reduce false positives, add player tracking, and plenty beyond that. The possibilities with these tools are exciting.