Out of all the ways you could have communicated, you chose that particular one? The one that invokes a long history of racist overt aggression? What’s wrong with “quit horsing around”? Or just “settle down”?
This is the essence of what’s being called micro-aggressions. You can’t be overtly racist anymore, unless you’re in a literal skinhead camp, nobody’s going to lend you social support in your quest to keep the black man down, so instead you choose smaller, less visible ways to make them feel less socially welcome.
Yes, this is absolutely racist, and no, you shouldn’t do it. Find a different way to put it. Better yet, don’t try to control people’s behavior and stop them from having fun. If someone, anyone, wants to “monkey around,” just let them. If you have to say something, keep it classy. I personally prefer “easy there,” but only when their fun threatens to, say, turn over a table full of drinks.