“We were embarrassed,” the father of four said with a laugh, recalling the awkward moment in public

Mark Wahlberg is a good sport about his kids’ constant jabs.

The actor joked about how his four kids — Grace, 13, Brendan, 15, and Michael, 17, and Ella, 20 — “always make fun” of him while appearing on Wednesday’s episode of The Late Show with Stephen Colbert. He then opened up about how he once took Brendan with him to the Super Bowl, though the father-son duo left at halftime.

“I went to do some stuff to promote the movie and then I promised my wife I would be home,” he explained. “I took my son and I was a little worried because my son, the last time we went to a Super Bowl, it was the Patriots versus the Falcons.”

He continued, “And people always wonder why I left at halftime, but he was 8 years old and he was screaming and cursing at the top of his lungs because they were down 28-3 and we were embarrassed. So I didn’t know what he was going to do at the game.”

Wahlberg noted the family lives “literally 15 minutes away from the stadium” and was able to watch the game with “no stress.”