Here’s a dirty secret about the world of sports: We have no idea if people truly know what they’re talking about.

Some people sound smarter. They can craft compelling narratives and come armed with reams of tables and numbers and charts. That doesn’t mean they are right.

Some people have experience. They can tell you lots of stories about a past they’ve seen and learned from, times they’ve made the correct call and times they’ve been wrong and adjusted accordingly. That doesn’t mean they are right.

Some people are confident. They can assert with authority that their opinion is accurate, and that anyone who doesn’t see things their way is lost. They can be so strong and so loud that it’s hard to think they don’t know what they’re talking about. That doesn’t mean they are right.

Look at any type of person in any corner of the sports industry—former players, so-called media “experts,” head coaches, GMs, statistical analysts—and there’s no good way to determine if anyone truly knows what they’re doing. GMs or coaches might have good won/loss records, but how much of that is due to factors outside of their control? What would we think of Steve Kerr if he had taken the Knicks job? What would we think of R.C. Buford if he hadn’t come up through the Spurs’ system with Gregg Popovich?

So much of the allure of sports is that it is a meritocracy, that those who truly are better at what they do get rewarded for it on the scoreboard. But study sports closely and it becomes clear that the industry doesn’t work that way. Sports may be more meritocratic than other areas of life, but it still has quite a bit of messiness. Which means it’s an extraordinarily difficult task to determine who we should listen to and who we should tune out.

That’s the exact problem that Phil Tetlock, a professor at the University of Pennsylvania, has studied for over 30 years. How do we actually know what experts know? How do we know who is right? Tetlock’s focus has been on political judgment, but the political world is not too dissimilar from the sports world. Discouraged about the way political debates were conducted, Tetlock set out to measure the accuracy of pundits. It turned out this was an incredibly difficult task, because our discussions naturally use vague language. So he created a framework for making forecasts that could be evaluated. As Tetlock wrote in Superforecasting¹, the excellent book summarizing his work on the subject: “Forecasts must have clearly defined terms and timelines. They must use numbers. And one more thing is essential: we must have lots of forecasts.” If you have those ingredients, you can score the accuracy of forecasters, and begin to pin down just who knows what they’re talking about.

Tetlock’s work is about turning conjecture into science. With his framework, if you could indeed make better predictions, if you were indeed more right than others, you could prove it. If done correctly, it would mean no more relying on age, experience, title, or appearance to value an opinion. You’d be able to see a real track record of someone’s predictions.

We all know the feeling when you get into a sports debate and are convinced you’re right. The Lakers are overrated! The Celtics are a lock for the Finals! Markelle Fultz is a bust! You feel it in your bones, and everyone who disagrees with you is just missing the obvious. But there’s no good way to resolve that debate. There’s no way to know who to trust more. At best, it turns into a dinner bet, an attempt for people to put their money where their mouth is.

That’s where Predict comes in.

Predict is a basketball forecasting competition, open to everyone. It works like this: I will post questions on the site that are difficult to forecast and require basketball knowledge to answer accurately. For example: Will the Los Angeles Lakers make the 2018-19 NBA playoffs? Or: Will Kawhi Leonard be on the Toronto Raptors’ roster as of October 1, 2019? Or: Will the Boston Celtics beat the Philadelphia 76ers on opening night? Questions will be a mix of different timelines, with some lasting a year or more and others revolving around just a single game. I will be adding questions over the course of the season as interesting debates pop up.

Anyone can sign up and enter their predictions in response to these questions. Predictions take the form of a probability. If you think there is no way the Lakers could possibly miss the playoffs, you’d predict 100% for that question. But if you think that’s contingent on LeBron’s health, and you think there’s a 5% chance that he gets significantly hurt this year, you might predict 95% for that question.

Predictions can be updated at any time, but you’re still scored in part on your past predictions. If you put the Lakers’ chance of making the playoffs at 100% for the first half of the season and then LeBron gets hurt and you drop it to 25%, that 100% prediction is still counted for half of your score. Updating is a key part of accurate forecasts: it’s not just about your initial opinion, but how you respond to new information as it comes in. If the Lakers look great in the preseason or make a big trade, you might want to bump up your initial prediction. How much? That’s part of the game.

The site will keep track of your predictions and award points for accuracy. I’ll save you the nitty-gritty math of how you’ll be scored², but just know the following: you’ll get the most points if you predict as close as possible to the true probability of each event. For example, if the questions were about flipping coins, you’d be best off guessing 50% each time. Maybe on one flip you’ll be wrong, and someone who said 100% heads will jump into the lead. But as the flips add up, that overconfident heads-guesser will pay a steep price whenever the coin lands on tails.

We also will compare your guess to the consensus of all forecasters. So if the Lakers are one game away from clinching a postseason berth and you come in and forecast at that point that they’re 99% likely to make the playoffs, you won’t get any bonus points because the rest of the community will know the same thing. You’re scored relative to the consensus, so we’re judging your unique insight, not whether you can predict the obvious. The goal is to be as right as possible earlier than everyone else.

As the predictions are finalized, we’ll create a leaderboard to rank the forecasters so you know where you stand. Note that your individual predictions aren’t public unless you choose to make them so, but you will appear on the leaderboard if you have made a forecast on a question that has closed.

Don’t be gun-shy. A lot of the point of Predict is to improve over time as you get feedback on what you’re getting wrong. So don’t think if you make some bad predictions early that you’re forever stuck at the bottom of the leaderboard. We’ll have various versions of the leaderboard that only take into account recent success, and perhaps even show the leaders in improvement over time. So don’t be afraid to forecast. Put some real thought into the questions, make a prediction, update over time, see how you do, learn, and improve.

I will lead the way on this. I am going to be making forecasts along with you. Many of my individual forecasts I’ll make public, writing up my thoughts on how I arrived at those forecasts. There’s risk here. I certainly don’t anticipate I will be at the top of the leaderboard, and I may even end up pretty far down the list. That might make my readers doubt my expertise. But, to be honest, I’d welcome that, because it will mean I’ll have learned something, and I can examine what I got wrong and improve from there.

One of my hopes for Predict is that this is something you can use as proof of your knowledge and ability. Want others to take you seriously? Show them a clear track record of accuracy. I’d like to think it would even hold weight on applications for basketball jobs, whether with a team or a media outlet. Shoot—I’ll certainly reach out to the best forecasters to see if they would write for Cleaning the Glass and detail how they’re making their predictions so successfully.

Check out the open questions now and start working on your predictions! Compete with me and others, and prove what you really know—or learn what you don’t.

Questions? Email ben@cleaningtheglass.com or find me on Twitter @bencfalk or @cleantheglass.