Daren, first let me say that I truly appreciate the article you wrote for us and I think you've tapped into a really important theme not just in football but in 21st century society. Books galore are literally (pun intended) being written on this subject although too few in my opinion focus on football specifically.
Let me say right out in the open that I'm a football loving math nerd. In fact my background is as a data scientist and I routinely work with AI/ML algorithms. I would like to politely enter the discussion on the topic at hand and wish to do so without pissing anybody off. ("hope in one hand...)
"Did analytics cause the decision for Dan Lanning to go for it late in the game with a 4th and 1 from his own 29-yard line? He doubled down on his 4th down from his own territory call against the Huskies. He wanted to prove that going for it in your own territory is the right thing, and he failed." More on this statement later.
Let me open with agreeing with the theme of your statement that going for it on 4th and 1 from our own 29 was the wrong call in a tough rivalry game.
My position on data analytics in sports is a bit more nuanced and to explain it I need to get some history out on the table.
Mathematicians have been studying games for a very long time. The famous mathematician Blaise Pascal helped to discover the concept of "expected value" around 1650 and that discovery has underpinned optimal strategies for games that revolved around chance ever since (dice, cards, roulette, etc.). There would be no professional gamblers without this area of math as games of chance that pay out are purpose built to favor the house. There are a lot of mathematicians employed in Vegas working tirelessly to stay ahead of these developing counter strategies.
The science of games really took a major leap forward however in the 20th century. John von Neuman published a revolutionary paper on the theory of games of strategy. Along with Alan Turing, von Neuman is the Einstein level genius that invented the modern science of computers. His theory on games of strategy was established as the new field of study named "game theory" and was a heavily funded area of research by the 1950's due to the cold war.
The US and Soviet governments spent BIG money trying to figure out what was the optimal strategy for a nuclear armed power and game theory and computer simulation was the primary methods employed. The obvious concern was that "going with the gut" would get everyone killed. Eventually it was game theory that proved that nuclear conflict was in reality a no-win game and both countries separately arrived at the conclusion to agree to arms control discussions. That's an oversimplification of course but the point is game theory occupied some of the greatest minds of the 20th century.
That's all fine and dandy but what does it have to do with football? My point with the nuclear discussion was to say that game theory is pretty well developed and it's not a hair brained flat earther scheme sitting out on the dark web. In fact it quickly (1950's) moved into the area of strategic games of pleasure and almost nobody was happy with that. Well maybe some math nerds.
Everyone over age 10 mostly understands that Tic-Tac-Toe isn't a very fun game to play, the reason? If both players are paying attention nobody wins. In fact game theory proved that Tic-tac-toe was unwinnable if both players play an optimal strategy. When a proven optimal strategy is found for a game we call it solved. Connect four, battleship, checkers are all solved games. All of these games however have something in common, they are deterministic zero sum games with perfect information. Meaning if I take a move in the game I always get exactly that move (deterministic). Zero sum means there are no cooperative solutions where both players win, tie counts as a mutual loss. Perfect information means you can see every piece on the board exactly at any point in time.
There are deterministic zero sum games with perfect information that we haven't solved such as chess and go. The reason they aren't solved is the number of possible moves is to big to account for and we haven't found patterns where we can reduce those possible moves to a manageable number for an algorithm to work on. That doesn't mean however that game theory isn't used to win these games.
Before checkers was a solved game we had developed computer algorithms that could beat any human player in the world. In 1996 Gary Kasparov lost the first round to IBM's deep blue chess computer which was the first time a human chess champion ever lost to a computer. Kasparov battled back to win the match 4-2 and said "... I sensed a new kind of intelligence". In 1997 he lost the match to deep blue and today the worlds best mostly refuse to play computers as they find it demoralizing.
Sports in general are a LOT harder for computers to form winning strategies with. They are stochastic games instead of deterministic meaning when I call a jet sweep sometimes I get a jet sweep and sometimes I get a fumble. Perfect information is debatable but what I can say is that there are a lot of hidden variables for a football player. Is the RB going to accelerate into the gap at 7 m/s^2 or did he go out on the town last night and today it's 6 m/s^2? Sports in general might have something that "looks like" a player turn (aka a down) but the play itself is real time so observing variables in real time is in itself a very hard problem.
My point is we aren't there yet where computers are spitting out the "perfect play call" given all observable data. Will we ever be? Almost certainly barring human extinction. When? That's really hard to say any we have a word for people that predict when algorithms will accomplish some goal, "marketing". The truth is innovation comes in bursts and they are mostly unpredictable so we could be 5 days or 5 centuries away from that capability. It's why full self driving cars is a bad investment, the technology has promise but it's not quite there yet so keep your hands on the wheel.
Baseball saw the rise of data scientists with Sabermetrics and it's proven to be a competitive advantage for teams but not as big a one as huge salary pools. Football absolutely isn't baseball. Baseball is checkers and Football is Starcraft. Statistics in baseball are more relevant due to the highly repeatable situational effort (pitches, hitting, fielding, etc.). You have a pitcher literally on an island performing a task. A batter trapped in a box countering the pitchers move. It really does lend itself quite well to stochastic methods in game theory. Even so sometimes players get the yips and the data models fail.
Does all this mean that Football is immune to data analytics? Unfortunately no, it doesn't. Data analytics is a very useful tool for making competitive decisions of all sorts. Business uses it, the DoD uses it, the Yankees use it, Manchester United uses it and so do the Oregon Ducks. The real key here is understanding that data analytics is just one tool in the box and while it has a purpose it's not a swiss army knife... at least not yet.
Now I return to Daren's quote "Did analytics cause the decision for Dan Lanning to go for it late in the game with a 4th and 1 from his own 29-yard line?" Daren answered this question himself "He wanted to prove that going for it in your own territory is the right thing, and he failed." He in this instance is Dan Lanning and the fact that he wanted to see a particular outcome other than just winning the game means he DIDN'T use data analytics to make the decision. He used his emotions and possibly justified it with ad hoc data analytics.
How the human mind generates a decision is extremely complicated and we absolutely don't understand it. We do however think we might understand some of it and if you're interested in getting a non-technical overview it can be found in the book "Thinking Fast and Slow". The main idea is that "slow" rigorous logical reasoning is extremely calorically expensive (mental exhaustion anyone?) and while it's a competitive advantage in nature in some situations over-thinking will actually get you killed. Enter the "fast" thinking of the human "gut" that we came pre-equipped with. Fast thinking takes a lot of logical short cuts and arrives at a cheap but usually right answer for areas where the individual has a LOT of experience.
The idea then is if you are doing something you are an expert in, don't overthink it just make the call and move on. If you are in new territory you should slow down and proceed cautiously. This rule of thumb, follow your gut, is generally speaking the optimal solution for humans. Where the problem can arise is that thinking "fast" will sometimes make errors. In the case of the optimal nuclear strategy those errors have devastating costs. How do we avoid them? We force ourselves to slow down and think it through when the risks are too high.
Game theory, data analytics, computer simulations, computer decision algorithms are all just tools that are meant to help us slow down and think it through. In the end they are just tools (for now anyway) and it's how the people in charge use them that matters in getting to the right situational decisions. This concept is called human-machine teaming and we've been using it since we learned how to make pointy sticks. The stick isn't the thing, it's the hand holding it, the eye aiming it and the mind deciding on what to do with it that has mattered. The machines have certainly gotten fancier but we really haven't. Same hand, same eye, same mind.
In football when it's 4th and 1 on your own 29-yard line and you are a new head coach late in a competitive rivalry game you are in an area of devastating consequences. Those are exactly the times you need to force yourself to take 15 seconds to slow down and use some tools to help you make the right decision. Data analytics is one of those tools and the danger is that the "thinking fast" system never stops so it's already made a call on what to do. It says "go for it" because we didn't get here without being aggressive, aggressive has paid off for us.
The hardest thing to do in that moment is to force yourself to be aware that part of you has already decided and the calorically efficient thing to do with an exhausted mind is to use a ready tool to tell yourself a story on why that decision is the right decision. This is hard stuff. This is hard stuff for football coaches, this is hard stuff for squad leaders in the Rangers, this is hard stuff for me when I'm debating on writing 2k words on the forum or getting started on that project I'm supposed to be working on.
It helps to work with people you respect that can challenge your instincts and give you pause when you have a tough decision. Mistakes will happen, mistakes were made.