people often focus on why AIs will never be perfect and how how hard a lot of problems are. it's not about solving NP-hard problems and finding the best solution - it's about finding a competent course of action that can present a challenge to the player, and making sure that challenge fits within the design parameters of the game.
In a symmetrical game, how well the player is challenged is what determines "good" versus "bad" playability. For AI purposes, "good" and "bad" are just a measure of competence, so if two AIs provide the same challenge to the player, the one that cheats less is the stronger artificial intelligence.
The point is that for almost all strategy games, good AI = good playability and bad AI = bad playability. This is because the games are symmetrical and are completely designed around opponents playing by the same rules.
Games like Civ simply stop working with bad AI because the level of cheating is so ridiculous that it completely changes all the gameplay elements. When a game is about racing (racing for technology, racing for wonders, racing for land, racing for military strength), then the race kind of needs to be fair - or at least seem that way in the eyes of the player.
Without the possibility of loss, there cannot be challenge. And when losing is not perceived to be player-driven, then the game sucks. Designers are absolutely horrible at making the player think that they lost because of poor play when they actually lost because some AI got 75% cheaper everything
10 years ago, there was 1 CPU and it basically had to be entirely devoted to just running the simulation and rendering all the fancy graphics. There was at least an excuse for the horrible AI, even for the turn-based games. Now that excuse is gone for the turn-based games, even if resources are still too limited for really great stuff.