I'm well aware that the title of this post is going to insight more anger than it will thought, but you're going to have to hear me out on this one.
To start, I've been a gamer since the summer of 1986. For those that can't do the math, I've been gaming for 25 years as of this post. My first gaming experience was Castle Wolfenstein, which I played on my aunt's Apple II. That is one of my earliest memories. I talked to my mom, and she thinks I played some Pong before then, but I don't consider it in the equation since neither of us know for certain if I first played that before or after the summer of 1986. My first console gaming experience was The Legend of Zelda in the fall of 1987. I've been a hardcore gamer since. I've played games ranging from Mr. Do! and Cops'n'Robbers on the Atari to Road Rash and Advanced Dungeons and Dragons: Slayer on the 3DO to The Death and Return of Superman and Maximum Carnage on the Genesis to Final Fantasy VII and Metal Gear Solid on the Playstation to Full Throttle and Diablo on the computer to Call of Duty 4: Modern Warfare and Mass Effect on the Xbox 360. I've played on almost every console ever created (I never could find a Pippen), but I'm not here to list out my full gaming references. I shared a sampling of my range of gaming experience to show the breadth of my experience. I want you to understand that what I have to say comes from an experienced gamer who played most of the classics when they were originally released.
There is a common misconception that older games are harder, that today's developers have gone soft on the gaming community. The idea actually stems from the idea that frustration is indicative of difficulty. The measuring stick should actually be based on the level of challenge that the game provides.
Older games are naturally limited by the technology that they were designed to be played on. Wolfenstein 3D couldn't have been played on the Atari 2600 because of the capabilities of the system at the time. Developers used what capabilities that were available at the time. I'm sure that the developers of Adventure wanted to make a game that was more like Skyrim, but it just wasn't technically feasible at the time. In the same way that technical capabilities are a barrier, accepted gameplay mechanics have changed the landscape of gaming. Time Crisis introduced the cover system utilized in Gears of War. Customization and RPG elements weren't new to First Person Shooters when Modern Warfare came out, but the inclusion of so many features and Prestiging to extend the online life of a shooter was revolutionary. Halo 2 popularized the Regenerative Health system, which had a major impact on the online experience, and completely flew in the face of the Health Pack system, the accepted norm.
With technological improvements and gameplay techniques constantly evolving, game designers haven't changed their mindset as to how to provide a better challenge to the gamer. The previous mentality was to increase difficulty by increasing the health of enemies, decreasing the player's defensive or offensive capabilities, or to increase the number of enemies the player has to fight. Those are simple solutions due to technical limitations brought on by older generations of gaming machines.
Clunky controls, old gaming mentalities, and limited AI capabilities formed into the best way to simulate difficulty: Make difficulties based on how frustrated a gamer is, so they think that it's more challenging. In short, old games suck, but gamers would challenge themselves to get better so the game would be less frustrating. That is a significantly different viewpoint than the games being more challenging themselves.
I'm not without solutions. Highlighting a problem without offering a solution is nothing more than complaining. They may not be easy, but gaming is a multi-Billion Dollar Industry. It seems to reason that investing some money into developing a better challenging experience, as opposed to just making things harder, would only serve to improve sales.
I'm not going to write a 60 page thesis on ways that game designers could improve the challenge of their games, but I will list a few solutions as examples to show how possible it is to come up with innovative ways to challenge a gamer and not frustrate them.
Solution #1: Build the game around the hardest difficulty and offer more options for easier levels. This can be the hardest or easiest way to improving the challenge for a gamer, depending on the type of game. For puzzle games, it means that a level has to be designed with a whole lot of flexibility in mind. For a game like Sudoku, this option is easy. Sudoku puzzles are rated on difficulty based on the numbers given to a player. For a game like Portal, however, each level is designed to be inherently at a specific challenge level, and adding or subtracting creates a significantly more complex problem for the designer than the player. In a platformer, the hardest difficulty could have one very complex way to scale a building, with the player being forced to rely on leaps of faith and extreme three dimensional spatial awareness to figure out how to progress. The easier difficulties could add more and more shortcuts that require less complex thought in order to traverse.
Solution #2: Adjust the enemy AI to account for more or less complex actions. This is going to be the hardest of all of my solutions. AI is a very tricky subject. It's already something hard to program, then when you try to work with easier or harder difficulties, it becomes very easy to make too hard or too easy. My suggestion would be to create a baseline AI that all enemies work on, and include more advanced tactics as the difficulty increases. As an example, on Normal, the enemies are not stupid by any means, but they don't work in concert with their teammates to try to kill the player. On Hard, they work as a team much more, and will cover each other to limit the player's actions. On Expert, they utilize choreographed actions, like Breach and Clear tactics instead of just filling in to the room. And on Master, they have a series of tactical techniques that they use in succession or in concert to outmaneuver the player. In this example, the individual enemy isn't dumbed down for easier difficulties, so they can still provide a challenge, but the enemies have more complex tools at their disposal to increase the challenge.
Solution #3: Adjust the enemy's range of tools. This idea is very similar to Solution #2, but is much easier to program for. Also note that I didn't say to just add more tools, because that's no different than giving them more health or improving their attack strength. The idea here is that each enemy has a specialty, but their choice in specialty grows with the difficulty. This way, the player doesn't necessarily have to deal with stronger enemies, but has to know how to deal with a wider array of enemies. It also creates more complex combinations of enemies, so the player has to be good enough to be able to adjust their tactics for each unique situation. As an example, the easier difficulties could outfit basic Rogues, Warriors, Mages, and Hunters for the player try to figure out how to defend against. In the harder difficulties, Rogue class enemies that trained in dual wielding with stealth based techniques could specialize in flanking tactics, disarming tactics, subduing tactics, or poisoning tactics could provide very specific challenges, especially when there are 256 different specifications of enemies that all with their own strengths and weaknesses. It becomes 1024 different types enemies if there are Humans, Elves, Dwarves, or Lizardmen. Mix and match races, classes, specializations, techniques, and tactics, with a wide variety in the number of enemies, and each fight challenges the player to constantly develop new techniques to deal with a wide variety of tactics. It's also possible to cross-cue classes and create a large number of combinations without as much of a wide variance in skills. Middle grade difficulties offer differences of Black Mages, White Mages, and Summoners, where harder difficulties have Black Mages that can heal or Warriors that can use stealth techniques.
It's easy to come up with one solution to a problem. It's another thing all together when a game designer has to figure out how to implement multiple techniques to the problem of challenge. Unfortunately, the best way is to make a unique blend, because the biggest challenge for a gamer is when they are challenged in different ways by different developers and different games.