Preface: This is largely just from personal experience, so your mileage may vary.




Damn you, shoddy QA!

I hear a lot of comments about how terrible Bethesda QA must be. The most important thing to remember about QA is that it is not, and never will be, QA's responsibility to ensure a bug free release. QA's only function is to assist stakeholders in judging the quality of the product.

In essence, QA can only highlight issues caused by developer mistakes (whether they are faults caused by bugs, or missed or misunderstood requirements) so, QA's effect on the actual quality of the release is largely passive, so to blame QA for post release issues is at best misguided.

Another thing you'll hear a lot of, is 'All software releases contain bugs.' this is perfectly true. No matter how much effort you put in to development or QA, you will always release with some bugs. Moreover, not only will the release contain bugs, but first releases will always contain known bugs. That is to say, just because a release contains bugs does not mean they were not identified ahead of time by QA. In fact it's quite normal for a first release to contain hundreds, if not thousands, of known issues of one kind or another.

That will perhaps shock some people, but this is because no matter how good the development group is, project resource is finite. It's generally just not possible to fix them all in time for release. What happens is that, given the necessary information by QA & Dev, an appropriate stakeholder (normally a project manager) will anolyse each issue, judge the business risk of release against the project risk of the fix and then prioritise accordingly. The output of this is a list of 'to fix for release' and 'to fix in maintenance release 1', 'to fix in maintenance release 2', etc.

Damn you, shoddy developers!

As a developer, one of the first thing you have to come to terms with is that no matter how good you are, you're going to make mistakes. You're going to put bugs into software without realising it and that software more often than not will be released with your bug/s in it, and your customers are going to have to deal with them one way or another. So, it's all the developers fault, right? Not necessarily.

While it's true that any bug that a customer sees can by and large be traced back to an individual developer, no developer works in isolation with a margherita in one hand and a Faberge egg in the other. Not unless they write COBOL, anyway. When judging the work of a developer it's important to remember is that all developers work under the constraints put in place by development management, project management and stakeholders. One way of thinking about it is that a software project can by and large be broken down to three main aspects. Quality, Time and Resource. So, for example, if marketing commit to a release date, time becomes fixed. In most cases the stakeholders will have set the budget at project initiation, so your resource is fixed - and the only thing left that's variable? Quality. With finite time and finite resource the maximum achievable quality of the product is now effectively fixed, and the eventual realised quality of the release is probably going to be defined by the maturity of your project management, development and QA processes, and the level of skill in the respective areas.

So, what's that got to do with developers? Well, the constraints for development are now set. Now, if the stars have aligned and you've been given realistic deadlines and a generous budget then great. However, if your marketing have decided that, say, releasing on 11/11/11 looks really good on their branding boards then you're probably out of luck. You'll now have your work items set, if not in stone, then at least in a fairly sticky substance that you really don't want to go near.

What's that? Your timelines don't seem to have accounted for the time necessary to write good unit tests for your code, or appropriately document it, or clarify ambiguous requirements with the stakeholder? Unlucky! It's come around to patch time and because you weren't allowed the time for them before, there are no regression suites to run to check that your latest change hasn't made dragons fly backwards? Unlucky!


So, no-ones to blame?

Not exactly. Realistically, the project team (Steering, Marketing, Development, QA, etc) all share some degree of responsibility for the final quality of the product, in sickness and in health.

However, in my experience, the people most likely to negatively affect the quality of the final product, almost without exception, aren't the QA team, aren't the development group, but Steering, with unrealistic budgets, unrealistic timelines and a lack of emphasis on product quality and Project management, by accepting unrealistic timelines and not managing stakeholders properly.


Tl;dr?

In closing, please don't always be quite so mean to Development and QA.

Developers work harder than most people could believe and generally love the software more than anyone - and while QA will forever be the red-headed stepchild of software development they are still highly skilled professionals and generally work as hard as anyone, if not harder, to help ensure a high quality release.

I am a Software Engineer. For a long time I was a Software Engineer in Test (QA). I agree with your points in general, but disagree that your points apply to this product. In my opinion, the QA for the game was either lacking, the developers ignored QA, or the time schedule mandated release as is.

The quality of the finished product is not (and cannot) be a 'black-box' measure of the validity of the QA process/resource applied.

As I said, QA's role is purely informative. If a PM or stakeholder decides that there is not the time, money or necessity to fix an issue (which quantifiably is most often the reason for live bugs) then QA&D cannot be held responsible for that decision, in my opinion.

But as I said, YMMV.

@OP: Thanks for this great post. And I agree completly. :yes:

This may explain the bugs, but what about the fact that just about every element of the game is broken or done in the most lazy way possible?

There's a problem of QA when the main storyline has a gamestopping bug like :

First game, first character I fall on that. I don't see how it can possibly pass through QA. Obviously there's been a change that wasn't tested.

Every element of the game? Care to say which ones?

So far the only "universal" problems i have seen are:
-Magical resistances borken.
-Dragons acting wierd.
-Graphical bugs.

I agree here. I think the OP is spot on in general when it comes to the SW industry. However, I think it's fairly obvious that general play testing was an afterthought for this product. Perhaps the game was tested, but as the OP said, a PM or stakeholder rammed the feature through.

It depends what you mean, really. Perceived quality in a game is much more subjective than in, say, an order processing system. For example while I (as an old RPG gamer) might want deep complexity, a 12 year old gamer on an xbox might prefer a more shallow 'whizz-bang' experience. Neither is more valid than the other really, so it would probably result in a trade off swayed by who your core demographic (defined by the stakeholders) is.

Well, I will say that it's always possible for any given bug to get 'past' quality controls (whether that's quality control in the actual QA team, or in the dev team). In fact, I'd go further and say that you can absolutely guarantee that no matter how good the QA team, it's literally impossible for them to find all bugs in any complex system.

However, (as an 'external' customer) to assume that the reason a bug exists post-release is that there's been a mistake by QA would be wrong, as QA just report the issues.

Whether or not that bug gets fixed is a project decision, not a QA decision.

Let's start with lack of branching dialogue/quest choices, shallow marriage/companion system, horrendous item and creature scaling, short and inconsequential quest lines, lack of NPC or game mechanic acknowledgement of completed questlines or achievements, extremely poor balance, lack of spelltypes, AI issues, tiny town and city sizes, and failure to include basic gameplay options such as resting and eating requirements.

The game was rushed, and incompetently handled.

And now that you have struck the management hiding in the corner with a Magelight, it is only a matter of time before they send the Dark Brotherhood a-knocking. "We know."

In all seriousness, I agree with you completely. The laws of economics and the principles of management. freakin DB.

These sound more like personal opinions than actually bugs.

Skills are lazy(most perks is just +% damage, some trees are almost identical like Light/Heavy and One-Handed/Two-Handed), broken(Destruction, Lockpick, Smithing) or just worthless(Speech), attributes are broken(You can make Magicka completely unnecessary, Stamina is a dump stat), races are unbalanced as usual and most powers are worthless, level scaling is AGAIN working with non-combat skills, enemies scale forever and everything scales, armor is broken because of the hyperbolic damage reduction and because Light Armor provides better protection than Heavy, weapons are broken because of the upgrade damage bonus being linear and the same no matter the weapon type or material and because of the low damage values, overall protection is way too easy to acquire, hell everything is way too easy to acquire, there are stupidly exploitable things like chopping wood which provide infinite money at no effort from the player whatsoever...

Rushed yes, Incompetent? No. Clearly the developers of this game are competent. Clearly they are capable of producing a high quality product. I don't think the problem with Skyrim is the downstream quality control, but the upstream quality control. The inconsistencies in the game tell me that somebody in a decision making role decided that "Good Enough" was the watchword. It should have been "Good enough isn't".

Please follow the quote chains to see what stemmed the conversation before adding pointless observations.

The incompetence is more on the backs of the lead developers and decision makers, as you note. Incompetently handled, not in the individual execution of tasks by the programming team, but at the macro level.

I did that and maintain my "pointless" observation.

Erm, say what? QA stands for "Quality Assurance", doesn't it? Doesn't that imply that their job is to ensure the QUALITY of a product? Like, not having game/quest breaking bugs? Or, patches not breaking things like ALL resistances? Or, free exploration in an "Open World" game, breaking quests?

I will grant that it isn't possible for QA to find every bug in a game of this scope, but, those two glaring examples seem to indicate that whatever the QA team was doing, they need to rethink their methods.

That does confirm my suspicions then.

All Bethesda games are bugged. Every single one, and they keep releasing patches, to fix the game, and patches to fix patches.

That, sir, is an opinion.

It is a fact that there are specific elements that are broken (i.e. not functioning as intended): this includes the invincible backwards-flying dragons, the 2-7 second stutters for those that run on XP, certain quests not functioning/activating, the favorites menu unhotkeying an item that is stacked in the inventory, and the magic resistance bug, among others.

Saying that everything is "broken or done in the most lazy way possible" is a complete lie, though.

I call troll.

Having a system which increases the strength of enemies based on the character level and then allowing for said level to be increased with skills that do not benefit combat in any way is blatant incompetence.

Not only is this a blatant incompetence, Bethesda already did it once(with Oblivion) and everyone criticized it and then they go and make SAME mistake AGAIN.

It's a joke. Skyrim is so full of rookie game mechanic mistakes it comes off as a mediocre indie product.



Broken does not mean "not functioning as intended". That's a bug or a glitch.

Generic post attempting to excuse bad QA? Check.

Here's ianpatt's description of the problem, you tell me that the QA/devs did their job after reading it:


http://www.gamesas.com/index.php?/topic/1288179-wipz-skyrim-script-extender-skse/page__st__120

In my opninion, 'QA' is something of a misnomer. People can (and do) argue names and definitions til the cows come home, but QA as a process is not something as simple as a "Oh this has a bug in it, this can't go live" bottleneck. It's a collective effort applied throughout the project lifecycle to increase the quality of a final product. Not just a static testing phase at the end of a development cycle.

Think of it this way, wouldn't logic dictate that the more visible a bug is to a user, the more visible a bug is to a highly skilled professional tester supported by a well thought out quality process?

The truth is that just because a bug is found does not mean it'll be fixed. It's also true that even with a great project team and well informed stakeholders, you simply can't find and fix all known bugs (especially in a new release) to any kind of timeline that's going to satisfy all users.

As I mentioned in the OP, it eventually comes down to a risk management decision and unfortunately that means that some issues have to be left to be fixed at a later date.

That's not Bethesda's fault, it's a fundamental fact of software development across the industry.