The Beta. The stage in the development cycle that generally begins when the product in question is (usually) feature complete. At this point, the product will have bugs that need to be caught and eliminated prior to progression to the next phase of development. It’s a term many in the industry – ranging from consumers to professionals – have come to understand differently.
While absolutely essential internally to development studios, the role and importance of the “public beta” has regularly been called into question, especially when it comes to games with multiplayer portions. Microsoft (and 343 Industries) recently confirmed that the upcoming Halo 4 would not have a public beta. Ubisoft also announced similar, for Assassin’s Creed III. Activision studios working on the Call of Duty franchise traditionally don’t have public betas for their multiplayer games, with 2007’s Call of Duty 4 being the last title to have held one.
So, are public betas still relevant in this age? Are we seeing a trend back towards the traditional development cycle, with fully internal testing being the norm? To answer those questions, we need to understand how and why public betas are run, and the benefits and costs of them to both developers and publishers.
Public betas are by definition, beta tests that are open to members of the public. These usually take the form of either open or closed tests. Closed versions are released to a selected group of individuals, by invitation. Open versions usually encompass a larger group, and are generally available to anyone with an interest. In an ideal world, public beta testers report the bugs that they find and use the platforms provided to suggest additional features, fixes or tweaks that should be made by the developer in time for the final release of the product. There is an old saying that “two eyes are better than one.” By that reasoning, making a beta release available to more eyeballs should yield considerably better results, right? Not necessarily.
Back when I was a part of the Community Team for Infinity Ward, the developer took the unpopular decision to not hold a public beta for the then-upcoming Modern Warfare 2. Creative Strategist at the time – Robert Bowling – delivered an explanation of the decision:
“The internal beta will allow us all the benefits of a public beta test, without the impact it takes on the time required to prep, distribute, and manage a public beta during this crucial time in development.” he said. “That fact alone allows us to have a smoother and faster process of iteration when addressing issues and pushing fixes out to beta participants.”
It is no secret that it takes some considerable time to prepare, distribute and manage a public beta, a fact confirmed to me by the developers I was able to speak to about this subject. It also costs a fair bit of money. I spoke to a developer who was able to shed some more light on the process. For obvious reasons, they wished to remain anonymous. “I can’t give you a number. What I can tell you is that when a publisher wants to deploy a public beta on consoles, they have to pay a certain price per download. In the case of a closed beta, they pay a fee for each key issued.” My source went on; “Microsoft and Sony do things quite differently. For instance, there is an additional fee to host a beta on the PS3, even if nobody downloads the client.”
I tracked down another source who also agreed to speak to me on the condition of anonymity. They went into a bit more detail. “It [deploying a beta] isn’t cheap. The cost for first parties is around £200,000, or at least it was when we ran our public beta. Whether that’s changed or not, I don’t know.” They continued; “In a marketing budget, that can be a drop in the ocean though. And for a development with a big online focus that needs server issues ironing out – well, we all know how much testers get paid! A beta lasting a month can get a hell of a lot more exposure than hiring 100 testers, and without the overheads of office space, desks, paperwork, PCs, consoles and so on.”
Both sources did agree that public betas were worth the work and initial expense, and more often than not, easily paid for themselves. “So long as the beta client is loaded with enough ways to diagnose issues players encounter, and the servers log things, then it’s all good. It can pay for itself to tighten multiplayer, and fix server issues that would kill a big launch.”
With that in mind, why would any publisher or developer choose not to have a public beta? For starters, public betas are treated by most development teams as a separate part of the project, with its own milestones and deadlines. As such, in order to prepare one, a lot of teams have to split their workforce to create “beta teams”. These new teams then devote their time and focus to the beta project; handling submission, monitoring, and data collection. So, valuable manpower and resources are being diverted away from the main project.
The often private tussle between developers and publishers also plays a key role here. Again, one of my sources was able to educate me on the matter. “There are usually two different reasons for holding a public beta. The developer reason revolves around not knowing if the servers/clients will hold up in bigger environments (online) and as such, absolutely have to be tested with more than just internal QA/QC.” they said. “The publisher reason revolves around generating a bit of hype to get people onboard for a game that isn’t necessarily top of the public’s buying agenda, based on their research.”
More often than not, the publisher usually comes out on top when it comes to making such decisions. And if they do decide to go ahead with a public beta (from a marketing standpoint), then the timing of the beta in relation to the development cycle becomes doubly important. “Arguably the sooner after Alpha [you hold a beta], the better. By then, most functionality should be working, and the game is still loose enough to allow changes based on that feedback.” my source explained. “Having internal builds with play testers under NDAs is one thing; having a build in an Alpha state in people’s homes is another. There will be a great many bugs, and things might not look as pretty as finished. So, most publishers wait until a game hits beta. That means all the game is missing is final debugging, and polish. But at that point, it means the course is pretty much set, and changes become a lot harder to make too.”
There is no set time period for when a project enters beta, or how long it stays in that phase either. As such, public betas have been known to be made available anything from as little as 3 weeks, to as long as 5 months before release. In the case of more recent development cycles (and usually amongst third parties), public betas generally get made available a month before the game’s release. “As a rule of thumb, at the stage of 1 month before release – the build is done, approved, and is going to manufacturing.” said my source; “So a beta at that point in time is handy to start to dictate work for patches, or for server load testing. But don’t expect us to be making any changes to the code that will go on the disks. And of course, the free advertising that a beta brings then is another matter entirely.”
Having looked at the issue of public betas from the viewpoint of those selling the games, we should also acknowledge what public betas mean to those that purchase games. As mentioned earlier, the “no public beta” announcement for a popular game is usually met with fierce disappointment by fans. While many gamers now view alpha/beta access as a badge of honour, a growing number use public betas to make purchase decisions. It has been a slow but perceptible shift in the mindset of many gamers that has undoubtedly had an effect on the number of developers and publishers releasing public betas primarily for marketing purposes. Those who still choose to do so find that they are running an increasing risk of being accused of releasing “glorified demos”, with knowledgeable gamers already being aware that the beta version has been polished to be as bug free as possible, in order to prevent putting off undecided customers from the final game.
It will be interesting to see how things change regarding this issue over the next few years, especially with the transition to the next generation of console gaming. While the likes of Microsoft and Ubisoft appear to be moving away from public betas for their marquee franchises, others such as EA and Capcom continue to offer these tests for public consumption. It would appear that public betas are still relevant to the industry. The question now is, for how long?
