Analytics in the Real World: Successful Production in Video Games

This was an essay I wrote for my MBA regarding the use of data analytics in the production of video games.

graph

By Estelle Tigani

The video games industry is one of the most unstable forms of entertainment. While those dedicated to developing games do so because they are passionate about the medium and its potential to influence decision-making and reactions towards their players, they also trade in an extreme amount of job uncertainty due to high project failure. One of the biggest problems faced in the industry is the ability to schedule and plan out a video game under what is fondly known as ‘The Iron Triangle’: within budget, on schedule, and at a high quality. It is rare for a game to succeed at all three, and the result is either a great game that cost millions of additional dollars in resources and time, a poorly-received title delivered on time within budget (this can occur especially with movie tie-ins and holiday releases), or the worst kind, a canceled game. The last type can often lead to layoffs, which is what makes employment so unstable.

In the video games industry, every game is treated as its own project, with funding usually given to the development studio by the publishing company. The publisher establishes a contract in tandem with the development studio, which lists the requirements for the game, the amount specified in the budget and dates when slices of the budget are given to the studio (usually around milestone developments), and the time when the game is expected to be launched to the public (thereby establishing a time limit for project). While this seems straight forward, the reality is that it is incredibly rare for any project to meet all three criteria (time, budget, and quality) due to the lack of perfect information and unforeseen future developments. A studio can plan for contingencies as much as it can, but so much money and therefore politics is on the line with these contracts, that it is hard to be so flexible and prepared. Any change to a contract usually requires an amendment and renegotiation period which can stall payments while the amendment is being done, causing further instability for a studio relying on those payments.

The role of the producer in such a studio is to create and assign tasks to their teams (art, design, visual effects, audio, etc.), as well as set milestones (based on agreements with the publisher) and time durations for these tasks. This in turn informs the times when slices of the budget will be received from the publisher, and how to invest in that budget, through paying employees a decided salary as well as purchasing software and hardware licenses. The other factor which makes developing games an unstable venture is how expensive these software and hardware licenses are per computer, especially if the game is to be developed at a high quality with state-of-the-art technology and graphics. Additionally software and hardware are being updated and changed less than annually. With all of these factors in mind, as well as the fate of employees’ job security, the pressure placed on the production team is extensive. Their decisions ultimately determine the success or failure of the milestone deliverables, and thus the project. In addition, the day-to-day emergencies within the studio and mediating conflict and personnel pipelines can also affect tasking and meeting project milestones.

The main goal is to optimize allocation of key resources (employees, licenses, work hours, technical demand) within a set of constraints (that being the three described above). Careful resource management is so important that consequences become more dramatic as the standard error increases. That is to say, if the producer incorrectly allocates resources too far from where they should, the negative results will be felt harder in the long term. Ensuring that the right developer is working on the right task is also a part of this craft. The producer needs to know their developers so they can play to their strengths, maximizing the capabilities of these resources. No amount of data or numbers can assist the producer there, because it requires a specialized understanding of people, behavior, and work ethics. One can read many books on this, but such people skills need to be changed and adapted for every different studio and every different employee.

People skills aside, producers can use analytics techniques and production tools which mine data, to make more informed and successful choices about task and time allocation. For example, by requiring all developers to check-in their times for starting and finishing certain tasks, records can be collected on how long it takes for a specific developer to complete their task compared to the next developer. Similarly, using a counter and percentage tracker to analyze how many tasks a specific developer is assigned on a daily or weekly basis can address if the developer is over-allocated. Graphs and charts which show ratios of tasks in total compared to completed can inform the producer on how successful a team is able to complete its tasks within a typical week. With this information, producers can understand what developers’ strengths and weaknesses are (are they completing some tasks faster and other tasks slower?), if a team needs to hire an additional employee for support, or which employees are working overtime compared to others. The metrics also inform HR on the performance of an employee; if everyone is working 8 hours a day but some are working faster than others, does that entitle them to a raise or promotion?

The ultimate measurements of studio-wide, team-specific, or individual-specific efficiency, are velocity and burndown rate. These are elements of the modern Agile Scrum methodologies that have been praised by more recent tech startups or Silicon Valley companies. Scrum demands the studio / team / group make a ‘backlog’ list of every task required in order move the project from start to end. Each task is then given a time estimate of how long it would take to complete. All these hours are then totaled and the result is the entire amount of time it will take to complete the project as a whole. Then, the amount of tasks completed are computed against this total number in order to create a burndown chart. If tasks are being completed faster than the total time, the burndown chart will end before the time initially expected (this is a good thing). If tasks are taking longer than the original estimated value/time, the burn rate is said to be high (often forcing a re-evaluation of the schedule and tasks).

Velocity refers to the slope of the burndown chart. If it is a straight line then tasks are being completed on par with expectations. If the line slopes upward then the burndown rate is becoming higher and the project is expected to be completed later than the original date. If the slope decreases then the tasks are being completed faster than originally computed, and the project will fall short of the estimated finish date (a good thing, but rare). Note: if more tasks are being added in the middle of the project that were not a part of the original summation of tasks (which often happens) then this will cause velocity to slope upward.

A few variables which come into play regarding velocity. The first is the total number of tasks which formed the original estimate. The second is the number of new tasks that have been added after the project has been commenced. Other variables include the amount of employees, the percentage of tasks per employee, and the number of new employees that have joined the studio since the commencement of the project. All these factors change the rate of the burndown chart, and the steepness of the velocity.

Naturally, a studio using a Waterfall methodology as apposed to a Scrum methodology will not utilize burndown charts. Instead, these companies use the Gantt chart, and thus calculate their total estimated time using the Critical Path method. This measurement takes into account tasks which are dependent on one another, that is, they cannot be started until a previous task has been completed. Dependencies and very specific time durations for each tasks are crucial variables in making accurate Waterfall estimations. New tasks that are added after project commencement, if they are dependent, will be slotted between the two appropriate tasks and therefore extend out the original critical path time, otherwise, independent tasks will start their own branch on the Gantt chart and their own path which may or may not be longer than the critical path.

Overtime is an occupational hazard in this industry, or more commonly referred to as ‘crunch’. Seldom has there been a project when some overtime has not been mandatory for employees. Crunch is so common in game development, that developers have listed cases of missing weddings, birthdays, and funerals. Sometimes, this overtime is unpaid, albeit rarely. While many are of the opinion that this sad truth is unavoidable, it can be reduced by using the data and metrics described previously. Crunch for extended periods does been shown to be counter-productive for quality (but not necessarily timetables). The burndown rate changes into simply… burnout. The less overtime and over-allocation, the happier and more efficient the developers, and the higher the chances of project success and studio payment. The same is true in the reverse, which instead spirals a studio down into bankruptcy. Some metrics can be so sophisticated that these production tools can even predict if a project will be completed on time or not (see burndown charts), which informs the studio ahead of time. While these are all helpful in reducing chance of failure, it does not fix the situation completely. A publisher can cancel a project that has been in development for 3 years for next to no reason, if they wanted to (although termination without cause is often a part of contracts).

Failure to meet The Iron Triangle criteria can be extremely punishing. Publishers are not forgiving when choosing to cancel a project, at the expense of the studio and the jobs of that studio’s employees. Sometimes, publishers even demand back the money they have given the studio from the beginning of the project, even if developers have already been paid with this money. The number of cancelled games compared to released games is staggering, so much that the typical game developer usually jumps between 5 studios in their career, with some cases as many as 13 studios. Developers know that their career is certainly one of passion for the medium and not for the luxurious perks and salaries (or lack thereof). Furthermore, developers know that they are entering an industry that is highly unstable, with low job security, and a fair chance that they could be let go even if their performance is high.

These problems are yet to be solved, and as stated previously, they are accepted as part of the industry and unavoidable, and can cause many employees to leave the industry altogether. However, there is ongoing research, debate, and media publications discussing how project failure, and the negative factors which cause it, can be reduced. By solving this problem, whole studios can avoid being shut down, employees can keep their jobs, and more games can be seen by the public rather than hidden on the hard drives of cancelled development teams. Considering that the games industry makes more than the music and film industries combined, there is definitely money available in the billions, but more often than not this money ends up in the publisher’s pockets than the developer studio’s. Behind the scenes, the development of video games is a very risky and ugly career choice, but if these problems are solved, it can be an extremely fun and rewarding one.

 

References

Agile Game Development – Sprint Burndown Charts for Feature Teams

http://blog.agilegamedevelopment.com/2007/09/sprint-burndown-charts-for-feature.html

 

Hansoft – Relevant Metrics to a Game Producer

http://www.hansoft.com/en/insights/what-metrics-are-relevant-for-a-game-producer

 

Kotaku – Why Game Developers Keep Getting Laid Off

http://kotaku.com/why-game-developers-keep-getting-laid-off-1583192249

 

Kotaku – The Horrible World of Video Game Crunch

http://kotaku.com/crunch-time-why-game-developers-work-such-insane-hours-1704744577

 

Gamespot – A Sad History of Cancelled Games

http://www.gamespot.com/gallery/a-sad-history-of-cancelled-games/2900-134/

 

Kotaku – What It’s Like to Crunch on a Video Game

http://kotaku.com/what-its-like-to-crunch-on-a-video-game-1701998016

 

Games Industry.biz – Game Devs: When Does Crunch Cross the Line?

http://www.gamesindustry.biz/articles/2013-10-23-game-devs-when-does-crunch-cross-the-line

 

Gamasutra – Working in the Games Industry a Job to Die For?

http://www.gamasutra.com/blogs/KatherineRogers/20140403/214684/Working_in_the_Games_Industry_a_job_to_die_for.php

 

The Guardian – Crunched: Games Industry Exploiting Workforce

http://www.theguardian.com/technology/2015/feb/18/crunched-games-industry-exploiting-workforce-ea-spouse-software

 

VG247 – Inside the Development Crunch Pandemic: Part One

http://www.vg247.com/2014/01/13/inside-the-development-crunch-pandemic-part-one/

 

VG247 – Inside the Development Crunch Pandemic: Part Two

http://www.vg247.com/2014/01/14/inside-the-development-crunch-pandemic-part-two/

 

Libcom – When is Comes to the Crunch: Unpaid Overtime in the Games Industry

https://libcom.org/library/when-it-comes-crunch-unpaid-overtime-games-industry

Leave a Comment

Awesome! You've decided to leave a comment. Please keep in mind that comments are moderated.

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>