"We did all of our testing internally" ???? what was beta test then???

[wodinoneeye]

from the interview
http://www.immortalcities.com/cotn/gameinfo/design_insights_02042005.php

"We did all of our testing internally"

?????

What was the beta test then ???


I test software for a living. And beta 'testing' IS a type of testing and since it was distributed outside of 'inhouse', this statement is false.


You might want to correct the interview.

[Keith]

....better "inhouse" than "outhouse". http://members.aol.com/dheitm8612/animated/roflao.gif

[imhotep3147]

@ Keith:

*snorts, then tries not to giggle :D


BTW I love those smileys you always get!!

[Keith]

@ Keith:

*snorts, then tries not to giggle :D


BTW I love those smileys you always get!!
I keep my eyes peeled for smilies that are a bit different. http://members.aol.com/dheitm8612/animated/3-eyed-fish.gif

http://members.aol.com/dheitm8612/animated/smilewinkgrin.gif

[Jeff Fiske]

We never could have done it without the beta-testers.

So I definitely should clarify the statement.

One of the first things that happen when studios merge with a major publisher is the loss of internal testing. To me, this means there is no longer a significant amount of testing, or bug management being done in a room down the hall from the development staff.

Often testers are a group of people who were not there when the game was designed they don't know who the designers are. This means they can start designing and complain about the game as their own community and never embrace what the designers are trying to do, often because they never had a chance to understand the goals of the game. Compound this with the fact that games are not fun for months & months before release, the testers can get dis-enfranchised and just hang with each other and try and force the direction of the game. This can be especially damaging if they have easy access to the publishers producers, or the marketing and PR staff at their location. You have $10/hour fans of some shooter or RTS trying to force a game into being something they want to play. What if I am a tester, I have no documentation, the game looks like it will never reach its stated goals and the producer who works upstairs comes in and asks me how is the game? Well I can answer my boss who works upstairs frankly, or I can try and be loyal to a develper 2,000 miles away, so what am I going to say? Can you imagine trying to develop a game like Fable, or Lords of Magic, if you did not have direct contact with the testers to prevent the design from being hijacked?

In our case, all of the testing feedback went through an internal staff of testers at our location. The information we fed to you was carefully managed to the best of our ability, and testers were constantly talking with the designers. (Note: very rarely the programmers or artists). There was no feedback that we received about play balancing that was filtered from a supervisor in a room hundreds or thousands of miles away. To me, that is the difference between internal and external testing. Are the main decisions about what goes onto the bug list and ‘what is wrong with the game’ being made at the developers address or at some other address?

Now that you got me started however-

As far as managing the beta test goes, lets compare the way we did our test to the way we frequently see these beta test go.

The worst are the tests that are awarded to x subscribers, such as fileplanet, or members of a certain community such as GameSpy. What if this group does not include the type of gamers that actually will enjoy the game? What kind of diverse group and systems are you going to get if you sample from a single community. One begins to wonder if this just a PR stunt? Especially if you get into the test and there is no method for feedback to the developer. Submit your bugs to some big black box and never hear back from them? That is not a beta test, in my mind.

Depending on the publisher, the quality of their testing can vary tremendously. I take great pride in our game coming out side by side with Zoo Tycoon 2 and RTC3. From the comparisons of the issues, one might think we were the ones with extensive testing labs and hundreds of professional testers. I assume that the folks at Blue Fang or Frontier have testing staff, but most of the work is being done at MS or Atari facilities and being fed back to the developer.

But for CotN, the entire bug lists and management was handled at our facility in Wayland- internally. There was a reason the game changed as much as it did in such a short time during the beta, and there was a reason so many testers said, "Wow, I can't believe you guys are listening to us so much."

We really love our fans and our community. We wish we could spend more time interacting with the fans and it is really fun to interact and make games we hope you will enjoy.

You all did such a great job, we almost considered you extensions of the team, and thus I do consider the testing to have been done 'internally'. There was no group of people outside of Wayland that we were beholden to, or that we were receiving our bug mandates from. (Of course both Myelin & Sega gave us feedback and worked with us appropriately to resolve issues, but our staff and the beta test did a vast majority of the important testing work.)

I think this interaction and the implementation of your feedback, is what makes for a loyal community, and makes me feel that we did our testing internally, especially compared to the way it is normally done these days.

[vic_4]

I can confirm that there was always a good dialog with developers, since almost every problem I reported was addressed and I had the impression that what I did really part of the completing of the game.

[angelisis]

Dito...and not only that but jeff mentioned that he/they felt that the testers were considered and exstention of the company...that is so true for not one second did anyone feel that we were not being listened to or our suggestions being ignored....would we have gotten that as testers anywhere else? maybe, maybe not.
Many of the testers were a part of this community (myself included) and even many before me, months before the game was even ready I would even hazard a guess to suggest that many things discussed between us all those many months ago were impletemented into the game without us even knowing it.
The members of this community are more a part of Tilted Mill than you will ever know.
I have always thought I was part of the team. :D


so whenever you ready to send that big fat cheque David :p

[Meli]

So, in a nutshell, it really was more about 'insider' versus 'outsider' testing, rather than 'internal' versus 'external.' :)

[wodinoneeye]

Internally means by the company that creates/produces the game.

Its definitely 'Internal' if done by the programmers/designers/developers themselves or the projects integrated testing department.

I suppose some may consider having the testing done by the publisher's
centralized/seperate resources as external.

Outshopping to a seperate testing company definitely is 'external'.

Beta testers (as done for COTN) are outside of that. TM paying attention
to bugs reported by beta testers doesnt make it 'internal'.

Maybe because inhouse personel had to do alot of work to recreate and isolate (beta) reported bugs it seemed like it was 'internal', but the problems were originally found and communicated by the beta testers (an external resource).

[tobing]

Maybe it should be phrased something like 'external testing done by insiders' or similar. ;)

[arcan]

Yep, 'cause even if I'm external to the company, Thanks to TM and the way they treated me(us) during the Beta-test, I feel almost like a team-member. I wouldn't object at being called an "insider" or a "rignt-next-to-the-border internal member" :D

[Helle]

I didnot beta test this game before it was released for sale. But I have never experience a forum with so much participation from the developers as this. I never seen the first patch reasled 9 days after the game release.

This shows me that the developers actually listen to the customers too.

If i didnt live abroad from the US i would volonter to beta test for this company anytime.

I think they made a beatifull game and i look forward to the next one. i would even buy it unseen.

[angelisis]

do not living abroad stop you from trying helle. I live in Australia and I got my beta CD all expences paid just like everyone else...the testers of this game came from all corners of the earth my dear :D


@Woody does it really matter whom or how or what tested or did not test a game...is a kid rocking up to the local game store with his/her cash really going to care about "internal v's external" testing....stop looking at the brush strokes hun and take a step back and look at the HOLE picture. :)

[Josh]

I didnot beta test this game before it was released for sale. But I have never experience a forum with so much participation from the developers as this. I never seen the first patch reasled 9 days after the game release.

This shows me that the developers actually listen to the customers too.
This is exactly why this group of gamemakers has a fanbase that will follow it no matter what their company name is. :)

[Helle]

do not living abroad stop you from trying helle. I live in Australia and I got my beta CD all expences paid just like everyone else...the testers of this game came from all corners of the earth my dear :D





I wouldnt work for me anyway, im in a wheelchair and wouldnt be able to travel across the world. But im not surpriced that it could be done.

[wodinoneeye]

do not living abroad stop you from trying helle. I live in Australia and I got my beta CD all expences paid just like everyone else...the testers of this game came from all corners of the earth my dear :D


@Woody does it really matter whom or how or what tested or did not test a game...is a kid rocking up to the local game store with his/her cash really going to care about "internal v's external" testing....stop looking at the brush strokes hun and take a step back and look at the HOLE picture. :)



Sorry, but a false statement is still a false statement.

Beta testers did the company a favor. They should get recognition for it.

Saying that ALL the testing was done internally doesnt do that.

They found alot of bugs fairly quickly that good internal testing should have found (and been fixed before a beta was done).

The 'Whole Picture' doesnt change that. Many years as a professional software tester/programmer has made it hard for me to miss 'Holes'.

[mouse]

Helle to beta test you don't have to travel just to the door to accept the package shipped by TM to testers. Being in a wheelchair wouldn't prevent you from taking part in a beta test. Beta testers sat in front of their very own computers in their own homes where ever they live in the world.

[angelisis]

I wouldnt work for me anyway, im in a wheelchair and wouldnt be able to travel across the world. But im not surpriced that it could be done.


hunny I ment the beta CD was shipped to me all expenses paid...not me shipped to the beta. :D
anyway if that were the case we would not all be here talkin about the game as TM would have went broke...there were 1000 beta testers and I would rather bleed from the eyeballs and have my guts pulled out via my ears than catch a plane EVER again :D

[angelisis]

Sorry, but a false statement is still a false statement.

Beta testers did the company a favor. They should get recognition for it.

Saying that ALL the testing was done internally doesnt do that.

They found alot of bugs fairly quickly that good internal testing should have found (and been fixed before a beta was done).

The 'Whole Picture' doesnt change that. Many years as a professional software tester/programmer has made it hard for me to miss 'Holes'.



Woody...... you make my head hurt. :rolleyes: :(

[Helle]

hunny I ment the beta CD was shipped to me all expenses paid...not me shipped to the beta. :D
anyway if that were the case we would not all be here talkin about the game as TM would have went broke...there were 1000 beta testers and I would rather bleed from the eyeballs and have my guts pulled out via my ears than catch a plane EVER again :D


Thanks for ruining the pretty picture of people sitting in a big room playing all day and eating cake with the TM people. :D

Think i will blame the misunderstanding on english not being my first langue, and not understanding what internal ment.

HEY TM i volonter for next beta, i have really much time on my hand to play, so i can play a lot. *trying not to beg too much*

:D :p

[Josh]

The testing was run in house

They didnt hire anyone else to run the test, they didnt outsource to their publisher, anything we as beta testers said went directly to the tilted mill staff, not to gamespy or some hired firm. The beta was done under TM; therefore In house, we were just an extention of that.

"They found alot of bugs fairly quickly that good internal testing should have found (and been fixed before a beta was done)." -wodineye

You make that comment unaware of what testing was done to get it do the state that we saw it in; You have no idea what the game was like in the alpha version. You also dont know if stuff we reported was already a well known fact to TM they just hadnt gotten around to fixing it. Just because your report it doenst mean its news.

Your complaints have all been addressed by a very well written post by Jeff, now you just seem to be complaining for the sake of complaning. :(

[NanaBanana]

There were probably some "carryovers" of buggy things from the alpha, but like Josh said, no one knows for sure except TM and they ain't talking. ;)

CotN was my second "real" beta. (I also beta test for Real Arcade.) I can truly say that the Sirs were very interactive during the beta. They gave us loads of help and really listened to our thoughts, complaints and suggestions.

I really wish that the NDA would be a little more relaxed and that we could tell you at least one BIG change that was made as a result of the beta test.

TM was running on a short schedule wanting to get the game out for the Christmas season, but they extended the beta well beyond what we signed up for. They were working around the clock, and interacting on the forum with the testers 24/7. Sure, some bugs slipped by TM and us testers, but that happens a lot. (refers to the burial goods bug in Cleopatra) Some are minor bugs that might go unnoticed while others are game killers (see reference).

@Helle Don't worry about being across the pond! The two betas I was involved in spanned many continents. I remember chatting with people in Denmark, Australia, France, Slovenia and Brazil to mention a few. It was great fun.

@Jeff Fiske Thanks for the clarification. While it would have been great fun to have my name in another manual and CD, testing the game was fantastic. I thank TM for the opportunity and would love to do it again!

[vic_4]

This was my first beta test so I have no other experiences, but when I signed for beta it was as a choice I did without inducements or promises, not even to be cited in manual or anywhere else. I did only since I thought I would enjoy it and as I already said I enjoied it far more then I expected. So I don't believe TM owes any of us anything and that what Jeff wrote clayfied the issue.

[wodinoneeye]

The testing was run in house

They didnt hire anyone else to run the test, they didnt outsource to their publisher, anything we as beta testers said went directly to the tilted mill staff, not to gamespy or some hired firm. The beta was done under TM; therefore In house, we were just an extention of that.

"They found alot of bugs fairly quickly that good internal testing should have found (and been fixed before a beta was done)." -wodineye

You make that comment unaware of what testing was done to get it do the state that we saw it in; You have no idea what the game was like in the alpha version. You also dont know if stuff we reported was already a well known fact to TM they just hadnt gotten around to fixing it. Just because your report it doenst mean its news.

Your complaints have all been addressed by a very well written post by Jeff, now you just seem to be complaining for the sake of complaning. :(




Beta test is supposed to be done when a product is close to release and fundamental problems are all fixed (leaving outstanding errors just obscures additional errors that could otherwise be found). The patch that came
immediately following the release showed that they pushed the schedule regardless of significant problems.

The post just tried to redefine the term 'internal' which in the industry does not refer to a Beta as was done with COTN.

You can define 'complaining' to be when someone makes a statement that disagrees with you, if you want.

[Jeff Fiske]

Beta test is supposed to be done when a product is close to release and fundamental problems are all fixed (leaving outstanding errors just obscures additional errors that could otherwise be found). The patch that came
immediately following the release showed that they pushed the schedule regardless of significant problems.

First of all, no two companies define what a Beta is the same way.
Second, no two companies run a beta test, closed or open, the same way.

Pushed the schedule regardless of "significant problems".

Are you kidding me? Name one title that came out this Xmas in better shape than CotN. Did you even try to play Zoo 2 or RTC3? HL2 (sound stutters & crashes)? (Maybe Pirates came out with fewer issues but look at the complexity of our game.)

The changes we made were mostly in the tutorial during the beta test. The obvious bugs that beta testers were finding, we were finding as well.

The bugs that were found by the testers that were most important were obscure path finding bugs that did not always repeat and stuck figures, something that we had a minor problem with at release, but something that neither Zoo2 or RTC3 has fixed to this day. (Z2 gives you a picker to move stuck animals, a luxury we don't have.)

You can criticize CotN for many things about what you would have liked the interface to do, the pacing or things, what the challeges were or things about city building that you wish we had not done the way we did. But to criticize the condition of its release, particularly compared to who we are, what the game technically accomplishes with the number of figures & AI, the resources we had, is really barking up the wrong tree.

Feel free to disagree, and I will be the first to acknowledge the shortcoming of the game in certain areas. To say it was clearly pushed out the door, is a term that should be applied to many products this past Xmas, but not CotN. Was CotN ready to go gold when it did? Name me a publisher that would have said, "Nope, version 1.0 is not ready to go gold".

[Helle]

I will have to say that i complety dissagree with the critic of the way the game was tested and released.
I have never been in a forum like this, where the developers of the game is so active. I have been waiting for this game for 9 months since i heard about it the first time. And yes all new games have problems but have many games have you played where a lot of the beginners problems was fixes in 9 days?

I have been playing an online game just before i started playing CotN and the developers still hasnt fixet serious problems that was found a year ago.

The tone and the way people behave themselves in this forum is like no other forum i ever been to. I am empressed by the game, by the forum and very empressed by the dedication of TM.

I think the only right way to beta test a game is to choose people who are truly interested in playing the game and not people who test random games at new releases. Only a true citybuilder knows what a citybuilder game is all about.

[athkatla]

I didnot beta test this game before it was released for sale. But I have never experience a forum with so much participation from the developers as this. I never seen the first patch reasled 9 days after the game release.

This shows me that the developers actually listen to the customers too.

If i didnt live abroad from the US i would volonter to beta test for this company anytime.

I think they made a beatifull game and i look forward to the next one. i would even buy it unseen.

I think this is a sign of things to come, and the way ahead for developers. I am currently involved in beta testing two games from home, 'Football Manager 2005' (currently testing the 2nd patch for the released game), and 'Supreme Ruler 2010' , which is due for release approximately April/May of this year.

With both these games, the developers are on the Forums several times a day answering all questions and not just on the Beta Testers forum, but on the public forums too. It is very encouraging to see this kind of interaction between developers and fans of their games.

It is also interesting to note that with the games I am testing, and CotN, it is the smaller companies that appreciate feedback from their fans, and also realise that these same fans are the ones who pay their wages and, potentially, will do for some time to come. :)

[Son of Moose]

Athkatla:

It is also interesting to note that with the games I am testing, and CotN, it is the smaller companies that appreciate feedback from their fans, and also realise that these same fans are the ones who pay their wages and, potentially, will do for some time to come.

Yes .... it does seem that it is the smaller companies that really listen to their future fans. :) Unfortunately, the same cannot be generally said about the larger and (often) more established companies which seem to have somehow mysteriously managed to "outgrow" their fan base. :(

There is precious little official information available about either Civilization IV or AOE III (except that it is already virtually a finished product).

[vic_4]

Usually big company think they know better, they do not need hard core fans, they prefer to address to not committed public. I do not think it is wise, but apparently it must be either business sound or they are out of touch with reality.

[arcan]

I'll go to the dark side, but I can see at least one reason not to call for HC fans.
If you try to change the gameplay, for ex the same difference as between pharaoh and CotN, hardcore fans would only compare the two games, and declare the latest "unplayable" or "unworthy", or so on. This way they can be free to make the changes they wish with no reference to the former games...

Ok now back to the good side! :D

[Son of Moose]

Arcan:

for ex the same difference as between pharaoh and CotN, hardcore fans would only compare the two games, and declare the latest "unplayable" or "unworthy", or so on

An excellent example ... as I certainly remember several respondents making exactly those claims on these forums before the eventual release of CotN. :)

I suppose that it can be an extremely tricky issue. :( Certainly (for example) the devs of SR2010 have publicly noted that they have greatly benefited from the not inconsequential input of fans at their forum.

A case of .... Different horses for different courses. :cool: