Forum:Relax manual of style verifiability requirements in limited circumstances

I think this manual of style tweak is an almost existential need for the ME3 parts of the wiki. Because of the high bar set for verification even for gameplay mechanics, a non-trivial amount of the information in the ME3 parts of the wikia is either incomplete or incorrect, and a quick play-through of ME3 multiplayer can quickly show this. Moreover, the verification requirement creates an odd situation where "strategic" information can be put on the wiki that can make un-verifiable or invalid assumptions (as present in virtually all of the ME3 singleplayer/multiplayer class guides), but an actual listing of gameplay mechanics backed up by extensive community testing and experimentation cannot be used.

I hope this doesn't proposal doesn't come off as antagonistic; I realize the ME wiki has been here for a long time before I came here and the community has a way of doing things. I earnestly want to help improve the ME wiki (namely a lot of the ME3 mechanics stuff) but the bar of mechanics verification is literally impossibly high. And as someone who relies on wikia sites for all sorts of different games, I really don't want to be in the sad situation where anytime I am trying to refresh my memory on what a power does I have to ignore all search results that come from masseffect.wikia.com.

(Thelee (talk) 18:01, May 14, 2013 (UTC))

Support

 * 1) Kind of a "moral support" vote, see below in discussion. Cattlesquat (talk) 20:18, May 14, 2013 (UTC)
 * 2) obviously i should vote for my own proposal (Thelee (talk) 20:25, May 14, 2013 (UTC))
 * 3) --71.53.155.38 20:29, May 14, 2013 (UTC)

Neutral

 * 1) i agree with this proposal in theory, my problem is that BSN, like any forum, is a minefield. I would prefer to use proven information from walkthrough guides such as IGN for mechanics references, if they are applicable, but I cannot support using information from random BSNers. Garhdo (talk) 23:34, May 14, 2013 (UTC)
 * 2) Somewhat the same as Garhdo, but as long as it's thoroughly checked by someone who has expertise in programming, and is backed, and confirmed by the devs, it'll work, along with Commdor's suggestion below. --Nord Ronnoc (talk) 23:30, May 14, 2013 (UTC)

Oppose

 * 1) LilyheartsLiara (talk) 18:57, May 14, 2013 (UTC)
 * 2) If the dev's make a statement on the BSN, that would be reliable. As stated below by Lily, random strangers on the internet aren't accurate 100% of the time, especially on the BSN. Avg Man (talk) 20:01, May 14, 2013 (UTC)
 * 3) -- Commdor  (Talk) 20:08, May 14, 2013 (UTC)
 * 4) --Bluegear93 (talk) 20:25, May 14, 2013 (UTC)
 * 5) --Legionwrex (talk) 20:30, May 14, 2013 (UTC)
 * 6) --JediSpectre117 (talk) 22:49, May 14, 2013 (UTC)

Discussion
To pick apart this policy proposal flaw by flaw:

"Is supported by in-game testing with explicit numbers and test parameters." – As I've said elsewhere, how can anyone know if a.) tests were actually done, b.) tests were done in a manner that ensures that the perceived results are accurate, and c.) that the results are actually what the person is claiming them to be? There is nothing special about BSN members—they are random people on the internet, not people who have actually programmed the game. Their word should not be taken as undeniable fact just because they say so.

"Must be the the last or latest discussion on the matter." – This is basically saying that earlier opinions are less valid based solely on the fact that people have expressed other opinions later on.

"There must be general consensus in the follow-up discussion, which implies that... The discussion must be a reasonably high-trafficked thread." – This part simply boils down to "if a bunch of people think it's right, then it must be". The opinions of a few people do not dictate reality.

"Non-BSN sites cannot be conditionally valid, as other sites do not receive as much attention for gameplay mechanics discussion, and generally any non-BSN discussion on gameplay mechanics should still source a BSN discussion." – Users on other forums are random people on the internet. Who are BSN users? Random people on the internet. People are not automatically more trustworthy based on where they post their opinions.

This policy boils down to taking the word of random people on the internet for no good reason—taking their word that they've actually run tests, taking their word on these extremely precise numbers they're claiming, taking their word that they are basing their information off of game data, taking their word on factors that are impossible to accurately measure in-game. What someone claims about their opinions or where they post their opinions does not negate the need for solid proof of information that is posted on the wiki. LilyheartsLiara (talk) 19:12, May 14, 2013 (UTC)


 * 1. "Their word should not be taken as undeniable fact just because they say so." We aren't. I don't know if you're representative of editors on the ME wiki, but have we forgotten what wikis are? The whole idea of a wiki is a source of knowledge that anyone can edit, and in so doing eventually we attain a general consensus of "truth." On wikipedia itself, this is referred to as NPOV (neutral point of view). So no, we don't take their word for it, but visibility of test data, etc. allows for verifiability and falsifiability. This is how you learn things, not just for gameplay mechanics (since literally no original IP computer game has completely disclosed 100% of their mechanics) but in knowledge in general.


 * 2. "This is basically saying that earlier opinions are less valid based solely on the fact that people have expressed other opinions later on." No, this is not. This is to account for the fact that discussions are ongoing, and pulling out an early test run of data for something ignores later test runs that have been done with corrected hypotheses or improved parameters.


 * 3. "This part simply boils down to "if a bunch of people think it's right, then it must be"." No, it doesn't. This is basically peer review. This is how scientific theories are made. IF someone posts a bunch of test data and no one comments then we have no verifiability of its accuracy. If, however, someone posts test data, other people talk about it, find flaws in it, corrections are made, then we are better able to ascertain its accuracy.


 * 4. "Users on other forums are random people on the internet. Who are BSN users?" BSN users are the ones actively making gameplay mechanics discussions. BSN is also where official bioware reps live, so it becomes convenient for BSN users to quickly link to devconfirmed equations and numbers. BSN is also significantly more heavily trafficked than yahoo answers.


 * 5. "This policy boils down to taking the word of random people on the internet for no good reason—taking their word that they've actually run tests, taking their word on these extremely precise numbers they're claiming, taking their word that they are basing their information off of game data, taking their word on factors that are impossible to accurately measure in-game. What someone claims about their opinions or where they post their opinions does not negate the need for solid proof of information that is posted on the wiki." No, it doesn't. The policy explicitly stipulates mechanisms to verify the data that is being posed (either by test data or self-evidentness). Refusal or lack of desire to actually test the data does not make the data incorrect. (Thelee (talk) 19:31, May 14, 2013 (UTC))
 * "The whole idea of a wiki is a source of knowledge that anyone can edit, and in so doing eventually we attain a general consensus of 'truth.'" – Except how can you have a consensus on "truth" when it is impossible for people to actually measure these things? Does "consensus" mean that you should ignore any nay-sayers?


 * "This is to account for the fact that discussions are ongoing, and pulling out an early test run of data for something ignores later test runs that have been done with corrected hypotheses or improved parameters." – And what guarantees that later tests wouldn't have good hypotheses or poor parameters? What guarantees that a later test is invariably more trustworthy than a later test?


 * "No, it doesn't. This is basically peer review. This is how scientific theories are made." – Okay, what proof is there that people who say that someone on BSN is right have actually done any testing of their own?


 * "BSN users are the ones actively making gameplay mechanics discussions. BSN is also where official bioware reps live, so it becomes convenient for BSN users to quickly link to devconfirmed equations and numbers." – Anyone on the internet can discuss gameplay mechanics in any forum. Anyone on the internet can access developer-confirmed data on the BSN. This argument boils down to "BSN is better than everyone else because".


 * "No, it doesn't. The policy explicitly stipulates mechanisms to verify the data that is being posed (either by test data or self-evidentness). Refusal or lack of desire to actually test the data does not make the data incorrect." – And the sources for the information you've attempted to add doesn't do any of that. Furthermore, it is simply impossible to test the data because it is impossible to measure things in-game to the degree of precision that these threads claim. It is simply impossible to determine that power combos do x(A) damage on (B) difficulty because there is nothing in the game that displays enemy health to the degree of precision that you can say "yes, it definitely does x(A) damage on (B) difficulty, and certainly not x(A+0.00001) or x(A-0.00001)". LilyheartsLiara (talk) 20:00, May 14, 2013 (UTC)


 * 1. "Except how can you have a consensus on "truth" when it is impossible for people to actually measure these things? Does "consensus" mean that you should ignore any nay-sayers?" It's not impossible. Why is it impossible?
 * 2. "And what guarantees that later tests wouldn't have good hypotheses or poor parameters? What guarantees that a later test is invariably more trustworthy than a later test?" There are no guarantees. This is why this is a wiki that people can edit and not an encyclopedia that we're about to publish in hardcover form.
 * 3. "Okay, what proof is there that people who say that someone on BSN is right have actually done any testing of their own?" Verifiability! If you don't think it's right or it doesn't sound right, you can verify it. And people do! That's why Overload (which keeps coming up as a topic of mine) has been through many test runs and various threads as people tweaked the parameters and their theories as to how it worked.
 * 4. "Anyone on the internet can discuss gameplay mechanics in any forum. Anyone on the internet can access developer-confirmed data on the BSN." I can weaken the policy position to be any community site that otherwise meets the requirements. Though from my own perusal, people on ign/gamespot/gamefaqs/xbox rarely ever actually test anything or provide test data for their theories or simply quote BSN.
 * 5. "Furthermore, it is simply impossible to test the data because it is impossible to measure things in-game to the degree of precision that these threads claim." I made this response on the talk page, but has anyone here actually played with the Mass Effect 3 console? (Thelee (talk) 20:18, May 14, 2013 (UTC))

Why should BSN be a reliable source? How are these tests made on the BSN? --Nord Ronnoc (talk) 19:28, May 14, 2013 (UTC)


 * BSN is "more reliable" because it is heavily trafficked. Which increases the likelihood that other people will verify data and offer corrections if things do not match up. (Thelee (talk) 19:32, May 14, 2013 (UTC))

I think what immediately crops up in this discussion is that no one seems to mind yet that there is literally information that is wrong or incomplete on the ME3 wiki. Sometimes it can be trivially shown that the information is wrong, sometimes you need test data to show that it is wrong, but the point is there is literally incorrect or wrong information that cannot be proven solely from devconfirmation. Editors are OK with that? (Thelee (talk) 19:33, May 14, 2013 (UTC))


 * Unsurprisingly, you're making a claim, not providing any evidence whatsoever, and insisting that people should simply take your word for it. LilyheartsLiara (talk) 20:00, May 14, 2013 (UTC)


 * Actually, I did provide evidence. Tell me, using this wiki and only devconfirmed sources, what the rank 6 evolution of Overload does. (Thelee (talk) 20:05, May 14, 2013 (UTC))


 * Or give me devconfirmed proof that the Fire Explosion power combo dot doesn't do anything (an edit I made that was not reverted). (Thelee (talk) 20:05, May 14, 2013 (UTC))


 * Or what about Snap Freeze's cryo explosion damage? (Thelee (talk) 20:06, May 14, 2013 (UTC))


 * Or the fact that there is no discussion (at least post-revert) about Neural Shock's mechanics with Overload? (Thelee (talk) 20:08, May 14, 2013 (UTC))


 * Using only this wiki/devconfirmed sources, how do headshots work? Do all weapons headshot? (Thelee (talk) 20:08, May 14, 2013 (UTC))


 * Not to mention that virtually all discussion in Mass Effect 3 Multiplayer/Character Customization talks about stuff that is not devconfirmed anywhere. Shouldn't we be basically stripping all those pages of basically any discussion of strategy? (Thelee (talk) 20:11, May 14, 2013 (UTC))


 * Either you don't grasp the difference between subjective statements regarding gameplay strategies and objective statements regarding gameplay mechanics, or you're just disregarding the difference. LilyheartsLiara (talk) 20:18, May 14, 2013 (UTC)


 * Many of the subjective strategy suggestions assume unverifiable claims about how certain powers and weapons work. Such as "The geth's light melee strike is actually slower in execution than the heavy melee shield pulse. The heavy melee also affects a larger radius and inflicts more damage. It is generally better to use the shield pulse." Where's the devconfirm on that? Or for "A maximum of 13 Grenades can be carried: the Demolisher benefits immensely from Gear Packs that increase the number of grenades that a player can carry, in addition to choosing all available Extra Grenade evolution. This also applies to your grenade count bonuses from powers. For instance, having both grenade powers at rank 2 will give you two extra grenades." Where's the devconfirm for that? Obviously those stick around because those can be verified, right? Which is what I'm getting at. (Thelee (talk) 20:24, May 14, 2013 (UTC))
 * Those are statements that can be confirmed from in-game data and/or information on the wiki. There is absolutely no way that you can measure the exact radius of a Tech Burst in-game. There is absolutely no way that you can measure the exact damage modifiers of power combos based on difficulty. If there is, please explain, because the people who have come up with these numbers haven't bothered to give readers any reason to believe that they have some way of actually confirming these numbers. LilyheartsLiara (talk) 20:29, May 14, 2013 (UTC)
 * "and/or information on the wiki" i.e. other information on the wiki that has not been devconfirmed but is on there for reasons of trivial verifiability. It's not the exact radius of Tech Burst, hence a range. Exact damage modifiers are possible, using combination of console or just linear approximation of health bars or looking at game ini files (health modifiers) and linking it to devconfirmed info (that combos scale with health scaling). Oh and ".5833333" is not some random decimal, it's a fraction, so it's no "super precise" it's just the decimal representation of a fraction carried out to a specific point (referring to a comment you made on the talk page for the manual of style). (Thelee (talk) 20:42, May 14, 2013 (UTC))


 * And actually, now that you have reverted the DOT-based note on Power Combos, that description is now misleading at best, incorrect at worst. (Thelee (talk) 20:14, May 14, 2013 (UTC))

And to unpack another thing that Lily has brought up - anyone can verify the test data. The policy proposal is written to require only sources that have easily verifiable test data. And the wiki itself has information on console commands and links to a python script with an even more detailed list of more console commands. And regardless of who you are, your own personal indifference, refusal, lack of desire, or lack of technical ability to verify the data yourself does not make the data invalid. I mean, is any editor here constantly working to verify basic proofs of physics or math? Stuff like gravity doesn't cease to be true just because another scientist chooses not to personally verify the data. (Thelee (talk) 20:02, May 14, 2013 (UTC))


 * And the data can be interoperated in multiple ways. This kind of subjectivity can be problematic. --Nord Ronnoc (talk) 20:16, May 14, 2013 (UTC)


 * Some data can be subjective. But others, even if it is as wonky as frame rate analysis for measuring cooldown effects, is verifiable and has a mechanism for repeatability. And I have to reiterate this point again: what editor opposing this policy from a stance of subjectivity or unverifiability has actually played around with the Mass Effect 3 console? (Thelee (talk) 20:28, May 14, 2013 (UTC))
 * You still haven't told us how it's tested or measured. Why should BSN be reliable aside from traffic? --Nord Ronnoc (talk) 20:35, May 14, 2013 (UTC)
 * I obviously can't speak for everyone on BSN because everyone has their own methods. Some people use recordings and use their in-game listed frame rate and calculate how quick cooldowns or other effects operate (with a margin of error equal to the frame rate, ie if 20 fps than the margin of error is +- 1/20 of a second). Some people tweak numbers in the various config file to see what happens in game (such as the global speed modifier which is something like 1.075 for Gold and anyone can trivially edit it--as the poster did--to a setting like 3 and note how everything flows three times faster). Some people use the console to spit out statistics or tweak statistics (some of which are documented right here: PC Tweaks (Mass Effect 3) and others inside the linked .py script, like "Damage TargetName xxx" or "DebugHealth." Some others use editors/decompilers (http://www.youtube.com/watch?v=1IGWIk_V5MU or http://me3tweaks.blogspot.com/2013/01/mass-effect-3-advanced-multiplayer.html). Point is, the difference between a random BSN speculation of "I think X does Y" is different from "I believe X does Y and here is the data that shows that" because the latter gives you a mechanism for verifying that X does Y yourself. (Thelee (talk) 21:00, May 14, 2013 (UTC))

So I don't think the proposal as stated is likely to pass as currently stated, and I agree that as currently stated it could lead to negative consequences, but I'm sympathetic to the overall goal so I'm going to vote support. (Thelee did you forget to vote for your own proposal?) The place where I think there is wiggle room is when something can be shown to be independently verifiable - e.g. if you can post a set of steps I can do on my own Xbox or whatever and I can see that Overload definitely does less than x3 damage or whatever, then that seems in line with our evidence requirements. And maybe policy doesn't even need to change for that, since I often see independently verifiable listed as something important (e.g. it's why we don't accept private messages from developers but we do accept publicly available tweets). Cattlesquat (talk) 20:24, May 14, 2013 (UTC)

After re-reading the policy, I've decided I at least would be prepared to support it on the condition that one more requirement be added to the lists: a minimum of three wiki users, including the user making the initial claim, must verify the data for themselves (similar to our MoS requirement for listing gameplay bugs in articles). The current confirmation policy as it pertains to gameplay data is admittedly strict, and as the devs move on to new projects, getting their confirmation for existing games will only become more difficult. We could be facing a time when this kind of data (assuming it's correct, of course) will be lost to the wiki. I think that the three-user requirement combined with the original proposal would be an effective enough screen to ensure the gameplay data presented on the wiki is solid. Even if that fails and some bad info does get in (which I doubt will happen often as player attention shifts away from ME3), we still have the tried-and-true process of a user noticing, starting a talk page discussion, and the community checking and removing it. -- Commdor (Talk) 21:08, May 14, 2013 (UTC)


 * Am I allowed to edit the policy proposal after it's been created? Or should I wait until this is rejected and do a different one? (Thelee (talk) 21:10, May 14, 2013 (UTC))


 * The proposer is free to alter his or her policy proposal during the vote as long as the changes don't affect the policy's original intent (for instance, you can't change a policy about allowing image files of a new format in wiki articles to a policy about requiring all screenshots be from the Xbox 360 version of the games) and the change is pointed out in a comment in the Discussion section. It's also considered highly improper to change a policy during the last days of voting since there's not enough time to allow all existing respondents to evaluate it and change their votes if need be, but this is day one. -- Commdor (Talk) 21:26, May 14, 2013 (UTC)


 * I'm 100% behind Commdor's plan. Cattlesquat (talk) 21:17, May 14, 2013 (UTC)
 * I'm backing up Commdor's plan. Sounds like a fair compromise.--Nord Ronnoc (talk) 23:36, May 14, 2013 (UTC)


 * Actually, the three-wiki user confirmation might be a very high bar if we want to confirm things exactly and not just make sure that they "make sense." I'm guessing the majority of editors here do not have the PC version of the game (which enables you to toy around with configuration files, enable the console, etc.) which would make precise verifiability by multiple editors difficult, if not de facto impossible. Surely there's some other mechanism for verifying? Or just have three wiki users confirm that something "makes sense." (Thelee (talk) 21:18, May 14, 2013 (UTC))
 * Not everyone has it on the PC, yes. In the likelihood, a bug could be an isolated case for a single user (or exclusive to a console). The only bug I've encountered in my SP game was the Blue Suns glitch. I think it only occurs in a new game plus. And to answer your question of the lore vandalism, yes. When the PC version of the first game came out, there were speculation of a new race added in. However, it turned out to be false, but there was quite an edit war here, if I recall correctly.--Nord Ronnoc (talk) 23:36, May 14, 2013 (UTC)

On another note, why are so many editors here so "loss averse," i.e. so worried about wrong information filtering on to the wiki? Wrong information filters onto all wikis all the time, but they get corrected. If I pull up a random Fallout wikia weapon page and write in "THIS WEAPON WILL INSTAKILL DEATHCLAWS" it might stick around for a while, but someone will revert it because it is obviously verifiably untrue. Conversely, if I pull up http://fallout.wikia.com/wiki/Fallout:_New_Vegas_combat#Combat_damage and add in equations (as I actually have done long ago), others can verify it, make sense of it, pose questions, etc and as it stands it has stood the test of time because even if it is not 100% accurate, it is effectively accurate enough (since only Obsidian would know the precise equation). In more business parlance, it appears the ME wiki has a culture of the status quo instead of a "bias for action." Was the lore vandalism mentioned in the "speculation" section of the manual of style really that lastingly-traumatic? (Thelee (talk) 21:23, May 14, 2013 (UTC))


 * If Commdor's addendum is included then you may have my support, HOWEVER the fact remains that a lot of this data can only be obtained and verified via the PC version of the game. IMO opinion that actually lessens its relevance to the wikia, especially as I believe that the PC is also running different patches to the console versions, creating even more problems. I really appreciate the spirit of what you are trying to do here, my friend, but I don't not believe such data can be completely accurate across all versions of the game.
 * Also a minor issue: How will we be presenting this information? I don't particularly want to scroll through reams of numbers and tables when all I am looking for is what, for example, the options are on Incinerate's Rank 5. And I definitely don't want those same reams of notes on enemy and weapon pages. I would ask that such data be given separate pages if it is of significant length. Garhdo (talk) 23:43, May 14, 2013 (UTC)

Possible Conclusion
Alright guys, digging around, I'm awfully discouraged by the editing culture at the ME wiki.

Two primary BSN users are responsible for an overwhelming amount of the fact-finding for gameplay mechanics: peddroelmz and corlist. Both have had user pages here: peddroelm and corlist. Both have clearly documented how they've gone about finding their numbers. At least one Bioware developer has confirmed that their methodology is sound (direct memory scanning using external development tools) and accurate: http://social.bioware.com/forum/1/topic/343/index/14601238/11#14635887. And from the looks of corlist's user talk page, this wasn't enough for editors here and at least scared off peddroelm from further contributing to this site (see note on corlist's talk page about his "predecessor").

Combined with the flabbergasting--in fact, what I would even deem anti-intellectual--attitude amongst a few of the editors/voters here against the utility of verifiable, repeatable, and testable data and an inherently prejudical bias against any source of internet data (even despite the merits of that internet data being verifiable and repeatable) and in fact what I would consider "missing the point entirely of a wiki," I'm heavily discouraged. Can so few of you actually see how such a restrictive policy for gameplay mechanics edits is actually toxic to the reliability and completeness of this wiki? Wild gesticulative fan speculation about lore/canon is one thing, but how can so few of you see the difference between that and the numerous threads on BSN (and in fact an entire google doc at https://sites.google.com/site/me3mpd/home) that have users repeatedly testing their theories in-game (not just in the abstract) and opening it up for public discussion and comment? How many potential helpful editors has this manual of style policy killed off in the bud? I know at least I'm contemplating an exit. (Thelee (talk) 22:25, May 14, 2013 (UTC))


 * For the record, here's peddroelmz on BSN going into a bit more detail about how it works: http://social.bioware.com/forum/1/topic/343/index/14033200/3#14035292


 * As a programmer, I can attest that this would be fairly accurate, and I'm pretty sure that's also why Brenon Holmes (of Bioware) also confirmed that this is an accurate method (because he would also know of the technical merits).


 * And as for devs being effectively the "Word of God," corlist's user talk page links to this thread: http://social.bioware.com/forum/1/topic/343/index/14601238/10 (I linked to a specific post in this thread earlier). If you're not familiar, basically corlist was able to prove to the developers that they had a bug in ME3, because the developers' own basic profiling tools had bugs in them and it took apparent PMs between corlist and Brenon to figure out that Bioware had a bug in being able to detect a bug (and direct memory scanning is by definition more accurate than the home-brewed profilers that Bioware were using). I mean, if you don't consider that an accurate enough methodology when you're able to find bugs for the developers when their own tools don't see a bug, I don't know what to tell you. (Thelee (talk) 22:49, May 14, 2013 (UTC))


 * I hope you'll stay, contribute, and continue to argue for what you believe in. Cattlesquat (talk) 22:54, May 14, 2013 (UTC)

Sorry to walk in kind of clueless-sounding, but, don't we have some Users who are skilled in the arcane arts of digital analysis and game mechanics around here? Maybe we can take the information that a BSN individual discovers, run it by our tech-savvy brethren around here, and place it into the appropriate articles accompanied with an appropriate amount of references to the original BSN gentleman/gentlewoman? Could that work? I know it seems rather. . . dis-trustful, towards the BSNittes (By the way, what is BSN exactly, Bioware Social Network?) but I'm not really seeing anything positive popping out of here beyond Administrator squat and Com's ideas, just seems like a straight steam-roll over Mr. Thelee's first proposal. Thoughts about such a mutual partnership? (talk) 22:56, May 14, 2013 (UTC)