Doctor Archive

---

Obata wrote:

I started a thread on the medic forum about this (this was the bi-weekly medic question). My post includes data from a study I did to determine the affect of crafting station and tool ratings. In a nutshell: my droid gave the worst results, my mid-rated station gave the best, and a positive rated tool was better than a negative. If you want to see the numbers, read the post.

Message Edited by Obata on 04-27-2004 11:42 AM

---

I read the post, and you'll forgive me for pointing this out, but that data is not at all conclusive.

100 samples is but a drop in the bucket compared to the actual number of experiments being performed across all servers on a daily (or even hourly) basis. Any trend noticed in such a limited sample would have to be suspect. You might do 1,000 samples and find 4 trends that agree with your posted results, and 6 that don't. Take it out to 10,000 samples and it might swing the other way.

Also, your data doesn't include any mention of the results of assembly, which can also be anywhere from failed to amazing.

It's an interesting experiment, but it does not at all -prove- what effect the ratings of stations or tools actually have on the results.

---

Obata wrote:
All I needed was enough samples to 'smooth' the random rolls.

---

And there is the flaw: How many samples does it take to "smooth" or "average out" the effect of the randomness? One-hundred? One-million?

That depends on the degree of randomness that the system introduces.

And while the code which they have written to govern that is running on a computer and thus theoretically "predictable", it is also possible, and more importantly probable that the degree of randomness is (due to the very ease of doing so with a computer program) enormous.

Obviously, if they were limiting themselves to a "1d 6" or "2d 20" type of roll model (pun intended ), then it would be much easier to define the degree of randomness.

Are they using such a simplified random generator? I don't know (and I suspect neither do you) but I, for one, highly doubt it.

---

bioshock wrote:


And there is the flaw: How many samples does it take to "smooth" or "average out" the effect of the randomness? One-hundred? One-million?

That depends on the degree of randomness that the system introduces.

---

Obviously. Let's consider the difference in the ratings of the stations I used. The 'good' station is rated about 65% higher than the next best station. Assuming that the rating is a multiplier in the formula (a safe assumption,I would think) the average result with the high rated station should be 65% better than the low rated one, so long as the other variables are controlled. The 100 samples that I did with each station show that this is not the case. Of course more sample would smooth the variability even more, but it is highley unlikely that the conclusion would be any different. If the ratings are factoring in the way they

---

Obata wrote:
Assuming that the rating is a multiplier in the formula (a safe assumption,I would think) the average result with the high rated station should be 65% better than the low rated one, so long as the other variables are controlled. The 100 samples that I did with each station show that this is not the case.

---

Which only proves that your assumption was wrong.




Which is fine, since I don't think your assumption that the number is used as a multiplier is as safe as you seem to think. The rating could be applied as an additive just as easily. Or it could be a multiplier which is dependant on other influences, which seems likely to me.

Depending on how they built the formula, the station rating could have a A) a large effect, B) a small effect, or C) no noticable effect except over a very large number of samples.

I happen to think C is what is actually happening, and your experiment of experimenting provides nothing to counter that opinion.

---

Obata wrote:
Look at it this way, if 100 samples is not enough to demonstrate a trend, then the magnitude of the randomness is so high that the ratings are negligible.

---

As I believe they are.

---

bioshock wrote:

---

Obata wrote:
Assuming that the rating is a multiplier in the formula (a safe assumption,I would think) the average result with the high rated station should be 65% better than the low rated one, so long as the other variables are controlled. The 100 samples that I did with each station show that this is not the case.

---

Which only proves that your assumption was wrong.

No, it proves only that the ratings are not having the affect that the devs said they do, and that was the question I was trying to answer.


Or it could be a multiplier which is dependant on other influences, which seems likely to me.

Certainly a strong possibility, and it would mean a smaller differential between stations, but would not explain a lower rated station having double the number of amazing successes.

---

What's next? ALL CAPS?

---

bioshock wrote:

---

Obata wrote:
Look at it this way, if 100 samples is not enough to demonstrate a trend, then the magnitude of the randomness is so high that the ratings are negligible.

---

As I believe they are.

---

As do I, now. Before doing this study, I would not have thought that. As crazy as it may be, I was willing to take the devs at their word until I see numbers that tell me otherwise.

---

Obata wrote:
Certainly a strong possibility, and it would mean a smaller differential between stations, but would not explain a lower rated station having double the number of amazing successes.

---

Having double the number of amazing successes - over a short duration.

Again, 100 samples is almost certainly not enough to reveal anything useful about trends. Yes, you got a trend where the lower station got more amazings, but over the course of 1,000 or 10,000 samples you might have (and probably would have) gotten very different results.

At any rate, since this is beginning to become a circular argument, and I'm having to repeat myself, I'm going to go ahead and bow out now.

Message Edited by Obata on 04-27-2004 10:35 PM

---

bioshock wrote:

Having double the number of amazing successes - over a short duration.

Again, 100 samples is almost certainly not enough to reveal anything useful about trends. Yes, you got a trend where the lower station got more amazings, but over the course of 1,000 or 10,000 samples you might have (and probably would have) gotten very different results.

---

I was 100 samples at EACHof 4 different points. If it doesn't convince you, then put in some work and do a study that will.

---

Obata wrote:

---

bioshock wrote:

Having double the number of amazing successes - over a short duration.

Again, 100 samples is almost certainly not enough to reveal anything useful about trends. Yes, you got a trend where the lower station got more amazings, but over the course of 1,000 or 10,000 samples you might have (and probably would have) gotten very different results.

---

I was 100 samples at EACHof 4 different points. If it doesn't convince you, then put in some work and do a study that will.

---

Seeking to convince others that my own personal wild-ssed guesses are correct isn't something that I'm willing to put much time into.

I leave the achievement of that goal to you.