Futuremark & NVIDIA Make Nice

Futuremark has released a statement to follow-up on the recent controversy over whether NVIDIA was cheating on their drivers to provide better 3DMark03 scores. The statement was sent along to a couple of sites (Bjorn3D was the first to send word), and while the conclusion states: "However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source," the main thrust of the release says: "Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat ."
View : : :
23 Replies. 2 pages. Viewing page 1.
Newer [  1  2  ] Older
23.
 
Re: Slanderous
Jun 3, 2003, 23:16
23.
Re: Slanderous Jun 3, 2003, 23:16
Jun 3, 2003, 23:16
 
I'd check this out for details:

http://www.futuremark.com/companyinfo/3dmark03_audit_report.pdf


It's painfully obvious that they _were_ cheating. There's no room for interpretation here. It seems ATI did a bit of cheating too, that part is quite toned down (hmm, wonder if it has anything to do with ATI being in their beta program).

In any case I couldn't care less about 3dmark (or any other benchmark for that matter). A company creating a benchmark, that earns a good chunk of money from one or more of the "target" companies just reeks IMO.

It doesn't matter if they actually do anything wrong or not in the end, for me their credibility (the tiny fraction that was there to begin with) is straight out the window.


22.
 
Re: Slanderous
Jun 3, 2003, 15:38
22.
Re: Slanderous Jun 3, 2003, 15:38
Jun 3, 2003, 15:38
 
That's some real selective quoting Anvil.

Well, he did refer to the part that you quote. The question he asked is which did nvidia do? The kind that is ok, or the kind that isn't ok?

"The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts." -- Bertrand Russell (I think...)
Avatar 9540
21.
 
Re: Slanderous
Jun 3, 2003, 12:57
21.
Re: Slanderous Jun 3, 2003, 12:57
Jun 3, 2003, 12:57
 
That's some real selective quoting Anvil. Here's the actual post: http://slashdot.org/comments.pl?sid=65617&cid=6051216

I noticed you left this part out in particular:

Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible.

-B

This comment was edited on Jun 3, 13:04.
20.
 
Re: Slanderous
Jun 3, 2003, 10:22
20.
Re: Slanderous Jun 3, 2003, 10:22
Jun 3, 2003, 10:22
 
Mind posting a URL, I seem to have missed this.
Although it is in the most popular threads section

JC says
When the output goes to a normal 32 bit framebuffer, as all current tests do, it is possible for Nvidia to analyze data flow from textures, constants, and attributes, and change many 32 bit operations to 16 or even 12 bit operations with absolutely no loss of quality or functionality. This is completely acceptable, and will benefit all applications, but will almost certainly induce hard to find bugs in the shader compiler.

He also says what is unacceptable. At no point does he say what it is that Nvidia have done.

Anvil
Anvil - from the land of warm beer and mad cattle.
19.
 
Re: Slanderous
Jun 3, 2003, 10:16
19.
Re: Slanderous Jun 3, 2003, 10:16
Jun 3, 2003, 10:16
 
You're wrong, Carmack did clearly state Nvidia cheated. Reread his post.

Mind posting a URL, I seem to have missed this.

This is a programmer's issue, I wished non-programmers would stop trying to do their own technical analysis.

I'm afraid I disagree, even though I program stuff. The technical question is "Has x piece of software been optimized for y application". It appears that the answer to this is unequivocably Yes.

the question "Does optimising x for y amount to a cheat " is essentially a moral one, not a technical one. It doesn't mean that programmers can't have a view, but it also means that consumers can have a legitimate and informed view as well.

Finally, we know from JC's .plans and elsewhere that DOOM , and indeed earlier iD games, had different code paths for different hardware. It's reasonable to assume that other software companies do this as well _ I'm almost 100% sure it applies to the Unreal engine for example. This isn't cheating in my book. So when we then come to benchmarks which have as a stated aim to indicate how current applications will perform on different hardware do we subject all hardware to the same code and see how it works best, or do we allow each software provider to optimise their offering in the same way as development companies would. How you answer this question determines wether you think NVIDIA "cheated", but bothe answers are perfectly reasonable.




Anvil
Anvil - from the land of warm beer and mad cattle.
18.
 
Re: Damn straight they cheated!
Jun 3, 2003, 08:15
18.
Re: Damn straight they cheated! Jun 3, 2003, 08:15
Jun 3, 2003, 08:15
 
..This "about face" by futuremark is the result of one thing and one thing only..

Lawyers, and good expensive ones. The kind a huge company like Nvidia can sick on a small company like Futuremark, period. What other possible explaination can thier be? The mud slinging has been particularly intense, with Futuremark releasing a patch (330) almost immediately after the discovery. Then we all wake up one day and they are all kissy kissy? I think not.

17.
 
Re: Damn straight they cheated!
Jun 3, 2003, 05:27
17.
Re: Damn straight they cheated! Jun 3, 2003, 05:27
Jun 3, 2003, 05:27
 
Of course, if you were Nvidia, you'd hardcode in some clip planes around the horse to hide it, some new shaders to take some shortcuts in displaying the horse, then accuse the stables of unfairly expecting them to fully display the horse honestly, only to be followed with a 24% hit in rendering the horse after someone comes along and disables your cheats.

16.
 
Re: Damn straight they cheated!
Jun 3, 2003, 05:03
16.
Re: Damn straight they cheated! Jun 3, 2003, 05:03
Jun 3, 2003, 05:03
 
It's not dead, it just needs new drivers.
I can probably get another 3 furlongs/fortnight with 44.03...


Avatar 13987
15.
 
Re: Damn straight they cheated!
Jun 3, 2003, 04:58
15.
Re: Damn straight they cheated! Jun 3, 2003, 04:58
Jun 3, 2003, 04:58
 
Give it up UnderLord, you are beating a dead horse.

[beating it off no less]

14.
 
Damn straight they cheated!
Jun 3, 2003, 04:34
14.
Damn straight they cheated! Jun 3, 2003, 04:34
Jun 3, 2003, 04:34
 
So you create a Hardware Abstraction Layer and the DirectX API, specifically so that the app' can be unaware of the hardware. Then the Hardware guys make a mockery of it by specifically being aware of the application for the sole purpose of getting better performance scores, EVEN THOUGH they produce inferior images than they would if 'un-optimized', and that's not cheating?
And if you're a newbie game developer, will Nvidia rush out driver patches to improve your game performance for free?

Avatar 13987
13.
 
Bullshit!
Jun 3, 2003, 04:05
13.
Bullshit! Jun 3, 2003, 04:05
Jun 3, 2003, 04:05
 
Nvidia and Futuremark are both talking complete crap. Drivers that produce a different result to what the application expects are [b]not[/b] optimisations. Not only that, the benchmark results are immediately invalidated because the cross-application result is contaminated by the so-called single-application optimisation. Do they think we were born yesterday? At the bottom of this deal is greed and corruption.
"Save the attitude for someone who cares." - Judge Dredd.
12.
 
Pie
Jun 3, 2003, 03:20
12.
Pie Jun 3, 2003, 03:20
Jun 3, 2003, 03:20
 
They're all pumpkin eaters.


Midget soothsayer robs bank. Small medium at large.
http://www.mythicarealms.com
"And then, suddenly and without warning, it turned into a real-life case of hungry, hungry hippos."
- Stephen Colbert
11.
 
Finally...
Jun 3, 2003, 02:44
11.
Finally... Jun 3, 2003, 02:44
Jun 3, 2003, 02:44
 
...has FutureMark understood that the performance of present and future games depends heavily on optimization and not as they believed in standard code like the one measured in 3dmark03.

As for CPUs, optimizations is the thing to do if you want to get the most out of the hardware (like with SSE2 for the P4s). In any case you won't play the unoptimized 3dmark03 but you will play optimized games and games is all that matters...

This will be a big benefit for nvidia since optimizations is the aspect where they shine but I'm just wondering the FutureMark will release a 3dmark03 second edition where hardware vendor specific code changes are allowed...

Nevertheless the problem of comparing FP24 (ATI) to FP16/FP32 (nvidia) still remains and needs to be adressed in order to make this benchmark a fair one for all graphiccard hardware manufactures...

10.
 
Re: Slanderous
Jun 3, 2003, 02:16
10.
Re: Slanderous Jun 3, 2003, 02:16
Jun 3, 2003, 02:16
 
From the sound of it, everyone cheats on the benchmarks. It's just that Futuremark is ok with it as long as you're paying them.

"The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts." -- Bertrand Russell (I think...)
Avatar 9540
9.
 
Re: Slanderous
Jun 3, 2003, 01:31
9.
Re: Slanderous Jun 3, 2003, 01:31
Jun 3, 2003, 01:31
 
You're wrong, Carmack did clearly state Nvidia cheated. Reread his post.

This is a programmer's issue, I wished non-programmers would stop trying to do their own technical analysis.

8.
 
Re: Slanderous
Jun 3, 2003, 01:13
indiv
 
8.
Re: Slanderous Jun 3, 2003, 01:13
Jun 3, 2003, 01:13
 indiv
 
No, it was more like:

Futuremark: "Nvidia cheated!"
Nvidia: "We'll look into it, but we think it's probably a driver bug."
Futuremark: "No, Nvidia cheated!"
John Carmack: "Nvidia didn't cheat."
Futuremark: "Ok, Nvidia didn't cheat."

Chalk up another goal for the Carmack!

7.
 
yeah..
Jun 3, 2003, 01:03
7.
yeah.. Jun 3, 2003, 01:03
Jun 3, 2003, 01:03
 
and ati admitted to damnear the same thing.

6.
 
Damn
Jun 3, 2003, 00:50
6.
Damn Jun 3, 2003, 00:50
Jun 3, 2003, 00:50
 
Always depressing to see someone you know is in the right have to apologize to someone else because the other person has more money than valor.

5.
 
Re: NVIDIA
Jun 3, 2003, 00:35
5.
Re: NVIDIA Jun 3, 2003, 00:35
Jun 3, 2003, 00:35
 
actually they were in the beta, they pulled out like last moment or somesuch.

but thats all moot, nvidia makes fast cards. so does ati

i buy the cheap ones.

since you are waiting for it DSR, -->bash, slander, and a mom joke...

Doin' it Big
4.
 
NVIDIA
Jun 3, 2003, 00:22
4.
NVIDIA Jun 3, 2003, 00:22
Jun 3, 2003, 00:22
 
The jury is still out for me on this one, but being the naive rainbow-eyed ball of happiness that I am, I don't think NVIDIA cheated.

After all, they weren't in the Futuremark beta, which they would have needed to really understand how the benchmark works (alrhough I'm not sure how easy those are to reverse engineer). The way the detections worked were such a generic texturing catch that it would be impractical to implement such a snag just to hurry a benchmark.

Again, these are just my first impressions, and the inevitable investigative follow-ups will let me form a more accurate decision, but I've always had faith in NVIDIA as a company, and I hope they continue to produce those purty cards that make my games so fancifully tantalizing

-DSR <--waits for the 20 some odd posts asking him if he works for NVIDIA and then proceeding to bash him even though he doesn't.

23 Replies. 2 pages. Viewing page 1.
Newer [  1  2  ] Older