TheKeyboardDemon

@Geno said:

3DMark 11 is a graphics benchmark. The reason why the results are so markedly different is because on Extreme settings the GPU becomes the bottleneck, equalizing CPU performance. On Performance, the CPU are the bottlenecks and that's where the results delineate. I'm not sure why HH tested with Extreme, as they're usually a fairly reliable review site, but perhaps they're not used to testing CPUs. Some amount of random deviation accounts for the differences in results from different websites, but the main reason is because of different testing methodologies. Different sites will use different methods with different system setups with different programs. Some of the testing methodologies, like the HH example given above, are erroneous or unmeaningful. Generally for CPU benchmarks you want to look at tests with a) base clocks, b) mix of multi and single threaded apps, and c) minimum setting game benchmarks. I chose the HC example previously because HC has testing methodology down pat for both CPU and GPU, always testing for the most relevant areas.

I was talking specifically about the tests they ran on the CPU, with 3d Mark they usually only look at the Physics scores to see how well the CPU does, but when tested at Extreme this seems to even the results favouring the Bulldozer by putting it closer to the i7, at first glance I coudn't see that. But when I looked again more closely I could see that it is getting 1782 points in a physics test where my Phenom II X4 965BE was getting around 4000 I started wondering why, this is one of those things that I had been missing, I usually look at the results and assume that the tests were conducted in a fair, open and honest way, but it looks like some sites are cherry picking results to make the Bulldozer look better than it really is and not, as I was thinking, doing it the other way round to favour the competition. Before I noticed this I did notice how the scores on Hardware Heaven had the i7 getting 1762 and the Bulldozer was at 1782 all I saw was a 20 point difference, which is close right? Then on the Hardware Canucks the Bulldozer is behind by a shade under 2000 points, why? Because they ran the test at Performance settings and I guess I really wanted the Hardware Heaven site to be right, but I don't think you can put a difference as big as 4600+ points down to random deviation, and SlasherMan has already tried to point out how some of the findings on Hardware Heaven were not

@SlasherMan: I now see why you think we should disregard the Hardware Heaven tests, their approach is flawed and inconsistent when examined in the context of how other sites have tested the same products. I'm just glad I'm not in a hurry to buy a CPU at the moment, my rig is running fine as it is and I was not planning any upgrades until after Ivy Bridge hits the shelves, there was a time when I would have seen the Hardware Heaven review and then just bought the CPU.

@Ghooble said:

@TheKeyboardDemon: Well regarding your question about how different people running the same tests can be getting different results is because each cpu off the line has its own variations. Generally they aren't too extreme but they're there. I wish there was a 100% clone of the original for every single one but that wish I'm certain won't come true. So when you go to test something, even if you're using the same motherboard, ram, and cpu if they all have their mini variations it could stack to make a noticeable difference. At least that's what I assume is rearing its head here.

-Ghooble

I think that is only part of the problem, another part will no doubt be the software, some coding just suits some CPUs better than other coding. The same is true of some games, Crysis 2 for instance runs better on nVidia cards than it does on ATI cards.

The biggest issue is the spread of results and the different means in which the CPU was tested, Hardware Heaven did the CPU benchmark on 3D Mark 11 at Extreme settings while Hardware Canucks did the test at Performance settings, Hardware Heaven shows the CPUs as being almost level with a 20 point difference, Hardware Canucks puts the 2600k about 2000 points about the FX8150. And that's just one example, Benchmarks that I used to rely on are starting to look completely random, adding to the general confusion, so right now the only thing that's really clear is that the Bulldozer has not met with expectations, I'm not sure that means it has completely failed, but fairly certain that it is not an outright success either.

@SeriouslyNow said:

@TheKeyboardDemon: Not buying it. On the one hand you're the guy talking about 'heavily multi-threaded tests' and on the other you're saying you need these things explained to you. Both can't be true.

Well yes, I was talking about heavily multi-threaded tests because that's what I understood the chip was made for, I was also confused about why being tested for single threaded operations was important, at least until SlasherMan pointed out how the average home user uses their PC at home and how this means that single threaded operations need a greater weighting in reviews than I had thought they needed. I can't actually think of any single threaded applications that I use where performance is an issue, I use MS Office for many tasks but I have found this to run well enough even on Netbooks, I browse the web using Firefox and IE and find this only occasionally gives me grief when the flash player crashes, I use Adobe Reader, iTunes, Mumble, I'm just starting to learn about Photoshop with my wife's student licence for CS5.1 (which is multithreaded) and I play games (my wife says too many, I say not enough!) and I plan to try using Pinnacle video editing software at some point, just to learn about it.

I'm still confused over how several people can essentially be running tests for the same things and be getting different results, I'd like to believe that some results are not being externally influenced to make it look better for one camp or another but I get the impression that some people think this. I can see what SlasherMan was saying about the HardwareHeaven and Legit Reviews tests, even about the CPU/GPU balance side of things as clearly tests have not been run at enough resolution settings (I can't believe that this has been down to time constraints either). But the thing that I am most confused about is how I can look at the same tests as everyone else and not see something that they can see, so I can only assume this means there is a gap in my knowledge, which frustrates me because I can't work out what I need to focus on to find the missing link.

@SeriouslyNow: @SlasherMan: Like a lot of people, I base my knowledge on the end results, I have never really understood what the benchmarks actually mean, so often it is the final round up results that matter more to me, on Tom's Hardware for instance, they have various comparitive tests the one I use most is the one that combines all the results into a single ranking and then looks at the relative performance which up until my last build worked fine, not so much on this build though. So in answer to the question, 'What more do I want?', what I really want is a balanced look at where the new CPU is doing well, where it is not doing well and how this will make or break a new build should someone go with it, and seeing someone say this is no good and then not explaining why doesn't help me get that, while reviews that show good performance in areas where other reviews show bad performance are just as confusing.

@SlasherMan: Your answers to my points in your last reponse have given more than the rest of thread and all of the 'random' benchmarks or commentaries from other forums, up until that I was starting to think that you just wanted to bash AMD or hate the platform, it doesn't look that way anymore.

@SlasherMan said:

This is precisely the conclusion every other website got to as long as we're talking about heavily multithreaded applications! Notice how every app the LegitReviews article used was an application that used 100% of all cores. And even then it's not always guaranteed to be consistent, just look at the x.264 benchmarks. Not only that, but the 2500k wasn't even included in every benchmark, which seems rather odd. It was there in some, missing in others.

I'm not sure I understand your point, AMD release an 8 core CPU, Legit reviews conduct tests with heavily multithreaded applications to see how well the 8 cores perform together and you're saying what exactly? That this is a bad test? Surely the fact that it only did well in some tests and not all tests highlights both the strengths and weaknesses of the CPU/platform, or are you saying that we should ignore all the tests where the CPU/platform did well and just focus on those where it did badly?

@SlasherMan said:

The HardOCP benches are of games stressing the GPU, notice the choice of resolution and game settings. Granted, those are the more realistic scenarios one would probably face in a GPU-heavy game, however, you may as well be testing GPUs there. There is a reason why most sites test at lower settings and resolution, that's where CPU bottlenecks show most prominently and that's how you can tell which CPU is actually faster. You don't crank the AA and resolution and expect to still see the difference in CPU performance, do you?

Here you are saying that tests conducted in the 'more realistic scenarios' are not realistic despite then saying that this is the way the majority of gamers will be playing their games and to get a realistic measure of performance the tests should be conducted using settings that were only really relevent when we were playing on 15" CRT monitors not 22" or bigger 1080p monitors or moitors that go up to 2560x1600 resolutiions.

@SlasherMan said:

And that last one, HardwareHeaven, are they for real? Not only is it the same thing with games as HardOCP, but they're a little bit lacking in the number of processors compared (no Thuban, no 2500k, etc..).

I can tell you didn't read any of the articles I linked, I mean not even the introductions or the conclusions, these are the bits I read first by the way, if you had you would have seen this in the introduction to the article:

Today we get to see for the first time how the top model in AMDs new FX range of CPUs, previously known as Bulldozer, performs against the existing Intel equivalent as we put the two CPUs head to head.

@SlasherMan said:

@TheKeyboardDemon said:

Now it looks like the marketing blurb might have had more to it than we had prevously thought!

Not really. Sorry.

What do you mean not really? Do you mean once we disregard every good result that the platform has achieved? Or do you genuinely think that there is not one single area in which this platform can actually meet with expectations?

Personally I have yet to make my mind up, I really want to know what these new CPUs and the Scorpius platform are truly capable of, I want to understand this so that when Ivy Bridge comes out in a few months time I will see how Intel have improved on their platform and I will be asking the same questions and looking at the same tests for answers, I'm assuming that I should ignore the tests where the Ivy Bridge platform does well as no doubt they will be flawed just like the Bulldozer tests. However I will want to know about multicore/multithread performance, I'm interested in the results from FPU and video encoding tests, in seeing CPU utilisation across everyday tasks and for me the most important thing that I want from a gaming platform is gaming performance that tests the hardware in the same way I want to use it.

@Deathmachine117 said:

@Evilmetal: I DONT get WHY some of YOUR words are in capitals WHILE others are not. Are you trying to EMPHASIZE a point ?

It's because of those capitals that I didn't read it all, I mean wtf is he shouting for.

@AlisterCat said:

My thoughts are, I just bought an i7 2600K. That is all.

I think that was a bad move on your part, firstly the 2700k will be out very soon and secondly this is likely to have an impact on the 2600k price, lastly I have just seen reviews of the Bulldozer that has achieved some very different results. Now every review so far has relied on the review pack sent by AMD, this meant that they were using a 990 based mobo, an FX8150 and an nVidia GTX580 and not surprisingly they have all beed getting the same results. The guys at Hardware heaven used a different motherboard, they used a Sapphire HD6950 and 1866mhz ram and in many of the tests where the FX8150 had been losing it was either beating or matching the 2500k and also the 2600k in some of the tests as well.

These reviews seem to show the FX8150 in a somewhat better light, they have opted to test the CPU by electing to use their own choice of mobo and GPU instead of the GTX580 and Asus mobo sent to them by AMD, these were actaully posted on another forum so all credit to the OP for finding these, here they are:

Harware Heaven

Hard[OCP]

Legit Reviews

Now it looks like the marketing blurb might have had more to it than we had prevously thought!

@SeriouslyNow said:

I'm no fanboy but the failure of Bulldozer has been well publicized. That was my point. And frankly, the way you shaped the discussion from the start seems to indicate that you were trying to deny reality.

I think you have a good point, by adding the 'record breaking' part in to the question it does kind of unbalance any views people might bring in to the thread. I think a lead into a discussion should have had as many negatives points in there as it does positives or better still the question should be neutrally phrased with no indications of preference so as to let the participants make their own minds up.

@SlasherMan said:

I get where you're coming from, but if I'm honest and a bit blunt, you're grasping at straws here.

Nothing but marketing and PR. "Hey, buy matching AMD parts from AMD!" I will bet you anything that the end result is no different regardless of what type of GPU is being used (and as I've previously shown, even with their 1866 RAM spec). This isn't some MMO where a matching set nets you bonus stats, hardware doesn't work that way. Neither Dragon nor Leo magically worked miracles for Phenom II, this isn't going to be any different.

I was rooting for AMD all along, I wanted them to be back in the game big time, I wanted to be able to recommend AMD CPUs again for something other than budget machines, but denying that this is anything but a disappointment for almost everyone is denying facts at this point. They will seriously have to reconsider their pricing structure if they actually want to compete.

Marketing or not, if someone makes a claim about their hardware, that claim should be tested, if there is a new HDD that a company claims will outperform an SSD on a 6gb/s connection you wouldn't expect reviewers to test it against another HDD on a 3gb/s connection, they would test the claim fully by testing all of the components within the 'marketing' specifications, then when the product fails or succeeds it is the product that was tested and not the guy in the marketing office that came up with the spin. On the other hand if they didn't follow the marketing guys spin then he will say, yeah of course those bad results are just because they didn't test it properly or he will claim victory if the test make his product look good.

@Ghooble said:

@TheKeyboardDemon: God I hope so because my heart sank when I read the reviews

-Ghooble

I hope so too, but at the moment, I think I have to agree with SlasherMan, I think that AMD have put more efford into delivering marketing spin then they have into delivering new ground breaking technology, right now all the benchmarks I have seen indicate that Bulldozer has a lower IPC than previous gen CPUs and that the only way to get round this is by overclocking the CPU, look at the massive leap in the results from even modest overclocks to see this for yourself. This means the FX8170 might be better, but I thnk the next gen of this chip is where we will see the design properly.

I'm not going to cancel my Pre-order, I really enjoyed the Beta, knowing it was a Beta.

Thanks, but I couldn't get any to work, I might be in the wrong country!