DXOMark's Overall score is useless....

[SkaGoat]

I was replying in another thread, and instead of hijacking that thread, I decided to start a new one.

I must admit to start, I shoot Canon.

I've been reading a lot of press on the 7D Mark II, and few weeks ago I see the headlines of "7D Mark II image quality no better than Nikon D300s according to DXOMark".

DXOMark gives the Canon EOS 7D Mark II a overall score of 70.
The Nikon D300S is also a score of 70, and the Nikon D7100 has a overall score of 83. This is where most Nikon (and Sony) fan boys stop.

But from all the reading I had done up until this point touted how good the high ISO noise, and Dynamic Range were on the new 7D Mark II. I was confused, so I dug into the graphs.

First you have to take into account what the 7D Mark II was designed to do. It is designed to be a Sports/Wildlife camera. It's industry leading AF, mixed with a fast burst rate, and the 1.6x crop ratio make it ideal for shooting sports and animals. It was never designed to shoot landscapes, or portraits, and will most likely people who buy this camera will very rarely use it for these things.

The First graph is comparing the Signal to Noise ratio between the 7D Mark II, D300S, and D7100.

This graph clearly shows that not only does it have a higher max ISO than either camera, noise wise it is clearly better than the D300S, and it matches the D7100.

The next graph is the Dynamic Range between the 7D Mark II, D300S, and D7100.

It simly destroys the D300S for DR at all ISO except for base ISO, the 7D Mark II, being used for what it's designed to do will very rarely be used to shoot at base ISO. Base ISO is important for landscapes, getting a shot of that bald eagle during golden hour, with a 600mm f/5.6 lens is not going to be shot at base ISO. There is under 1 EV difference between the 7D Mark II, and the D7100 by the time you get the 1600 ISO. And again, the D300S is not even close to the 7D Mark II anywhere but base ISO.

Lets also remember the 7D Mark II is much more expensive, has a much faster burst rate, and a much better AF system than both the Nikon cameras.

DXOMark clearly rates base ISO DR highly in the algorithm it uses to come up with the overall score, my argument would be most users are not shooting base ISO.

DXOMark goes so far in their review to recommend users purchase the Sony A77 Mark II instead, there are pluses and minuses, Canon has a better lens line up, the Sony is faster, and cheaper, but it's AF isn't as good. It's also a little noisier. Here are a couple graphs

My point is basically the Canon receives a much lower score then it's contemporaries simply because they don't concentrate on base ISO noise performance. The 7D Mark II beats the A77 Mark II in IQ, but still receives a lower score.

 

[havoc315]

Excellent breakdown! As I said in the other thread, the graphs are helpful, the overall scores are useless.

In their overall scores, the D7100 is basically tied with the A77ii, and they are both well ahead of the 7dii.
Upon examination of the graphs, one can see that the A77ii and 7dii are actually the ones that should be tied. The D7100 does have some IQ advantages, but not nearly to the degree the scores would suggest.

As you accurately stated, DXO scores the IQ mostly at base ISO. The A77ii and D7100 both have noticeably better dynamic range than the 7Dii at base ISO --- And as you accurately stated, this isn't meaningful for all types of photographers. I do shoot some landscapes, and I therefore appreciate having a ton of dynamic range at base ISO. But once you move up just a little bit in the ISO range, the differences evaporate. And for sports and wildlife shooters, base ISO is rarely used.

You did leave out the 3rd prong in their scoring -- Color sensitivity, where the Nikon has a bit of an advantage through the entire range.

Still, these differences overall are minor.

It would suggest that the D7100 will indeed give you better IQ for landscapes, possibly better for portraits. But for the intended use, sports and wildlife -- there is no IQ advantage.

The comparison between the A77ii and 7dii is a very interesting one. They both have excellent AF systems -- One would need to see more real world testing, but the 7dii may be slightly better due to more cross points. But I suspect very similar AF performance.
The A77ii has a faster maximum frame rate -- 12fps vs 10fps. But for full exposure control, the A77ii is limited to 10fps, equal to the Canon.
They both have nice deep buffers for sports shooting.
As noted above, the A77ii does have an image quality advantage AT LOW ISO.. so in terms of multi-purposing the camera, the A77ii is potentially better than the 7Dii.
Canon has a much more complete set of full frame lenses than Sony, including more affordable options.
The A77ii is much cheaper than the 7Dii. Considering how similar they are in camera body performance, I do find the 7Dii overpriced and/or the A77ii a great bargain -- But as noted, one might do better buying Canon lenses.
Then the big OVF versus EVF debate -- For 90% of uses, I personally prefer an EVF. Being able to judge your white balance and exposure before the shot, being able to chimp your shots in the viewfinder, using the viewfinder for video... Lots and lots of advantages for the EVF of the A77ii. BUT, for the intended purpose of sports -- At high burst shooting, the EVF lags the action. Thus, tracking action at a high burst, is easier done with the 7Dii.

Truthfully, someone who doesn't have brand loyalty/brand investment, would be well served to check out both the 7Dii and the A77ii if they are looking for a sports/wildlife setup. Nikon simply doesn't have an APS-C camera that is really competitive in that category currently.

 

[Geoff_M]

"DXOMark's Overall score is useless...."

I disagree. It's one measure, and in order to be able to apply the number, you need to know what the number is... and isn't. As for your complaint that DXO overlooked the fact that the 7D is designed to be more of a sports/action camera, no they didn't. That's why they have the Sports metric that clearly shows that there is a high-ISO advantage to the Canon body over the D300s.

No one should buy a camera based on a single numeric evaluation.

 

[havoc315]

"DXOMark's Overall score is useless...."

I disagree. It's one measure, and in order to be able to apply the number, you need to know what the number is... and isn't. As for your complaint that DXO overlooked the fact that the 7D is designed to be more of a sports/action camera, no they didn't. That's why they have the Sports metric that clearly shows that there is a high-ISO advantage to the Canon body over the D300s.

No one should buy a camera based on a single numeric evaluation.

Problem is, for their "overall score," their choice of measurement is very misleading.
You're a hockey fan... What if the Rangers were playing the Red Wings..
In the first period, the Rangers outscore the Red Wings 2-1. In the second period, the Red Wings outscore the Rangers 1-0. And in the third period, the Red Wings out score the Rangers 3-1.

Game isn't even close -- the Red Wings won the game 5-2. But, now you ask DXO.. what was the score of the Rangers-Red Wings game.. And DXO tells you, "The Rangers won, 2-1."

Now, if you really fully understand how they do the scoring, you understand that the score is only reflecting the first period. But that seems a rather silly way to compute an "overall score."

I think that's the objection -- They call it an overall score, but it's actually a very very narrow score. Just a score of the first period of the game.

 

[SkaGoat]

I disagree. It's one measure, and in order to be able to apply the number, you need to know what the number is... and isn't. .

Which no one knows because DXOMark hasn't revealed the algorithm they use to come up with that number. The simple fact that it appears the overall score is so highly weighted to base ISO performance makes it useless, as hardly anyone shoots at ISO 100 all of the time.

 

[Geoff_M]

Which no one knows because DXOMark hasn't revealed the algorithm they use to come up with that number. The simple fact that it appears the overall score is so highly weighted to base ISO performance makes it useless, as hardly anyone shoots at ISO 100 all of the time.

Again, I disagree. It's one metric. Even DXOMark doesn't claim that it represents the be-all-and-end-all of evaluating a sensor:

To better use Sensor Scores:
- Identify your preferred use case: General purpose, Portrait, Landscape, and/or Sports.
- Choose the resolution that you need for the kind of photography you do.
- If you shoot in RAW, Sensor Scores will help you rank the best cameras according to the resolution and use case(s) that you have chosen.

As for their methodologies, they too are very open about it: An in-depth case study of the use of DxOMark data

No single metric algorithm is "perfect", which is one of the reasons that DXOMark examines and includes different use cases. But if you want to pick nits, you certainly can.