Mirrorless Camera ISO Signal/Noise Shootout: NEX5, NX10, E-P2, E-PL1, GF1, G2, GH1

Amin

Hall of Famer
Having compared the size of several different mirrorless interchangeable lens systems, we'll now take a look at how much detail relative to noise is captured at any given ISO by each of these systems.

Special thanks to
B&H Photo for providing the following cameras used in this comparison: Panasonic G2, Olympus E-PL1, Olympus E-P2, Sony NEX5, Panasonic GF1. The Panasonic GH1 is mine.

If you're reading this on the blog main page with resized images, click here to view this post with all images at full size.

Design and Methods:


  • The goal here was to show the relative potential of these systems, hence I will present only RAW conversions and not in-camera JPEGs. There are many sites with excellent in-camera JPEG comparisons.
  • All images were made using Sony, Samsung, and Panasonic (14-45mm) kit zooms at f/5, which was chosen because it represents a setting of good center sharpness with adequate depth of field for all systems.
  • All kit zooms were used at their widest setting, and the subject distance was constant. The Panasonic lens at 14mm actually has a slightly wider angle of view than the Sony/Samsung at 18mm, but the reverse is true after Lightroom correction of barrel distortion. For practical purposes, the diagonal angle of view was fixed.
  • Since the purpose here was to look at sensor rather than lens performance, only center crops from the within sharp focus portion of the image were used for comparison.
  • To present a valid comparison of detail relative to noise at any given ISO, exposure times were matched for all systems as follows: 1) ISO 100: 2s; 2) ISO 200: 1s; 3) ISO 400: 1/4s; 4) ISO 800: 1/4s; ISO 1600: 1/8s; ISO 3200: 1/15s; ISO 6400: 1/30s. Lighting sources were kept constant.
  • Each shot was done in triplicate to ensure reproducibility.
  • All system settings were image stabilization off, self-timer, sturdy tripod.
A few details about the RAW conversions:

  • RAW files were processed in Lightroom 3.2 RC, which supports all of the cameras tested.
  • Custom white balance was applied during RAW conversion, and each RAW file was white balanced off of the gray paper in the image. Due to my imprecise white balance matching and differences in the LR 3.2 color profiles for each camera, I could not match colors perfectly. The colors on the GH1 crops are particularly off and do not represent a sensor problem but rather a limitation of my testing methodology.
  • Some of these cameras differ in image brightness (in Lightroom) given the same nominal ISO, f-number, and shutter speed. In order to present as controlled a comparison as possible, "Auto tone" was used in Lightroom to more closely match the final brightness of each image.
  • Sharpening settings were Lightroom default (same for all cameras), and both luminance and color noise reduction were disabled entirely.
  • In order to provide as fair a comparison as possible, the Sony and Samsung images were downsized during export from Lightroom such that their diagonal image dimension would match that of the Micro 4/3 cameras. Showing all images at their native size would have penalized the Sony and Samsung for using more megapixels, while the chosen method is akin to comparing equal-sized prints from each camera.
  • RAW files provided for download have been converted to DNG for the sake of conserving storage/bandwidth.

Here is the test scene with yellow rectangles indicating the selections shown in the crop comparisons which follow:

Scene.jpg



ISO 100 crops (NEX 5 not shown since it has no ISO 100 setting):

Picture-25.jpg


Picture-24.jpg


Picture-23.jpg



ISO 100 RAW files:


ISO 200 crops:


Picture-22.jpg


Picture-21.jpg


Picture-20.jpg



ISO 200 RAW files:

 
The majority of color noise can be removed without impacting apparent detail. Let's look at some high ISO crops which have undergone color noise (but not luminance noise) removal using Noise Ninja and an "Auto Color" step in Photoshop to try for a slightly better color match.

In these comparisons, I am using the Panasonic G2 as representative of the G2/GF1/E-P2/E-PL1, which all share the same sensor and as shown above, produce similar results.

ISO 1600 crops:

1600-1.jpg


1600-2.jpg


1600-3.jpg



ISO 3200 crops:

3200-1.jpg


3200-2.jpg


3200-3.jpg



ISO 6400 crops:

6400-1.jpg


6400-2.jpg


6400-3.jpg
 
Analysis:

  • The NEX 5 has the best performance in terms of detail relative to noise. This is apparent even at low ISO, and the gap widens with increasing ISO.
  • At very high ISO, the NEX5 has a stop or more advantage over the G2/GF1/E-PL1/E-P2. You can see that by looking at these side-by-side crops (color noise removed):
one-stop.jpg


These observed results correlate well with data measured by DxOmark:


dxo.jpg


  • The Panasonic G2, GF1, E-PL1, and E-P2 show very similar* results. This is consistent with my understanding that all of these cameras use the same sensor (with minor differences in the low pass filter).
  • The Samsung NX10 is a little disappointing at very high ISO. Based on sensor size, one would expect the NX10 to achieve much better results than the Micro 4/3 cameras. While it outperformed the G2/GF1/E-P2/E-PL1 in this shootout, the magnitude of difference was less than I expected, and the results at the highest ISOs were disappointing.
  • The Panasonic GH1 is, to my eye, second best in terms of signal relative to noise in these comparisons. One issue with the GH1 is that the files are susceptible to banding at high ISO. As a GH1 user, I've found that Nik Dfine 2.0 does a terrific job of addressing banding in GH1 files.

*Splitting hairs, it looks to me like the E-PL1 and E-P2 crops are slightly noisier than the G2 and GF1 crops at high ISO. I believe this is due to the Olympus cameras actually applying less analog gain than the Panasonics for any given nominal ISO. DxOmark data (not shown) suggest for nominal ISO 1600, the Olympus image is an underexposed ISO 1000 image which has been "pushed" to ISO 1600 equivalent brightness, whereas the Panasonic image is an overexposed ISO 2000 image which has been "pulled" to ISO 1600 equivalent brightness.

If that is true, what one would expect as is slightly more noise for the Olympus images (when shutter speed, nominal ISO, f-number, and final image brightness are equalized), and significantly more highlight headroom for the Olympus files (under the same circumstances). As it turns out, that's exactly what we have here. Don't take my word for it, open the Olympus RAW files in your favorite RAW converter, pull back the apparent exposure by, and those blown highlights will be salvaged. Try the same thing with the Panasonic files, and those highlights are gone beyond recovery.

This doesn't mean that the Olympus cameras produce noisier files (or higher dynamic range files) than the Panasonic cameras. They use the same sensor, and the Olympus files are only noisier with better highlight retention if you match the nominal ISO, aperture, and shutter speed as I have done here. If you set the E-P2 to ISO 3200, 1/8s, f/5 and the Panasonic to ISO 1600, 1/8s, f/5, the RAW data (including signal relative to noise and highlight headroom) should be the same, and the appearance should match up after you "pull" the E-P2 file during RAW conversion.



These comparisons take lots of time, energy, bandwidth, and server space, and would not be possible without B&H Photo providing the gear for testing.

Please help support Serious Compacts by clicking the button below before you make your next purchase from B&H Photo.


View attachment 29251
View attachment 29252


Direct links to check pricing and availability at B&H:

 
Thank you so much for all your efforts. I can't even begin to fathom all the time and effort that has gone into these tests and comparisons. And many thanks to B&H Photo, too!
 
Srange for me as I have owned a G1 (two to be precise) and niether could touch my NX10 for noise comparison. I find that my NX10 is able to be 'cleaned' better at high ISO than the G1. I also found that my NX10 is easily better at ISO800 and 1600 than my G1. Unless I had two 'bad' G1's I cannot understand this 'not so small' difference in noise control.

(assuming that the G1 has a similar sensor to the other Lumix cameras here)

Thanks for the tests anyway.
 
Thanks for all the work and the summary. I agree with you about the NEX-5 - impressive results without a doubt. As I am in transition from a {G1+E-P2} combo to a GH1 - I am happy about how it performs. It seems to have a different colour rendition in the brown area - not sure if it is more or less accurate.
 
Thanks, Bill. As I said in the post, I think the color differences with the GH1 were due to my difficulties in getting a proper white balance with the Lightroom WB dropper and maybe due to the specific Lightroom color profile used, not a more or less accurate color response intrinsic to the GH1 sensor.

Toonman, the NX10 outperformed the G2/GF1/E-PL1/E-P2 here, but the magnitude of difference was less than I expected, especially at very high ISO (above ISO 1600). I slightly changed my wording in the analysis to reflect this and understand that your impression varies with mine.
 
Thanks Amin, great report ! The Sony NEX outperforms the competitors. I'm in need for high ISO performance, but I also need an OVF / EVF, must I go NEX as my next camera?
 
Another very sophisticated comparison, Amin. I'm a little surprised that the NX10's performance wasn't closer to that of the NEX-5. The Sony really looks good, although until there is at least one faster lens (like the Lumix 20/1.7) it can't really take advantage of the sensor's low light capability.
 
Thanks for the comparisons. For me, I find even NEX's 3200 ISO not useable, so only up to ISO1600 interests me, where NEX is useable, but m43 only in B&W.

My issue with NEX (or even m43) remains lenses. I've picked up the Oly 14-150 for a vacation to the mountains. There's no option like that for NEX, and kit lenses are boring (though some might like the 16mm prime, I shoot long more often). So, it's still m43 for me. Also, neither system yet has a portrait prime. :(
 
A forum criticism posted at DPReview and my response:

panamamike wrote:
> Why do I say it's not fair? If you look at the DXOMark site, you'll see the NEX-5 true ISO ratings are closer to 3/4 of a stop lower than the Sony indicated stop, for example ISO 200 is actually closer to ISO 130. On the other end the of the spectrum, the GH1 ISO ratings are about a quarter stop higher. That is at ISO 100 the GH1 is more along the lines of ISO 130.


It's a little unfair, but unfair to the Sony not the GH1. In comparing processed files of similar brightness at nominal ISO 200, the Sony is shooting at ISO 130 (per the DxOmark definition of ISO) and those files are being "pushed" to ISO 200-equivalent brightness during RAW conversion, while the Panasonic files are being shot at ISO 270 (per the DxOmark definition of ISO) and being "pulled" down to ISO 200-equivalent brightness during RAW conversion. Remember that these sensors are being given the same amount of light (same f-number and shutter speed) for any given nominal ISO, so the camera which uses more analog gain rather than digital gain (pushing) to arrive at a matched final apparent exposure is at an advantage in a noise contest.

> Also, how did they get GH1 at ISO 6400?

That was explained in the test methods. Cameras without native ISO 6400 settings were shot at ISO 3200 underexposed by a stop and pushed a stop during processing.

Using DxOmark terms of ISO for the nominal ISO 6400 shootout, the GH1 is being shot at DxOmark-measured ISO 4200 (nominal ISO 3200) and pushed to ISO 6400-equivalent brightness while being given the same amount of light (f/5, 1/30s) as the NEX5 being shot at DxOmark-measured ISO 4300 (nominal ISO 6400) and pushed to ISO 6400-equivalent brightness. Couldn't be much more fair than that.
 
Great job! Thanks! I know this took a lot of time and careful effort in setting up and examining the results. Makes me that much happier about my decision to buy the used GH1 a year ago for $1000!
 
The NEX-5 uses a bigger chip, thus should have better high ISO performance, and indeed it has! The NEX-5 seem to have performance much like my K-x, in other words, and no surprise, as I seem to recall that it has roughly the same size chip (different brand, maybe, but who made the chip in the K-x has not been revealed)!
 
The NEX-5 uses a bigger chip, thus should have better high ISO performance, and indeed it has! The NEX-5 seem to have performance much like my K-x, in other words, and no surprise, as I seem to recall that it has roughly the same size chip (different brand, maybe, but who made the chip in the K-x has not been revealed)!

It's open secret who made Kx sensor. Sony did. Before K20D (Samsung sensor), all Pentax DSLRs's used Sony sensors too. Except D3s/D700, all Nikon cameras have Sony sensors. It's Sony's 12 MP CMOS sensor in Kx.
 
Back
Top