Mirrorless Camera ISO Signal/Noise Shootout: NEX5, NX10, E-P2, E-PL1, GF1, G2, GH1

Discussion in 'Featured Articles, Reviews & Posts' started by Amin Sabet, Aug 14, 2010.

  1. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    Having compared the size of several different mirrorless interchangeable lens systems, we'll now take a look at how much detail relative to noise is captured at any given ISO by each of these systems.

    Special thanks to
    B&H Photo for providing the following cameras used in this comparison: Panasonic G2, Olympus E-PL1, Olympus E-P2, Sony NEX5, Panasonic GF1. The Panasonic GH1 is mine.

    If you're reading this on the blog main page with resized images, click here to view this post with all images at full size.

    Design and Methods:


    • The goal here was to show the relative potential of these systems, hence I will present only RAW conversions and not in-camera JPEGs. There are many sites with excellent in-camera JPEG comparisons.
    • All images were made using Sony, Samsung, and Panasonic (14-45mm) kit zooms at f/5, which was chosen because it represents a setting of good center sharpness with adequate depth of field for all systems.
    • All kit zooms were used at their widest setting, and the subject distance was constant. The Panasonic lens at 14mm actually has a slightly wider angle of view than the Sony/Samsung at 18mm, but the reverse is true after Lightroom correction of barrel distortion. For practical purposes, the diagonal angle of view was fixed.
    • Since the purpose here was to look at sensor rather than lens performance, only center crops from the within sharp focus portion of the image were used for comparison.
    • To present a valid comparison of detail relative to noise at any given ISO, exposure times were matched for all systems as follows: 1) ISO 100: 2s; 2) ISO 200: 1s; 3) ISO 400: 1/4s; 4) ISO 800: 1/4s; ISO 1600: 1/8s; ISO 3200: 1/15s; ISO 6400: 1/30s. Lighting sources were kept constant.
    • Each shot was done in triplicate to ensure reproducibility.
    • All system settings were image stabilization off, self-timer, sturdy tripod.
    A few details about the RAW conversions:

    • RAW files were processed in Lightroom 3.2 RC, which supports all of the cameras tested.
    • Custom white balance was applied during RAW conversion, and each RAW file was white balanced off of the gray paper in the image. Due to my imprecise white balance matching and differences in the LR 3.2 color profiles for each camera, I could not match colors perfectly. The colors on the GH1 crops are particularly off and do not represent a sensor problem but rather a limitation of my testing methodology.
    • Some of these cameras differ in image brightness (in Lightroom) given the same nominal ISO, f-number, and shutter speed. In order to present as controlled a comparison as possible, "Auto tone" was used in Lightroom to more closely match the final brightness of each image.
    • Sharpening settings were Lightroom default (same for all cameras), and both luminance and color noise reduction were disabled entirely.
    • In order to provide as fair a comparison as possible, the Sony and Samsung images were downsized during export from Lightroom such that their diagonal image dimension would match that of the Micro 4/3 cameras. Showing all images at their native size would have penalized the Sony and Samsung for using more megapixels, while the chosen method is akin to comparing equal-sized prints from each camera.
    • RAW files provided for download have been converted to DNG for the sake of conserving storage/bandwidth.

    Here is the test scene with yellow rectangles indicating the selections shown in the crop comparisons which follow:

    Scene.jpg


    ISO 100 crops (NEX 5 not shown since it has no ISO 100 setting):

    Picture-25.jpg

    Picture-24.jpg

    Picture-23.jpg


    ISO 100 RAW files:

    • G2-1.dng
    • E-PL1-1.dng
    • NX10-1.dng
    • GH1-1.dng
    • E-P2-1.dng
    • GF1-1.dng

    ISO 200 crops:

    Advertisement

    Picture-22.jpg

    Picture-21.jpg

    Picture-20.jpg


    ISO 200 RAW files:

    • E-PL1-2.dng
    • G2-2.dng
    • NEX5-2.dng
    • E-P2-2.dng
    • GH1-2.dng
    • NX10-2.dng
    • GF1-2.dng
     
    • Like Like x 6
  2. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    ISO 400 crops:

    Picture-19.jpg

    Picture-18.jpg

    Picture-17.jpg


    ISO 400 RAW files:

    • G2-3.dng
    • GF1-3.dng
    • GH1-3.dng
    • NEX5-3.dng
    • E-PL1-3.dng
    • NX10-3.dng
    • E-P2-3.dng

    ISO 800 crops:

    Picture-16.jpg

    Picture-15.jpg

    Picture-14.jpg


    ISO 800 RAW files:

    • G2-4.dng
    • GF1-4.dng
    • GH1-4.dng
    • E-PL1-4.dng
    • NX10-4.dng
    • E-P2-4.dng
    • NEX5-4.dng
     
    • Like Like x 6
  3. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    ISO 1600 crops:

    Picture-13.jpg

    Picture-12.jpg

    Picture-11.jpg


    ISO 1600 RAW files:

    • G2-5.dng
    • GF1-5.dng
    • GH1-5.dng
    • NEX5-5.dng
    • E-PL1-5.dng
    • NX10-5.dng
    • E-P2-5.dng


    ISO 3200 crops:

    Picture-10.jpg

    Picture-9.jpg

    Picture-8.jpg


    ISO 3200 RAW files:

    • GH1-6.dng
    • NX10-6.dng
    • NEX5-6.dng
    • GF1-6.dng
    • G2-6.dng
    • E-PL1-6.dng
    • E-P2-6.dng



    [​IMG]

    [​IMG]
     
    • Like Like x 5
  4. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    ISO 6400* crops:

    Picture-7.jpg

    Picture-6.jpg

    Picture-5.jpg


    ISO 6400* RAW files:

    • E-PL1-7.dng
    • G2-7.dng
    • GF1-7.dng
    • NEX5-7.dng
    • NX10-7.dng
    • GH1-7.dng
    • E-P2-7.dng


    * Cameras lacking an ISO 6400 setting were used at ISO 3200 underexposed by 1EV and pushed to ISO 6400 equivalent during RAW processing. Ie, all files shown here as ISO 6400 equivalent were made at f/5 and 1/30s with a constant light source.
     
    • Like Like x 4
  5. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    The majority of color noise can be removed without impacting apparent detail. Let's look at some high ISO crops which have undergone color noise (but not luminance noise) removal using Noise Ninja and an "Auto Color" step in Photoshop to try for a slightly better color match.

    In these comparisons, I am using the Panasonic G2 as representative of the G2/GF1/E-P2/E-PL1, which all share the same sensor and as shown above, produce similar results.

    ISO 1600 crops:

    1600-1.jpg

    1600-2.jpg

    1600-3.jpg


    ISO 3200 crops:

    3200-1.jpg

    3200-2.jpg

    3200-3.jpg


    ISO 6400 crops:

    6400-1.jpg

    6400-2.jpg

    6400-3.jpg
     
    • Like Like x 5
  6. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    Analysis:

    • The NEX 5 has the best performance in terms of detail relative to noise. This is apparent even at low ISO, and the gap widens with increasing ISO.
    • At very high ISO, the NEX5 has a stop or more advantage over the G2/GF1/E-PL1/E-P2. You can see that by looking at these side-by-side crops (color noise removed):
    one-stop.jpg

    These observed results correlate well with data measured by DxOmark:


    dxo.jpg

    • The Panasonic G2, GF1, E-PL1, and E-P2 show very similar* results. This is consistent with my understanding that all of these cameras use the same sensor (with minor differences in the low pass filter).
    • The Samsung NX10 is a little disappointing at very high ISO. Based on sensor size, one would expect the NX10 to achieve much better results than the Micro 4/3 cameras. While it outperformed the G2/GF1/E-P2/E-PL1 in this shootout, the magnitude of difference was less than I expected, and the results at the highest ISOs were disappointing.
    • The Panasonic GH1 is, to my eye, second best in terms of signal relative to noise in these comparisons. One issue with the GH1 is that the files are susceptible to banding at high ISO. As a GH1 user, I've found that Nik Dfine 2.0 does a terrific job of addressing banding in GH1 files.

    *Splitting hairs, it looks to me like the E-PL1 and E-P2 crops are slightly noisier than the G2 and GF1 crops at high ISO. I believe this is due to the Olympus cameras actually applying less analog gain than the Panasonics for any given nominal ISO. DxOmark data (not shown) suggest for nominal ISO 1600, the Olympus image is an underexposed ISO 1000 image which has been "pushed" to ISO 1600 equivalent brightness, whereas the Panasonic image is an overexposed ISO 2000 image which has been "pulled" to ISO 1600 equivalent brightness.

    If that is true, what one would expect as is slightly more noise for the Olympus images (when shutter speed, nominal ISO, f-number, and final image brightness are equalized), and significantly more highlight headroom for the Olympus files (under the same circumstances). As it turns out, that's exactly what we have here. Don't take my word for it, open the Olympus RAW files in your favorite RAW converter, pull back the apparent exposure by, and those blown highlights will be salvaged. Try the same thing with the Panasonic files, and those highlights are gone beyond recovery.

    This doesn't mean that the Olympus cameras produce noisier files (or higher dynamic range files) than the Panasonic cameras. They use the same sensor, and the Olympus files are only noisier with better highlight retention if you match the nominal ISO, aperture, and shutter speed as I have done here. If you set the E-P2 to ISO 3200, 1/8s, f/5 and the Panasonic to ISO 1600, 1/8s, f/5, the RAW data (including signal relative to noise and highlight headroom) should be the same, and the appearance should match up after you "pull" the E-P2 file during RAW conversion.



    These comparisons take lots of time, energy, bandwidth, and server space, and would not be possible without B&H Photo providing the gear for testing.

    Please help support Serious Compacts by clicking the button below before you make your next purchase from B&H Photo.


    [​IMG]

    [​IMG]


    Direct links to check pricing and availability at B&H:

     
    • Like Like x 10
  7. BBW

    BBW Administrator Emeritus Subscribing Member S.C. Charter Member

    Joined:
    Jul 7, 2010
    Messages:
    13,210
    Location:
    NW corner of CT
    Real Name:
    BB
    Thank you so much for all your efforts. I can't even begin to fathom all the time and effort that has gone into these tests and comparisons. And many thanks to B&H Photo, too!
     
    • Like Like x 1
  8. pictor

    pictor Lounge All-Pro S.C. Charter Member

    Joined:
    Jul 14, 2010
    Messages:
    1,128
    That's an interesting and very well done test! I enjoyed reading it very much, thanks for your efforts!
     
    • Like Like x 1
  9. Toonman

    Toonman Lounge Regular S.C. Charter Member

    Joined:
    Jul 22, 2010
    Messages:
    107
    Location:
    North East England (UK)
    Srange for me as I have owned a G1 (two to be precise) and niether could touch my NX10 for noise comparison. I find that my NX10 is able to be 'cleaned' better at high ISO than the G1. I also found that my NX10 is easily better at ISO800 and 1600 than my G1. Unless I had two 'bad' G1's I cannot understand this 'not so small' difference in noise control.

    (assuming that the G1 has a similar sensor to the other Lumix cameras here)

    Thanks for the tests anyway.
     
  10. bilzmale

    bilzmale Super Moderator Emeritus Subscribing Member S.C. Charter Member

    Joined:
    Jul 17, 2010
    Messages:
    5,064
    Location:
    Perth, Western Australia
    Real Name:
    Bill Shinnick
    Thanks for all the work and the summary. I agree with you about the NEX-5 - impressive results without a doubt. As I am in transition from a {G1+E-P2} combo to a GH1 - I am happy about how it performs. It seems to have a different colour rendition in the brown area - not sure if it is more or less accurate.
     
  11. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    Thanks, Bill. As I said in the post, I think the color differences with the GH1 were due to my difficulties in getting a proper white balance with the Lightroom WB dropper and maybe due to the specific Lightroom color profile used, not a more or less accurate color response intrinsic to the GH1 sensor.

    Toonman, the NX10 outperformed the G2/GF1/E-PL1/E-P2 here, but the magnitude of difference was less than I expected, especially at very high ISO (above ISO 1600). I slightly changed my wording in the analysis to reflect this and understand that your impression varies with mine.
     
    • Like Like x 1
  12. Herman

    Herman The Image Stimulator S.C. Charter Member

    Joined:
    Jul 11, 2010
    Messages:
    2,484
    Location:
    The Netherlands
    Real Name:
    Herman
    Thanks Amin, great report ! The Sony NEX outperforms the competitors. I'm in need for high ISO performance, but I also need an OVF / EVF, must I go NEX as my next camera?
     
  13. Björn Utpott

    Björn Utpott Lounge Rookie S.C. Charter Member

    Joined:
    Jul 17, 2010
    Messages:
    18
    Another very sophisticated comparison, Amin. I'm a little surprised that the NX10's performance wasn't closer to that of the NEX-5. The Sony really looks good, although until there is at least one faster lens (like the Lumix 20/1.7) it can't really take advantage of the sensor's low light capability.
     
  14. wt21

    wt21 Lounge All-Pro

    Joined:
    Aug 15, 2010
    Messages:
    1,983
    Thanks for the comparisons. For me, I find even NEX's 3200 ISO not useable, so only up to ISO1600 interests me, where NEX is useable, but m43 only in B&W.

    My issue with NEX (or even m43) remains lenses. I've picked up the Oly 14-150 for a vacation to the mountains. There's no option like that for NEX, and kit lenses are boring (though some might like the 16mm prime, I shoot long more often). So, it's still m43 for me. Also, neither system yet has a portrait prime. :(
     
  15. Amin Sabet

    Amin Sabet   Administrator Subscribing Member

    Joined:
    Jul 3, 2010
    Messages:
    2,478
    A forum criticism posted at DPReview and my response:

     
  16. deezeen

    deezeen New to the Lounge S.C. Charter Member

    Joined:
    Aug 6, 2010
    Messages:
    1
    Thank you Amin for your effort. You did a great job as always.
     
  17. photoSmart42

    photoSmart42 New to the Lounge S.C. Charter Member

    Joined:
    Aug 9, 2010
    Messages:
    4
    Great job! Thanks! I know this took a lot of time and careful effort in setting up and examining the results. Makes me that much happier about my decision to buy the used GH1 a year ago for $1000!
     
  18. Tord S Eriksson

    Tord S Eriksson Lounge Regular

    Joined:
    Aug 16, 2010
    Messages:
    49
    Location:
    Gothenburg. Sweden
    The NEX-5 uses a bigger chip, thus should have better high ISO performance, and indeed it has! The NEX-5 seem to have performance much like my K-x, in other words, and no surprise, as I seem to recall that it has roughly the same size chip (different brand, maybe, but who made the chip in the K-x has not been revealed)!
     
  19. Djarum

    Djarum Lounge All-Pro Subscribing Member S.C. Charter Member

    Joined:
    Jul 10, 2010
    Messages:
    1,382
    Location:
    Huntsville, AL
    Real Name:
    Jason
    Excellent write up Amin. I agree, the NEX system looks the best. I am really suprised with the NX10. I thought it would have done better.
     
  20. oneguy

    oneguy New to the Lounge

    Joined:
    Aug 16, 2010
    Messages:
    1
    It's open secret who made Kx sensor. Sony did. Before K20D (Samsung sensor), all Pentax DSLRs's used Sony sensors too. Except D3s/D700, all Nikon cameras have Sony sensors. It's Sony's 12 MP CMOS sensor in Kx.
     

Share This Page