How to determine final image quality quantitatively?

[ram]

I'm a big believer in creative decision making yet when I first integrate images (i.e., human judgement), I'd like to be able to throw out versions that are "worse" than others. I know they are worse because of the data I put in but yet sometimes they produce final outputs that visually look very very similar. Upon close inspection, one can convince themselves one is better than the rest but I honestly wonder if I'm fooling myself.

For instance, I collected 56 frames of an object last night and it was very cloudy. Using blink in PI, I could see that there was close to nothing in ~10 images and another 9 (subjectively) had weak signal but perhaps useable. So about a total of 10-20 frames that could be discarded. I created three final integrations after calibrating and registering all but 3 of the images (which failed to register): all 53 registered images, 46 images throwing out the worst 10, and 37 images throwing out the worst 10 plus another 9 I felt were bad.

You'd have to look very hard but you may be able to argue that the last version with the best set of 37 images is the best one but only at this stage (since it is Ha and mono). The middle one seems as good as the last one and of course the one with all the crappy data looks a tiny tiny fraction bit softer but once it becomes a SHO image I really doubt even these very nitpicky differences would be visible.

This is a testament to PI's integration capabilities but nonetheless I'm not sure which image I should use downstream from now and I was wondering if there was a quantitative measurement of image quality here. Is there anything like "frame selector" (like subframe selector) that looks at overall SNR ratio based on the total stack, etc.?

--Ram

PS: One thing I noticed is that using quantitative metrics to throw these out like FWHM, eccentricity, median, stars, etc. proved difficult in this case except for # of stars, which did largely correlate with the 19 frames I throw out (the worst 19 were the ones with the fewest stars detected but aside from the worst 3 not registering). But in the stacked images, all the stars will be nearly identical.

PPS: In case anyone is interested, the three images are uploaded to here. Feel free to guess which is which (let alone which is better). One is all 53 frames, one is 46 best frames (#stars), and the other is 37 best frames (#stars).

http://www.ram.org/images/space/downloa ... _Ha.v0.jpg
http://www.ram.org/images/space/downloa ... _Ha.v1.jpg
http://www.ram.org/images/space/downloa ... _Ha.v2.jpg

[yobbo89]

I use pi to process but i only stack in deep sky stacker, the most important stack is the luminance stack, sometimes i'll let data with defects go through just to get signal up for the colour channels.

deepsky stacker has a analyst quality readout , one of them i think is star quality which indicates overall quality of the image/nebula ectt., it will give them a low score for smeared/oval stars/data/ tracking problems.. and it also has a sky background score which gives you an indication on signal, the higher the percentage the more washed out the data is. there is a filter option ie to list worst to good visa versa so you can manually find the bad frames and remove them.

i manually process my b/w frames, giving them noise reduction, background extractions and a slight stretch to try and balance the data out before i assign them to l rgb in pi

[ram]

@yobbo89 - PI also has subframe selector which evaluates the SNRratio and a bunch of other stats on each frame. The issue I'm getting at I believe is that when you filter on one or even a multiple of these quantities, you don't know if the end integration is the best in that quantity, let alone in some combined/integrated fashion. For instance, if I integrate the 10 best individual SNRratio frames, and the 20 best SNRratio frames, it's possible that the 20 frames version is the "better" one (i.e., has overall better SNRratio). I could rescore my final integrations back in subframe selector but I don't think this is the correct answer since it doesn't take the added value of the integration into account and doesn't produce a "holistic" result.

I suppose one criteria could be that if you do the combination and find that things are similar/get worse in those quantities like FWHM, eccentricity, SNR, etc. in the integrated image relative to the MEDIAN/WORST of the frames (which is what I saw in my m45 integrations that included some frames with moonlight) then the integration wasn't done correctly. So we're always trying to improve upon the median/average of the stack.

Also the difficulty is when things are very close - if it's obvious visually then I think the existing metrics largely work... but I'm talking about close calls. I bet many of you are perfectionists who struggle with this issue, like knowing when to stop tweaking...

See the examples I posted. I can actually discern the difference between three images that look very very similar. Yet most would argue that while my discernment is real, it is subjective as to which is better. You could assign "better" to the version that has the "best" data but that seems rather subjective.

--Ram

[yobbo89]

I leave it to someone else to articulate the info you have posted maybe they have a methode., the end results of astrophotography is that we are really recording photons from space then processing the data and giving it an interpretation/and are making art,since our eyes are really poor at seeing in a lot of different wavelengths, everything is subjective and we process things that are visually striking for oneself and others, if it looks good then that is good enough for a quality image and move on to another target. given that there are some ground basis on a quality image which should be something like , round stars, no speckles from noise,bright histogram to easily see target a nice colour ectt,ectt. I guess it's human nature to make things neat, even, round,not too bright,not too dark, to perfection things, ect..

[Don Quixote]

Hello Ram. Thank you for this post. I am ignorant of the processes involved, but I understand the question you are asking.
I am examining these images and it is difficult to judge.
I think I like the first one in the list. :-)
This image has a tiny bit more grain, but I like the crispness that it gives to my view.

I am curious about this one as to your process details.

Thank you for this interesting post.

[yobbo89]

PPS: In case anyone is interested, the three images are uploaded to here. Feel free to guess which is which (let alone which is better). One is all 53 frames, one is 46 best frames (#stars), and the other is 37 best frames (#stars).

http://www.ram.org/images/space/downloa ... _Ha.v0.jpg
http://www.ram.org/images/space/downloa ... _Ha.v1.jpg
http://www.ram.org/images/space/downloa ... _Ha.v2.jpg

i guess ha v2 is the 53 stack ,good
ha v 1 is the 46 stack , bad
and ha v0 is the 37 , worst

i just look in only the low signal areas in the corners,

with astrophotography ,there is a limit to amount of data collected and the signal doesn't improve much at all, i could say that with an extra 5-10 frames here and there an image quality improvement can almost be unnoticeable to the human eye much like subtle comparison to say telescopes 10-11 inch newtonian,a bit of a difference but not worlds apart. you really need to double down on target time to get that wow factor!!! and compare it to it's lesser half of data collect from a previous time,it will give you a good indication on what kind integration length you need or would like to do,

There are targets that no matter how much time you throw at it they really don't improve from say 1hr to 5-6 hrs, in my position i live in a bortle 6 and that is the case for most galaxies for me, a subtle difference but 1hr to 6 hrs difference isn't worlds apart for some of those targets due to lp, i'm really just collecting photons from the lp from the near by city

edit: i think i'll add onto part of the question, how to determine the final quality image, when the darker regions in space that are hazed in low signal /noise in your images come to life and produce a shape from its very faint signals of photons from it's nebula/gas then that is i would say the target goal for an astro image, to capture every faint signal into your cmos/ccd chip and produce an image that is almost populated from every pixel on your screen.

[ram]

So the answer is v0 is 53, v1 is 46, and v2 is 37. I'd say the 37 stack contains the best data - the 53 stack contains all data which for instance in the case of m45 actually proved to be a problem.

Which contains the good and bad is the question I'm unable to answer especially as it varies by target. For instance, yobbo and Don like the v0 data (brighter overall, more data even if a lot of it is noisy). The v2 contains all the bad frames thrown out and produces a slightly more contrasty image but it is missing perhaps an hours worth of total data.

yobbo89, I agree with your "add on" part - it is about the maximum dynamic range of the fainest bits, at least that's the goal for me. I've been playing with Sharpcap smart histogram to achieve this and like my IC1396 image (https://www.astrobin.com/409845/B/) these were the same settings used, which is what the brain recommended for the Ha filter. Normally I've been using higher gains with lower exposure length.

In the end, esspecially once the NB image is combined with the other colours, it probably doesn't matter which one I pick...

Again, this is a testament to PI's ability in integration IMO. These images had no right to look this similar!

--Ram

[fatboy1271]

I personally like v2 the best. I think the bright star in the middle of the cloud has a good "graininess" to it, not overdone, and I think the pillar has a good amount of sharpness to it. My opinion should be taken lightly; however, there it is

fat

[Graeme1858]

Hello Ram

They all look the same to me, not quite similar but in fact identical! So one of three things is true; the resolution of my laptop screen is insufficient to differentiate, the effort and time we put into pre processing is irrelevant or you're having a bit of a ruse!

It's likely to be a bit of both of the first two options.

Nice image though.

Regards

Graeme

[JimMinCT]

I think part of the problem, (or solution, depending on your outlook), is PI itself...
It treats low Signal to noise images with less weight automatically, so it will, by itself, reduce the effect of bad images.
I've also left in less than optimum quality images, (usually for lack of time on target), and PI will use what data it can.

It may very well be giving those bad images low, to no weight.
Watch the processing screen as it works and check the weights it gives to each frame.
It may tell you a bit more about what's going on with the program.
Cheers

[ram]

For some reason, I didn't see the replies to this thread (usually I get notified of them). I apologise for that.

Anyways, I think there's some truth to all these comments. The images do look very similar but I can see that they are different upon close inspection. I use my 11 year old daughter a lot to distinguish between images and she'd say there was no difference between them and usually I listen but there's a faint amount of brightness in the core that is slightly different. It is a very very nitpicky thing to observe and only matters to the person doing the AP I think. Sometimes there is a preference as to which difference is "better" but in this case, it's unclear since it is very slight.

I think the bottom line from this is that JimMinCT is right - PI handles all of this properly - if it can register an image it probably means there's something "there". It seems the best thing to do is leave it all in and let PI sort it out when it does it well. The bottom 20 frames are terrible - in terms of blink I'd throw out at least 10 of them since they are so cloudy yet PI takes it well.

I went and collected double the amount of data now (so almost 13 hours total just for Ha alone). Here're a couple of resulting images of 108 x 7 min frames. The first is with is with minimal processing (HT) and then the other is a bit more, particularly LHE (local histogram equalisation). Compared to the previous data you can see the signal is WAY better even though I had clouds this time too (though every single frame registered). You can make out the skull shape clearly. But I bet if I threw out the worst 10-20 frames, the difference would be even less than before, since the good frames outweigh the bad ones by a greater fraction now.

http://www.ram.org/images/space/downloa ... a.v3_p.jpg
http://www.ram.org/images/space/downloa ... a.v3.1.jpg

However you can see that the core is starting to get blown out now that there's more and more data. It will be a challenge to process these two components (the outer loop and the core).

--Ram

[ram]

So I'm doing NGC1499 now and I'm seeing the same phenomenon - I captured 59 frames and taking out the worst 19 frames makes no difference in the final image that I can see visually. I'm not doing anything different in terms of processing - integrate with same parametres, auto histogram, and then histogram transformation. Looks near identical , even though there were 19 frames of mostly cloudy data in one case. The weights for these are not high but they are not 0 either.

Happy to post the images in case people are interested, but this wasn't just a target issue (since the NGC7822 was a target that required a lot of exposure to get the details out so one could argue small differences in # of frames didn't matter, but here this is a magnitude 5 target).

--Ram

[Stuart]

The way I do it is start with blink and I throw out the bad ones. I'm actually pretty brutal, and I am working with a dataset where I threw out 60% just by looking. Usually it's not that bad.
Then I'll use SFS using the formula available on the extra material in the Inside PIxinsight book.
springer.com/us/book/9783319976884