Frustrated - my histogram is misbehaving (flattening)!

[ram]

So I'm working on the processing this difficult data set I have for a difficult target: even though it's 60 hours of total data, it's probably worth about half that due to clouds/moon/etc. Given I've put in so much time I don't want to throw it out*. The thing is that the initial processing makes it look good (Dynamic Crop, DBE). I really like how the mono images look especially compared to what it looked like prior to DBE and in my head if the mono images channel combined the way I thought they would, it would look pretty decent. But after I combine, it looks very "thin" and when I do a histogram transformation the histogram collapses to become flat instead of a hill shape: as I stretch and move it to the left to make it darker it the hill collapses and becomes flat rapidly.

I've not had this problem before though early on in my imaging days I did have flattened histograms due to things being overexposed but I've kept my gain the same throughout. Has anyone dealt with this before?

Suggestions? *Should I just integrate only the best images (80% or so?) and see if THAT will work better? Since it seems like adding all the extra data has only made it worse or I've forgotten how to process images.

Any other thoughts on how I can salvage this? It's really weird how some targets turn out beautifully and some others suck even after putting in some time. I earlier did one with 76 hours and it was also a difficult target but you could see the resulting data was "thick" and I could've processed it better but it was there. But here I only see signs of positiveness in the mono images but not in the colour combined images.

In case anyone wishes to take a look and/or help out to let me know what I'm doing wrong, I'd greatly appreciate it. The v1.2 (after DC and DBE) will be the fastest download since they're smallest and stored using compression. I've put up the integrated master lights from both my manual integration (v1) and from WBPP (v2) as well as where I am after DBE (v1.2) so you can see the individual frames especially O3.v1.2 look quite salvageable. All are in linear form.

From manual integration, no processing:

http://ram.org/images/space/downloads/s ... Ha.v1.xisf
http://ram.org/images/space/downloads/s ... O3.v1.xisf
http://ram.org/images/space/downloads/s ... S2.v1.xisf

From manual, after DC and DBE (this is when you start to see things pop out in mono):

http://ram.org/images/space/downloads/s ... .v1.2.xisf
http://ram.org/images/space/downloads/s ... .v1.2.xisf
http://ram.org/images/space/downloads/s ... .v1.2.xisf

WBPP:

http://ram.org/images/space/downloads/s ... Ha.v2.xisf
http://ram.org/images/space/downloads/s ... O3.v2.xisf
http://ram.org/images/space/downloads/s ... S2.v2.xisf

Thanks a lot!

--Ram

[STEVE333]

Hi Ram - That's a lot of work. You definitely want to get this fixed!!!

1) I downloaded your DC and Crop SHO files.
2) Without doing any Deconvolution, noise reduction, etc. (because this is only a test), I stretched each of the files with HistogramTransformation.
3) I put the stretched SHO images into the SHO-AIP script.
3a) I used the option to create a Luminance image (using all three SHO images).
3b) I clicked on the Mix L-SHONRVB button and created the image (used 50% L).

Below is the image that was created. Looks pretty cool. Can't wait to see what you do with it.



I'm still new to NB processing, but, this looks pretty normal to me. I could stretch it with HT if desired. I don't know if you stretch before combining, but, I did.

Below is a link to the .xisf file for this SHO-AIP image in DropBox.

https://www.dropbox.com/s/wc6teq79xmbnx ... .xisf?dl=0

One comment, I had to stretch the image about three or four times with HistogramTransformation to get it up to a usable level. This is more stretching than I remember doing on any of my images. Don't know if that means anything.

Hope this helps,

Steve

[ram]

Hi Steve, thanks so much for your attempt - I'm able to produce something like that but I'm wondering why the nebula portion isn't brighter (or really I need to figure out how to make it brighter while keeping the background the same, so probably some clever masks since the top portion is stronger than the bottom portion).

You've seen the mono images of the DC and DBA but for others who're perhaps following I've attached a screenshot of the Ha and O3 master lights side by side and you can see it looks pretty bright and could easily be made brighter too. Yet when I combine, it appears to become "thin" instead of the usual brightness that comes from combining NB images. On top of it, the histogram flattens - I looked at your histogram and it also has flattened. Almost all my image histograms are peaking at the top whereas if you look at the histogram of your image (mine goes down even more) it is a small hill instead of a big hill.

I can get the histogram to become a big hill by I have to move it to the left but then the image gets darker. What I want is a big mountain for my histogram while the appearance is bright.

It's a faint nebula, especially the bottom portion. I was happy to have captured the mesh like structure that is the lion's body/legs but still the histogram behaviour is frustrating. Furthermore, there's both Ha and O3 data for this region yet it seems like for the bottom O3 dominates and for the top Ha dominates with no in between.

I guess what I'm saying is that the final NB image doesn't match my expectation of what the individual filter master lights look like. Plus the histogram behaviour is weird. Maybe masks are the answer, or maybe I should integrate with only my best set of lights instead of ALL of them. So many things to try.

Thanks for looking into this - I really appreciate it!

--Ram

[STEVE333]

Sorry that didn't help Ram. I really don't see the HT problem, but, maybe just my age!? When I slide the middle HT slider to the left the RGB "mountains" all slide to the right and keep their same heights.

Have you considered trying to create a starless image which can be enhanced and then reinserting the stars. That might help to enhance the nebula's appearance. If you consider it I would do it on each channel individually, and, only after doing noise reduction and HistogramTransform stretching and maybe a bit more noise reduction (TGVDenoise). Just a thought. That approach works well for me.

Steve

[ram]

Thanks Steve - I will try it out. I've been meaning to try out your new starless technique anyway in PI so this would be a good test of that.

Sorry, you're right about the HT slider going left. You need to slide the middle HT slider to the right. You'll see it peak up at the far right and my images normally look like this when the HT slider is in the middle.

But I just noticed something - when I did the DBE, I didn't check the "normalise" box which I normally do and it turns out checking the box makes a difference in terms of brightness and some other things that didn't work the usual way. So that just have been it. I'm redoing now with normalisation checked.

--Ram

[ram]

Sorry, I have my left and right mixed up. So the middle slider to the right in HT causes peaks to the far left. Whereas if you did the DBE with "normalise" it behaves differently and it works more consistently with other things in the LVA tutorial (esp. the next part of making linear images non-linear).

[STEVE333]

Thanks Steve - I will try it out. I've been meaning to try out your new starless technique anyway in PI so this would be a good test of that.

Sorry, you're right about the HT slider going left. You need to slide the middle HT slider to the right. You'll see it peak up at the far right and my images normally look like this when the HT slider is in the middle.

But I just noticed something - when I did the DBE, I didn't check the "normalise" box which I normally do and it turns out checking the box makes a difference in terms of brightness and some other things that didn't work the usual way. So that just have been it. I'm redoing now with normalisation checked.

--Ram

Good catch on finding the DBE setting. So many ways to get in trouble in this hobby.

I just sent you a PM showing an HOO version of your data both with and without stars. I didn't do a SHO version because I just wanted you to see what the starless approach can do to brighten the nebula. Hope it is helpful.

Really look forward to seeing your final version on this beauty.

Steve

[ram]

Like I said in the PM, what you did was simply stunning and if I produced an image like that I would be satisfied (and there's not a lot of S2 data - there was 18 hours of time on there and compared to Ha and O3 you can see it's not as much but there is some and we could use it in different combinations, i.e., red could be O3+S2 in ratio). I will definitely try out the starless technique now that I've worked out at least one issue that I did differently from before (and on top of it, you worked with the bug with the non-normalised DBE but perhaps these things, starlessness letting you brighten up the nebula and DBE normalisation are related - since that was the problem I was having if I tried to brighten everything I would get the background to come up also but by making it starless you effectively remove the background?).

Is this the workflow you used? "I would do it on each channel individually, and, only after doing noise reduction and HistogramTransform stretching and maybe a bit more noise reduction (TGVDenoise)."

Thanks Steve - you've given me a lot of hope which is what matters! I was getting very frustrated*** so this gives me some hope to keep going with this knowing the outcome can be at least as good as what you've produced.

--Ram

***Trying to do too many things at once. I'm writing a grant for COVID-19 research while having processing going on in the background. That's my life so between not having processed a image for the last 8 months and being swamped with this pandemic

[yobbo89]

I had a bit of a play , i hope you don't mind, i can run down with you my process if you like , (using pixinsight)

it certainly would be good to get that tail end of the nebula to brighten up but that is a little bit beyond my technique.

[ram]

yobbo89, absolutely - that's why I posted the data -DBE version (v1.2) it was not normalised after DBE which caused an assumption about the background which I think resulted in later problems. Steve's idea of removing the stars and then brightening up the nebula is really promising so I will try it out once I find some time.

--Ram

[STEVE333]

Like I said in the PM, what you did was simply stunning and if I produced an image like that I would be satisfied (and there's not a lot of S2 data - there was 18 hours of time on there and compared to Ha and O3 you can see it's not as much but there is some and we could use it in different combinations, i.e., red could be O3+S2 in ratio). I will definitely try out the starless technique now that I've worked out at least one issue that I did differently from before (and on top of it, you worked with the bug with the non-normalised DBE but perhaps these things, starlessness letting you brighten up the nebula and DBE normalisation are related - since that was the problem I was having if I tried to brighten everything I would get the background to come up also but by making it starless you effectively remove the background?).

Is this the workflow you used? "I would do it on each channel individually, and, only after doing noise reduction and HistogramTransform stretching and maybe a bit more noise reduction (TGVDenoise)."

Thanks Steve - you've given me a lot of hope which is what matters! I was getting very frustrated*** so this gives me some hope to keep going with this knowing the outcome can be at least as good as what you've produced.

--Ram

***Trying to do too many things at once. I'm writing a grant for COVID-19 research while having processing going on in the background. That's my life so between not having processed a image for the last 8 months and being swamped with this pandemic

Ram - The starless image still has ALL the background. It is literally the original image but without the stars.

Yes,I process each channel individually up through TGVDenoise and then create a starless version of each channel. The starless versions (just HOO for the image I shared with you) are then combined using the SHO-AIP script. This combined image can be further enhanced (ExponentialTransformation, Increase saturation, etc) before putting the stars back in.

You sound like a busy man Ram. Hope your research grant gets accepted.

I played a little bit more with the HOO version of your image. It is shown below.



Thanks for allowing us to look at your beautiful data. The HOO colors are perfect for the Lion's fur!!

Steve

[ram]

If I could give your posts several thumbs up I would - no wonder your images look so stunning in your signature. I have a lot to learn...

--Ram

[STEVE333]

If I could give your posts several thumbs up I would - no wonder your images look so stunning in your signature. I have a lot to learn...

--Ram

Thanks Ram - I appreciate your images too.

I'll get started on the "Updated New Starless Approach".

Steve

[ram]

Getting back to this and applying this to the Wizard, I'm wondering about the use of the SHO-AIP script - it is deprecated as of PI 1.8.5 - I could get it but is it the same as the NBRGB script? I mean it combines the images and shows you how things look like? I normally use channel combination or pixelmath to do this since the SHOAIP script was removed from PI.

Basically if I follow you right, you're just stretching each channel and do noise reduction, etc. (i.e., do everything normally) and then remove the stars and then do the SHO combination and then manipulate and put the stars back?

--Ram

[STEVE333]

Getting back to this and applying this to the Wizard, I'm wondering about the use of the SHO-AIP script - it is deprecated as of PI 1.8.5 - I could get it but is it the same as the NBRGB script? I mean it combines the images and shows you how things look like? I normally use channel combination or pixelmath to do this since the SHOAIP script was removed from PI.

Basically if I follow you right, you're just stretching each channel and do noise reduction, etc. (i.e., do everything normally) and then remove the stars and then do the SHO combination and then manipulate and put the stars back?

--Ram

The NBRGB script doesn't allow you to add more than one NB channel into a given RGB channel. For example you couldn't add 70% O3 + 30% Ha into the G channel. That would be a problem for me because that is my normal starting point when doing HOO. The SHO-AIP has a difficult user interface, but, does allow "mixing" NB data.

You described my NB processing approach correctly.

As an aside, I do the following to get stars to put into the "manipulated" SHO image:
I create a separate straight HOO image using the stretched NB data just before removing the stars. This gives an HOO image with stars. These stars have some color to them. I use these stars to put back into the "manipulated" SHO image.

Just FYI.

Steve

[ram]

Thank Steve - I was thinking of doing exactly that. Either the master light SHO image or the best Ha, O3, S2 frame and combine into SHO and extract the stars. I plan to try out a few things ONCE I get to that stage. I'm still at doing a DBE for my ngc7380 (I've done that for sh2-132). It's slow going.

--Ram

[STEVE333]

Patiently looking forward to your results.

Hope you got some sleep.

Steve

[Graeme1858]

There's some top level stuff going on here!

So that mere mortals and newbies can have a go Ram, could you clarify what software you use to process an xisf files please. And file under additional newbie alert but what's DBE and WBPP?

Cheers

Regards

Graeme

[ram]

Hi Graeme, it's PixInsight (PI) and DBE is Dynamic Background Extraction, and WBPP is Weight Batch Pre-Processing - the former is a process in PI and the latter is a script.

--Ram

[Graeme1858]

Thanks for the clarification Ram.

I'll delete the file I downloaded then!

Can PI export as fits or a tifs?

Regards

Graeme