Flats trouble?

[bobharmony]

I decided that my Iris data could benefit from fresh darks and a new master bias, as I can use any help available in getting at the dark dustiness there. My current master Bias was done in January, and I haven't redone my dark library in some time.

So last night the camera went out on the porch with lens cap on, and I fired off 180 darks over 12 hours, of which 79 were at the matching temperature for my light. I then did 203 bias exposures, and was ready to go with a new stack. The result is that the background is brighter than ever and there is some funky color creeping in, not to mention tehre is more graininess than the original had.

My first thought was that I screwed up either the new darks or the bias. I loaded the masters up into PhotoShop and checked them out. They displayed the usual hot pixels, and the histogram showed everything way over to the left, as expected. So, if it isn't the darks, or bias, then I decided I need to look at my flats. I loaded the master that DSS generated into PS, and it looks rather dark to me. The histogram supports this, peaking about 1/8 of the way from the left edge. My assumption is that a too-dark flat, will make the image appear lighter than it should, as division by a smaller number will create a lighter result than division by a larger number, if I interpret this correctly.

As an experiment, I took the master Flat and applies a straight-line curve to lighten it to see if I could improved the final stacked image, saved it with a new name, and tried to run the DSS stack again. DSS failed to recognize the modified tif file saying "the checked pictures are not compatible (width, height, number of colors, only one master dark, offset and flat)."

I have two questions - 1) is my thinking on adjusting the flat reasonable, and 2) does anyone have a way to "fool" DSS into accepting the adjusted master? Any help is appreciated!

Bob

[bobharmony]

I think I have found the root of the inability to use the modified flat. The CFA column in DSS is listed as "No", while for all the other master files (including the original master flat) the value is "Yes". Now I have to figure out how to tweak that in PhotoShop.

[bobharmony]

Still poking around - DSS lists the file as a Flat file, not a Master flat. IS there anyone with experience switching this file type?

Also if there are opinions on "lightening" the flat, I'd love to hear them.

Bob

[Stuart]

Hi Bob,
I'll tell you what I tell everybody about flats. The histogram is misleading and almost always leads to flats that are way too dark. The best way to do your flats is to measure the ADU with software and make sure it's 50% of your full well depth. Can you please post how you take your flats? Also, can you upload a RAW flat? I'll be happy to take a look at it for you.
Stu

[Juno16]

Very interesting thread. I’ve always just shot flats using a histogram. Pretty new in this game though and eager to learn.

Thanks,
Jim

[bobharmony]

Hi Bob,
I'll tell you what I tell everybody about flats. The histogram is misleading and almost always leads to flats that are way too dark. The best way to do your flats is to measure the ADU with software and make sure it's 50% of your full well depth. Can you please post how you take your flats? Also, can you upload a RAW flat? I'll be happy to take a look at it for you.
Stu

Stu - I agree that my flats are too dark (at least that's what I suspect). One of my original questions in this thread was about trying to lighten the master flats from DSS and the trouble I am having with that process.

In any case, I take my flats by throwing one of those light tables used for tracing over top of the OTA at the end of each session, setting the camera to AV-priority, and snapping away. I have uploaded one of my RAW flats to Google Drive. It can be found here:



I appreciate any and all help getting this squared away for the long term, and for any insights on saving the flats I took for the Iris processing.

Bob

[Stuart]

I took your flat, extracted the lightness and measured it. WAY too dark. You need to increase your exposure time by 10X and try again. Your goal ADU for a 16 bit scale is about 30,000

[bobharmony]

I took your flat, extracted the lightness and measured it. WAY too dark. You need to increase your exposure time by 10X and try again. Your goal ADU for a 16 bit scale is about 30,000

Thanks, Stu. I suspected that too dark was the issue, it is nice to get a quantitative readout. I will try to get some longer exposures next time out. Also they are rather blue, which was not something I expected from the light box.

In the meantime I am going to try to translate the current set into something usable. We'll see how that goes!

Bob

[Stuart]

Bob,
Don't worry about the blue hue. It doesn't matter. As long as the field is even.
Stu

[bobharmony]

I took your flat, extracted the lightness and measured it. WAY too dark. You need to increase your exposure time by 10X and try again. Your goal ADU for a 16 bit scale is about 30,000

Stu, looking at the stats you pulled, I see the maximum value measured is 34702. Is that 37402 of a possible 64K? If that's the case, increasing the exposure 10x, would clip the peak of the flat, potentially rather severely. In that scenario I would be able to increase exposure time slightly less than 2x. The AV priority setting selected an exposure time of 1/250th. I can see going to 1/160th within the parameters as I interpret the data. Do I have this right?

Bob

[Stuart]

Bob, you are interested in the mean ADU amount. Don't worry about an occasional high amount.
Do one that is 10X as bright and repost. I'll measure it for you.

[bobharmony]

Thanks, Stu. It will take me a few days to get this set up and shot, but I will post here when it is ready. I'll shoot at 1/25 and we'll see see where that takes the numbers.

Bob

[Stuart]

Bob, just to throw a wrench in your plans a bit: I recommend shooting at least 0.5 seconds. That way you avoid flicker artifact from your light source and shutter shadow. You'll ask, how do you keep it from oversaturating? First, lower your ISO. There is no reason why you have to be shooting at ISO 800 for your flat--you can go down to unity gain, whatever it is for your camera, or ISO 100. Second, put a couple of sheets of white paper in front of your OTA to decrease and diffuse the light.

Now, I'll hasten to add--that's what I do. YMMV.

Stu

[bobharmony]

Stu, wrench away. I'll see how much longer I can make the exposure. I was thinking about going back to the iPad for my light source as well. I am feeling that my trouble may have started when I went to the light table as a source.

I think I am going to dig out one of my old master flats made with the light pad and see if it improves the general output any better. I tend to use the same camera orientation every time out, unless there is need for special framing, so I should be able to match something up. I ran a stack without flats, and the noise looked better, but the vignetting was fierce.

All experimentation that doesn't take away from imaging time is good, right?

Bob

[bobharmony]

I finally got around to generating a new flat (81 of them, actually). I used a piece of white paper between the light source and the scope as a diffuser and dropped my ISO to 100. I used 1/50 of a second, when I went to 1/30, the RAW blue component looked clipped in PhotoShop. I am including a link to a single flat here, if Stu would like to take a crack at it.



In the meantime, I am running a new stack in DSS to see whether the result on the Iris can be improved. The new flats are much lighter than the old, so I am hoping for some improvement when they get divided out - we will see!

Bob

[Baskevo]

Still very low (I'm helping ):

Along with [mention]Stuart[/mention]'s advice, this website helped me out a lot: https://www.myastroscience.com/proper-flats-with-dslr

[bobharmony]

Thanks, James. A little progress, but still a long way to go. If I go to 1/20 second, it looks like the blue saturates. I suppose it doesn't hurt to experiment with the .5 second time and see what it looks like. I will take a look at the link you provided as well.

Processing the image with the 1/50 flats went better than with the original. Less noise reduction was required, and I liked the result better. Some day I'll figure this out for real!

Bob

[Baskevo]

1/50 might be a little low... This is from the website I shared: "Your flat panel might be too bright and sky might be too bright which will cause your ISO1600 flats to be of very, very short exposure, short as fraction of a second. That will lead to a shutter shadowing effect (sensor will record shutter movement) and cause to non-uniform illumination across the chip. You should aim for at least 0.5 second exposure to eliminate this effect, better off, if you can do 1-2 seconds exposures for flat and not slower."

I use my iPad, and I had to decrease the brightness significantly so I could do around 2 - 3 seconds, which works very well for me. What happened to me, and what the website explains, is that when my brightness is low enough to work at 2-3 seconds in AV mode on my DSLR, it exposes perfectly at half-way in the camera histogram, but still not where you need it to be in PI, which is showing a mean ADU of only 3500.

First thing I did was take an overexposed flat at 15 seconds at maximum brightness, and loaded the image in PI to look at the maximum mean ADU value (where it is equal to the median). I measured my own maximum mean ADU and I saw that my max mean value was around 16000, so I wanted my flats mean ADU to be half of that around 8000. So what I did was (and this is where the website came in), while in AV mode, I increased the "exposure composition" to above 2, because I needed my ADU to be a little above twice as large as it was.

So you're exposing at around 4,500 mean ADU, and lets say you need the ADU to be around 8,000 like mine (for example), you would increase the exposure composition while in AV mode to 2. But you need to find out what your Max mean ADU is, first.

This is why Stu told you to post a flat at 10x the exposure (I think). This will allow us to look at the maximum mean ADU value to tell you what you SHOULD be exposing at.I don't see Pixinsight under your software, and I don't know if you have the ability to look at ADU with other software, so if you want to take an over exposed flat at the maximum level brightness with "exposure composition" at the default of 0, you can post it here and we can tell you what your ADU should be.

I hope this makes sense! I just spent a month figuring all this out myself haha

The website has pictures to, which might help with explaining all of that ^

[bobharmony]

Thanks, James. I read through the information on the website and it makes sense. My camera, the 60D, has a 14 bit depth, so math tells me the max ADU is 16384 (2 raised to the 14th power), which matches up well with yours.

At 1/50th, the reading you took indicates a mean of just over 4000, so I am half way there. That indicates to me that with the set-up I am using, going to 1/25 would get to the 8000 mean we seem to be striving for. To get beyond that, I am going to have to dim the input, or put additional paper sheets between the light source and the OTA to further increase the exposure time, or potentially set exposure compensation on the minus side to get longer exposures.

I think I am on the right track here, and the info you and Stu have supplied has been very helpful in getting me there. The image I calibrated with 1/50 flats was easier to process and came out better than my original attempt, I expect further improvement when I get the ADU tuned even better!

Bob

[Baskevo]

again, I would try to go for longer exposures by decreasing your light source quite a bit Otherwise you might end up with shutter shadowing affect, which affects the flats. You should at least try to get it down to 1/2 seconds. I believe this is why people shoot in AV mode, because it is easier to get to the half way point in your histogram, and you can figure out what's the correct brightness without changing the exposure time. My iPad seems really dark when I shoot my flats, but that's okay.

If this time it doesn't work, I would take the test shots to post. If you can shoot in AV mode to get to that half way point in your camera's histogram and post an image, then hopefully we would just need to increase your exposure composition by however much you need, and that should settle it.