Flats trouble?

[Stuart]

James,

Can you upload two examples that I can take a look at? The first being what you think is a good flat and the other your "maximum" flat (with your pixels saturated and the resulting image white?). I need to see it so I can figure it out.

Stu

[JayTee]

Have you guys tried a piece of software called Astrophotography Lab? It is freeware and was promoted by the author over on AF about 2 years ago. It is a very powerful title for image and sensor analysis as long as you have your sensor detailed specs. Give it a try and see what you think.
http://lars-frogner.github.io/Astrophot ... index.html

Cheers,
JT

[bobharmony]

Can't you just use the camera's LCD screen to judge if the lighting was sufficient? This is what I do, regardless of whether I take them at night or the morning after and it seems to work fine. If it looks medium grayish with the sides a bit darker, it's good.

At night I use a white stucco wall that I aim my headlamp at. Since the scope does not have the wall in focus it looks evenly white to the camera. That way flicker is als no problem. In the morning I use a flattened paper towel.

It sounds crude but I have no reason to believe it's wrong, from experience. I have a feeling that there's a good deal of tolerance or else it would not work.

But thanks for raising the problem, and I would like to know if you get noticeably better results eventually.

Henk - I have been operating under the same assumption as you for a long time. Lately I have seen people suggesting that using what we have been doing may be underexposing the flats and I am checking out that possibility for myself. My main issue is the noise level in my stacked images, I am seeing if brighter flats can reduce that as it seems they will add less noise to the final result. For sensor and cc spot removal they have been working well.

I will report on my results. So far it seems that processing with brighter flats allows me to either stretch the data further with the same apparent noise level, or to get the same level of detail with less noise.

Bob

[bobharmony]

Here is your ADU value:

Screen Shot 2019-11-04 at 10.12.42 PM.png

Did you find out what your maximum ADU value is, though? Because yours could be 30,000 in which it is still too dark, or it could be 15,000, in which it is too bright, from my understanding

James - I would love to see what you come up with by changing the image to 14 bit. If the mean ADU drops to around 4K, that will tell me how much further I need to go. Your numbers are falling pretty close to Peters, so I am not sure what to expect.

Bob

[Baskevo]

James,

Can you upload two examples that I can take a look at? The first being what you think is a good flat and the other your "maximum" flat (with your pixels saturated and the resulting image white?). I need to see it so I can figure it out.

Stu

This is a full exposure, maximum brightness, 20 second exposure. Same numbers as my 15 second exposure. No shirt or anything to block the light, so it is the maximum mean ADU value I can obtain, measuring at 16-bit:

This is a flat I took recently, it's more than 51% of the maximum mean ADU value, so I was a little worried about it, but it worked perfectly and cleared up my dust motes. I got this method from the website I shared earlier in this post, which I followed step-by-step:

I thought my camera is measured in 16-bit because it is a canon (I don't know, that stuff kind of confuses me). at 14-bit, the over-exposed flat mean ADU value was around 4,000.

JT thank you for sharing that! I'm going to check it out!

[Stuart]

James,
Can you upload the actual files to dropbox or something? It's hard for me to tell with the screen shots.
Stu

[Baskevo]

James,
Can you upload the actual files to dropbox or something? It's hard for me to tell with the screen shots.
Stu

sure here you are: https://photos.app.goo.gl/FT5g3s3zWR5DTEmX7

[bobharmony]

Have you guys tried a piece of software called Astrophotography Lab? It is freeware and was promoted by the author over on AF about 2 years ago. It is a very powerful title for image and sensor analysis as long as you have your sensor detailed specs. Give it a try and see what you think.
http://lars-frogner.github.io/Astrophot ... index.html

Cheers,
JT

JT, thanks for that link. I will give that a shot, it's about time I start doing my own lifting on measuring my flats

Bob

[SkyHiker]

Can't you just use the camera's LCD screen to judge if the lighting was sufficient? This is what I do, regardless of whether I take them at night or the morning after and it seems to work fine. If it looks medium grayish with the sides a bit darker, it's good.

At night I use a white stucco wall that I aim my headlamp at. Since the scope does not have the wall in focus it looks evenly white to the camera. That way flicker is als no problem. In the morning I use a flattened paper towel.

It sounds crude but I have no reason to believe it's wrong, from experience. I have a feeling that there's a good deal of tolerance or else it would not work.

But thanks for raising the problem, and I would like to know if you get noticeably better results eventually.

Henk - I have been operating under the same assumption as you for a long time. Lately I have seen people suggesting that using what we have been doing may be underexposing the flats and I am checking out that possibility for myself. My main issue is the noise level in my stacked images, I am seeing if brighter flats can reduce that as it seems they will add less noise to the final result. For sensor and cc spot removal they have been working well.

I will report on my results. So far it seems that processing with brighter flats allows me to either stretch the data further with the same apparent noise level, or to get the same level of detail with less noise.

Bob

Thanks Bob

I took a look at some of my flats. I don't have many because I toss them after they are used to save disk space. But they vary quite a bit. In the end it's all about the master flat, those I keep. I looked the histogram in the corners vs. the center and the largest difference I found in the count was about a factor 2. In other cases it was less, this may depend on the scope used.

I think the proper way to look at it is to brighten the flat using a linear curve from the origin at zero (black point) to the maximum of the histogram (set the white point there). If that stretched image looks smooth to the eye it ought to be good enough because the effect is multiplicative. Well with the inverse of course but in that range of a factor 2 max, it would look smooth too if the flat itself looks smooth.

Many of the flats I have look too close to the origin (left) but some of the later ones are further to the right. Despite that, most of them look smooth enough that I would trust them to do the job. So, I'm still skeptical about how critical this is.

With darks it's quite different because its effect is by subtraction, so noisy darks can be really bad for the end result because you are subtracting small noisy numbers from small noisy numbers. With multiplication of flats, a simple smoothness check should be good enough and it's not all that sensitive. But true, further to the right is better but I don't think it is the bottleneck in most cases.

Again, thanks for looking into it. If you find kmprovement I would like to see some actual results and comparisons, if that's possible.

[Stuart]

James? Are you shooting in .jpg? The files you uploaded were .jpg files. If so, that's your problem. You should be shooting in RAW. If you're shooting in .jpg none of this will work.

[Baskevo]

James? Are you shooting in .jpg? The files you uploaded were .jpg files. If so, that's your problem. You should be shooting in RAW. If you're shooting in .jpg none of this will work.

No, they are not jpg? They are .CR2 files which are canon raws.. why do you think they are .jpg..?

hmm I just downloaded the files from google photos and for some reason they download as .jpg files. They are not .jpg files though. let me try uploading them some where else

[Baskevo]

Okay try these:

Over-exposed flat:
https://1drv.ms/u/s!AnLt545x1S3QiFZINVJrD3kxEttq

Normal flat:
https://1drv.ms/u/s!AnLt545x1S3QiFe2MlxI1DcjLPzk

[Stuart]

Those worked.
I extracted the lightness from each so I'd get a single value as opposed to an RGB value
The over exposed one I get a value of 65,533 which is what I would expect from a saturated chip
The regular flat I get a value of 41,517.6. That's a bit bright. Looking at your FITS header you were at 3.2 seconds (not sure that's accurate) But assuming it is, and you want a value of 30,000 then I'd go 31,000/41,517 X 3.2 = 2.3 seconds.

I can post screen shots if you want.

[Baskevo]

Those worked.
I extracted the lightness from each so I'd get a single value as opposed to an RGB value
The over exposed one I get a value of 65,533 which is what I would expect from a saturated chip
The regular flat I get a value of 41,517.6. That's a bit bright. Looking at your FITS header you were at 3.2 seconds (not sure that's accurate) But assuming it is, and you want a value of 30,000 then I'd go 31,000/41,517 X 3.2 = 2.3 seconds.

I can post screen shots if you want.

Yeah can you?

How are you extracting the lightness? I'm measuring it as a pure RAW image, so there is no RGB to extract. Is that why you are getting such drastically different numbers than me?

[Stuart]

Okay.

Step 1: Extract the lightness. (There is a button at the top that can shortcut this also)

Step 2: Open statistics process, find the file, click 16 bit, and measure

Does this help? I only did the one file.

[Baskevo]

Okay.

Step 1: Extract the lightness. (There is a button at the top that can shortcut this also)
extract lightness.jpg

Step 2: Open statistics process, find the file, click 16 bit, and measure
Statistics.jpg

Does this help? I only did the one file.

Yes I did that earlier. I had to switch back to RGB from gray to be able to do it, but the reading wasn't much higher than 16,000. Are you using the Pure Raw setting under format explorer?

[Stuart]

Uhh.....the what under what now? I didn't mess with anything under that--I just used whatever defaults came with the program. I'm honestly not sure what setting you're referring to. Now I think you're going to have to do a screenshot.

[Baskevo]

Uhh.....the what under what now? I didn't mess with anything under that--I just used whatever defaults came with the program. I'm honestly not sure what setting you're referring to. Now I think you're going to have to do a screenshot.

Haha Another screenshot I am currently stacking an image on the final step (the longest step), but I will take a screenshot after it is done.

It is under format explorer -> RAW -> edit preferences, and then I believe on the bottom-left corner you select "pure RAW." apparently this is supposed to make sure PI has the right format for DSLR raw images and doesn't debayer your photos at the wrong time or something? I'm not sure why you are supposed to do it, but every tutorial I've seen recommends it for working with DSLR images. Here's a link showing how to turn on the "right" settings for DSLR raw images: https://dslr-astrophotography.com/dslr- ... ixinsight/

I've never tried it without it, so I can't tell you if it makes a difference or not, but I suspect that is why we are getting different values for ADU. I believe we still had the same concept though

[Juno16]

Have you guys tried a piece of software called Astrophotography Lab? It is freeware and was promoted by the author over on AF about 2 years ago. It is a very powerful title for image and sensor analysis as long as you have your sensor detailed specs. Give it a try and see what you think.
http://lars-frogner.github.io/Astrophot ... index.html

Cheers,
JT

Astrophotography Lab really looks like an amazing piece of FREE software!

I have been experimenting with it some and it has a multitude of very valuable tools. I just need to spend more time and learn how to use its features.

Thanks JT!

Jim

[Stuart]

Hey James,
When I'm back at that computer I'll check. I'm sure it's some setting somewhere. Anyway, your saturated one is saturated, so you want to be 50% of that number.
Stu