AP Software Experiences

[ram]

So you know I've played around with PI a lot and I use SNR/SNRWeight to determine my choices and what I've learnt is that even though there are small differences here and there for most options, I can't tell the difference visually. If I can tell the difference, it's usually something that shouldn't be done but if I'm doing something right (and there are multiple ways of doing it) then the small differences in SNR aren't significant. Even though it has felt good to extract out every db of SNR in my recent project, the final difference in the between my best and worst options (out of more a hundred evaluated) is about 5 dbs (this is using the ImageAnalysis->SNR script in PI). I think there are also aspects of PI like the adaptive/local normalisation that go beyond look at pixels independently during integration. Nonethless I take your point about Startools and it's why I downloaded it to try out - in theory it should magically produce better images than all the fussing around we do in PI. In practice... I need to spend more time with it. I mean as programmers we really want the tedium of programming to go away, right? We should be able to say "take this data and make me the best image you can with it". But it seems to make every photon count, there are no short cuts (yet).

What would be good is having comparable use cases: a dataset where Startools made a clear difference vs. the PI approach (or more robustly, how often does it make a difference to using one approach or the other for a large number of datasets of different quality). I went on a quest to extract the most signal in my latest project and while there are better and worse options in terms of SNR, it's either the choices of how much data to use or which artistic choices we go with that influences the final look of the image more than small changes in the SNR.

--Ram

PS: My first programming on a personal basis was also on a Z80 (and a VAX/VMS mainframe). I actually was into hardware first when I was a teenager and I built a simple computer using a Z80 chip complete with making the entire PCB and using ferric sulphate to etch out the copper according to the design I copied from a magazine... the only "program" it ran was a telephone call redialler programmed using a calculator keyboard.

[startoolsastro]

...I can't tell the difference visually

Nail on head! The core tenet of the approach to noise in StarTools is that noise is only a problem if you can see its effects in the final image. All algorithms in StarTools work backwards from what the user sees in the final, heavily processed, stretched image.

Once you have that ability - to retroactively determine how SNR per-pixel develops - then something like, for example, MURE Denoise is quickly revealed to be sub-optimal; how can a noise reduction routine you run at the very beginning of your processing, know where noise grain is visually objectionable if it has no idea of how the image will be processed and stretched?

It's the same thing with deconvolution in PI (or other apps); how can its regularization algorithm accurately determine what is the optimal balance between ringing/grain artefacts and detail, if it has no idea how visible noise grain and artefacts will be once the deconvolved dataset is stretched?

if I'm doing something right (and there are multiple ways of doing it) then the small differences in SNR aren't significant.

Of course, SNR analysis/comparisons is fraught with caveats and is only useful under very strict conditions. That said small differences in signal fidelity of the source material can have a significant impact on algorithms that are extremely susceptible to noise (e.g. deconvolution).

I mean as programmers we really want the tedium of programming to go away, right? We should be able to say "take this data and make me the best image you can with it". But it seems to make every photon count, there are no short cuts (yet).

Apart from automating tedious tasks that require no subjective inputs, and throwing away less signal by respecting it better, facilitating a human's interpretation of what a "best" image constitutes, is definitely extremely high priority.

What would be good is having comparable use cases: a dataset where Startools made a clear difference vs. the PI approach

The benefits of back and forward propagating signal throughout the full workflow chain should readily demonstrable on almost any dataset that is reasonably clean and not too oversampled, particularly when deconvolution and noise reduction is involved. Any dataset where you don't see marked improvements, do let me know (this would actually be considered a bug!)

PS: My first programming on a personal basis was also on a Z80 (and a VAX/VMS mainframe). I actually was into hardware first when I was a teenager and I built a simple computer using a Z80 chip complete with making the entire PCB and using ferric sulphate to etch out the copper according to the design I copied from a magazine... the only "program" it ran was a telephone call redialler programmed using a calculator keyboard.

Ha! That's awesome!
I still dabble with 6502 assembly now and then for fun (here is a dynamically generated Commodore 64 version of some of the StarTools content ). Why does this exist? Long story...

[ram]

So I've signed up for the startools forum - once I get approved for posting there are some issues I've run into which I'll write about if I don't solve them ahead of time. I'm really not a big fan of learning by video tutorials - I prefer to read text. My understanding so far is that all the preprocessing up to creating a stacked image must be done outside of StarTools?

--Ram

[startoolsastro]

So I've signed up for the startools forum - once I get approved for posting there are some issues I've run into which I'll write about if I don't solve them ahead of time. I'm really not a big fan of learning by video tutorials - I prefer to read text. My understanding so far is that all the preprocessing up to creating a stacked image must be done outside of StarTools?

--Ram

You were just approved! Indeed, StarTools is focused on post-processing only. You will want to give these dos and don'ts a look over before processing your first stack in StarTools. Some things may be obvious to a veteran user, though some may not be. If useful, you will also find a PI <-> ST translation table on the forum.

One thing not immediately obvious if you are otherwise quite familiar with AP processing, is that ST (preferably) processes your luminance and chrominance signal separately (yet simultaneously). Looking forward to your findings/questions on the forum!

[STEVE333]

So I've signed up for the startools forum - once I get approved for posting there are some issues I've run into which I'll write about if I don't solve them ahead of time. I'm really not a big fan of learning by video tutorials - I prefer to read text. My understanding so far is that all the preprocessing up to creating a stacked image must be done outside of StarTools?

--Ram

You were just approved! Indeed, StarTools is focused on post-processing only. You will want to give these dos and don'ts a look over before processing your first stack in StarTools. Some things may be obvious to a veteran user, though some may not be. If useful, you will also find a PI <-> ST translation table on the forum.

One thing not immediately obvious if you are otherwise quite familiar with AP processing, is that ST (preferably) processes your luminance and chrominance signal separately (yet simultaneously). Looking forward to your findings/questions on the forum!

I'll be watching this "marriage" with interest. Who knows, you may convert me Ivo.

Steve

[startoolsastro]

Who knows, you may convert me Ivo.

I'd rather have the results speak for themselves Steve, rather than me telling you this or that is "better".

To help with a side-by-side, apples-to-apples comparison, it is - to a certain extent - possible to "inject" another program's stretch into StarTools' Tracking engine (there may be some minimal signal loss, but nothing too bad), so if you're interested in comparing the results of ST's tools working with the exact same stretched data as some other software (for example PI), then that might be of interest.

E.g. you would get some clean, not too oversampled (but not undersampled!), mono data. Clean it up with DBE/ABE (this is where some signal loss may occur), save that virgin linear data away. Then do a global stretch (or a screen stretch you plan on making permanent) in PI and save that virgin stretched image away. Then in PI, undo that stretch, do your - for example - decon and anything else that requires linear data, then redo that same stretch you performed earlier.

In ST, open the virgin linear data you saved away in PI and switch Tracking on, go into the Layer module, Open the virgin stretched PI image into the foreground and Keep that. Now StarTools will be able to tell how you got from A (linear image) to B (stretched image) - though minus the DBE/ABE step - and it can build on this.

Continue processing as normal in ST (and PI), testing out the different behaviors of the different modules when using them on this "stretched" data in isolation or in conjunction. For example, try doing some Wavelet Sharpening with Decon on top and notice how Decon is completely unfazed by the stretch or the subsequent wavelet sharpening (in fact it uses the extra knowledge to its advantage). Or you can turn off Tracking immediately and evaluate how noise reduction is performing, etc.

Happy to take anyone on a tour with such a dataset privately, or in another thread!

[Juno16]

Very interesting!

[Juno16]

So I've signed up for the startools forum - once I get approved for posting there are some issues I've run into which I'll write about if I don't solve them ahead of time. I'm really not a big fan of learning by video tutorials - I prefer to read text. My understanding so far is that all the preprocessing up to creating a stacked image must be done outside of StarTools?

--Ram

Hi Ram,

Are you using the name "Ram" on the Startools forum?

I would like to keep up with the posts. This will be interesting! Thansk!

[ram]

My account name is ramdom. I don't see how I can set my name on there but "ramdom" should work.

I am still trying to see if I can follow the ST Quick Start guide on my (difficult) data set to see if I can even produce something reasonable but so far it has been difficult. I will post by the end of the night if I don't make progress. I can post it here as well in my Ou4/Squid thread - it is a challenge to process my 53 hour master light into something better than what I've done so far.

--Ram

[Juno16]

My account name is ramdom. I don't see how I can set my name on there but "ramdom" should work.

I am still trying to see if I can follow the ST Quick Start guide on my (difficult) data set to see if I can even produce something reasonable but so far it has been difficult. I will post by the end of the night if I don't make progress. I can post it here as well in my Ou4/Squid thread - it is a challenge to process my 53 hour master light into something better than what I've done so far.

--Ram

Thanks Ram. I will look out for your posts.

Given the amount of data that you have along with the beautiful images that you have produced, it will be really cool to see some comparison work done in Startools. Perhaps Ivo can take a whirl at the data too since he has all of the tools.

[STEVE333]

Who knows, you may convert me Ivo.

I'd rather have the results speak for themselves Steve, rather than me telling you this or that is "better".

To help with a side-by-side, apples-to-apples comparison, it is - to a certain extent - possible to "inject" another program's stretch into StarTools' Tracking engine (there may be some minimal signal loss, but nothing too bad), so if you're interested in comparing the results of ST's tools working with the exact same stretched data as some other software (for example PI), then that might be of interest.

E.g. you would get some clean, not too oversampled (but not undersampled!), mono data. Clean it up with DBE/ABE (this is where some signal loss may occur), save that virgin linear data away. Then do a global stretch (or a screen stretch you plan on making permanent) in PI and save that virgin stretched image away. Then in PI, undo that stretch, do your - for example - decon and anything else that requires linear data, then redo that same stretch you performed earlier.

In ST, open the virgin linear data you saved away in PI and switch Tracking on, go into the Layer module, Open the virgin stretched PI image into the foreground and Keep that. Now StarTools will be able to tell how you got from A (linear image) to B (stretched image) - though minus the DBE/ABE step - and it can build on this.

Continue processing as normal in ST (and PI), testing out the different behaviors of the different modules when using them on this "stretched" data in isolation or in conjunction. For example, try doing some Wavelet Sharpening with Decon on top and notice how Decon is completely unfazed by the stretch or the subsequent wavelet sharpening (in fact it uses the extra knowledge to its advantage). Or you can turn off Tracking immediately and evaluate how noise reduction is performing, etc.

Happy to take anyone on a tour with such a dataset privately, or in another thread!

Thank you Ivo for the very generous offer of some of your time Things are a bit hectic here at the moment, but, when I have the time I would like to look into Star Tools. I'm only 79, so, no time like the present to keep learning about alternate AP processing possibilities.

Does Star Tools offer a trial of some sort?

Steve

Steve

[startoolsastro]

53 hour master light

53 hours! Nice! Here to help if there's no progress.

Does Star Tools offer a trial of some sort?

StarTools is freely downloadable from the website. The only thing you cannot do in trial mode is save your work.
You may also be interested in the 1.7 alphas (also available in the download section) with lots of new functionality as well as full GPU acceleration.

[pakarinen]

To me, learning the user interface and the "particulars" for each piece of post-processing software is more difficult and more of a PITA than was my differential calculus class in college!

LOL. When I saw that itty-bitty green textbook for Diff Eq, I knew it was going to be painful.

[XCalRocketMan]

I too will be watching this with interest. StarTools seems like a very viable product (if I can only find the time to spend with it).

[XCalRocketMan]

To me, learning the user interface and the "particulars" for each piece of post-processing software is more difficult and more of a PITA than was my differential calculus class in college!

LOL. When I saw that itty-bitty green textbook for Diff Eq, I knew it was going to be painful.

Back in my undergraduate college days (and we are talking WAY back - '70s), I had just picked up my Quantum Mechanics text book from the bookstore. Now mind you, we're talkin' the 1970's. The book was hardback, only about 1/2" thick but cost about $90! The opening paragraph in the very first chapter read: "It is assumed the student has a thorough knowledge of Tensor analysis" ---- I knew I was in trouble

[STEVE333]

To me, learning the user interface and the "particulars" for each piece of post-processing software is more difficult and more of a PITA than was my differential calculus class in college!

LOL. When I saw that itty-bitty green textbook for Diff Eq, I knew it was going to be painful.

Back in my undergraduate college days (and we are talking WAY back - '70s), I had just picked up my Quantum Mechanics text book from the bookstore. Now mind you, we're talkin' the 1970's. The book was hardback, only about 1/2" thick but cost about $90! The opening paragraph in the very first chapter read: "It is assumed the student has a thorough knowledge of Tensor analysis" ---- I knew I was in trouble

How did we ever make it through those classes? Now I know what happens to old scientists. They become astrophotographers!!!

Cheers

Steve