PIER-05 Star halos again

Dear Subscribers, Steve, Phil and Peter

I have been looking at as much of the data as I can over the last three months to see if these star halos are an isolated incidence, or something more common.

My estimate is that mostof the images in the last month are quite severely affected by bright star halos - even if the most obviously cloud affected sub-frames are removed.  

There are some wonderful images that I have reduced and posted on Astrobin e.g. M45, Cave Nebula and the IC1396 that suffer much less (if at all) but all come from data taken in September.

More recent data from October and November appears to have quite severe halos, including all of the objects I have proposed, e.g. Jellyfish, IC348, Shark, Helping Hand.  Of these I have only posted one on Astrobin (IC348) even though it had quite a marked halo around the bright star in the centre.  I have not posted the others, because I believe it doesn't necessarily reflect well on Roboscopes.  Or not least until I understand better the origin of the problem.  I could also ask for more data on each on these objects.  Indeed, I did for this Helping Hand - to no avail, as the hals second time again were, if anything, worse.  Having more-or-less a 100% fail rate one's proposed objects may also a bit dispiriting...

Of course, it is really difficult to tell whether this is due to cloud, optics, stellar profile or, indeed, my poor processing.  But I did look at the recent M42 data, which was taken in Sep, Oct and Nov.  Compared to the others, the November data has very marked halos (see attached - and comparison with the early Oct data).  The September and October data is much better from this viewpoint, with the early Oct data marginally better than the Sep data (at least in my estimation).   Of course, comparing individual data set like this is fraught with difficulties as so many variables are at play here.  

But it would be helpful to have other Pier 5 subscribers thoughts.  

Am I being too picky about data quality?  Have we just had a bad month with high cloud/haze?  Is this just something I have to live with [limitation of optics and/or site], and either avoid bright stars and or get better as post-processing and removing data halos? Even if this were true, it would also serious implications for what imaging is possible, given so much light is being scattered out of the PSF.  

Despite all this, I am enjoying myself - but I would dearly love to understand what is going on a little better.  Either to fix the problem or reset my own expectations.

Grateful for your collective wisdom...  

Thanks 

Brian

Hi Brian

I will answer more fully later when I am more awake

We are as I have said several times already in the middle of our wet season so yes our skies are not as good this
time of year, as for the rest I will as I said go over it this weekend

In the mean time this is Mikes version of your same dataset you have
highlighted above. he sent it my privately to critique a couple of
nights ago. The first one is obviously clipped a little which I informed him about
but the second is not and much better.

I know Mike will not mind me sharing them which is why I have not asked him

The point I am making is its a great image from the data and Mike is very
fussy. when we have dew, haze etc he is normally a very good barometer
and lets me know.  So I am somewhat surprised

None the less I will look into it with Pete this weekend and come back to you properly

Steve

Hi Steve, 


Thanks for the response.  Mike’s data reduction is very impressive, and if I were able to achieve that final result, I would be extremely happy. Although my images only had the minimal of post-processing, I doubt there is any way I could have achieved those results - particularly with the Nov data.  

This suggests to me, that the issue might simply that I have to work harder with the data reduction.

However, it also means that the intrinsic quality of the site at this time is likely to be worse than my backyard, since these are not skills I have had to deploy with data from my own observatory (at least not after I cleaned by secondary!) 

I am also aware that this time of the year is not good.  I know that you have “said this many times” but my questions are driven more by where to set my expectations as a new user. Is a 0% success rate acceptable? How much more do I have to lift my post-processing skills in order to deal with the quality of the data coming from the scope in Oct/Nov? Indeed, I am a little confused about sky quality at the moment as only last week you were calling for more submissions due to the unexpectedly good weather. 

Brian

Brian

while I think about it may I ask your to run through your workflow with images a little please, from when you receive the data to how you select it, calibrate the data etc please ?

I have just had a very fast look at the helping hand and I can go way way deeper and cleaner than you without any trouble is why I ask.

Basically I want to see if there is something being missed in the steps that we can point you towards 

HTH

Steve

Hi Steve, 

thanks for the help. Your comment that you can do substantially better both excites and troubles me simultaneously!

i am a PIxInsight user. I use Blink and sub-frame selector to weed out bad frames (defined in subframe selector as approx 50% worse than the mean in FWHM/starcount/noise ratio/eccentricity).

I then use I the Weighted Batch Pre-Processing script on the selected good subs to do the initial processing. I used to do all the pre-processing steps “by hand” but I leave it to WBPP now. 

After that the post-processing steps in PI are:

Dynamic Crop
Automated Background Extraction
Photometric Colour Calibration
MLT noise reduction (linear)
HistrogramTransformation 
TGVNR noise reduction (non-linear)
HDR compression (sometimes)
Local Histogram Equalisation (sometimes) 
Morphological Transformation
MLT sharpening 

All steps done with the appropriate Luminance or StsrMasks (straight or inverted). 

And then a few tweaks with histograms/Noise reduction/colour saturation as needed.

I guess if there is a gross effect on the depth I am getting, most likely its something I am doing wrong in WBPP - but I thought it was fairly foolproof. 

i suppose I could buy APP and try that.  

 

Hi Steve, 

thanks for the help. Your comment that you can do substantially better both excites and troubles me simultaneously!

i am a PIxInsight user. I use Blink and sub-frame selector to weed out bad frames (defined in subframe selector as approx 50% worse than the mean in FWHM/starcount/noise ratio/eccentricity).

I then use the Weighted Batch Pre-Processing script on the selected good subs to do the initial processing.

I use the flats, bias and darks from your calibration files. I used to do all the pre-processing steps “by hand” but I leave it to WBPP now.  I guess the inly odd thing about the Roboscopes data in WBPP is that it doesn’t pick up the debeyer pattern automatically  and I have to tell the script manually it is RGGB. 

After that the post-processing steps in PI are:

Dynamic Crop
Automated Background Extraction
Photometric Colour Calibration
MLT noise reduction (linear)
HistrogramTransformation 
TGVNR noise reduction (non-linear)
HDR compression (sometimes)
Local Histogram Equalisation (sometimes) 
Morphological Transformation
MLT sharpening 

All steps done with the appropriate Luminance or StsrMasks (straight or inverted). 

And then a few tweaks with histograms/Noise reduction/colour saturation as needed.

I guess if there is a gross effect on the depth I am getting, most likely its something I am doing wrong in WBPP - but I thought it was fairly foolproof. 

i suppose I could buy APP and try that.  

 

No need to trouble you, processing is like life, the day you stop learning new skills is the day you become arrogant or dead  

Myself and Pete will take a look this weekend 

As I said we are here to help

Steve

Hi Steve, 

Many thanks.  

Are you able to share your quick reduction of the Helping Hand (well named) so I may see how much deeper you were able to get? 

it might give me something to aim for over the weekend too?

brian 

Ps

Automated batch processing is the worst thing you can do, you will loose half your data!

You should speak to Pete as he is far better than I at processing but none the less leaving pixinsight to choose is a massive big fat NO 


Pixinsight is a very powerful tool but needs a humans touch and an artists eye at times to get the best from it  

Steve

I tend to do deliveries for my company of a weekend as the traffic is easier hence why I said I would answer more fully later.

I have just spoken tonPete and he said he has more time this weekend so he will take a proper look and let you know 

HTH

Steve

Hi Steve et al,

Encouraged by your observation that you were able to to way way deeper on the Helping Hand that I was able to, I thought I would see whether I would have any better luck with AstroPixelProcessor than WBPP in PI.

Although pre-processing is new to me in APP, it was quite a straightforward - if long - process.  I used the full data set of 260 frames, the top 80% in APP's quality measure and the top 25% in quality.  The final images went no deeper than I was able to achieve with PI (and the 25% data-set was, of course, significantly less deep).   The star halos were just as pronounced in each data set.  So whatever I am doing wring in PI WBPP, I must also be doing wrong in APP.  

I also processed the latest N13333 mosaic panel 1 released - and again I found a significant halo around the bright star Atik, slightly worse that the one in the same area imaged  a month ago (job 1237).

I think I have taken this about as far as my (limited) ability will allow.

Its either 
a) A problem with my processing
b) A problem with the sky
c) A problem with the telescope

More than happy to have it demonstrated it is a).

Finally I would note that Astrobin's IOTD is, in fact the Helping Hand nebula.  Almost 17hours on a Sharpstar 150mm f2.8.  Not an OSC image, but it is not far off the 18hours on the Tak obtained here.  All things being equal, it should be possible to get to something approaching the quality of this image - most likely in the hands of a more skilled APer.  Consequently, I look forward to Peter's analysis of the situation.  

Regards

Brian

Hi All, 
I saw that data for  Cone nebula popped up on the ftp site, which I had already taken  at a similar focal length (550mm) albeit with a Skywatcher Esprit 100mm telescope from my home observatory back in January. It was nominated for a top pick on Astrobin, so I figured it was a reasonable comparison to make.  

Both use an OSC camera, although the the Espirt data is taken with an ASI2600MC using larger 5.76um pixels, so the plate scale on the Tak is about 30% smaller than the Esprit.   The collecting area of the Tak is 3.24 x that of the Esprit, but the pixels on the Esprit are 2.4 x bigger.  My site is Bortle class 3-4  equivalent to Roboscopes site.   The seeing at my site is rather poor: 3-4arcsec at best.  

Both observations were taken under similar conditions (moon down) and had similar exposure times 2h30 for the Esprit 100 and 2h50h for the Tak 180.  While "all comparisons are odious" and we are talking about refractor v reflecctor - they should be "sort of" comparable. 

I resduced the Tak data in *exactly* the same way as I had done with the Esprit data, and here's the result.

Both are cropped to reduce the file size.  The star bloat in the Tak data is clear.   Furthermore, the definition and depth in the Tak data look slightly down on the Esprit data.  This is exactly whet I might expect in hazy conditions or with dewed optics.  However, the Tak data was taken on 7 Nov, during an apparently good run of weather.  

Please be assured, I am not trying to say the "my site/teepscope is better than your site/telescope".  I want both to be as good as possible.  And who better to provide feedback that the customers?

Equally, if I am doing something really stupid, I would love to know.  Steve's comment that he was able to go way way deeper than my image processing has me intrigued and excited.  But at least in this comparison my sub-optimal image processing was applied equally to both sets of data.

Brian

Hi,  

I attach the N1333 mosaic (binned x2 to fit). .Again its characterised by blue star halos and noise.

In my limited experience these later data sets (late Oct/November) appears to be worse than the earlier once to which I have access, but the technical team and other Pier 5 subscribers would be in a much better position to know.    Happy for someone more experienced in data reduction to show me that it is just me and that the data is fine.   With improved SNR and no halos, it does make for a stunning mosaic.  Kudos to Vikas for proposing it. 

Brian 

Hello Brian,

Don't never use automated background extraction.
Use  dynamic background extraction, it's far more superior than automated background extraction.
Second use the script Normalize Scale Gradient to compute the accurate weights of the individual subs.
The computed weights in the image integration process are mathematically not very robust. 
your worst subs could get the highest weight in other words your worst subs could be the best during image integration.

See:
PixInsight NormalizeScaleGradient Part 1: Introduction - YouTube
PixInsight NormalizeScaleGradient Part 2: Setup - YouTube
PixInsight NormalizeScaleGradient Part 3: Output and Results - YouTube

Hi Mike,

Many thanks for the tip on NSG.  I tried it out  on the cone nebula and it definitely improved the SNR.  Not massively, but enough to be noticeable.  I plan to use it in future as part of my workflow.  

It did not, however, remove the star halos.  

Re ABE v DBE.  I completely get where you are coming from here.  I actually use both, but would prefer to use neither.  My flat fields should be good enough.  [I think this is a hangover from my professional career!]  Nevertheless the scale of the ABE/DBE correction is at least 10-100 x times greater than the scale of the halos.  

These more I look at these halos, the more they look like dewed optics to me.  Not least the multiple "legs" to the halos.  

Brian

Hi Brian,
Sorry Its taken so long but I thought I would throw my hat into the ring and offer my opinion.

I have been looking at these halos on Pier 5 and wanted to check what was going on. All locations are generally seasonal , the autumn is the worst season for our location and we do suffer from high haze and dew. I will say that this year we have been caught out with the dew on several of the piers as we have never experienced dew like this here before. The optics were cleaned in the summer, and will be cleaned again on the next maintenance visit.

Some of the subs are definitely showing high haze and dew. When data sets are sent over we do send all of the frames collected good or bad. These need to be carefully checked and sometimes brutally culled as part of the pre-process routine to get the best out of them.
These hazy artefacts will become more apparent them more the data is pushed and some objects react differently depending on what they are and how aggressively they are processed. 

I now refer to the Helping hand data set. I have gone over and can see very clearly the haze moving in and out depending on the frames and dates. I have removed the bad frames out to the set and processed it very quickly and pushed it more than I would and especially around the stars.
The first thing I noticed were the choice of exposure time for a dark subject I think 300s subs would have been a better choice rather than the 180s. I am assuming the 180s choice was based on the bright star in the centre of the field? IMO the 2600MC is 16bit even at 300s has the dynamic range to handle it and would have helped more with the dark nebula. 

Here is my test process of the data set, I did force the stars more than I would normally to see how bad the halos were. Bearing in mind the type of subject this is and how much I pushed it the haze is minimal.



Regards
Peter

Hi Peter,

Thanks for looking at this for me.  Very helpful.  Your image certainly looks better than mine.  I hope you don't mind me asking, but what did you use to reject the bad subs?  And how many did you end up throwing away?  I actually culled to 25% of the total (using WBPPWGHT) to determine best frames for my version of the Helping Hand, and still had worse halos. 

So either I am removing the wrong frames or I am doing something sub-optimal is processing/post-processing. 

I will try the NSG routine routine suggested by Mike.  But it will take a fair amount of time to go through 360 frames!

Good to hear that you will look at the dew problem on the next visit.  Until then, I will probably stick to fields without bright stars and 300secs subs*

Brian

*You are right - I chose t180secs for "star-taming"purposes.  I am interested to hear your experience that 300sec subs are better for DN.  I wonder why this is, as mathematically, the S/N on each 180sec sub is massively sky-dominated, with the ReadNoise being only around a few per cent of the sky (at least according to my calculations).  

Hi Peter, 
Encouraged by your improved image of the Helping Hand, I tried a little experiment.  I simply removed all subs taken after midnight.  My rationale being that, if it is dew, then it is mostly likely to occur later in the night, once the optics cool to the dew point.  Midnight was an arbitrary call, but it provides a good a priori position with which to test the dew hypothesis. [There is at least three nights data in there, so there would have to be a little bit of a coiindence for it to cloud over on the second half of all three nights - unless there is local atmospherics eg ridge cloud that I don't know about.]

Having done that, I was left with 110 frames out of the 315 taken.   I attach the final reduced image below.  The halos are still there but much, much reduced.  And the SNR - while still not brilliant - is not significantly worse (if anything slightly better) that the full 315 frame stack. 

This leads me to think that:
a) There is a significant dewing of the optics. Any existing dew control on the  primary or secondary may need investigation.
b) I suspect that the optics could do with a clean/wash, since halos remain even it the frames before midnight.  Based on my own experience, they do look like a dirty/dewed secondary.  
c) I am confident that my data processing is not causing as bad as I was beginning to think, and I am pretty sure it is not the cause of the issue
d) I can't rule out high cloud/thin haze, but everrything I have done to date points to dew as the primary culprit at least.

Shades of the Gremlins movie, but I am beginning to suspect that "Don't observe after midnight" should be the approach until the dew issue goes away.