Sunday, 05 December 2021
  25 Replies
  674 Visits
0
Votes
Undo
  Subscribe
There are a number of remote observatory companies in Spain apart from Roboscopes and overall we are very similar in how and what we do, one
small difference between us however is when humidity is very high we do
not close the roof. We still gather data rather than simply close the
roof. The man reason for this as UK astronomers is dew and humidity is
something we deal with 12 months a year and in the UK and if we closed
the roof for humidity we would never open the roof a single night in all
365 of them. So we firmly believe that some data is far better than no data. Others amongst you may disagree with this philosophy and that's the point of this post

This approach mind does come benefits and issues

Let’s start with issues
  • Not all subs are good[/*]
  • The user has more work to sorting through and removing below par subs[/*]
  • Overall the data is not as good as it is the rest of the year due to atmospherics[/*]
  • You get all the data warts and all so yes some manual labour is 100% required[/*]
Benefits
  • The roof is open so you get data you would never receive with a closed roof[/*]
For the staff at Roboscopes the wet seasons are far more intensive for
us work wise because we have to keep on top of High humidity, dew, high
haze, not to mention sub standard data where we simply have to just
resubmit it for the group as a whole.
Every data set get checked with blink and we monitor the weather very
closely this time of year because high haze has a tendency of slipping
past the IR based weather systems because it may only alter the sky
temperature but 1º above clear and for the most part we leave a 3º
differential so the roof is not bouncing open/closed all night like a
yoyo.

Roboscopes staff will always do our best to keep on top of things and
let you know when we have foibles etc however you must also take into
account remote diagnosis of issues takes time unlike popping out into
your backgarden/yard with mark1 eyeball.

So at present we are in a quandary, do we mirror all the other remote
observatory operators we know and close the roof when the humidity is
high or do we leave it open and keep collecting data ?

Obviously for us closing the roof is by far the easier option but in the
very beginning Roboscopes chose not to do this as in our opinion even
with the extra work involved for us and you the members, you can still
get some good data/images during the wet season.

I know some of you are long standing members and will remember last
season was very extending rain wise and the roof remained firmly shut
for long periods. This season seams to be high haze and humidity instead
so far.

However we feel the time has come to draw a line under it being our
decision to make alone and as the members we want your opinion.

So a vote from each member would be fantastic please

High humidity times

1 - Roof open and keep imaging despite it being extra work for all concerned etc etc?

Or

2 - Close the roof until the nudity and other atmospherics get to a lower level

Whichever option gets the most votes on each pier we will simply go with what you ask

Ray, Mike and other dual subscribers may we be so bold as to presume
that your vote would count on any pier you are a subscriber ?


Foot note:

I will say that this year has been the worst of 5 years I have been in
Spain, Pier 3 as well as some others we have never had to even turn the
due control on in previous years so its been a little bit of an eye
opener, Not to mention the complexities of getting extra items stuck in
Spanish customs in this wonderful post Brexit world we now live in  :(

Extra dew control was posted out weeks ago and fingers crossed it should land very soon.

However if we don’t get it right or we mess up then you have our
apologies but believe me it’s not for the want of hard work at our end
but we are only human as dedicated as we are :)

Steve and the team

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


0
Votes
Undo
Hi, 

Phil that's very much appreciated. Having now spoken to the boss the one remaining bit I couldn't understand has been clarified. 

To anyone interested, this is what will take place if we agree to the change. First a review of the current procedure. 

Data is normally only being checked by Peter when a job reaches 100 percent. There is an exception to this when it is suspected that the data may have been affected by an unforeseen change in the weather conditions, or a suspicion that there may have been an equipment malfunction. On these occasions the data, or more likely part of it, is checked and if found to be crap that data will be rejected. That really helps and will continue to be done. :) 

So the  change being voted on will be that as soon as each job completes the data will be made available to us. The benefit is that there won't be the current delay in delivery whilst waiting for the data to be checked, which at productive times can be quite lengthy given the number of Syndicate piers. On the down side this may occasionally result in us having to resubmit more data as a result of dropping the final check. 

Nothing you are currently able to see in the job list or how data will be delivered to you will change. 

That is what can be done. To get data back any quicker will require a major change in how we submit our jobs. This has been briefly discussed before with suggestions such as limiting the number of hours per individual submission. This quite frankly will never work as that is contrary to each members freedom to, within boundaries, do what they want. 

I'm happy to go with the majority vote on this, so put me down as a spoilt ballot paper. Something I've always wanted to do. :) 

Cheers, 

Ray 

Ray
Roboscopes Guinea Pig


2 years ago
·
#4104
0
Votes
Undo
[size=100]The current process is:

[/size]
  1. [size=100]Data is input into the system by Pete.[/size][/*]
  2. [size=100]We wait until dataset shows 100% completion (This is a daily scheduled task that notifies us each day)[/size][/*]
  3. [size=100]When Pete is able he then checks the data with Blink (This could be the same day, or over a week depending how busy he is)[/size][/*]
  4. [size=100]Once Blinked if most of the data is below standard Pete will resubmit either part of, or the whole of a job. (Cases where job goes from 100% to 50% etc etc)[/size][/*]
  5. [size=100]Once Petes happy theres enough good frames he notifies me with a list and removes the datasets from the queue.[/size][/*]
  6. [size=100]I go through my list in systematic order (Pier 1-14) and zip each set up[/size][/*]
  7. [size=100]Once Zipped I transfer them to our NAS for backup purposes.[/size][/*]
  8. [size=100]I then FTP them upto the server into each individual folder.[/size][/*]
[size=100]Sometimes Pier 14 / Pier 6 sets get delayed due to the shear size of them so they usually get uploaded within a couple of days of me zipping them.[/size]

Phil McCauley
Roboscopes Website Admin


0
Votes
Undo
I'm still awaiting a response to my first post today. This laid out what I believe to be how things currently operate when any job runs over more than one night. What is described is backed up by observations. 

Waiting patiently, 

Ray 

Ray
Roboscopes Guinea Pig


2 years ago
·
#4102
0
Votes
Undo
I am happy to test out supplying un-blinked raw data.

I agree with Florent. It would be great if we should see the data arriving in real-time, but if we can't (and I fully understand the reason why not), I don't see the added value of holding up the data for 3-5 days for checking, when we are going to check the data we use anyway. I am sure there are more valuable things the staff could be doing.

The only case I can see would be that an initial check showed that the data was so poor an immediate resubmission is required and would be handled without our intervention. But if the raw data was made available immediately capture was finished, one of us is just as likely to spot the problem.

old_eyes
2 years ago
·
#4101
0
Votes
Undo
For me to zip daily and send Syndicate jobs as they reach 100% is no issue.

Phil McCauley
Roboscopes Website Admin


2 years ago
·
#4100
0
Votes
Undo
Hi all 

In my proposition , i ve also say that data transfert to the NAS should be automatic to spare time for roboscope crew . 

if you have to do the transfer to NAS manualy, you cant do it each day, i understand. 

it was just my opinion , you dont have to change all you workflow for me :) . As we are not behind the curtain , i can image there is some issue we dont see.  maybe a lot of increase traffic on you bandwith ? 
0
Votes
Undo
Hi Steve, 

Hopefully we are all on the same page as Florent, however, has something been overlooked? 

Have a chat with Phil before committing to anything right now, and discuss how that will work. 

 I foresee a few issues which I'll gladly share, but not until you've had a word with the poor soul with more problems to deal with. If he is happy, then fair enough, but I believe it'll require more than a couple of daily butties to convince him. :) 

Cheers, 

Ray 

Ray
Roboscopes Guinea Pig


2 years ago
·
#4098
0
Votes
Undo
Honestly, it makes a huge difference and my preferred option if I am honest. far faster for all concerned

We can trial on P1 & 14 for now if you wish ?

Cheers

steve

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


2 years ago
·
#4097
0
Votes
Undo
Good morning Steve, 

I was responding likewise. :) 

You can clear up my misunderstanding very easily.

Florent, if I have it correct, is asking for all data collected nightly, unblinked, to be made available. That may well put pressure on another of your overworked staff members to download all syndicate data after any night data was collected. 

You, if I have it correct and probably not, say data will only be made available, unblinked, upon 100% completion.

That is a big difference. If I am understanding you correctly then we need Florent to confirm that you are delivering what he requested. I'll be happy to give my response then. 

Cheers, 

pita


That means I missed his full meaning also :) As I thought he was suggesting once data is finished rather than waiting for the data to be blinked by Pete (which can take 3/4/5 days) we simply zip and send to members

This would speed things up for members obviously

Steve


Yes, if the data is not blinked until the collection is complete, then I think it is a waste of your valuable time to blink it. We are going to blink the data anyway, and we all have different thresholds for what we think is acceptable/good data.

The current procedure is clearly not catching all the weak data (cf the M45 dataset - job 1141), so is it really worth your effort and the delays it introduces into the process?

I am with Florent. Give us the 'warts and all' raw data and we can decide whether to resubmit or put in for additional data.

old_eyes
2 years ago
·
#4096
0
Votes
Undo
Good morning Steve, 

I was responding likewise. :) 

You can clear up my misunderstanding very easily.

Florent, if I have it correct, is asking for all data collected nightly, unblinked, to be made available. That may well put pressure on another of your overworked staff members to download all syndicate data after any night data was collected. 

You, if I have it correct and probably not, say data will only be made available, unblinked, upon 100% completion.

That is a big difference. If I am understanding you correctly then we need Florent to confirm that you are delivering what he requested. I'll be happy to give my response then. 

Cheers, 

pita


That means I missed his full meaning also :) As I thought he was suggesting once data is finished rather than waiting for the data to be blinked by Pete (which can take 3/4/5 days) we simply zip and send to members

This would speed things up for members obviously

Steve

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


0
Votes
Undo
Good morning Steve, 

I was responding likewise. :) 

You can clear up my misunderstanding very easily.

Florent, if I have it correct, is asking for all data collected nightly, unblinked, to be made available. That may well put pressure on another of your overworked staff members to download all syndicate data after any night data was collected. 

You, if I have it correct and probably not, say data will only be made available, unblinked, upon 100% completion.

That is a big difference. If I am understanding you correctly then we need Florent to confirm that you are delivering what he requested. I'll be happy to give my response then. 

Cheers, 

pita

Ray
Roboscopes Guinea Pig


2 years ago
·
#4094
0
Votes
Undo
Ray

I was responding to Florent's suggestion ?

Cheers

Steve



Hi all 

i understand that option 2 is not workable , you cant blink each day
data from 14piers .....    At home , if weather os ok for 2 days , i m
overwhelm by data with one or 2 rig....

my own opinion is that as part of a syndicate , roboscope crew dont have
to blink the data .  So you could send each day all the new data and we
, syndicate user , have to select the data.    One good point of it is
that we can process data more quickly and have some partial result that
can be ok or even enought


in other remote obsevatory , crew dont check the data as long that no error on equipement occurs during the night. 

for paid per hour data , i t is a good think to be sure to deliver good data each time as we dont plan the observation. 




florent

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


0
Votes
Undo
Morning Steve, 

Probably I'm misunderstanding this but I do not see any great benefit from the revised procedure you have described, and is not what I understood was being requested. 

The following is my current understanding of how data captured nightly is being checked for us.

After a night of data capture, time permitting, it is blinked by roboscopes to identify very obvious bad data. When this happens roboscopes (Peter) reduces the cumulative hours captured accordingly so that additional subs will automatically be taken. That is a tremendous benefit, saving us having to do so when we finally receive the data. This is most welcome and appreciated. :) This out of necessity can only be a quick check by Peter and still requires us to do a more thorough check and requires us to resubmit accordingly. 

Before wasting more of your time by continuing this post, is my current understanding correct? 

Cheers, 

Ray 

 

Ray
Roboscopes Guinea Pig


2 years ago
·
#4092
0
Votes
Undo
Morning all,

I wondered if you had thought on the suggestion of us not blinking the data from our end and as it completes we simply zip it up and send it to members. This would make a massive difference in the speed you get the data as Phil can just zip it as with impunity as it sees the 100% complete icon :)

Just let me know one way or another if you can please

Steve

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


2 years ago
·
#4091
0
Votes
Undo
Hi all 

i understand that option 2 is not workable , you cant blink each day data from 14piers .....    At home , if weather os ok for 2 days , i m overwhelm by data with one or 2 rig....

my own opinion is that as part of a syndicate , roboscope crew dont have to blink the data .  So you could send each day all the new data and we , syndicate user , have to select the data.    One good point of it is that we can process data more quickly and have some partial result that can be ok or even enought

in other remote obsevatory , crew dont check the data as long that no error on equipement occurs during the night. 

for paid per hour data , i t is a good think to be sure to deliver good data each time as we dont plan the observation. 




florent


Good morning Florent

We have no problem with blinking or not blinking but obviously not blinking saves us a lot of time and will speed up things :)

Piers-1 & 2 do not really suffer with dew
Pier - 3 a small amount (This I think has now been sorted)
Pier-5 a small amount for the first time ever (This I think has now been sorted)
Pier-14 I think we have now sorted it (testing last nights data today)

I have added an image below from Pier-10 this pier has been suffering quite badly for the last few weeks with the heavy humidity, howevert now the extra dew systems have finaly arrived in Spain and been added to the systems. This is B33 blue channel from last night with the dreaded Alnitak. I think this is acceptable data :)

So to sum up if all members are happy for us not to blink and you all self check and re submit as needed, then we are happy to do this option as it will speed things up a hell of a lot on subscription piers?

Steve

2021-12-14 11.43.29.jpg

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


2 years ago
·
#4090
0
Votes
Undo
Hi all 

i understand that option 2 is not workable , you cant blink each day data from 14piers .....    At home , if weather os ok for 2 days , i m overwhelm by data with one or 2 rig....

my own opinion is that as part of a syndicate , roboscope crew dont have to blink the data .  So you could send each day all the new data and we , syndicate user , have to select the data.    One good point of it is that we can process data more quickly and have some partial result that can be ok or even enought

in other remote obsevatory , crew dont check the data as long that no error on equipement occurs during the night. 

for paid per hour data , i t is a good think to be sure to deliver good data each time as we dont plan the observation. 




florent
2 years ago
·
#4089
0
Votes
Undo
I think Option 1 is the most practical for all concerned.

The only frustration as has been mentioned is not knowing if you have good data until the end of a run. But then if you keep capturing, you never know which subs will be poor quality anyway.

We have to realise that being part of a syndicate is not the same as owning a remote observatory. We can't 'look over the shoulder' of the Roboscopes piers while they are running.

There are interesting approaches to checking the data as it is captured (for instance https://www.youtube.com/watch?v=a4IkAUZkXH0), but all that does is relieve us of a quality control check when we receive the data.

I normally use blink to screen out any obviously defective subs, but I noticed with the M45 data that it was not until I had done the full WBPP on the subs that looked OK that the problems were visible. I probably need to include a sub-frame selector step in my process and be more data-driven instead of relying on visual inspection
2 years ago
·
#4088
0
Votes
Undo
[*]Data collected on main pier PC SSD drive[/*][*]Pete then checks using Blink[/*][*]Data is then zipped up[/*][*]Data is then transferred to our on site NAS drives as a hard backup copy[/*][*]Phil then uploads a copy of data to the server[/*][*]Data is then shared[/*][/list]Suggested system above lets call it - Option 2

  1. Data is taken partially on main pier PC SSD drive
  2. [/*]
  3. Pete checks partial data using Blink[/*]
  4. Partial data is zipped up[/*]
  5. Partial data is then transferred to on-site NAS[/*]
  6. Partial data is then uploaded to server[/*]
  7. Data is then shared
    [/*]

More data is then taken

we repeat above steps
  1. Data is taken partially on main pier PC SSD drive
  2. [/*]
  3. Pete checks partial data using Blink[/*]
  4. Partial data is zipped up[/*]
  5. Partial data is then transferred to on-site NAS[/*]
  6. Partial data is then uploaded to server[/*]
  7. Data is then shared
    [/*]
More data is then taken

we repeat above steps
  1. Data is taken partially on main pier PC SSD drive[/*]
  2. Pete checks partial data using Blink[/*]
  3. Partial data is zipped up[/*]
  4. Partial data is then transferred to on-site NAS[/*]
  5. Partial data is then uploaded to server[/*]
  6. Data is then share[/*]

To sum up


Option 2 is very very labour intensive not to mention difficult to keep track of what data has been sent out of a data set already that could be taken over the course of say 10 nights of a 30 day period, We are a team of 5 people and honestly there are not hours in the day for use to keep checking small finished amounts of finished data then uploading it and keeping track of it. We could at times have to repeat this 5/10 times for one data set.

I hope this explains my reticence because the logistics for us would be impossible for what is a wet season data blip that normally does not have a bearing upon the rest of the years data collection

However, I will keep my thinking cap on to see if I can come up with a better solution :)

Steve

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


2 years ago
·
#4087
0
Votes
Undo
yes , i understand , but may be changing the way to deliver the data . May be having some automatic rolling copy fo data from the shooting pc to the delivry NAS 

then we will do ourselves the sorting and selecting ? 
2 years ago
·
#4086
0
Votes
Undo
Hi  all 

i m for option 1 , i prefer data than no data (as now in my locations , 4weeks cloudy time , i m highly depressed ..)

But  , the problem is that we check data at the complet end of a job which may finish in a month or more . And if we resubmit , it will last another month or so . Which make project very hard to follow . 

may be we need a way to check more often the raw data so to be able to reschedule earlier , only the bad subs , and not all of them ? 

may be to have dirrectly acces to the rwa file each day so we will make our own project file ? 


Evening :)

Option 1 is fine, the only issue with data access early is the data is stored in the interim locally on the PC and does not go onto a network share until it is finished, We can't let customers have direct PC access through the firewall system I am afraid as I am sure you will understand

Remember this is only applicable while while the weather is changeable and for most of the year is not an issue

Steve

Please ignore my dylexia wherever possible, just be thankful I can control my Tourettes ;)

Things to do, so little time!

Steve
Roboscopes Tea Boy


  • Page :
  • 1
  • 2
There are no replies made for this post yet.
Be one of the first to reply to this post!

Follow Us

Newsletter

Proud to use

  • FLI

  • 656 Imaging

  • 10 Micron

  • Planewave

  • ZWO

Company Details:

Roboscopes

802 Kingsbury Road
Birmingham
B24 9PS
United Kingdom


Roboscopes is a trading name of ENS Optical LTD ¦ Copyright© 2020 Roboscopes
Cron Job Starts