Use new sim_telarray metadata system

[maxnoe]

In the latest experimental release sim_telarray allows adding metadata to the output files.

It allows both exporting the final configuration options per telescope but also setting custom user attributes.

This seems very handy.

E.g. adding this:

METAPARAM GLOBAL SET production=LST-MAGIC Prod2  
METAPARAM TELESCOPE ADD mirror_degraded_reflection nightsky_background  

to CTA-PROD5-LaPalma-baseline.cfg you get this with read_simtel -s -h <file> (support in pyeventio is on the way):

Meta parameters for ID=-1 (global) with 12 name/value pairs.
   *PRODUCTION = LST-MAGIC Prod2
   CONFIG_RELEASE = Release "2021-12-16 (development beyond prod-5)" from 2021-12-25T19:33:19 UTC.
   CONFIG_VERSION = 
   ARRAY_CONFIG_NAME = LaPalma-baseline-prod5
   ARRAY_CONFIG_VARIANT = LST/MST-NectarCam (including other MST camera type) at CTA North prod-5/prod-5b (extended)
   ARRAY_CONFIG_VERSION = 2020-11-20
   RANDOM_GENERATOR = mt19937
   RANDOM_STATE = auto
   RANDOM_SEED = auto
   SIMTEL_VERSION = 2021-12-16 18:31:13 CET (konrad@wizard4)

Meta parameters for ID=1 with 8 name/value pairs.
   MIRROR_DEGRADED_REFLECTION = 1 
   NIGHTSKY_BACKGROUND = all: 0.238006 
   OPTICS_CONFIG_NAME = LST
   OPTICS_CONFIG_VARIANT = LST-1 prototype
   OPTICS_CONFIG_VERSION = 2020-04-29
   CAMERA_CONFIG_NAME = LST
   CAMERA_CONFIG_VARIANT = LST-1 prototype, with nsb_autoscale_airmass
   CAMERA_CONFIG_VERSION = 2020-11-24

[jsitarek]

Hi @maxnoe

before we start using this version I think we should make sure that we have no problems reading those files. While you did the changes in pyeventio, it can still happen that there is a clash between the ctapipe and pyeventio versions and we might end up with needing to make some changes in lstchain to make it compatible with the ctapipe version that is compatible with the updated pyeventio. It might all work out of the box, but well, better to check it.

[maxnoe]

@jsitarek Since this only added new fields to the SimTelFile, it is compatible with the current ctapipe versions (otherwise I'd have made it version 2.0, not 1.6.

It's actually the opposite: since the metadata are added by default using the current sim_telarray we already need the latest pyeventio, otherwise we will get the same error that you got when trying to analyze the files.

[jsitarek]

Hi @maxnoe
Let me clarify: I did not mean this change in pyeventio, but if there were some other changes in the meantime that might not be compatible.
About the change in sim_telarray, If I understand correctly for the moment is only in the "Testing" version. and I guess we are using one of the releases for the sim_telarray production. Of course I fully agree with you that if the used sim_telarray has this already implemented we need the new pyeventio either way.

[maxnoe]

Let me clarify: I did not mean this change in pyeventio, but if there were some other changes in the meantime that might not be compatible.

I don't understand what you mean by this. What other changes in the meantime?

[jsitarek]

I checked in detail. As far as I can see ctapipe needs eventio 1.5.0 - 2.0, lstchain actually requires >=1.5.1, which is basically the same since I do not see the release 1.5.0. After 1.5.1 there was also 1.5.2 and then 1.6.0 with your latest change. My worry was e.g. in 1.5.2. there could have been something added that is not compatible with the current ctapipe or lstchain, and before it was working because it can always downgrade to 1.5.1. This would not be possible if we require 1.6 version. But I see that my lst environment is actually using 1.5.2 version, and there are no more changes in 1.6 for the moment, so indeed it should be all fine, you can disregard my earlier comment.

[maxnoe]

The current lstchain tests are already running with eventio 1.6. The CI (and anybody setting up an environment) will always get the newest compatible version.

As said, we use (or at least try) semantic versioning, so any breaking change would have resulted in being 2.0, which would then be excluded by the ctapipe requirements. Of course, I could by accident make a breaking change without realizing it..

[maxnoe]

See also discussion here: cta-observatory/ctapipe#1853

[maxnoe]

With Prod6, the metadata system will be used and future versions of ctapipe will very likely start relying on some of the metadata information being available. This includes for example using the telescope metaparams OPTICS_CONFIG_NAME and CAMERA_CONFIG_NAME to fill the subarray information and also using the global metadata SITE_CONFIG_NAME ALTITUDE *LONGITUDE, *LATITUDE for array location.

Here is how the LST metadata looks like in a current (2022-06-30) prod6 example run:

In [8]: s.telescope_meta[1]
Out[8]: 
{b'NIGHTSKY_BACKGROUND': b'all: 0.244637 ',
 b'NSB_AUTOSCALE_AIRMASS': b'0.84 0.29 ',
 b'MIRROR_CLASS': b'0 ',
 b'FOCUS_OFFSET': b'6.55 0 0 0 ',
 b'FOCAL_Length': b'2800 ',
 b'EFFECTIVE_FOCAL_Length': b'2923.7 0 0 0 0 ',
 b'MIRROR_LIST': b'mirror_CTA-S-LST_v2020-04-07.dat',
 b'CAMERA_DEGRADED_EFFICIENCY': b'1 ',
 b'CAMERA_DEGRADED_MAP': b'none',
 b'MIRROR_DEGRADED_REFLECTION': b'0.8 ',
 b'PRIMARY_DEGRADED_MAP': b'none',
 b'DEFAULT_TRIGger': b'AnalogSum',
 b'CAMERA_CONFIG_FILE': b'camera_CTA-LST-234_analogsum21_v2020-04-14.dat',
 b'PARABOLIC_DISH': b'1 ',
 b'DISH_SHAPE_Length': b'2800 ',
 b'MIRROR_FOCAL_Length': b'0 ',
 b'DISCRIMINATOR_PULSE_SHAPE': b'LST_pulse_shape_7dynode_high_intensity_pix1s.dat',
 b'ASUM_THRESHold': b'260.7 ',
 b'ASUM_CLIPping': b'9999 ',
 b'ASUM_SHAPING_FILE': b'none',
 b'OPTICS_CONFIG_NAME': b'LST',
 b'OPTICS_CONFIG_VARIANT': b'LST Prod-6',
 b'OPTICS_CONFIG_VERSION': b'2022-06-15 (Prod-6)',
 b'CAMERA_CONFIG_NAME': b'LSTcam',
 b'CAMERA_CONFIG_VARIANT': b'LST Prod-6',
 b'CAMERA_CONFIG_VERSION': b'2022-06-09 (Prod-6)'}