Problems with sensors in "Configuration" and "Diagnostic" in Home Assistant

[HansRemmerswaal]

emsesp_info (3).json

PROBLEM DESCRIPTION

I noticed in the EMS-ESP the sensor VR4 and TW1 since I implemented EMS-ESP-3_6_4-dev_2-ESP32.

Normally after a firmware upgrade I go to Home Assistant, open the History tab and add the devices ems-esp Boiler and ems-esp Thermostat to see if there are new sensors added or some sensors not there anymore. I was surprise to see that the VR4 and TW1 where not shown. I then then checked in the History tab if I could add the VR4 and TW1 directly as entity, and that worked.

So somehow the VR4 and TW1 are not considered, in the history tab, to be part of the device ems-esp Boiler.

I then went to the Settings -> Devices & Services -> Devices -> ems-esp Boiler. In there I see 4 groups;

• Controls
• Sensors
• Configuration
• Diagnostic

In that block Diagnostic I see 5 sensor values. And all those 5 sensor values I do not see in History -> Choose device -> ems-esp Boiler...

EXPECTED BEHAVIOUR

I would expect to see all the values from the ems-esp Boiler in History -> Choose device -> ems-esp Boiler.
So also the values from the group Diagnostic.

Not sure why some sensor values are part of Diagnostic and not part of Sensors. Maybe there is somehow the root cause...? Maybe that Diagnostic is reserved for something else in Home Assistant?

SCREENSHOTS

[proddy]

which version did you upgrade from? Those are new entities which I think were added a few weeks ago. Diagnostic sensors in EMS-ESP are required now (see #1382 (comment))

[HansRemmerswaal]

On Nov 13th I upgraded to EMS-ESP-3_6_3-test_7a-ESP32. Not sure if at that time we already had that Diagnostic.
On Nov 23rd I upgraded to EMS-ESP-3_6_4-dev_2-ESP32 and I noticed that Diagnostic with sensors which do not show up in History -> Choose device -> ems-esp Boiler but only in History -> Choose entity.

Is this Diagnostic the same what this help document refers to? As that more sounds like diagnostic data of the ems-esp gateway itself and not about the sensors it's reading.

The diagnostics integration
provides a way to download diagnostic data from a device or integrationIntegrations connect and integrate Home Assistant with your devices, services, and more.
[Learn more], that can be shared in issue reports. Sharing diagnostics data when reporting an issue allows developers to diagnose and fix your reported problem quicker.

[HansRemmerswaal]

To me it's not consistent that I see 20 EMS-ESP temperatures in "sensors" and 2 in "diagnostic".
And is it then a defect in HA that these temperature sensors in "diagnostic" are not visible in History -> Choose device -> ems-esp Boiler?

[proddy]

ok.

[HansRemmerswaal]

OMG... I upgrade from HA 2023.11.2 to HA 2023.11.3
I rebooted.

And now almost all the ems-esp entities are part of Configuration and Diagnostic. So using History -> Choose device -> ems-esp Boiler is only showing now a few entities...

Not happy. Is this a defect in HA or is this a configuration issue in EMS-ESP?

For now I have rolled back release by release, and in between I rebooted HA.
I'm now back on 3.6.2 and see my entities again in HA.

[HansRemmerswaal]

@proddy @MichaelDvP

I raised a ticket on the Home Assistant form and their they indicate that it's by design that sensors part of "Configuration" and "Diagnostic" are not part of the history anymore in Home Assistant.

To me it sounds like all the EMS-ESP32 sensors should not be part of "Configuration" and "Diagnostic". Maybe only a few of the ESP device itself like "uptime".

Can you please move all the sensors back to the original classification?

[proddy]

can you share the ticket? Did you ask on HA's forum? I'd like to follow the HA's standards so if its wrong we'll fix it.

[proddy]

I can't reproduce. Take for example on my system 'Boiler Actual boiler temperature'. The MQTT Discovery payload shows the HA Entity Category as Diagnostics (i.e. not a writeable sensor):

And I can see it as Diagnostics in HA under Devices & Sensors/MQTT:

And I can see all the History in HA:

Do you have use-case you can share?

[HansRemmerswaal]

In your screenshot showing the history you have the ems-esp Boiler device and the ems-esp Boiler Actual temperature.
The only sensor shown is the ems-esp Boiler Actual temperature...

So in the history you can show individual sensor values from "Configuration" and "Diagnostic" if you select them specifically. But they are not shown when you select the device. As in your example, no sensor values are shown for the device anymore...
A few firmware releases back they where not classified as "Configuration" and "Diagnostic" and then you could see in one go all the sensors of the ems-esp Boiler device in history.

home-assistant/core#104451 (comment)

[HansRemmerswaal]

I have to say that the help is not very clear... but they give as example for diagnostics the RSSI or MAC address of the device and for configuration the background illumination of the device. So more related to the status and the settings of the EMS-ESP32 device itself.

https://developers.home-assistant.io/docs/core/entity/

Classification of a non-primary entity. Set to EntityCategory.CONFIG for an entity which allows changing the configuration of a device, for example a switch entity making it possible to turn the background illumination of a switch on and off. Set to EntityCategory.DIAGNOSTIC for an entity exposing some configuration parameter or diagnostics of a device but does not allow changing it, for example a sensor showing RSSI or MAC-address.

[HansRemmerswaal]

How do we move on with this? Not sure if you are in contact with Paulus for clarification.
Maybe till then we rollback the change made after 3.6.2 with respect to the entity_category?

[proddy]

not sure - it's a huge change just to be able to show the full history in HA. What you could try is manually removing the ent_cat from the MQTT tropics using MQTTExplorer and see if that works, and that there are no errors. Also see if you can scout around to other programs like Tasmota and ESPHome to see how they solved it. I'm out for work so won't have time to look into this before next week

[HansRemmerswaal]

not sure - it's a huge change just to be able to show the full history in HA.

Without doubt, but it's not only to see it in the fully history in HA. Diagnostic and configuration entities aren’t added to auto generated dashboards either. But most important is that we get the entity_category configured the way it should be, whatever that that is.

Also see if you can scout around to other programs like Tasmota and ESPHome to see how they solved it.

There it's solved the way I would expect it...

[proddy]

ok - think you're right - I made a quick 3.6.5-dev-1 build to try https://github.com/emsesp/EMS-ESP32/releases/tag/latest

[HansRemmerswaal]

I installed the new firmware, restarted Home Assistant and all is working fine!

Thanks for the quick support!
I will see if I can find a clear definition of the 4 categories, and with some luck get the Home Assistant help updated with it....

[HansRemmerswaal]

I'm still struggling to understand your use case though. When an EMS-ESP entity is tagged as Configuration or Diagnostics there is historic data kept (per my screenshot below). Are you saying in HA if you select only the device in the History page it doesn't show all the entities? Why would you do that anyway?

The screenshot below is what I see when I rollback to 3.6.2. When you select in the history the device you see all the entities that exist for the device, all the 135+ entities. All the temperatures and settings of the boiler etc. Compare that with your screenshot, there you don't see anything.

Showing all the entities for a device is a lot of information but it's useful if you do not know for what you are looking. E.g. you see something strange and you try to explain it. By looking at all the entities you might see a kind of pattern.

This sounds like only special entities should have the configuration, i.e. the status and heartbeat mqtt form system device. But this blogpost is 2 years old and the definition could have been changed since.

This is also my understanding. More the entities of the device itself. Typically what we have in ems-esp device itself like the Free memory, MQTT, uptime...

[HansRemmerswaal]

And last but not least, it’s very useful to see in one go all the entities after a firmware release. You easily see if entities are added, removed or renamed.