Troubleshooting

Symptoms seen on real installations, with the cause and the fix. Each entry names where to look first.

403 Forbidden on CMDB, Integrations, AI Control or Agents

Cause. The user’s group lacks the permission for the module: cmdb_access, integrations_access, ai_access or agents_access. This is the normal state after an upgrade from 1.5.x, because /etc/energy-monitor/auth_groups.yml is kept from the previous version.

Fix. Manage -> Configuration -> Group Rights, enable the permission for the group and save. The defaults of the current version are in auth_groups.yml.rpmnew next to the file. See the Upgrade chapter.

The API answers 500 and “auth_users.yml cannot be read”

Cause. The API runs in a container that mounts /etc/energy-monitor/auth_users.yml. The web server inside the container runs as www-data (uid 33), while the file belongs to naemon:apache with mode 660, so the process cannot read it.

Fix. Grant read access to that uid without changing the mode:

setfacl -m u:33:r /etc/energy-monitor/auth_users.yml

Check with a request that needs authentication, for example GET /api/config/host/localhost with the administrator’s credentials.

A page returns 200 but shows an error inside

Cause. Nacoma configuration forms, CMDB and the agent builder are rendered inside an iframe. The surrounding page is served by Ninja and returns 200 even when the framed component fails.

Fix. Open the framed address directly (for example /monitor/energy/nacoma/windows_agents.php or /cmdb/) to see the real response, and check the Apache and php-fpm logs for that path.

“The database is currently being exported” and no configuration change is possible

Cause. Nacoma holds a lock in the objlocks table of the nacoma database while the configuration is exported. A failed export, for example through the API, can leave the lock behind (object_type = all).

Fix. Make sure no export is running (ps aux | grep export), then remove the stale row:

DELETE FROM nacoma.objlocks WHERE object_type = 'all';

Pending configuration changes disappeared

Cause. Every call to the Nacoma API (/opt/energy-monitor/nacoma/api/monitor.php), including calls made by the BPI status service, imports the configuration files into the database when the files are newer than the last import. Unsaved changes in the database are discarded by that import.

Fix. Save changes promptly. When scripting, use the Nacoma command line in one sequence: -a undo_config, the changes, -a save_config.

Check results are stale

Where to look. Monitor -> Scheduling queue shows the last and next check of every service. Performance information shows check latency. Rising latency means the core is not keeping up: too many checks, checks that hit the 60 s timeout, or too small a server.

A service is UNKNOWN

Cause. The check could not answer: an unrecognised command, a wrong command definition, an old agent, a module switched off in the agent, or a plugin that prints usage text instead of a result.

Where to look. The Status Information of the service is the raw plugin output. Run the command by hand with Test this command on the command’s configuration page.

Acknowledgements vanished after renaming a host

Cause. Acknowledgements and comments belong to the host name. Renaming the host creates a new object in the core and the old acknowledgements are gone.

Fix. Acknowledge again after the rename. Plan renames outside demo and reporting periods.

The geomap shows no tiles

Cause. The provider setting. The osm provider points at a discontinued tile service and the google provider needs an API key.

Fix. Geomap options: provider OpenStreetMap (stored as local) with the tile URL https://tile.openstreetmap.org. The settings live in /opt/energy-monitor/ninja/application/config/geomap/settings.xml, owned by apache.

Monitor Wall shows acknowledged problems

Cause. The default filter of the wall is [services] all, and the filter is stored in the browser, not on the server.

Fix. In the browser that drives the wall, set the filter to [services] state != 0 and acknowledged = 0.

After the upgrade OpenSearch does not start

Cause. Seen once after a 1.5.5 to 1.6.0 upgrade together with the intelligence package (a jar hell error in the OpenSearch log).

Where to look. systemctl status opensearch and /var/log/opensearch/. Without OpenSearch the interface shows no history, graphs or event log, and AI Correlations does not work.