Health Monitoring Dashboards

New PowerHA Modern Web Interface
We recently announced a modern web interface for PowerHA on IBM i 7.4 and 7.5. This new modern web interface will provide at-a-glance health information from a web browser. Additional information on this new interface will be available in December. We recommend utilizing the new web interface built into PowerHA instead of Fortra’s Insite and Insite Analytics. The instructions below will remain for existing installations and those looking for examples of utilizing PowerHA SQL services in other monitoring products.
Planned availability: December 16th, 2022

PowerHA by design requires minimal ongoing monitoring, management, and maintenance. With PowerHA dashboards, ongoing monitoring is simplified even more. Dashboards enable administrators to see any errors or warnings that may affect their business continuity. These dashboards can be configured to show PowerHA information alongside information from other products and can be configured to show multiple PowerHA environments on one screen.

Before You Begin

The dashboards are delivered as part of Fortra’s Insite and Insite Analytics - a free offering from Fortra. Insite and Insite Analytics must be installed prior to following the steps below. For more information on Insite see the following information:

These dashboards require the following PowerHA versions: 7.4 HA 4.3.1 7.2 HA 3.5

Procedure

Open Insite Analytics

Log in to the Insite Server
Open the Insite Analytics Product

Opening Insite Analytics

Note: For general usage of Insite Analytics, see the Insite Analytics Users Guide.

After logging in to the Insite server, the homepage is displayed.
Click the Select Product button and select Insite Analytics.
The Insite Analytics menu will appear.

Creating Data Connections

Data connections define the link between the SQL queries, the different systems and databases across the network where the data is stored. Follow the steps in this section to create data connections between Insite Analytics and the PowerHA cluster nodes.

Tip: You must add a Data Connection for at least one node in each PowerHA cluster. Data connections are not necessary for every node in a PowerHA cluster. Insite Analytics displays data as it is transmitted by the nodes with a data connection.

Add a data connection for at least one node in each cluster that will be on a dashboard

Creating a Data Connection

In the Insite Analytics Menu select Data Connections
Click +Add from the Data Connections panel.
Fill in the following options:
1. Name: Enter a name for the data connection. The IBM i Host Name is a good choice for the name of the connection.
2. Driver: From the dropdown list, select Db2/400
3. URL: Define the connection string that will be used to connect to the server. The connection string template is:
```
jdbc:as400://[serverName];prompt=false;extendedmetadata=true;translatebinary=true;libraries=*LIBL;naming=system
```
  Where: [serverName] = the network resolvable name or address of the IBM i server.
  For example, if the server had IP address 203.0.113.2, the connection URL would be: jdbc:as400://203.0.113.2;prompt=false;extendedmetadata=true;translatebinary=true;libraries=*LIBL;naming=system
4. Timezone: If the IBM i Server is in a different time zone to the Insite Server, set its UTC offset accordingly.
5. Permissions: By default, permission to edit or use a data connection is limited to the Admin user and the creator of the data connection. These options enable other users to be given permission to either edit or use the data connection. Possible values are:
  1. Users: Choose any of the listed users to provide rights to the data connection.
  2. Groups: Choose any of the listed groups to provide rights to the data connection.
  3. Permit All: Select whether to allow all users access to the data connection or to limit permissions to the Admin, creator, or any specified user or group.
6. Credentials: For the node defined in the URL above, enter a valid IBM i Username and Password of the user profile that will be used to process requests for this data connection.
Click the blue Test button to verify the connection
Click the green Save button at the top right to save the connection details.
If you have more than one cluster, additional Data Connections can be set up repeating the above instructions.

Queries

The Query Builder enables the creation of SQL queries, which are used to select the information to be displayed in Insite Dashboards. Follow the procedure in this section to create SQL queries with Query Builder:

In the Insite Analytics menu select Queries

Add the basic PowerHA Queries

Add the Cluster Nodes Query with the following information:

Query Name	PowerHA Cluster Nodes
Description	A list of all cluster nodes along with their status information
SQL	SELECT NODE,NODE_STATUS FROM QHASM.CLUSTER_NODES

For instructions to add queries to Insite Analytics, expand this section.

Adding a Basic Query to Insite Analytics

Click the green +Add button at the top right of the Queries panel. The New Query panel opens on the right.
Complete the options as described below using copy/paste where applicable. For example:
1. Name: Cluster Nodes
2. Description: A list of all cluster nodes along with their status information
3. Select Data Connection: Select the data connection(s) that you previously created
Click the green Edit Query button to continue.
Click the gray Advanced Mode button to allow direct entry of the SQL statement.

Enter the SQL statement. For example:

SELECT NODE,NODE_STATUS FROM QHASM.CLUSTER_NODES

Click the Preview button. The panel should look similar to the following, returning information from your environment:
Click the green Save button in the upper right.

2. Add the Administrative Domains Query with the following information:

Query Name	PowerHA Administrative Domains
Description	A list of administrative domains with their status information
SQL	SELECT ADMINISTRATIVE_DOMAIN,DOMAIN_STATUS FROM QHASM.ADMIN_DOMAIN_LIST

3. Add the Monitored Resources Query with the following information:

Query Name

PowerHA Monitored Resources Requiring Attention

Description

A list of monitored resources that are either failed or inconsistent along with additional node level information

SQL

SELECT DETAILS.MONITORED_RESOURCE,DETAILS.RESOURCE_TYPE,DETAILS.LIBRARY,DETAILS.GLOBAL_STATUS,DETAILS.NODE,DETAILS.LOCAL_STATUS,DETAILS.MESSAGE_ID,DETAILS.MESSAGE_TEXT 
  FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST()) LIST,
       TABLE(QHASM.ADMIN_DOMAIN_MRE_DETAILS(MONITORED_RESOURCE => LIST.MONITORED_RESOURCE, RESOURCE_TYPE => LIST.RESOURCE_TYPE, LIBRARY => LIST.LIBRARY)) DETAILS
  WHERE (LIST.GLOBAL_STATUS = '*INCONSISTENT' OR LIST.GLOBAL_STATUS = '*FAILED') AND
        DETAILS.LOCAL_STATUS != 'CURRENT'

Note: Since the Monitored Resources Requiring Attention query only returns monitored resources that require attention, it is likely that when previewing this query, no data will be returned.

4. Add the Unmonitored Resources Query with the following information:

Query Name

PowerHA Unmonitored Resources

Description

A list of resources not currently monitored by the administrative domain that could be monitored by the administrative domain

SQL

SELECT JOBD.OBJNAME AS "Unmonitored Resource", '*JOBD' as "Resource Type", JOBD.OBJLONGSCHEMA as "Resource Library"
    FROM TABLE(QSYS2.OBJECT_STATISTICS('*ALL','*JOBD','*ALLSIMPLE')) JOBD
    WHERE JOBD.OBJLONGSCHEMA != 'QSYS' AND JOBD.OBJLONGSCHEMA != 'QINSYS' AND JOBD.OBJLONGSCHEMA != 'QINPRIOR' AND JOBD.OBJLONGSCHEMA != 'QINMEDIA' AND
        NOT EXISTS (
        SELECT MONITORED_RESOURCE
        FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => '*JOBD')) MRE
        WHERE MRE.MONITORED_RESOURCE = JOBD.OBJNAME)
UNION
SELECT SBSD.OBJNAME AS "Unmonitored Resource", '*SBSD' as "Resource Type", SBSD.OBJLONGSCHEMA as "Resource Library"
    FROM TABLE(QSYS2.OBJECT_STATISTICS('*ALL','*SBSD','*ALLSIMPLE')) SBSD
    WHERE SBSD.OBJLONGSCHEMA != 'QSYS' AND SBSD.OBJLONGSCHEMA != 'QINSYS' AND SBSD.OBJLONGSCHEMA != 'QINPRIOR' AND SBSD.OBJLONGSCHEMA != 'QINMEDIA' AND
        NOT EXISTS (
        SELECT MONITORED_RESOURCE
        FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => '*SBSD')) MRE
        WHERE MRE.MONITORED_RESOURCE = SBSD.OBJNAME)
UNION
SELECT USRPRF.OBJNAME AS "Unmonitored Resource", '*USRPRF' as "Resource Type", USRPRF.OBJLONGSCHEMA as "Resource Library"
    FROM TABLE(QSYS2.OBJECT_STATISTICS('QSYS','*USRPRF','*ALLSIMPLE')) USRPRF
    WHERE NOT EXISTS (
        SELECT MONITORED_RESOURCE
        FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => '*USRPRF')) MRE
        WHERE MRE.MONITORED_RESOURCE = USRPRF.OBJNAME)
UNION
SELECT AUTL.OBJNAME AS "Unmonitored Resource", '*AUTL' as "Resource Type", AUTL.OBJLONGSCHEMA as "Resource Library"
    FROM TABLE(QSYS2.OBJECT_STATISTICS('QSYS','*AUTL','*ALLSIMPLE')) AUTL
    WHERE NOT EXISTS (
        SELECT MONITORED_RESOURCE
        FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => '*AUTL')) MRE
        WHERE MRE.MONITORED_RESOURCE = AUTL.OBJNAME)
UNION
SELECT CLS.OBJNAME AS "Unmonitored Resource", '*CLS' as "Resource Type", CLS.OBJLONGSCHEMA as "Resource Library"
    FROM TABLE(QSYS2.OBJECT_STATISTICS('*ALL','*CLS','*ALLSIMPLE')) CLS
    WHERE CLS.OBJLONGSCHEMA != 'QSYS' AND CLS.OBJLONGSCHEMA != 'QINSYS' AND CLS.OBJLONGSCHEMA != 'QINPRIOR' AND CLS.OBJLONGSCHEMA != 'QINMEDIA' AND
        NOT EXISTS (
        SELECT MONITORED_RESOURCE
        FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => '*CLS')) MRE
        WHERE MRE.MONITORED_RESOURCE = CLS.OBJNAME)

5. Add the Cluster Resource Groups Query with the following information:

Query Name	PowerHA Cluster Resource Groups
Description	A list of cluster resource groups along with their status
SQL	SELECT CLUSTER_RESOURCE_GROUP,CRG_TYPE,CRG_STATUS,PRIMARY_NODE FROM QHASM.CLUSTER_RESOURCE_GROUP_LIST

6. Add the Recovery Domain Nodes Requiring Attention Query with the following information:

Query Name

PowerHA Recovery Domain Nodes Requiring Attention

Description

A list of nodes in CRG recovery domains that are inactive or ineligible

SQL

SELECT RCYDMN.CLUSTER_RESOURCE_GROUP, RCYDMN.NODE, RCYDMN.NODE_STATUS, RCYDMN.SITE_NAME
  FROM QHASM.CLUSTER_RESOURCE_GROUP_LIST LIST,
       TABLE(QHASM.CRG_RECOVERY_DOMAIN(LIST.CLUSTER_RESOURCE_GROUP)) RCYDMN
  WHERE NODE_STATUS != 'ACTIVE'

Note: Since the Recovery Domain Nodes Requiring Attention query only returns nodes that are not eligible for a switchover, it is likely that when previewing this query, no data will be returned.

7. Add the Session Info Query with the following information:

Query Name

PowerHA Session Information

Description

A list of sessions along with current replication status

SQL

SELECT SESSION_INFO.SESSION_NAME,SESSION_INFO.SESSION_TYPE,SESSION_INFO.ASP_DEVICE,SESSION_INFO.SOURCE_NODE,SESSION_INFO.SOURCE_ASP_STATE,SESSION_INFO.COPY_STATUS,SESSION_INFO.TARGET_NODE,SESSION_INFO.TARGET_ASP_STATE
FROM QHASM.SESSION_LIST SESSION_LIST,
     TABLE(QHASM.SESSION_INFO(SESSION => SESSION_LIST.SESSION_NAME)) SESSION_INFO

Add the PowerHA Queries with Drilldown Query

1. Add the Count Cluster Nodes Requiring Attention Query:

Query Name	PowerHA Count Cluster Nodes Requiring Attention
Description	A count of cluster nodes that are not active
SQL	SELECT COUNT(*) AS COUNT FROM QHASM.CLUSTER_NODES WHERE NODE_STATUS != 'ACTIVE'
Drilldown Query	PowerHA Cluster Nodes

For instructions to add drilldown queries to Insite Analytics, expand this section.

Adding a Drilldown Query

Specific drill down queries are used to create color-coded indicator widgets showing a count of errors. Users can click on the widget to drill down into details of the errors.

Click Advanced Drilldown
Under choose query, select the drilldown query. For example: PowerHA Cluster Nodes
Select to Inherit Data connection from the main query.
Click the green OK button in the upper-right corner.
On the main Query Edit panel, click the green Save button in the upper-right corner.

2. Add the Count of Admin Domains Requiring Attention Query:

Query Name	PowerHA Count of Admin Domains Requiring Attention
Description	A count of administrative domains that are not active
SQL	SELECT COUNT(*) AS COUNT FROM QHASM.ADMIN_DOMAIN_LIST WHERE DOMAIN_STATUS != 'ACTIVE'
Drilldown Query	PowerHA Administrative Domains

3. Add the Count of Monitored Resources Requiring Attention Query:

Query Name	PowerHA Count of Monitored Resources Requiring Attention
Description	A count of monitored resources requiring attention
SQL	SELECT COUNT() AS COUNT FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST()) LIST WHERE LIST.GLOBAL_STATUS = 'INCONSISTENT' OR LIST.GLOBAL_STATUS = '*FAILED'
Drilldown Query	PowerHA Monitored Resources Requiring Attention

4. Add the Count of Unmonitored Resources Query:

Query Name	PowerHA Count of Unmonitored Resources
Description	A count of resources not currently monitored by the administrative domain that could be monitored by the administrative domain
SQL	SELECT COUNT() AS COUNT FROM (SELECT JOBD.OBJNAME AS "Unmonitored Resource", 'JOBD' as "Resource Type", JOBD.OBJLONGSCHEMA as "Resource Library" FROM TABLE(QSYS2.OBJECT_STATISTICS('ALL','JOBD','ALLSIMPLE')) JOBD WHERE JOBD.OBJLONGSCHEMA != 'QSYS' AND JOBD.OBJLONGSCHEMA != 'QINSYS' AND JOBD.OBJLONGSCHEMA != 'QINPRIOR' AND JOBD.OBJLONGSCHEMA != 'QINMEDIA' AND NOT EXISTS ( SELECT MONITORED_RESOURCE FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => 'JOBD')) MRE WHERE MRE.MONITORED_RESOURCE = JOBD.OBJNAME) UNION SELECT SBSD.OBJNAME AS "Unmonitored Resource", 'SBSD' as "Resource Type", SBSD.OBJLONGSCHEMA as "Resource Library" FROM TABLE(QSYS2.OBJECT_STATISTICS('ALL','SBSD','ALLSIMPLE')) SBSD WHERE SBSD.OBJLONGSCHEMA != 'QSYS' AND SBSD.OBJLONGSCHEMA != 'QINSYS' AND SBSD.OBJLONGSCHEMA != 'QINPRIOR' AND SBSD.OBJLONGSCHEMA != 'QINMEDIA' AND NOT EXISTS ( SELECT MONITORED_RESOURCE FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => 'SBSD')) MRE WHERE MRE.MONITORED_RESOURCE = SBSD.OBJNAME) UNION SELECT USRPRF.OBJNAME AS "Unmonitored Resource", 'USRPRF' as "Resource Type", USRPRF.OBJLONGSCHEMA as "Resource Library" FROM TABLE(QSYS2.OBJECT_STATISTICS('QSYS','USRPRF','ALLSIMPLE')) USRPRF WHERE NOT EXISTS ( SELECT MONITORED_RESOURCE FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => 'USRPRF')) MRE WHERE MRE.MONITORED_RESOURCE = USRPRF.OBJNAME) UNION SELECT AUTL.OBJNAME AS "Unmonitored Resource", 'AUTL' as "Resource Type", AUTL.OBJLONGSCHEMA as "Resource Library" FROM TABLE(QSYS2.OBJECT_STATISTICS('QSYS','AUTL','ALLSIMPLE')) AUTL WHERE NOT EXISTS ( SELECT MONITORED_RESOURCE FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => 'AUTL')) MRE WHERE MRE.MONITORED_RESOURCE = AUTL.OBJNAME) UNION SELECT CLS.OBJNAME AS "Unmonitored Resource", 'CLS' as "Resource Type", CLS.OBJLONGSCHEMA as "Resource Library" FROM TABLE(QSYS2.OBJECT_STATISTICS('ALL','CLS','ALLSIMPLE')) CLS WHERE CLS.OBJLONGSCHEMA != 'QSYS' AND CLS.OBJLONGSCHEMA != 'QINSYS' AND CLS.OBJLONGSCHEMA != 'QINPRIOR' AND CLS.OBJLONGSCHEMA != 'QINMEDIA' AND NOT EXISTS ( SELECT MONITORED_RESOURCE FROM TABLE(QHASM.ADMIN_DOMAIN_MRE_LIST(RESOURCE_TYPE => 'CLS')) MRE WHERE MRE.MONITORED_RESOURCE = CLS.OBJNAME))
Drilldown Query	PowerHA Unmonitored Resources

5. Add the Count of Cluster Resource Groups Requiring Attention Query:

Query Name	PowerHA Count of Cluster Resource Groups Requiring Attention
Description	A count of cluster resource groups that are not active
SQL	SELECT COUNT(*) AS COUNT FROM QHASM.CLUSTER_RESOURCE_GROUP_LIST WHERE CRG_STATUS != 'ACTIVE'
Drilldown Query	PowerHA Cluster Resource Groups

6. Add the Count of Recovery Domain Nodes Requiring Attention Query:

Query Name	PowerHA Count of Recovery Domain Nodes Requiring Attention
Description	A count of cluster resource groups that are not active
SQL	SELECT COUNT(*) AS COUNT FROM QHASM.CLUSTER_RESOURCE_GROUP_LIST LIST, TABLE(QHASM.CRG_RECOVERY_DOMAIN(LIST.CLUSTER_RESOURCE_GROUP)) RCYDMN WHERE NODE_STATUS != 'ACTIVE'
Drilldown Query	PowerHA Recovery Domain Nodes Requiring Attention

7. Add the Count of Sessions Requiring Attention Query:

Query Name	PowerHA Count of Sessions Requiring Attention
Description	A count of sessions where the copy status is not active
SQL	SELECT COUNT(*) AS COUNT FROM QHASM.SESSION_LIST SESSION_LIST, TABLE(QHASM.SESSION_INFO(SESSION => SESSION_LIST.SESSION_NAME)) SESSION_INFO WHERE SESSION_INFO.COPY_STATUS != 'ACTIVE'
Drilldown Query	PowerHA Session Information

PowerHA Status Dashboard

This section describes how to create a Insite Dashboard and add various widgets to show the status of PowerHA, highlighting issues that require attention. The example dashboard defined here includes all the queries you have defined. You can choose which of these to include and adjust the layout to suit your preferences.

Create an Insite Dashboard

Creating an Insite Dashboard

In the Navigation Pane, click Dashboards. If the menu is hidden, hover over the Navigation Pane to expand it.
Click the green +Add Dashboard button at the upper-right of the screen
Enter a unique name for the dashboard. For example: PowerHA Status Dashboard
Enter a Description. For example: Status for PowerHA Environments
Select a Color Theme. This theme applies to all widgets on this dashboard. In this example, we used the default Light Theme.
Set the Edit Rights and Share With sections as required or leave them as the default. Refer to the Insite User Guide for options.
Click Save.
From the dashboard in Edit mode, click Select Quick Layout.
From the layouts panel, select the 2 widgets tile.

Tip: The layout is customizable. Choose a layout that suits your needs.

2. Add PowerHA Dashboard Widgets

For instructions to add PowerHA widgets to Insite Analytics, expand this section.

Adding PowerHA Widgets to the Dashboard

There are 7 Power HA count of widgets. This example uses Indicator Widgets to show the count of each type. These indicator widgets will be grouped using a Group Widget, allowing a single title above the widgets.

Creating the Layout

Ensure Edit mode is on.
Hover your mouse over the top section and select the green +Add Group.
Click Edit Group.
Select to Show the Header.
Enter a Title such as PowerHA Status.
Click the green Save button in the upper-right hand corner.
Hover your mouse over the top section and select the Edit Groups Widgets button.
Use the < and > arrows to split the section into 8 as shown below. If you make a mistake, click the undo button at the top of the panel.

Adding Widgets to the Layout

Add the Cluster Nodes Widget
1. Click the three dots at the top right of section 1 and select Add Widget.
2. Select Insite Analytics for the Data Source.
3. For the Asset select PowerHA Count of Cluster Nodes Requiring Attention.
4. Select the cluster node to use for this connection. In this example, the data source is Prod.
5. Enter Nodes under Name.
6. Select Key Indicator as the Widget Type.
7. Select the only metric that is displayed. The preview on the left will now update to display a number.
8. In order to show the widget as green when the count is zero, and red when greater than zero, thresholds must be used. Select the Layout Tab.
9. Select Show for Use Status Color as Widget Background and Custom for Thresholds.
10. Click the green Add Rule button.
11. On the first rule, set the Condition to =, leave the Value as 0, and set the status to Good. This rule causes the widget to display in green when there are no cluster node errors.
12. On the second rule, set the Condition to >, leave the Value as 0, and set the status to Critical. This rule causes the widget to display in green when there are one or more cluster node errors.
13. Click the green Save button in the upper-right hand corner.
Repeat step 1 a-m for the remaining 6 sections in this Group:
1. For the 2nd section:

Asset	PowerHA Count of Admin Domains Requiring Attention
Name	Admin Domains
Rules	Condition	Value	Status
	=	0	Good
	>	0	Warning

c. For the 3rd section:

Asset	PowerHA Count of Monitored Resources Requiring Attention
Name	MREs
Rules	Condition	Value	Status
	=	0	Good
	>	0	Critical

d. For the 4th section:

Asset	PowerHA Count of Unmonitored Resources
Name	Unmonitored Resources
Rules	Condition	Value	Status
	=	0	Good
	>	0	Warning

e. For the 5th section:

Asset	PowerHA Count of Cluster Resource Groups Requiring Attention
Name	CRGs
Rules	Condition	Value	Status
	=	0	Good
	>	0	Warning

f. For the 6th section:

Asset	PowerHA Count of Recovery Domain Nodes Requiring Attention
Name	Recovery Domain
Rules	Condition	Value	Status
	=	0	Good
	>	0	Critical

g. For the 7th section:

Asset	PowerHA Count of Sessions Requiring Attention
Name	Sessions
Rules	Condition	Value	Status
	=	0	Good
	>	0	Critical

3. When all widgets have been defined, click on Stop Editing Group.

Other Suggestions

The dashboard described in this document is just one example of a PowerHA dashboard. Some suggestions for use include:

If you have multiple clusters to monitor, a single dashboard can be created with a group for each cluster or a separate dashboard can be created for each cluster.
If you have PowerHA and Robot HA, a single dashboard can be created that shows both the status of PowerHA and the status of Robot HA.
You can create queries to other data sources and combine PowerHA data with data from other applications.