1   Summary of the QAWG recommendations to SQuaSH¶

Recently, in DMTN-085 [1], the QA Strategy Working Group (QAWG) made specific recommendations to improve the SQuaSH metrics dashboard.

QAWG-REC-34

SQuaSH should issue alerts to developers and key stakeholders on regressions in important metric values.

QAWG-REC-35

Provide a single, reliable source of documentation describing the SQuaSH system and a vision for its use in DM-wide metric tracking.

QAWG-REC-36

The SQuaSH system should be closely coupled to the drill-down environment; in particular, the former should use the latter to enable drill-down functionality into particular metric values.

QAWG-REC-37

It must be possible to submit metrics to SQuaSH from arbitrary pipeline execution environments.

QAWG-REC-38

SQuaSH should be able to store and display appropriate metric values per DataId.

These recommendations help to define the scope and technical requirements for SQuaSH. In particular, QAWG-REC-37 and QAWG-REC-38 have implications on how metric values are stored and visualized, including metadata from the verification jobs and the execution environment. QAWG-REC-34 requires an automated regression detection and a notification system. QAWG-REC-36 suggests a workflow that starts in SQuaSH to find a misbehaving metric and then spawns an instance of the drill-down environment with enough information to look into why that particular metric is an excursion. Finally, QAWG-REC-35 requires better user documentation.

Another general recommendation made by the QAWG is that the drill-down capability should be removed from SQuaSH and implemented in external dashboards or the notebook aspect of the LSST Science Platform (LSP).

2   Adopting the InfluxData stack¶

In DM-16223, we investigate open source solutions that could be adopted to achieve the functionalities required in SQuaSH. Among them, the InfluxData stack presents the most exciting monitoring capabilities. This section describes the status of adopting the InfluxData stack in SQuaSH, in particular, InfluxDB, Chronograf, and Kapacitor components. We describe what has changed in SQuaSH so far, and propose a roadmap to implement what is pending.

SELECT "totalUnassociatedDiaObjects"
FROM "squash"."autogen"."ap_association"
WHERE  ("ccdnum"='10' OR "ccdnum"='5' OR "ccdnum"='56')
GROUP BY "ccdnum"

SELECT mean("totalUnassociatedDiaObjects")
FROM "squash"."autogen"."ap_association"
WHERE  ("ccdnum"='10' OR "ccdnum"='5' OR "ccdnum"='56')
GROUP BY time(1d)

var name = 'Association time alert'
var db = 'squash-prod'
var rp = 'autogen'
var measurement = 'ap_association'
var groupBy = ['visit', 'ccdnum', 'ci_dataset']
var whereFilter = lambda: TRUE

var message = '{{.Name}} is {{.Level}} on build #{{ index .Tags "ci_id" }}: AssociationTime = {{ index .Fields "value" | printf "%0.2f s" }} for {{.Group}}'

var triggerType = 'threshold'
var crit = 5

var data = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement(measurement)
        .groupBy(groupBy)
        .where(whereFilter)
    |eval(lambda: "ap_association.AssociationTime")
        .as('value')
var trigger = data
    |alert()
        .crit(lambda: "value" > crit)
        .message(message)
        .stateChangesOnly()
        .slack()
        .channel('#dm-squash-alerts')

3   Supporting multiple execution environments¶

To be generally useful for the verification activities, SQuaSH must support multiple execution environments.

The following project environments are currently supported:

• Jenkins CI
• LDF

SQuaSH captures environment variables from these environments and use them as metadata associated with the metric values.

SQuaSH has the concept of runs. A run may contain results from several verification jobs executed on a given environment. For example a GET request to /jenkins/<run_id> or to /lfd/<run_id> will retrieve all the verification jobs in that run.

In DM-18505, we add support for a local execution environment. That allows DM developers to run verification jobs in the notebook aspect of the LSP or from their laptop and dispatch results to SQuaSH. We plan to implement one execution environment per user to have have in SQuaSH auto-incremental run IDs per user. The user name registered in SQuaSH is going to be added as metadata to the environment. This implementation fulfills QAWG-REC-37.

4   Connecting SQuaSH to the drill-down environment¶

DMTN-085 [1] describes drill-down workflows to debug processing problems and investigate the effects of new algorithms. It recommends the implementation of a “browser-based interactive dashboard that can run on any pipeline output data repository (or comparison of two repositories) to quickly diagnose the quality of the data processing”. Here, the drill-down system is referred to merely as QA dashboard.

To connect SQuaSH and the QA dashboard in a meaningful way, they need to share a subset of metrics. Those metrics must have the same definition, must be computed by the same code, configuration, and run on the same data. Finally, metric values must be tested against the same specifications so that both systems indicate consistent metric regressions.

In other words, to accomplish the above we need to combine information from the Workflow System and the Verification Framework. The Workflow System runs the Pipeline Tasks on a controlled environment and associates code version and configuration with a run ID. The Verification Framework defines metrics and specifications and persists the metric values in verification jobs in the output data repository of that run. The verification jobs are submitted to SQuaSH along with the run ID and are used by the QA dashboard when analyzing the output data repository.

This way, the run ID links the SQuaSH and the QA dashboard. We also assume a service provided by the Workflow System in which the QA dashboard can introspect the path to the output data repository, code version, and configuration from the run ID.

SQuaSH’s primary job is then to discover run IDs that present misbehaving metrics. Given the run ID, the QA dashboard can get the information needed from the Workflow System for further investigation.

Given the assumptions above, to fulfill QAWG-REC-36, the minimum set of information that SQuaSH needs to store is:

• Run ID;
• Name of the execution environment;
• DataId’s associated with metric values.

The Run ID uniquely identifies the output data repository, the code version and the configuration run for a given environment. The name of the environment is used to distinguish runs executed on controlled environments (e.g., LDF) from runs performed on user local environments. Finally, DataId’s associated with metric values make it possible to spawn an instance of the QA dashboard on a particular DataId.

From the point of view of SQuaSH, QAWG-REC-36 is easy to implement, in fact for the Jenkins CI environment we already store the run ID, the name of the environment and DataIds as metadata tags associated with metric values (see 2.1   InfluxDB, a time-series database).

The InfluxDB HTTP API is the recommended API to query metric values in SQuaSH, and the SQuaSH REST API is the recommended API to query metrics definition and specifications. Currently this is done by interacting directly with the APIs using the Python Requests module. This situation can be improved by implementing a SQuaSH Python client as a more user friendly interface to retrieve metric data.

5   SQuaSH documentation¶

QAWG-REC-35 recommends a single, reliable source of documentation describing the SQuaSH system and a vision for its use in DM-wide metric tracking.

We understand QAWG-REC-35 requests SQuaSH documentation for DM developers. We plan to implement user documentation at https://squash.lsst.io. In addition, we need to complete the SQuaSH design documentation SQR-009 [2] and add deployment instructions to that as well.

6   References¶