Development Checklist

Review these items when releasing a Skill Template (CodeBundle) for general use. These are linting checks — functionality should be thoroughly tested separately. This list will eventually be converted into an automated tool.

Basic checks

Settings

• Documentation — a basic description of the Skill Template’s functionality
• Metadata
  • Author — GitHub username of the author
  • Display Name — the “pretty name” shown in the platform and registry
  • Supports — all platforms/technologies this Skill Template supports
• Suite Setup — the keyword called before any Task runs

*** Settings ***
Documentation       Triages issues related to a deployment and its replicas.
Metadata            Author    stewartshea
Metadata            Display Name    Kubernetes Deployment Triage
Metadata            Supports    Kubernetes,AKS,EKS,GKE,OpenShift

Suite Setup         Suite Initialization

Suite Setup

• RW.Core.Import User Variable — import all non-sensitive configuration values
• RW.Core.Import Secret — import sensitive data / configuration values
• Set Suite Variable — set all imported variables for use across all Tasks

*** Keywords ***
Suite Initialization
    ${kubeconfig}=    RW.Core.Import Secret
    ...    kubeconfig
    ...    type=string
    ...    description=The kubernetes kubeconfig yaml containing connection configuration used to connect to cluster(s).
    ...    pattern=\w*
    ...    example=For examples, start here https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/
    ${DEPLOYMENT_NAME}=    RW.Core.Import User Variable    DEPLOYMENT_NAME
    ...    type=string
    ...    description=Used to target the resource for queries and filtering events.
    ...    pattern=\w*
    ...    example=artifactory
    ${NAMESPACE}=    RW.Core.Import User Variable    NAMESPACE
    ...    type=string
    ...    description=The name of the Kubernetes namespace to scope actions and searching to.
    ...    pattern=\w*
    ...    example=my-namespace
    ${CONTEXT}=    RW.Core.Import User Variable    CONTEXT
    ...    type=string
    ...    description=Which Kubernetes context to operate within.
    ...    pattern=\w*
    ...    example=my-main-cluster
    Set Suite Variable    ${kubeconfig}    ${kubeconfig}
    Set Suite Variable    ${CONTEXT}    ${CONTEXT}
    Set Suite Variable    ${NAMESPACE}    ${NAMESPACE}
    Set Suite Variable    ${DEPLOYMENT_NAME}    ${DEPLOYMENT_NAME}

Tasks

• Documentation — describes the specific Tool’s function
• Tags — additional tags that help with indexing
• RW.Core.Add Pre To Report — for Runbooks (runbook.robot), content must be added to the report with this keyword
• RW.Core.Push Metric — for Monitors (sli.robot), a metric is pushed to the platform with this keyword

*** Tasks ***
Check Deployment Log For Issues with `${DEPLOYMENT_NAME}`
    [Documentation]    Fetches recent logs for the given deployment in the namespace and checks the logs output for issues.
    [Tags]    fetch    log    pod    container    errors    inspect    deployment
    ${logs}=    RW.CLI.Run Bash File
    ...    bash_file=deployment_logs.sh
    ...    cmd_override=./deployment_logs.sh | tee "${SCRIPT_TMP_DIR}/log_analysis"
    ...    env=${env}
    ...    secret_file__kubeconfig=${kubeconfig}
    ...    timeout_seconds=180
    ...    include_in_history=false
    ${history}=    RW.CLI.Pop Shell History
    RW.Core.Add Pre To Report    Recent logs from Deployment ${DEPLOYMENT_NAME} in Namespace ${NAMESPACE}:\n\n${logs.stdout}
    RW.Core.Add Pre To Report    Commands Used: ${history}

Advanced checks: Issues and next steps

The power of RunWhen Skill Templates comes from Issues and Next Steps. A Tool can raise an Issue that provides specific detail to a RunWhen AI Assistant or a service owner for further action.

Issue fields

Field	Description
severity	1 = Critical, 2 = Error/Major, 3 = Warning/Minor, 4 = Informational
expected	The expected state
actual	The actual state found
title	What and where — e.g. “Deployment cartservice in Namespace payments is unavailable”
reproduce_hint	The command or method used to determine this result
details	Specific details — usually a subset of the data posted to the report
next_steps	A \n-delimited list of suggested next steps. These should look like another Tool that may exist in a workspace

Simple issue

RW.Core.Add Issue    title=Errors in Flux Controller Reconciliation
...    severity=3
...    expected=No errors in Flux controller reconciliation.
...    actual=Flux controllers have errors in their reconciliation process.
...    reproduce_hint=Run flux_reconcile_report.sh manually to see the errors.
...    next_steps=Inspect Flux logs to determine which objects are failing to reconcile.
...    details=${process.stdout}

Advanced issue — conditional severity

IF    $deployment_status["available_condition"]["status"] == "False" or $deployment_status["ready_replicas"] == "0"
    RW.Core.Add Issue
    ...    severity=1
    ...    expected=Deployment `${DEPLOYMENT_NAME}` in namespace `${NAMESPACE}` should have minimum availability.
    ...    actual=Deployment `${DEPLOYMENT_NAME}` in namespace `${NAMESPACE}` does not have minimum availability.
    ...    title=Deployment `${DEPLOYMENT_NAME}` in Namespace `${NAMESPACE}` is unavailable
    ...    reproduce_hint=View Commands Used in Report Output
    ...    details=Deployment has ${deployment_status["ready_replicas"]} ready pods, needs ${deployment_status["desired_replicas"]}
    ...    next_steps=${item_next_steps.stdout}
ELSE IF    $deployment_status["unavailable_replicas"] > 0
    RW.Core.Add Issue
    ...    severity=3
    ...    expected=Deployment should have ${deployment_status["desired_replicas"]} pods.
    ...    actual=Deployment has ${deployment_status["ready_replicas"]} pods.
    ...    title=Deployment `${DEPLOYMENT_NAME}` has less than desired availability
    ...    next_steps=Inspect Deployment Warning Events for `${DEPLOYMENT_NAME}`
END

Complex issue — dynamic from script output

${recommendations}=    RW.CLI.Run Cli
...    cmd=echo '${container_restart_analysis.stdout}' | awk '/Recommended Next Steps:/ {flag=1; next} flag'
...    env=${env}
...    include_in_history=false
IF    $recommendations.stdout != ""
    ${recommendation_list}=    Evaluate    json.loads(r'''${recommendations.stdout}''')    json
    IF    len(@{recommendation_list}) > 0
        FOR    ${item}    IN    @{recommendation_list}
            RW.Core.Add Issue
            ...    severity=${item["severity"]}
            ...    expected=Containers should not be restarting for Deployment `${DEPLOYMENT_NAME}`
            ...    actual=Containers with restarts found for Deployment `${DEPLOYMENT_NAME}`
            ...    title=${item["title"]}
            ...    reproduce_hint=${container_restart_analysis.cmd}
            ...    details=${item["details"]}
            ...    next_steps=${item["next_steps"]}
        END
    END
END