Elastic Watcher not returning results

0

I am trying to simulate a watch and see if the actions are triggering fine. But my problem is the search returns no results. My query

  1. Checks for a particular index.
  2. Checks for a range
  3. Check for the servicename field to be a particular value.

This is my watch definition

    {
      "trigger": {
        "schedule": {
          "interval": "10m"
        }
      },
      "input": {
        "search": {
          "request": {
            "search_type": "query_then_fetch",
            "indices": [
              "datasolutions-svc-*"
            ],
            "body": {
              "query": {
                "bool": {
                  "filter": [
                    {
                      "term": {
                        "level": {
                          "value": "ERROR"
                        }
                      }
                    },
                    {
                      "term": {
                        "servicename": [
                          "Iit.Det.Urm.MepsSubscriber"
                        ]
                      }
                    },
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-60m"
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      },
      "condition": {
        "compare": {
          "ctx.payload.hits.total": {
            "gt": 0
          }
        }
      },
      "actions": {
        "notify-slack": {
          "slack": {
            "account": "elastic_watcher_alerts",
            "proxy": {
              "host": "proxy.dom",
              "port": 80
            },
            "message": {
              "from": "Error Monitor",
              "to": [
                "#det-errors"
              ],
              "text": "The following error(s) have been logged",
              "dynamic_attachments": {
                "list_path": "ctx.payload.items",
                "attachment_template": {
                  "color": "#f00",
                  "title": "{{msg}}",
                  "title_link": "https://elastic.mid.dom:port/{{index}}/doc/{{id}}?pretty",
                  "text": "{{msg}}",
                  "fields": [
                    {
                      "title": "Server",
                      "value": "{{host}}",
                      "short": true
                    },
                    {
                      "title": "Servicename",
                      "value": "{{service}}",
                      "short": true
                    }
                  ]
                }
              }
            }
          }
        }
      },
      "transform": {
        "script": {
          "source": "['items': ctx.payload.hits.hits.collect(hit -> ['msg': hit._source.message, 'service': hit._source.servicename, 'index': hit._index, 'id' : hit._id, 'host': hit._source.agent.hostname ])]",
          "lang": "painless"
        }
      }
    }

I am trying to now test it by using the simulate option and giving it an input. This input is copied from actual data that is in the index. I copied a json document from kibana (in the discover section), so the alternate input json should be ok

Here's the alternative input

{
  "_index": "datasolutions-svc-live-7.7.0-2021.01",
  "_type": "doc",
  "_id": "Hre9SHcB1QIqYEnyxSCw",
  "_version": 1,
  "_score": null,
  "_source": {
    "exception": "System.Data.SqlClient.SqlException (0x80131904): blabla",
    "agent": {
      "hostname": "SATSVC3-DK1",
      "name": "datasolutions-svc-live",
      "id": "8c826ae1-e411-4257-a31f-08824dd58b5a",
      "type": "filebeat",
      "ephemeral_id": "e355bf8a-be67-4ed1-85f4-b9043674700e",
      "version": "7.7.0"
    },
    "log": {
      "file": {
        "path": "D:\\logs\\7DaysRetention\\Iit.Det.Urm.MepsSubscriber\\Iit.Det.Urm.MepsSubscriber.log.20210128.log"
      },
      "offset": 17754757
    },
    "level": "ERROR",
    "message": "Error while starting service.",
    "@timestamp": "2021-02-17T10:00:28.343Z",
    "ecs": {
      "version": "1.5.0"
    },
    "host": {
      "name": "datasolutions-svc-live"
    },
    "servicename": "Iit.Det.Urm.MepsSubscriber",
    "codelocation": "Iit.Det.Urm.MepsSubscriber.MepsSubscriberService.OnStart:29"
  },
  "fields": {
    "@timestamp": [
      "2021-02-17T10:00:28.343Z"
    ]
  },
  "highlight": {
    "servicename": [
      "@kibana-highlighted-field@Iit.Det.Urm.MepsSubscriber@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1611833128343
  ]
}

But when I run "simulate", I get the ctx.payload.total.hits as null because apparently it does not find any results. Result of the simulate-

    {
  "watch_id": "_inlined_",
  "node": "eMS-E34eT4-zZhGwtPNSmw",
  "state": "execution_not_needed",
  "user": "sum",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2021-02-17T10:57:04.077Z"
    },
    "last_checked": "2021-02-17T10:57:04.077Z",
    "actions": {
      "notify-slack": {
        "ack": {
          "timestamp": "2021-02-17T10:57:04.077Z",
          "state": "awaits_successful_execution"
        }
      }
    },
    "execution_state": "execution_not_needed",
    "version": -1
  },
  "trigger_event": {
    "type": "manual",
    "triggered_time": "2021-02-17T10:57:04.077Z",
    "manual": {
      "schedule": {
        "scheduled_time": "2021-02-17T10:57:04.077Z"
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "datasolutions-svc-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "level": {
                      "value": "ERROR"
                    }
                  }
                },
                {
                  "term": {
                    "servicename": [
                      "Iit.Det.Urm.MepsSubscriber"
                    ]
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-60m"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "metadata": {
    "name": "datasolutions-svc-mepssubscriber",
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2021-02-17T10:57:04.077Z",
    "execution_duration": 0,
    "input": {
      "type": "simple",
      "status": "success",
      "payload": {
        "highlight": {
          "servicename": [
            "@kibana-highlighted-field@Iit.Det.Urm.MepsSubscriber@/kibana-highlighted-field@"
          ]
        },
        "_index": "datasolutions-svc-live-7.7.0-2021.01",
        "_type": "doc",
        "_source": {
          "exception": "System.Data.SqlClient.SqlException (0x80131904): blabla",
          "agent": {
            "hostname": "SATSVC3-DK1",
            "name": "datasolutions-svc-live",
            "id": "8c826ae1-e411-4257-a31f-08824dd58b5a",
            "type": "filebeat",
            "ephemeral_id": "e355bf8a-be67-4ed1-85f4-b9043674700e",
            "version": "7.7.0"
          },
          "@timestamp": "2021-02-17T10:00:28.343Z",
          "ecs": {
            "version": "1.5.0"
          },
          "log": {
            "file": {
              "path": "D:\\logs\\7DaysRetention\\Iit.Det.Urm.MepsSubscriber\\Iit.Det.Urm.MepsSubscriber.log.20210128.log"
            },
            "offset": 17754757
          },
          "level": "ERROR",
          "host": {
            "name": "datasolutions-svc-live"
          },
          "servicename": "Iit.Det.Urm.MepsSubscriber",
          "message": "Error while starting service.",
          "codelocation": "Iit.Det.Urm.MepsSubscriber.MepsSubscriberService.OnStart:29"
        },
        "_id": "Hre9SHcB1QIqYEnyxSCw",
        "sort": [
          1611833128343
        ],
        "_score": null,
        "fields": {
          "@timestamp": [
            "2021-02-17T10:00:28.343Z"
          ]
        },
        "_version": 1
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": false,
      "compare": {
        "resolved_values": {
          "ctx.payload.hits.total": null
        }
      }
    },
    "actions": []
  },
  "messages": []
}

I am not sure what can't it find the results. Can someone tell me what is it that I am doing wrong?

elasticsearch
kibana
elasticsearch-watcher

1 Answer

1

I was able to solve it using the "inspect" section of discover page of the index.

Finally my input for the watcher query had to be changed to

"input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "datasolutions-svc-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "must": [],
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "match_phrase": {
                          "servicename": "Iit.Det.Urm.MepsSubscriber"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                },
                {
                  "match_phrase": {
                    "level": "ERROR"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m",
                      "format": "strict_date_optional_time"
                    }
                  }
                }
              ],
              "should": [],
              "must_not": []
            }
          }
        }
      }
    }
  }
answered on Stack Overflow Feb 17, 2021 by Saugat Mukherjee

User contributions licensed under CC BY-SA 3.0