First steps with elastic watcher

-

Some time a go elastic introduced a new product called watcher. Watcher is product that lets you take action based on the outcome of queries executed against elasticsearch but also against other http end points. The main purpose for the product is creating notification on some condition.

In this blog post I will explain the basic concepts and you can follow a long using some example code that I present in the blog post. At the end of the post you should have a general idea about what you can do with the watcher product.

Installation

Watcher is a commercial plugin from elasticsearch. At the moment it is still in beta and you need to register with elastic to get beta access. More information can be found here:

https://www.elastic.co/downloads/watcher

Like with other commercial offerings from elastic, watcher comes as a plugin for elasticsearch. You have to install the license plugin as well as the watcher plugin. You can install the watcher plugin in your main cluster, but you can also install it in a seperate cluster. I have choosen to follow the second path. So I run two clusters on my local machine. The commercial cluster containing the commercial plugins runs on port 9300/9200 and the other cluster on 9301/9201.

After installation you can check that it works using curl.
Check that it works:

curl -XGET 'http://localhost:9200/_watcher/stats?pretty'

The response in my case is.

{
  "watcher_state" : "started",
  "watch_count" : 1,
  "execution_queue" : {
    "size" : 0,
    "max_size" : 20
  }
}

What we want to accomplish

We are going to log the amount of snapshots that we have in the repository. We also want to have a look at the watcher history

Structure of a watcher

Time to take one step further with the watcher. In this section we take a first look at the different elements of a watcher.

  • Metadata – Static data added to the watch (payload), can be used in scripts or templates
  • Trigger – Determines when a watch is checked. At the moment there are only scheduled triggers, but the future will most likely also bring us document event based triggers. All sorts of scheduled triggers are available. Think about hourly, dayly, weekly, etc. Each trigger can be configured to run on certain minutes every hour, or certain hours every day, etc.
  • Input – Three types of input, Simple for testing, search for querying the local cluster and http for calling a remote service which can be a remote elasticsearch instance.
  • Conditions – Four type of conditions: always, never, compare ad script. Compare enables you to do some basic comparisons on the watch payload values. If you need a real script you also need to enable dynamic scripting. Since elasticsearch 1.6 there is fine grained control for the scripting support. More on this late on.
  • Transforms – Three type of transforms: search, script and chain. Search replaces the current payload, script transforms the current payload and with chain you can add multiple search and script transformations in a chain. A good example is to determine the amount of errors and check if there is an action required based on the number of errors. Than in the transform we obtain all the errors and replace the payload with these obtained exceptions.
  • Actions – Watcher supports four types of actions email, webhook, index, and logging. Important to notice that you can have multiple actions. Log to the log service as well as send an email.

Let us create a watcher

Before we can add the watcher let us have a look at the command to obtain snapshots. The following command shows all available snapshots for the repository named “temp-bck”.

curl "http://localhost:9201/_snapshot/temp-bck/_all"

The actual response contains more items in the array, but each item looks like this snapshot.

{
  "snapshots" : [ {
    "snapshot" : "conferences-20150619094018",
    "indices" : [ "conferences-20150619092724" ],
    "state" : "SUCCESS",
    "start_time" : "2015-06-19T07:40:18.466Z",
    "start_time_in_millis" : 1434699618466,
    "end_time" : "2015-06-19T07:40:18.540Z",
    "end_time_in_millis" : 1434699618540,
    "duration_in_millis" : 74,
    "failures" : [ ],
    "shards" : {
      "total" : 2,
      "failed" : 0,
      "successful" : 2
    }
  }]
}

Now let us use this request and response to create the watcher

Input

We are using another cluster and we use a different elastic api than the search api. Therefore we have to use the http input.

{
  "input" : {
    "http" : {
      "request" : {
        "host" : "localhost",
        "port" : 9201,
        "path" : "/_snapshot/temp-bck/_all"
      }
    }
  }
}

The trigger

Every watch needs a trigger. Since we only have schedule based triggers at the moment, we create a schedule based trigger. The following code shows a scheduled trigger to run every 5 seconds. A bit to much for this use case, but for testing purposes better. We don’t want to wait to long for results. In reality you could be better of using one of the hourly/daily/weekly/monthly/yearly triggers. For this sample we stick with the interval based schedule.

{
  "trigger" : {
    "schedule" : {
      "interval" : "5s"
    }
  }
}

One feature for calling the action is called throtteling. Only one instance of a watcher can run at a time and the action is by default only performed if it did not perform the past 5 seconds. This behavior can be customized using the property:

throttle_period

. There is an additional concept for throtteling called acknowledgment. With acknowledgment the action will not be performed again after the first time before the action is acknowledged. More information about this can be found here.

actions-ack-throttle

Condition

For now we use the always condition, this means that every watch is passed on to the actions. This is the default condition when not providing a condition at all.

Transform

Before taking action we want to transform the payload into some interesting numbers. We want to transform using a script. Before being able to execute scripts you have to configure the cluster to support scripts. Check the following resource for more information:

modules-scripting

In this case I needed to add the following setting to the elasticsearch.yml file

script.engine.groovy.inline.plugin: on

{
  "transform": {
    "script": "return [total_snapshots : ctx.payload.snapshots.size()]"
  }
}

Action

The most simple action is a log action, for now let us stick with that. We want to log the amount of available snapshots at the time of running the watcher.

{
  "actions" : {
    "log" : { 
      "logging" : {
        "text": "Found {{ctx.payload.total_snapshots}} snapshots at {{ctx.execution_time}}"
      }
    }
  }
}

If you are learning about the response, you can also print out “ctx.payload” or even “ctx” using this action. That way you can get an idea what information is available in the watcher context.

Putting it all together

curl -XPUT "http://localhost:9200/_watcher/watch/monitor_snapshots" -d'
{
  "trigger": {
    "schedule": {
      "interval": "5s"
    }
  },
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9201,
        "path": "/_snapshot/temp-bck/_all"
      }
    }
  },
  "transform": {
    "script": "return [total_snapshots : ctx.payload.snapshots.size()]"
  },
  "actions": {
    "log": {
      "logging": {
        "text": "Found {{ctx.payload.total_snapshots}} snapshots at {{ctx.execution_time}}"
      }
    }
  }
}'

Now you can check the log of elasticsearch to see the log lines coming by.

[2015-06-21 12:22:42,899][INFO ][watcher.actions.logging  ] [Node-jc] Found 2 snapshots at 2015-06-21T10:22:42.882Z

Do not forget to remove the watcher if you are done, or else it keeps logging 🙂

curl -XDELETE "http://localhost:9200/_watcher/watch/monitor_snapshots"

Look at the watcher history

There is a complete REST api for the watcher. Using this api you can create new watchers, remove watchers but also get information about a specific watcher. Given the watcher we have created in this blog, the following command can be used to obtain information about the watcher.

curl -XGET "http://localhost:9200/_watcher/watch/monitor_snapshots"

What I did not expect was the status information of the watcher. This status information makes it easy to read the last time the watcher was executed, whether is was succesfull or not. Below I show you only the status part of the response

{
  "status": {
    "last_checked": "2015-06-21T21:14:28.547Z",
    "last_met_condition": "2015-06-21T21:14:28.547Z",
    "actions": {
      "log": {
        "ack": {
          "timestamp": "2015-06-21T21:14:18.495Z",
          "state": "ackable"
        },
        "last_execution": {
          "timestamp": "2015-06-21T21:14:28.547Z",
          "successful": true
        },
        "last_successful_execution": {
          "timestamp": "2015-06-21T21:14:28.547Z",
          "successful": true
        }
      }
    }
  }  
}

Test execution of a watcher

This is such a feature that you learn to appreciate from the elastic guys. They like to make it easier for the developers. There is an execution api to force execution of a watcher. There are a lot of things to configure, there is something like a dry run. But what is even more interesting is providing a watcher inline, so without actually storing it in elasticsearch first. More information can be found here.

Inline watch

There are lots of other endpoints to be used using the REST api. You can start/stop/restart the watcher process. You can ask for stats about the watcher, but you can also query the history of the watchers using plain queries against the following index

watch_history-yyyy.MM.dd

. This index is the trace of the watcher. It contains everything you need to find out what your watcher did, when it did it and what the resutls were. Since it is a normal index you can do everything you want with it. The following query just obtains the last 10 log entries. If you want to learn more about the response try it out yourself, you won’t be disappointed, I promise.

curl -XGET "http://localhost:9200/.watch_history-*/_search" -d'
{
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ]
}'

Additional reading and watching

For a product in beta there is a fair amount of documentation. Also check the video with an introduction into watcher.

https://www.elastic.co/guide/en/watcher/current/introduction.html
Webinar by Uri Boness and Steve Kearns