Creating an elasticsearch plugin, the basics

-

Elasticsearch is a search solution based on Lucene. It comes with a lot of features to enrich the search experience. Some of these features have been recognised as very useful in the analytics scene as well. Interacting with elasticsearch mainly takes place using the REST endpoint. You can do everything using the different available endpoints. You can create new indexes, insert documents, search for documents and lots of other things.

Still some of the things are not available out of the box. If you need an analyser that is not available by default, you can install it as a plugin. If you need security, you can install a plugin. If you need alerting, you can install it as a plugin. I guess you get the idea by now. The plugin extension option is nice, but might be a bit hard to begin with. Therefore in this blog post I am going to write a few plugins. I’ll point you to some of the resources I used to get it running and I want to give you some inspiration for your own ideas for cool plugins that extend the elasticsearch functionality.

Bit of history

In the releases prior to version 5 there were two type of plugins, site and java plugins. Site plugins were used extensively. Some well known examples are: Head, HQ, Kopf. Also Kibana and Marvel started out as a site plugin. It was a nice feature, however not the core of elasticsearch. Therefore the elastic team deprecated site plugins in 2.3 and the support was removed in 5.

How does it work

The default elasticsearch installation already provides a script to install plugins. You can find it in the binfolder. You can install plugins from repositories but also from a local path. A plugin comes in the form of a jar file.

Plugins need to be installed on every node of the cluster. Installation is as simple as the following command.

bin/elasticsearch-plugin install file:///path/to/elastic-basic-plugin-5.1.2-1-SNAPSHOT.zip

In this case we install the plugin from our own hard drive. The plugins have a dependency on the elastic core and therefore need to have the exact same version as the elastic version you are using. So for each elasticsearch release you have to create a new version of the plugin. In the example I have created the plugin for elasticsearch 5.1.2.

Start with our own plugin

Elastic uses gradle internally to build the project, I still prefer maven over gradle. Luckily David Pilato wrote a good blog post about creating the maven project. I am not going to repeat all the steps of him. Feel free to take a peek at the pom.xml I used in my plugin.

Create BasicPlugin that does nothing

The first step in the plugin is to create a class that starts the plugin. Below is the class that has just one functionality, print a statement in the log that the plugin is installed.

public class BasicPlugin extends Plugin {
    private final static Logger LOGGER = LogManager.getLogger(BasicPlugin.class);
    public BasicPlugin() {
        super();
        LOGGER.warn("Create the Basic Plugin and installed it into elasticsearch");
    }
}

Next step is to configure the plugin as described by David Pilato in his blog I mentioned before. We need to add the maven assembly plugin using the file src/main/assemblies/plugin.xml. In this file we refer to another very important file, src/main/resources/plugin-descriptor.properties. With all this in place we can run maven to create the plugin in a jar.

mvn clean package -DskipTests

In the folder target/releases you’ll now find the file elastic-basic-plugin-5.1.2-1-SNAPSHOT.zip. Which is a jar file in disguise, we could change the extension to jar, there is no difference. Now use the command from above to install. If you get a message that the plugin is already there, you need to remove it first

bin/elasticsearch-plugin remove elastic-basic-plugin

Then after installing the plugin you’ll find the following line in the log of elasticsearch when starting

[2017-01-31T13:42:01,629][WARN ][n.g.e.p.b.BasicPlugin    ] Create the Basic Plugin and installed it into elasticsearch

This is of course a bit silly, let us create a new rest endpoint that checks if the elasticsearch database contains an index called Jettro.

Create a new REST endpoint

The inspiration for this endpoint came from another blog post by David Pilato: Creating a new rest endpoint.

When creating a new endpoint you have to extend the class org.elasticsearch.rest.BaseRestHandler. But before we go there, we first initialise it in our plugin. To do that we implement the interface org.elasticsearch.plugins.ActionPlugin and implement the method getRestHandlers.

public class BasicPlugin extends Plugin implements ActionPlugin {
    private final static Logger LOGGER = LogManager.getLogger(BasicPlugin.class);
    public BasicPlugin() {
        super();
        LOGGER.warn("Create the Basic Plugin and installed it into elasticsearch");
    }
 
    @Override
    public List> getRestHandlers() {
        return Collections.singletonList(JettroRestAction.class);
    }
}

Next is implementing the JettroRestAction class. Below the first part, the constructor and the method that handles the request. In the constructor we define the endpoint url patterns that this endpoint supports. The are clear from the code I think. Functionality wise, if you call without an action or with another action than exists, we return a message, if you ask for existence we return true or false. This handling is done in the prepareRequest method.

public class JettroRestAction extends BaseRestHandler {
 
    @Inject
    public JettroRestAction(Settings settings, RestController controller) {
        super(settings);
        controller.registerHandler(GET, "_jettro/{action}", this);
        controller.registerHandler(GET, "_jettro", this);
    }
 
    @Override
    protected RestChannelConsumer prepareRequest(RestRequest request, NodeClient client) throws IOException {
        String action = request.param("action");
        if (action != null && "exists".equals(action)) {
            return createExistsResponse(request, client);
        } else {
            return createMessageResponse(request);
        }
    }
}

We have two utility classes that transform data into XContent: Message and Exists. The implementations of the two methods: createExistsResponse and createMessageResponse, can be found here.

Time to re-install the plugin, first build it with maven, remove the old one and install the new version. Now we can test it in a browser or with curl. I personally use httpie to do the following requests.

This way we can create our own custom endpoint. Next we dive a little bit deeper into the heart of elastic. We are going to create a custom filter that can be used in an analyser.

Create a custom Filter

The first part is registering the Filter in the BasePlugin class. We need to extend the interface org.elasticsearch.plugins.AnalysisPlugin and override the method getTokenFilters. We register a factory class that instantiates the filter class. The registration is done using a name that can later on be used to use the filter. The method looks like this

@Override
public Map> getTokenFilters() {
    return Collections.singletonMap("jettro", JettroTokenFilterFactory::new);
}

The implementation of the factory is fairly basic

public class JettroTokenFilterFactory extends AbstractTokenFilterFactory {
    public JettroTokenFilterFactory(IndexSettings indexSettings, 
                                    Environment environment, 
                                    String name, 
                                    Settings settings) {
        super(indexSettings, name, settings);
    }
 
    @Override
    public TokenStream create(TokenStream tokenStream) {
        return new JettroOnlyTokenFilter(tokenStream);
    }
}

The filter we are going to create has a bit strange functionality. It only accepts tokens that are the same as jettro. All other tokens are removed.

public class JettroOnlyTokenFilter extends FilteringTokenFilter {
    private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
 
    public JettroOnlyTokenFilter(TokenStream in) {
        super(in);
    }
 
    @Override
    protected boolean accept() throws IOException {
        return termAtt.toString().equals("jettro");
    }
}

Time to test my fresh created filter. We can do that using the analyse endpoint

curl -XGET 'localhost:9200/_analyze' -d '
{
  "tokenizer" : "standard",
  "filter" : ["jettro"],
  "text" : "this is a test for jettro"
}'

The response now is

{"tokens":[{"token":"jettro","start_offset":19,"end_offset":25,"type":"","position":5}]}

Concluding

That is it, we have created the foundations to create a plugin, thanks to David Pilato, we have written our own Jettro endpoint and we have created a filter that only accepts one specific word, Jettro. Ok, I agree the plugin in itself is not very useful, however the construction of the plugin is re-useable. Hope you like it and stay tuned for more elastic plugin blogs. We’re working on an extension to the synonyms plugin and have some ideas for other plugins.