The new elasticsearch java Rest Client - part 2

Search

In the previous blog post I started writing about using the new Rest based client for java in elasticsearch 5. In this blogpost I am going to continue that quest by experimenting with some layers of abstraction using spring features and other considerations.

Spring Factory Bean for RestClient

First I introduce the Client factory, responsible for creating the singleton client of type RestClient. This factory creates the client as a singleton. It also creates the Sniffer to check for available nodes, like we discussed in the first blog post (see references). The code is the same as before, therefore I am not going to show it again. Check the class: nl.gridshore.elastic.RestClientFactoryBean, git repo is referenced later on. The RestClientFactoryBean extends the spring AbstractFactoryBean. The result of using the spring Factory pattern is that this factory object creates the RestClient instance which can be injected into other beans.

The goal for the application we are going to create is an employee search tool. Therefore I have created and Employee object that I am going to use to index but also as the query result. But I do not want the elastic client execution code to be mixed with the employee business code. Therefore I have created an abstraction layer of interacting with the elastic client. The code to execute index requests for indexing employees and querying for employees is using this abstraction layer.

The EmployeeService knows the QueryTemplateFactory. It uses the factory to get an instance of a QueryTemplate. The QueryTemplateFactory knows the RestClientFactoryBean. The RestClientFactoryBean is responsible for maintaining the one instance of RestClient. The QueryTemplateFactory creates a new instance of the QueryTemplate for each call and injects the RestClient. The QueryTemplate could now be used to inject a string based query and handle the execution and response handling. Of course that is not really what we want, adding strings to create a query. There is a better way of doing this using a template engine. But first I want to talk about handling the response.

Handling the response

Handling a response can be challenging. For a basic query it is not to hard. But when interacting with nested structures and later on adding aggregations it is a lot harder. For now we focus on a basic response that looks like this:

java
{
  "took": 19,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.7373906,
    "hits": [
      {
        "_index": "luminis",
        "_type": "ams",
        "_id": "AVX57b0YY0m5tWxWetET",
        "_score": 0.7373906,
        "_source": {
          "name": "Jettro Coenradie",
          "email": "jettro@gridshore.nl",
          "specialties": [
            "java",
            "elasticsearch",
            "angularjs"
          ],
          "phone_number": "+31612345678"
        }
      }
    ]
  }
}

The QueryTemplate must to the boilerplate and from the Employee perspective we want to provide enough information to convert the _source parts into Employee objects. I use a combination of Jackson and generics to accomplish this. First let us have a look at the code to interact with the QueryTemplate:

java
public List<Employee> queryForEmployees(String name) {

    Map<String, Object> params = new HashMap<>();

    params.put("name", name);

    params.put("operator", "and");


    QueryTemplate<Employee> queryTemplate = queryTemplateFactory.createQueryTemplate();

    queryTemplate.setIndexString(INDEX);

    queryTemplate.setQueryFromTemplate("find_employee.twig", params);

    queryTemplate.setQueryTypeReference(new EmployeeTypeReference());


    return queryTemplate.execute();

}

As you can see from the code, the QueryTemplate receives the name of the index to query, the name of the twig template (see next section) and the parameters used by the twig template. The final thing we need to give is the TypeReference. This is necessary for Jackson to handle the generics. This type reference looks like this:

java
public class EmployeeTypeReference extends TypeReference<QueryResponse<Employee>> {
}


Finally we call the execute method of the QueryTemplate and notice that it returns a list of Employee objects. To see how this works we have to take a look at the response handling by the QueryTemplate.

java
public List<T> execute() {

    List<T> result = new ArrayList<>();


    this.queryService.executeQuery(indexString, query(), entity -> {

        try {

            QueryResponse<T> queryResponse = jacksonObjectMapper.readValue(entity.getContent(), this.typeReference);


            queryResponse.getHits().getHits().forEach(tHit -> {

                result.add(tHit.getSource());

            });

        } catch (IOException e) {

            logger.warn("Cannot execute query", e);

        }

    });


    return result;

}


Using jackson we convert the son based response from elasticsearch client into a QueryResponse object. Notice that we have a generic type ’T’. Jackson only know how to do this by passing the right TypeReference. The java object tree resembles the json structure:

java
public class QueryResponse<T> {

    private Long took;


    @JsonProperty(value = "timed_out")

    private Boolean timedOut;


    @JsonProperty(value = "_shards")

    private Shards shards;


    private Hits<T> hits;

}
 
public class Hits<T> {

    private List<Hit<T>> hits;

    private Long total;


    @JsonProperty("max_score")

    private Double maxScore;

}
 
public class Hit<T> {

    @JsonProperty(value = "_index")

    private String index;


    @JsonProperty(value = "_type")

    private String type;


    @JsonProperty(value = "_id")

    private String id;


    @JsonProperty(value = "_score")

    private Double score;


    @JsonProperty(value = "_source")

    private T source;
}

Notice what happens with the generic type ’T’, so the source is converted into the provided type. Now look back at the code of the execute method of the QueryTemplate. Here we obtain the hits from the QueryResponse and loop over these hits to obtain the Employee objects.

Using the twig template language

Why did I chose to use a template language to create the query? It is not really safe to just copy a few strings including user input together into one long string and provided that to the QueryTemplate executer. I wanted to stay close to json to make copy paste from the elastic console as easy a possible. However I also wanted some nice features to make creating queries easier. I found jTwig to be powerful enough and very easy to use in java. I do not want to write an extensive blogpost about jTwig. if you want to know more about it please check the references.

The following code block shows how we use jTwig:

java
public void setQueryFromTemplate(String templateName, Map<String, Object> modelParams) {

    JtwigTemplate template = JtwigTemplate.classpathTemplate("templates/" + templateName);

    JtwigModel model = JtwigModel.newModel();

    modelParams.forEach(model::with);


    this.query = template.render(model);

}

First you have to create the model and load the template. In my case I load the template from a file on the class path in the templates folder. Next I add the provided parameters to the model and finally I render the template using the model. The next code block shows the template.

java
{

    "query":{

{% if (length(name)==0) %}

        "match_all": {}

{% else %}

        "match":{

            "name": {

                "query": "{{ name }}",

                "operator": "{{ default(operator, 'or') }}"

            }

        }

{% endif %}

    }

}

This model supports two parameters: name and operator. First I check if the user provided something to search for. If the name is empty we just return the match_all query. If the name has length, a match query is created on the field name. Also notice that we can provide a default value for the operator parameter in this case. So if no operator is provided we make it or.

Concluding

That is it, now we have an easy way to write down a query in a jTwig template and parse the results using jackson. If would be very easy to query for another object instead of Employee if we wanted to. The indexing side is similar to the query side. Feel free to check this out yourself. All code is available in the github repo.

References

http://jtwig.org – Template language used for the queries.
https://amsterdam.luminis.eu/2016/07/07/the-new-elasticsearch-java-rest-client/ – part 1 of this blog series
https://github.com/jettro/elasticclientdemo – The sample project

Search