With more and more data finding its way to elasticsearch, some indexes become too large for their purpose. We do not need to keep all data of the past weeks and months. We just want to keep the data needed for aggregations we show on a dashboard. Think about an index containing logs of our web server. We have a chart with HTTP status codes, response times, browsers, etc. Now you can create a rollup configuration providing the aggregations we want to keep, containing a cron expression telling when to run and some additional information about how much data to keep. The result is a new index with a lot less data that you can keep for your dashboards.
More information about the rollup functionality can be found here.
Last year at Elasticon Canvas was already shown. Elastic continued with the idea and it is starting to look amazing. With Canvas you can create beautiful looking dashboards that go a big step further than the standard dashboards in Kibana. You can customise almost everything you see. It comes with options to put an image on the background, a lot of color options, new sort of data visualisation integrated of course with elasticsearch. In a next blog post I’ll come up with a demo, it is looking very promising. Want to learn more about it, check this blog post.
One of the areas I still need to learn is the Docker / Kubernetes ecosystem. But if you are looking for a solution to monitor you complete Kubernetes platform, have a look at all the options that elastic has these days. Elastic has impressive support to monitor all the running images. It comes with standard dashboards in Kibana. It now has a dedicated space in Kibana called the Infra tab. More information about the options and how to get started can be found here.
Presenting Geo/Gis data
A very cool demo was given on how to present data on a Map. The demo showed where all Elasticon attendees were coming from. The visual component has an option of creating different layers. So you can add data to give the different countries a color based on the number of attendees. In a next layer show the bigger cities where people are coming from in small circles. Use a logo of the venue in another layer. Etc. Really interesting if you are into geodata. In all makes use of the Elastic Maps Service. If you want more information about this, you can find it here.
Elastic Site Search
Up till now there was news about new ways to handle your logs coming from application monitoring, infrastructure components, application logs. We did not hear about new things around search, until showing the new product called Elastic Site Search. This was previously known as Swiftype. With Google naming its product google search appliance end of life, this is a very interesting replacement. Working with relevance, synonyms, search analytics is becoming a lot easier with this new product. More information can be found here.
Elastic cloud sizing
If you previously looked at the cloud options elastic offers, you might have noticed that choosing elastic nodes did not give you a lot of flexibility. When choosing the amount of required memory, you also got a fixed amount of disk space. With the upcoming release, you have a lot more flexibility when creating your cluster. You can configure different flavours of clusters. One of them being hot-warm cluster. With specific master nodes, hot nodes for recent indexes with more RAM and faster disks, warm nodes containing the older indices with bigger disks. This is a good improvement if you want to create a cluster in the cloud. More information can be found here.
Opening up X-Pack
Shay told a good story about creating a company that supports an open source product. Creating a company only on support is almost impossible in the long run. Therefore they started working on commercial additions now called the X-Pack. Problem with these products was that the code was not available. Therefore working with elastic to help them improve the product was not possible. Therefore they are now opening up their repositories. Beware, it is not becoming free software. You still need to pay, but now it becomes a lot easier to interact with elastic about how stuff works. Next to that, they are going to make it easier to work with the free stuff in X-Pack. Just ask for a license once instead of every year again. And if I understood correct, the download will contain the free stuff in easier installation packages. More information about the what and why in this blog post from Shay Banon.
Nice party but I had to sign a contract to prohibit me from telling stories about the party. I do plan on writing more blog post the coming two days.