Elasticsearch is a free, open search analytics engine for all types of data. Elasticsearch is the most popular search engine out there. It can be used to quickly index and retrieve and search among the entries. For example, GitHub uses Elasticsearch for user searches in the repository, users, and other documents and entries. Also, when you search for an item on an e-commerce website, you are probably performing a search query inside an elastic search instance. Besides these user-facing applications, many organizations use Elasticsearch for their internal analytics and monitoring jobs. Elasticsearch is distributed. It means that multiple instances of Elasticsearch on multiple servers act as one big index that can be scaled to hundreds of servers. In this document, we will talk about how to deploy a single instance of Elasticsearch on the Doprax cloud platform.
Add Elasticseaarch
To use Elasticsearch in your project, you have to add it as a service. Go to your services tab and click add service, on the next page chose Elasticsearch and click add. It will ask for a mandatory discovery_type environment variable. For a single node installation (which is our case) leave it blank.
Then, if you come back to your services tab, you can see the Elasticsearch listed under added services.
Environment variables
For the maximum performance of your Elasticsearch instance, you need to set two other environment variables also.
bootstrap.memory_lock: This environment variable controls memory allocation and prevents Elasticsearch from using SWAP memory. It gives the Elasticsearch instance maximum performance. Set this environment variable to true
ES_JAVA_OPTS: This option controls the size of the JVM heap. We in this example it has been set to 8GB by setting this environment variable to -Xms8g -Xmx8g. It sets the maximum and minimum heap sizes.
For more information about all the options, you can visit the https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html. At the end, the service should look like the below image in-service tab.
Add volume
By default, Elasticsearch uses the writable layer of the container to save its indexes. But just like every other container, all data will be erased when the container is stopped or restarted. Even if you don’t need to save data, using this writable layer reduces the performance. To solve this, we need to create a volume and mount it on the Elasticsearch container. Go to the volumes tab and click on create a volume button. Give the volume a name and for the mount on option choose Elasticsearch. The mount path for Elasticsearch should be /usr/share/elasticsearch/data
Run Elasticsearch
To run your newly added Elasticsearch service, you need to go to the deploy tab in your project. Then click the run button on the Elasticsearch service and everything will be built and run in a couple of minutes. But before running the Elasticsearch instance we should probably increase the resource size (RAM and CPU cores). Since I am going to have a big index size and I have already had a JVM heap size set to 8GB I need to give Elasticsearch at least 8GB of RAM and I will choose 2 vCPU cores. Click on cog shape to change the resource size:
After that, you can run the Elasticsearch container. Check out the logs to see if any problems and issues are there.
Permission problem
One issue that might arise when using a volume for Elasticsearch is the writing permission and ownership of the volume. If the Elasticsearch does not have the permission to write on the volume path (/usr/share/elasticsearch/data) it will fail to start. You can easily change the permission of the folder by opening a shell. The real problem is here; If the Elasticsearch does not have permission to write to the volume, it will fail to start. Shell is only possible for the containers that are at the running stage. Then how exactly should we open a shell into that?! One possible solution is to temporarily mount the Elasticsearch volume to another container (for example main) and use the shell of the main to change the permissions of the volume path. And then we will mount it back to the Elasticsearch container. Let’s do this. Go to the volumes tab and fine ES_data volume. Click on three dots (ellipses) and click settings. Then choose another container to mount to. I am going to choose the main container. It should look like this:
Then go back to the deploy tab and restart the main changes to take effect and the volume could be mounted to main. When the main container goes into the running mode. It is time to open a shell and change the permission of this folder. By default, the Elasticsearch container uses a user named elasticsearch and has the code of 1000. Open the shell on ES_data and enter the following commands to change the ownership to the use with code of 1000 and group of 1000 and then give it the writing permission
chown 1000:1000 /usr/share/elasticsearch/data -R chmod +w /usr/share/elasticsearch/data
Now that the volume path has appropriate permissions, it is time to mount it back to the Elasticsearch container. Go to the volumes tab and click on the three-dot (ellipses) of the ES_data volume and select Elasticsearch for mount on. No go to the deploy tab and restart the Elasticsearch container. This time there should be no problems. Check out the logs to see if there are any issues.
Check the health of Elasticsearch
To make sure that the Elasticsearch is healthy and without any problems, you should open a terminal on the Elasticsearch containers (in deploy tab) and enter the following command
curl -X GET "localhost:9200/_cluster/health?timeout=50s&pretty"
The result should be something like this: Notice the green status.
{
"cluster_name" : "docker-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}Now you can access the elasticsearch from the main and other containers on the Elasticsearch host (with the hostname of elasticsearch) and on the port of 9200. For example from the main container, you can run the previous command like this:
curl -X GET "elasticsearch:9200/_cluster/health?timeout=50s&pretty"
Notice that we have changed the localhost to elasticsearch since we are connecting to the elasticsearch from outside the elasticsearch container. That is possible because containers of the project are connected to the same private network. You can also use the assigned private IP of each container for communications. But every time a new service is started and stopped it may receive another IP. Because of this, it is much easier to use hostnames for communications. The hostname of each service by default is the name of the service (here it is elasticsearch).