\n The number of shards per node is one of the important things to monitor while monitoring the health of the Elasticsearch cluster. In this post, we will see how to get it programmatically.\n
\n\n\n Elasticsearch stores documents in indices. Each index is made up of one or more shards. Each shard is an instance of Lucene and can be considered as a self-contained index itself with a subset of the data. There are two types of shards: primary and replica shards. Primary shards are the ones where writing happens and replica shards contribute to the data redundancy and handling search queries. \n
\n There is no hard limit imposed by Elasticsearch on the number of shards per node. The number of shards on a node is proportional to the amount of heap memory. If you want to check the current heap memory you can use the following command:
\n
GET _cat/nodes?v=true&h=heap.current\n\n One of the good rules of thumb is to aim for 20 shards per GB of heap memory. So for example, if a node has 10 GB heap memory, then it should have a maximum of 200 shards. To check the shards per node we can use the shard API as shown below:\n
\nGET _cat/shards?v=true&format=json\n// Result of the above command is as following:\n[\n {\n \"index\": \".apm-custom-link\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"0\",\n \"store\": \"208b\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \"accounts\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"1000\",\n \"store\": \"382.3kb\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \"accounts\",\n \"shard\": \"0\",\n \"prirep\": \"r\",\n \"state\": \"UNASSIGNED\",\n \"docs\": null,\n \"store\": null,\n \"ip\": null,\n \"node\": null\n },\n {\n \"index\": \"posts\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"1\",\n \"store\": \"4.4kb\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \"posts\",\n \"shard\": \"0\",\n \"prirep\": \"r\",\n \"state\": \"UNASSIGNED\",\n \"docs\": null,\n \"store\": null,\n \"ip\": null,\n \"node\": null\n },\n {\n \"index\": \".kibana_1\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"50\",\n \"store\": \"10.4mb\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \"accounts_test\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"5000\",\n \"store\": \"1.7mb\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \"accounts_test\",\n \"shard\": \"0\",\n \"prirep\": \"r\",\n \"state\": \"UNASSIGNED\",\n \"docs\": null,\n \"store\": null,\n \"ip\": null,\n \"node\": null\n },\n {\n \"index\": \"ilm-history-2-000001\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": null,\n \"store\": null,\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \".kibana_task_manager_1\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"6\",\n \"store\": \"417.9kb\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \".apm-agent-configuration\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"0\",\n \"store\": \"208b\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n },\n {\n \"index\": \".kibana-event-log-7.9.3-000001\",\n \"shard\": \"0\",\n \"prirep\": \"p\",\n \"state\": \"STARTED\",\n \"docs\": \"4\",\n \"store\": \"21.6kb\",\n \"ip\": \"172.17.0.2\",\n \"node\": \"618c33cb6a53\"\n }\n]\n\n we can then count the number of shards per node.\n
\n\n\n The following code shows how we can achieve the same using the Java High Level Rest client. Please note that we are using the low level rest client here to fetch the details about the shards as I could not find support in high level client. Please have a look into this for the completeness of high level client.\n
\npublic Map<String, Integer> getNumberOfShardsPerNode() {\n try {\n final Response response = restHighLevelClient.getLowLevelClient()\n .performRequest(new Request(\"GET\", \"/_cat/shards?v&format=json\"));\n final var string = EntityUtils.toString(response.getEntity());\n final var shardDetails = new ObjectMapper()\n .readValue(string, new TypeReference<List<NodeDetails>>() {\n });\n final var shardsPerNode = shardDetails.stream()\n .filter(nodeDetails -> !\"UNASSIGNED\".equalsIgnoreCase(nodeDetails.getState()))\n .collect(Collectors.groupingBy(NodeDetails::getNode, Collectors.summingInt(value -> 1)));\n shardsPerNode.forEach((node, shards) -> log.info(\"Node : {}, Shard count: {}\", node, shards));\n return shardsPerNode;\n\n } catch (IOException ioException) {\n ioException.printStackTrace();\n }\n return Collections.emptyMap();\n }\n The above code prints number of shards per node as below:\n
\nNode : 34_b6Yiq, Shard count: 360\nNode : as0aDaPl_, Shard count: 360\nNode : 32_wkiqAf5, Shard count: 360\n\n In this short post, we saw how to calculate the number of shards per node in the Elasticsearch cluster. Happy learning š.\n
\n\n\n We can use Java High Level Rest Client to get the count of documents in an index. It can also be used to execute a query and get the count of matching documents.\n
\n\n\n The Elasticsearch Java High Level Rest Client provides count API which can be used to get the followings:\n
\n In our previous post we used account data and inserted in accounts index. I want to know the number of documents where the first name of the account holder is Kelley. The following code shows the JSON query for the same:\n
#Number of accounts with Firstname as Kelley\nPOST accounts/_count\n{\"query\":{\"match\":{\"firstname.keyword\":\"Kelley\"}}}\n\n For achieving the same using Java High Level Rest Client, we can build the query as shown below:\n
\nfinal var countRequest = new CountRequest(\"accounts\");\ncountRequest.query(QueryBuilders.matchQuery(\"firstname.keyword\", \"Kelley\")); \n\n After this we can use High Level Rest Client to execute the query as following. For configuring the client please follow this post.\n
\nfinal var countResponse = restHighLevelClient.count(countRequest, RequestOptions.DEFAULT);\nfinal var count = countResponse.getCount(); \n A working code example is present in this git repository. I hope it is useful for you. Happy learning :).\n
\n\n\n \n
", "image": "https://imteyazahmad.com/media/posts/8/hand-162127.png", "author": { "name": "Imteyaz Ahmad" }, "tags": [ "high level rest client", "Java", "Elasticsearch" ], "date_published": "2021-04-22T10:44:36-04:00", "date_modified": "2021-04-22T12:12:08-04:00" }, { "id": "https://imteyazahmad.com/elasticsearch-pre-indexing-transformation-using-ingest-pipelines/", "url": "https://imteyazahmad.com/elasticsearch-pre-indexing-transformation-using-ingest-pipelines/", "title": "Elasticsearch: A Short Introduction to Ingest Pipelines", "summary": "While indexing data we need to apply some changes that are common…", "content_html": "\n\n While indexing data we need to apply some changes that are common to all the documents. We can achieve that with some custom logic while indexing the documents. However, that can take more efforts and can be error prone. Elasticsearch also provide an inbuilt functionality called Ingest Pipelines to achieve some of the transformation.\n
\n Ingest pipelines provide mean to apply some common transformations before indexing the document. e.g If we want to set a field called writeTime in each of the document or converting a field to uppercase.\n
\n The minimum requirement to use ingest pipeline is at least one of the nodes in the cluster must be an ingest node. For more details please follow this.\n
\n The following block shows the structure of an ingest pipeline. The description is optional and then we can define a list of one or more processors. A processor is configurable task. \n
{\n \"description\" : \"This is optional\",\n \"processors\": [\n {\n //processor 1\n },\n {\n //processor 2\n }\n //\n ]\n}\n\n These processors run sequentially. \n
\n\n\n We need to add a field called writeTime in each document of an index in the Elasticsearch. The following pipeline achieves the same:\n
PUT _ingest/pipeline/write_time_pipeline\n{\n \"description\": \"Pipeline to add writeTime in the documents\",\n \"processors\": [\n {\n \"set\": {\n \"field\": \"writeTime\",\n \"value\": \"{{_ingest.timestamp}}\"\n }\n }\n}\n\n Once this is done we can use this pipeline in index request as shown below:\n
\nPOST my-events/_doc?pipeline=write_time_pipeline\n{\n \"reason\": \"foo\"\n}\n If you are using Java High Level Rest Client then we can use as following:\n
\nfinal Map<Sring, Object> source = new HashMap<>();\nsource.put(\"reason\",\"foo\")\nIndexRequest indexRequest = new IndexRequest(\"my-events\");\n// we are setting the pipeline here to the request\nindexRequest.setPipeline(\"write_time_pipeline\");\nindexRequest.source(source);\nrestHighLevelClient.index(indexRequest,RequestOptions.DEFAULT);\n\n Elasticsearch provides a lot of processors.To get a list of available processors we can use node info API as shown below or a detailed list can be found here.\n
\ncurl -X GET \"localhost:9200/_nodes/ingest?filter_path=nodes.*.ingest.processors&pretty\"\n\n I hope this is helpful. If you have any feedback, please feel free to comment.\n
\n\n\n \n
", "image": "https://imteyazahmad.com/media/posts/7/digitization-5140055_1920.jpg", "author": { "name": "Imteyaz Ahmad" }, "tags": [ "ingest pipeline", "Java", "Elasticsearch" ], "date_published": "2021-04-12T09:59:49-04:00", "date_modified": "2021-04-12T10:03:34-04:00" }, { "id": "https://imteyazahmad.com/elasticsearch-bulk-insert-using-java-high-level-rest-client/", "url": "https://imteyazahmad.com/elasticsearch-bulk-insert-using-java-high-level-rest-client/", "title": "Elasticsearch: Bulk Insert usingĀ Java High Level Rest Client", "summary": "In our previous post, we saw how to insert data into Elasticsearch…", "content_html": "\n\n In our previous post, we saw how to insert data into Elasticsearch using the curl command. In this post, we will use Java High Level Rest Client to achieve the same.\n
\n For any use case, data is essential. If we already have some data in Elasticsearch, we can use it. Otherwise, we will have to insert some data into it. In this post, we will see how to bulk insert data using Java High Level Rest Client. The details about instantiating and configuring the client can be found here.\n
\n\n\n We will use the accounts data present here. The following represents a single document in the accounts index in Elasticsearch:\n
{\"account_number\":1,\"balance\":39225,\"firstname\":\"Amber\",\"lastname\":\"Duke\",\"age\":32,\"gender\":\"M\",\"address\":\"880 Holmes Lane\",\"employer\":\"Pyrami\",\"email\":\"amberduke@pyrami.com\",\"city\":\"Brogan\",\"state\":\"IL\"}\n Let's create an Account class that represents a single document as shown above in the index. \n
public class Account { \nprivate static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); \n@JsonProperty(\"account_number\")\nprivate long accountNumber;\n@JsonProperty(\"balance\")\nprivate long balance;\n@JsonProperty(\"firstname\")\nprivate String firstname;\n@JsonProperty(\"lastname\")\nprivate String lastname;\n@JsonProperty(\"age\")\nprivate int age;\n@JsonProperty(\"gender\")\nprivate String gender;\n@JsonProperty(\"address\")\nprivate String address;\n@JsonProperty(\"employer\")\nprivate String employer;\n@JsonProperty(\"email\")\nprivate String email;\n@JsonProperty(\"city\")\nprivate String city;\n@JsonProperty(\"state\")\nprivate String state;\n//Getter and Setter removed for brevity\n\n// This method is used to convert an instance of Account class to map. \n// This is done because IndexRequest accepts a map as a source.\npublic static Map<String, Object> getAsMap(final Account account) {\n return OBJECT_MAPPER.convertValue(account, new TypeReference<Map<String, Object>>() {\n });\n } \n}\n\n\n We have one another class called AccountManager which reads the data from the file and writes into the Elasticsearch index:\n
/**\n * The type Account manager.\n */\npublic class AccountManager {\n\n private final RestHighLevelClient restHighLevelClient;\n\n /**\n * Instantiates a new Account manager.\n *\n * @param restHighLevelClient the rest high level client\n */\n public AccountManager(final RestHighLevelClient restHighLevelClient) {\n this.restHighLevelClient = restHighLevelClient;\n }\n\n /**\n * Read data from the file and return as a List<Account>.\n *\n * @param fileName the file name\n * @return the list\n * @throws IOException the io exception\n */\n public List<Account> readAccounts(String fileName) throws IOException {\n final File file = new File(fileName);\n final Account[] accounts = new ObjectMapper().readValue(file, Account[].class);\n return Arrays.asList(accounts);\n }\n\n /**\n * Write accounts data into Elasticsearch\n *\n * @param accounts the accounts\n * @param indexName the index name\n * @return the bulk response\n * @throws IOException the io exception\n */\n public BulkResponse writeAccounts(List<Account> accounts, String indexName) throws IOException {\n final var bulkRequest = new BulkRequest();\n accounts.forEach(account -> {\n final var indexRequest = new IndexRequest(indexName);\n indexRequest.source(Account.getAsMap(account));\n bulkRequest.add(indexRequest);\n });\n return restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);\n }\n}\n\n The important thing to notice here is how we are creating the BulkRequest (Line#39-44). For each Account in the list, we are creating a new IndexRequest with the given index name and then passing the current account data as a map to it, which is then added to the instance of bulk request.\n
\n Once the bulk request is prepared we are calling restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT) to insert data into Elasticsearch.\n
\n The following is a test to write data into Elasticsearch:\n
\n @Test\n public void writeAccountsValidAccountAndIndexNameShouldReturnBulkResponse() throws IOException {\n final URL resource = AccountManager.class.getClassLoader()\n .getResource(\"accounts.json\");\n Assertions.assertThat(resource).isNotNull();\n final List<Account> accounts = accountManager.readAccounts(resource.getFile());\n final var bulkItemResponses = accountManager.writeAccounts(accounts, \"accounts_test\");\n Assertions.assertThat(bulkItemResponses.hasFailures()).isFalse();\n Assertions.assertThat(bulkItemResponses).isNotEmpty().hasSize(1000);\n }\n In the previous post we saw how to configure Elasticsearch Java high level rest client. In this post we are going to discuss how to make search request using the same.\n
\n\n\n Once we have configured our Elasticsearch Java high level rest client, it won't be useful if we don't make it work. Here we are going to use it to make some queries and see the result. We will use some fake data here.\n
\n\n\n We can download data from Elasticsearch repository. If you are using Linux the we can download the data as shown below:\n
\nwget https://github.com/elastic/elasticsearch/blob/master/docs/src/test/resources/accounts.json?raw=true -O accounts.json \n\n Once downloaded, run the below commands to insert data into Elasticsearch \n
\n# Create an index named \"accounts\"\ncurl XPUT http://localhost:9200/accounts\n# Bulk insert data into this index\ncurl -s -H \"Content-Type: application/x-ndjson\" -XPOST http://localhost:9200/accounts/_bulk --data-binary @accounts.json\n# once this is done we can verify the data\ncurl -XGET http://localhost:9200/accounts/_count\n#this should return something like this \n{\"count\":1000,\"_shards\":{\"total\":1,\"successful\":1,\"skipped\":0,\"failed\":0}}\n\n With the data in place we want to know the followings:\n
\n\n\n First let's see the JSON query for the same:\n
\nPOST accounts/_search\n{\"size\":10000,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"balance\":{\"gte\":1000,\"lte\":2000}}}]}},\"sort\":[{\"balance\":{\"order\":\"desc\"}}]}\n Now, we will create the same query using Elasticsearch high level rest client.\n
\n//First we need a bool query with a range filter\nfinal var boolQueryBuilder = QueryBuilders.boolQuery();\n boolQueryBuilder\n .filter(QueryBuilders.rangeQuery(\"balance\")\n .gte(1000).lte(2000)\n .includeLower(true)\n .includeUpper(false));\n\n//After this we provide this to SearchSourceBuilder\nfinal var searchSourceBuilder = new SearchSourceBuilder();\nsearchSourceBuilder.size(1000);\nsearchSourceBuilder.query(boolQueryBuilder);\nsearchSourceBuilder.sort(\"balance\");\n\n//And pass this source builder to search request\nfinal var searchRequest = new SearchRequest();\nsearchRequest.indices(\"accounts\");\nsearchRequest.source(searchSourceBuilder);\n\n\n Once that's done, we need to use the HighLevelRestClient to call Elasticsearch\n
final ElasticsearchConfig elasticsearchConfig = new ElasticsearchConfig();\nfinal RestHighLevelClient restHighLevelClient = elasticsearchConfig.restHighLevelClient();\nfinal var searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);\n\n After this we can parse the result as shown below:\n
\nfinal var hits = searchResponse.getHits().getHits();\nfinal var customers = Arrays.stream(hits)\n .map(SearchHit::getSourceAsMap)\n // Here we are mapping the data to string. you can also map to some object like Account\n .map(stringObjectMap -> \"Name: \" + stringObjectMap.get(\"firstname\") + \" \"\n + stringObjectMap.get(\"lastname\") + \", Balance: \" + stringObjectMap.get(\"balance\"))\n .collect(Collectors.toList());\n A working code example is present in this git repository.\n
\n\n\n \n
\n\n\n \n
\n\n\n \n
", "image": "https://imteyazahmad.com/media/posts/4/hand-1248053_1920-2.jpg", "author": { "name": "Imteyaz Ahmad" }, "tags": [ "high level rest client", "Java", "Elasticsearch", "Analytics" ], "date_published": "2021-04-07T00:33:32-04:00", "date_modified": "2021-04-15T10:40:27-04:00" }, { "id": "https://imteyazahmad.com/configuring-elasticsearch-java-high-level-rest-client-2/", "url": "https://imteyazahmad.com/configuring-elasticsearch-java-high-level-rest-client-2/", "title": "Elasticsearch: Java High Level Rest Client", "summary": "This article describe how to configure the new Elasticsearch Java High level…", "content_html": "\n\n This article describe how to configure the new Elasticsearch Java High level rest client in a Java Application.\n
\n\n\n Elasticsearch 5.0.0 released it first Java REST Client. This client was called low-level client as the Java users had to build the JSON request and parse the results. Later, Elasticsearch 5.6 introduced Java High Level REST Client which accepts request objects and returns response objects for the most important APIs(info, get, index, delete, update, bulk, search, search-scroll, and clear-scroll). For other APIs, we still have to use the low-level client.\n
\n\n\n It depends of the following artifacts and their transitive dependencies:
\n
\n The minimum Java version required is 1.8. \n
\n If you are using maven then it can declared as follows\n
\n<dependency>\n <groupId>org.elasticsearch.client</groupId>\n <artifactId>elasticsearch-rest-high-level-client</artifactId>\n <version>${elasticsearch.version}</version>\n</dependency>\n\n The Gradle users can declare the dependencies as\n
\ndependencies {\n compile \"org.elasticsearch.client:elasticsearch-rest-high-level-client:${elasticsearch.version}\"\n} \n Replace the ${elasticsearch.version} as appropriate.\n
\n The RestHighLevelClient is built on top of the REST low-level client builder. The following code will instantiate a low level RestClientBuilder.\n
RestClientBuilder restClientBuilder = RestClient.builder(\n new HttpHost(\"localhost\", 9200, \"http\"),\n new HttpHost(\"localhost\", 9201, \"http\"));\n If the Elasticsearch is secured with user name and password then we need to provide the credentials using the CredentialsProvider as shown below. After that we can set a callback that allows to modify the http client configuration. \n
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();\ncredentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(\n <elasticSearchUsername>, <elasticSearchPassword>));\nrestClientBuilder.setHttpClientConfigCallback(new HttpClientConfigCallback() {\n @Override\n public HttpAsyncClientBuilder customizeHttpClient(\n HttpAsyncClientBuilder httpClientBuilder) {\n return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);\n }\n });\n/*\nUsing the Java 8 syntax, we can replace the above code like this\nbuilder.setHttpClientConfigCallback(\n httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));\n*/\n After this we can initialize the RestHighLevelClient as follows\n
RestHighLevelClient client = new RestHighLevelClient(restClientBuilder);\n Each APIs in High Level REST Client can be called synchronously or asynchronously. In case of synchronous call it returns a response object. It can throw an IOException if either of the following occurs:\n
SearchRequest searchRequest = new SearchRequest(\"posts\");\nsearchRequest.source(searchSourceBuilder);\nSearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT); \n\n The following shows how to call a method asynchronously. We need to define an ActionListener which define how to handle the response and error if any. The following shows the code example for that:\n
// Asynchronous call example\nActionListener<SearchResponse> listener = new ActionListener<SearchResponse>() {\n @Override\n public void onResponse(SearchResponse searchResponse) {\n //write the logic to parse the search response. \n }\n\n @Override\n public void onFailure(Exception e) {\n //Code to handle the exception goes here.\n }\n};\nclient.searchAsync(searchRequest, RequestOptions.DEFAULT, listener);\n\n\n Once we are done with the processing, the client can closed as follows:\n
\nclient.close();\n A working code example is present in this git repository.\n
\n\n\n The classic TransportClient is deprecated in Elasticsearch 7.0 and will be removed in version 8.0.\n