How to calculate disk space needed for each elasticsearch node in a cluster

Vote count:

0

I am planning a Elasticsearch cluster with around 5-6 node for estimated 250 million records. The database is around 100gb large in CSV format. I would like to test this setup on Azure with the Basic D12 VMs that have 4 cores, 28GB of ram and exactly a 100GB SSD to see how mutch I can improve the performance of some queries. Before I really start evaluation ES, I would like to unterstand a couple of things.

On this site I found the following calculation for 1 shard:

[...] if you have X GB of documents [...]:

• Elasticsearch = (30% X) x number of replicas, + 30%X for backup.

I deleted the other points on the list, because they are not directly related to ES.

And here I found this benchmark:

Conclusions for today: Use both standard compression as term vector compression: they improve index throughput and reduce storage by about 50% in our use case. For archiving your index, close it, (g)zip it, and move it. If you want to open it again: download it, un(g)zip it and open it.

Here comes the questions:

What does really happen with the data on the disk in ES?

As far as I understand how this works, they split the data in four portions in a index with 4 shards. So there would be 2 normal and 2 replica shards with each 50GB. So a single node would (simplified) need a SSD with around 200GB for 100GB of data (uncompressed). Is this correct?

Now, if I have 5 nodes, the 100GB get split in 5 pieces. So each node would store around 40gb of data with 2 normal shards and 2 replica shards.

So basically the calculation for each node would be something like:

NodeDisk = (((TotalData/NumberOfNodes)/(NormalShards))*(NormalShards+ReplicaShards))*CompressionRate

Is this correct?

The second point is: How to index them? I first thought to run first a node, index all the data, and then let ES do the rest of the work when I attach the other nodes to the cluster.

Or should I just create the 5 nodes and start indexing then?

My biggest problem is that it is quite expensive to run so many VMs just to do some testing. So I would like to be sure that I am planning in the right direction without getting a bill of X hundred dollars just because I choose the wrong VMs and did not understand correctly how ES really works. And the only VMs with SSDs larger then 100GB are very expensive.

asked 27 secs ago

Teko

328

How to calculate disk space needed for each elasticsearch node in a cluster