Elasticsearch使用小结之冷热分离
索引迁移
索引setting中的index.routing.allocation.exclude和index.routing.allocation.include可以用于指定索引分配与哪些节点。同时，这两个配置是可以在index存在的时候修改的，我们可以通过修改这两个配置的方式来迁移索引。
比如：
ES集群存在5个节点，ip分别为：
192.168.1.101，
192.168.1.102，
192.168.1.103，
192.168.1.104，
192.168.1.105
1. 我们先创建一个索引，使其分配在192.168.1.101，192.168.1.102上
PUT test_index
{
"settings":{
"index":{
"routing.allocation.exclude._ip":"192.168.1.101,192.168.1.102"
}
}
}
2. 使用cat shards API查看test_index的分片分布
GET _cat/shards/test_index
返回如下：
test_index 2 p STARTED 0 162b 192.168.1.105 Z1SgiFF
test_index 0 p STARTED 0 162b 192.168.1.103 qEu6eMp
test_index 3 p STARTED 0 162b 192.168.1.104 xlTltSO
test_index 5 p STARTED 0 162b 192.168.1.104 xlTltSO
test_index 4 p STARTED 0 162b 192.168.1.105 Z1SgiFF
test_index 1 p STARTED 0 162b 192.168.1.103 qEu6eMp
3. 修改配置
PUT test_index/_settings
{
"settings":{
"index":{
"routing.allocation.exclude._ip":"192.168.1.103,192.168.1.104,192.168.1.105",
"number_of_shards": 6
}
}
}
4. 查看分片
GET _cat/shards/test_index
test_index 2 p STARTED 0 162b 192.168.1.102 9OHYhSa
test_index 4 p STARTED 0 162b 192.168.1.101 -RdAJHx
test_index 0 p STARTED 0 162b 192.168.1.102 9OHYhSa
test_index 5 p STARTED 0 162b 192.168.1.104 xlTltSO -> 192.168.1.101 -RdAJHx
test_index 3 p STARTED 0 162b 192.168.1.101 -RdAJHx
test_index 5 p STARTED 0 162b 192.168.1.102 9OHYhSa
节点tag
在节点启动时，在192.168.1.101和192.168.1.102上指定了
bin/elasticsearch -Enode.attr.rack=rack1
其余三台指定了
bin/elasticsearch -Enode.attr.rack=rack2
使用 GET _nodes/stats 查看：
{
"_nodes": {
"total": 5,
"successful": 5,
"failed": 0
},
"cluster_name": "elasticsearch",
"nodes": {
"-RdAJHxHTf2kHNyfUwHHBw": {
"timestamp": 1559467108509,
"name": "-RdAJHx",
"transport_address": "192.168.1.101:9200",
"host": "192.168.1.101",
"ip": "192.168.1.101:9200",
"roles": [
"data",
"ingest"
],
"attributes": {
"tag": "rack1"
}
....
},
"9OHYhSaxRgKKu_H0q18KyA": {
"timestamp": 1559467108628,
"name": "9OHYhSaxRgKKu_H0q18KyA",
"transport_address": "192.168.1.102:9200",
"host": "192.168.1.102",
"ip": "192.168.1.102:9200",
"roles": [
"data",
"ingest"
],
"attributes": {
"tag": "rack1"
}
....
},
"qEu6eMp9SEK7mlB9HmjCFA": {
"timestamp": 1559467110507,
"name": "qEu6eMp",
"transport_address": "192.168.1.103:9200",
"host": "192.168.1.103",
"ip": "192.168.1.103:9200",
"roles": [
"data",
"ingest"
],
"attributes": {
"tag": "rack2"
}
....
},
"xlTltSOCQmibG9HICbnQyw": {
"timestamp": 1559467110507,
"name": "xlTltSO",
"transport_address": "192.168.1.104:9200",
"host": "192.168.1.104",
"ip": "192.168.1.104:9200",
"roles": [
"data",
"ingest"
],
"attributes": {
"tag": "rack2"
}
....
},
"Z1SgiFFrQMqrPKKGir1EXg": {
"timestamp": 1559467110507,
"name": "Z1SgiFF",
"transport_address": "192.168.1.105:9200",
"host": "192.168.1.105",
"ip": "192.168.1.105:9200",
"roles": [
"data",
"ingest"
],
"attributes": {
"tag": "rack2"
}
....
}
我们也可以通过指定attributes来控制索引的迁移，这里需要配置index.routing.allocation.exclude.tag或者index.routing.allocation.include.tag
冷热分离
结合上述，我们可以很方便的进行冷热分离，先将节点分类，划分为两块，分别用于存储热数据和冷数据，在新建索引时指定索引的index.routing.allocation.exclude.tag为冷节点(或者index.routing.allocation.include.tag为热节点，效果一样)，然后在索引不再有数据写入时指定index.routing.allocation.exclude.tag为热节点(与前面用的对应, 建议统一使用exclude或者include，两者混用时，由于两者不能覆盖，需要注意置空)
示例：
在上述的ES中，我们将rack1的作为热节点，rack2的作为冷节点。
1. PUT test_index
{
"settings":{
"routing.allocation.exclude.tag":"rack2",
"number_of_shards": 6
}
}
2. 数据写入
3. PUT test_index/_settings
{
"settings":{
"routing.allocation.exclude.tag":"rack1"
}
}