Elasticsearch集群数据备份与恢复 Snapshot & Restore

语言: CN / TW / HK

简介

  • Elasticsearch 拥有副本机制来保障集群的高可用,然而无法解决如下情况的数据丢失:
    • 主副本所在机器存储全部损坏。
    • 误删除索引数据。
    • 升级失败,数据无法回滚。
  • 定期对数据做备份,按需恢复可以很好的解决如上问题。
  • Elasticsearch 提供了 Snapshot 和 Restore API 用于对集群数据完成备份与恢复。
  • 数据备份的过程可以简单理解成将本地数据文件同步到远程仓库(repository) 的过程。
  • 支持全量和增量备份。
  • repository 常见有如下类型: fs / S3 / HDFS / Azure / Google Cloud Storage。

实战

1 创建原始索引并导入数据

PUT _template/blog-template
{
  "index_patterns": [
    "blog-*"
  ],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic_templates": [
      {
        "integers": {
          "match": "int_*",
          "mapping": {
            "type": "integer"
          }
        }
      },
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    ],
    "properties": {
      "title": {
        "type": "text"
      }
    }
  }
}

PUT blog-test
POST blog-test/_bulk
{"index":{}}
{"title":"elasticsearch best practice","author_firstname":"tony","author_lastname":"song","tags":"elasticsearch ,logstash","int_age":18,"locale":"zh,en"}
{"index":{}}
{"title":"elastic stack release V5","author_firstname":"shay","author_lastname":"banon","tags":"elk ,elasticsearch,release","int_age":28,"locale":"zhc,en"}
{"index":{}}
{"title":"kibana tutorial","author_firstname":"zen","author_lastname":"wei","tags":"","int_age":38,"locale":"zhc,en"}

GET blog-test
GET blog-test/_search

2 修改elasticsearch配置文件

修改 elasticsearch.yml 配置文件,增加如下配置:

path.repo: ["/home/elastic/backup"]

重启elasticsearch进程,查看创建的repo:

GET _cluster/settings?include_defaults&filter_path=*.path.repo

#输出结果:
{
  "defaults" : {
    "path" : {
      "repo" : [
        "/home/elastic/backup"
      ]
    }
  }
}

3 创建关联 repository

#创建关联 repository
PUT /_snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "/home/elastic/backup/my_fs_backup",
        "compress": true
    }
}

查看创建的关联:

GET _snapshot/my_fs_backup

#输出结果:
{
  "my_fs_backup" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/home/elastic/backup/my_fs_backup"
    }
  }
}

查看在哪个node上创建了关联:

POST _snapshot/my_fs_backup/_verify

#输出结果
{
  "nodes" : {
    "pMrJwVGSQcSgeTZdh61QRw" : {
      "name" : "node1"
    }
  }
}

4 执行 snapshot

  • indices:做快照的索引。
  • wait_for_completion=true:是否等待完成快照后再响应,如果为true会等快照完成后才响应。(默认为false,不等快照完成立即响应)
  • ignore_unavailable: 设置为true时,当创建快照时忽略不存在的索引。
  • include_global_state: 设置为false时,当某个索引所有的主分片不是全部的都可用时,可以完成快照。
#创建snapshot_1对blog-test索引做快照
PUT /_snapshot/my_fs_backup/snapshot_1?wait_for_completion=true
{
  "indices": "blog-test",
  "ignore_unavailable": true,
  "include_global_state": false
}

查看创建的快照:

GET /_snapshot/my_fs_backup/snapshot_*
#GET /_snapshot/my_fs_backup/_all 看所有的

#输出结果:
{
  "snapshots" : [
    {
      "snapshot" : "snapshot_1",
      "uuid" : "FEbAt3BiR1SAiBkO7pfoZg",
      "version_id" : 7020199,
      "version" : "7.2.1",
      "indices" : [
        "blog-test"
      ],
      "include_global_state" : false,
      "state" : "SUCCESS",
      "start_time" : "2021-02-06T03:28:40.001Z",
      "start_time_in_millis" : 1612582120001,
      "end_time" : "2021-02-06T03:28:40.213Z",
      "end_time_in_millis" : 1612582120213,
      "duration_in_millis" : 212,
      "failures" : [ ],
      "shards" : {
        "total" : 3,
        "failed" : 0,
        "successful" : 3
      }
    }
  ]
}

查看快照状态:

GET _snapshot/my_fs_backup/snapshot_1/_status

#输出结果:
{
  "snapshots" : [
    {
      "snapshot" : "snapshot_1",
      "repository" : "my_fs_backup",
      "uuid" : "FEbAt3BiR1SAiBkO7pfoZg",
      "state" : "SUCCESS",
      "include_global_state" : false,
      "shards_stats" : {
        "initializing" : 0,
        "started" : 0,
        "finalizing" : 0,
        "done" : 3,
        "failed" : 0,
        "total" : 3
      },
      "stats" : {
        "incremental" : {
          "file_count" : 12,
          "size_in_bytes" : 15608
        },
        "total" : {
          "file_count" : 12,
          "size_in_bytes" : 15608
        },
        "start_time_in_millis" : 1612582120074,
        "time_in_millis" : 108
      },
      "indices" : {
        "blog-test" : {
          "shards_stats" : {
            "initializing" : 0,
            "started" : 0,
            "finalizing" : 0,
            "done" : 3,
            "failed" : 0,
            "total" : 3
          },
          "stats" : {
            "incremental" : {
              "file_count" : 12,
              "size_in_bytes" : 15608
            },
            "total" : {
              "file_count" : 12,
              "size_in_bytes" : 15608
            },
            "start_time_in_millis" : 1612582120074,
            "time_in_millis" : 108
          },
          "shards" : {
            "0" : {
              "stage" : "DONE",
              "stats" : {
                "incremental" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5104
                },
                "total" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5104
                },
                "start_time_in_millis" : 1612582120143,
                "time_in_millis" : 39
              }
            },
            "1" : {
              "stage" : "DONE",
              "stats" : {
                "incremental" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5265
                },
                "total" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5265
                },
                "start_time_in_millis" : 1612582120074,
                "time_in_millis" : 25
              }
            },
            "2" : {
              "stage" : "DONE",
              "stats" : {
                "incremental" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5239
                },
                "total" : {
                  "file_count" : 4,
                  "size_in_bytes" : 5239
                },
                "start_time_in_millis" : 1612582120113,
                "time_in_millis" : 21
              }
            }
          }
        }
      }
    }
  ]
}

5 恢复 snapshot

  • rename_pattern: 正则匹配原索引名
  • rename_replacement: 将匹配到的字段用于重命名新索引
POST _snapshot/my_fs_backup/snapshot_1/_restore
{
  "indices": "blog-test",
  "ignore_unavailable": true,
  "rename_pattern": "(.+)",
  "rename_replacement": "restored_$1"
}

查询新索引的数据:

GET restored_blog-test/_search
分享到: