JoyLau's Blog

JoyLau 的技术学习与思考

发表于 更新于 分类于
Docker 删除 状态为Dead 的容器

错误信息

因为一些不正确的操作,导致容器的状态变成了 dead

1
2
3
4
5
6
7
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                      PORTS                                                              NAMES
c21c993c5107        34.0.7.183:5000/joylau/traffic-service:2.1.7          "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service
dfbd1cdb31c2        34.0.7.183:5000/joylau/traffic-service-admin:1.2.1    "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service-admin
8778a28ab120        34.0.7.183:5000/joylau/traffic-service-data:2.0.4     "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service-data
65a3885e08b5        34.0.7.183:5000/joylau/traffic-service-node:1.2.3     "/bin/sh -c './nod..."   2 weeks ago         Dead                                                                                           traffic-service-node
90700440e1df        34.0.7.183:5000/joylau/traffic-service-server:1.2.1   "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service-server

这类的容器删除时会报错

1
2
# docker rm c21c993c5107
Error response from daemon: Driver overlay2 failed to remove root filesystem c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64: remove /var/lib/docker/overlay2/099974dbeef827a3bbd932b7b36502763482ae8df25bd80f61a288b71b0ab810/merged: device or resource busy

解决方式

找到 filesystem 后面的字符串

1
# grep c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64 /proc/*/mountinfo

得到如下输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/proc/28032/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28033/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28034/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28035/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28036/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28037/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28038/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28039/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28040/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28041/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28042/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28043/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28044/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28045/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28046/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28047/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28048/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k

proc 和 mountinfo 中间的数字将其 kill 掉即可

写一个批量处理的脚本列出所有的 pid

1
grep c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64 /proc/*/mountinfo | awk '{print substr($1,7,5)}'

再 kill 掉

1
grep c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64 /proc/*/mountinfo | awk '{print substr($1,7,5)}' | xargs kill -9

print 是awk打印指定内容的主要命令

$0 表示整个当前行
$1 每行第一个字段,每个字段以空格隔开
substr($1,7,5) 每行第一个字段,第7个字符开始,截取5个字符

然后在 docker rm container

完美解决.

发表于 更新于 分类于
记录下 elasticsearch analyzer 和 search_analyzer 的区别

环境

  • elasticsearch 6.4.3

示例

下面一段文字用 ik 进行分词

http://34.0.7.184:9200/_analyze/ POST

1
2
3
4
{
  "analyzer": "ik_smart",
  "text": "关于加快建设合肥地铁七号线的通知说明"
}

分词结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
{
"tokens": [
{
"token": "关于",
"start_offset": 0,
"end_offset": 2,
"type": "CN_WORD",
"position": 0
}
,
{
"token": "加快",
"start_offset": 2,
"end_offset": 4,
"type": "CN_WORD",
"position": 1
}
,
{
"token": "建设",
"start_offset": 4,
"end_offset": 6,
"type": "CN_WORD",
"position": 2
}
,
{
"token": "合肥",
"start_offset": 6,
"end_offset": 8,
"type": "CN_WORD",
"position": 3
}
,
{
"token": "地铁",
"start_offset": 8,
"end_offset": 10,
"type": "CN_WORD",
"position": 4
}
,
{
"token": "七号",
"start_offset": 10,
"end_offset": 12,
"type": "CN_WORD",
"position": 5
}
,
{
"token": "线",
"start_offset": 12,
"end_offset": 13,
"type": "CN_CHAR",
"position": 6
}
,
{
"token": "的",
"start_offset": 13,
"end_offset": 14,
"type": "CN_CHAR",
"position": 7
}
,
{
"token": "通知",
"start_offset": 14,
"end_offset": 16,
"type": "CN_WORD",
"position": 8
}
,
{
"token": "说明",
"start_offset": 16,
"end_offset": 18,
"type": "CN_WORD",
"position": 9
}
]
}

  • 这个时候如果配置的 analyzer 为 ik_smart 或者 analyzer 和 search_analyzer 都为 ik_smart, 则短语中每一个字都能搜到结果,还可以设置高亮信息来着重看一下

  • 如果配置的 analyzer 为 ik search_analyzer 为 standard ,则 通知,说明,七号 这样的词是搜不到的,而 线 和 的 这样的词可以搜到,理解一下

http://34.0.7.184:9200/attachment_libs/_search POST

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "query": {
    "multi_match": {
      "query": "关于",
      "fields": [
        "fileName^1.0"
      ],
      "type": "best_fields",
      "operator": "OR",
      "slop": 0,
      "prefix_length": 0,
      "max_expansions": 50,
      "zero_terms_query": "NONE",
      "auto_generate_synonyms_phrase_query": true,
      "fuzzy_transpositions": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
      "fileName"
    ],
    "excludes": [
      "data"
    ]
  },
  "highlight": {
    "pre_tags": [
      "<span style = 'color:red'>"
    ],
    "post_tags": [
      "</span>"
    ],
    "fields": {
      "*": {}
    }
  }
}

返回的结果为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": [ ]
}
}

而搜索 线 返回的结果为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.2876821,
"hits": [
{
"_index": "attachment_libs",
"_type": "attachment_info",
"_id": "fd45d5be-c314-488a-99d3-041acc015377",
"_score": 0.2876821,
"_source": {
"fileName": "关于加快建设合肥地铁七号线的通知说明"
},
"highlight": {
"fileName": [
"关于加快建设合肥地铁七号<span style = 'color:red'>线</span>的通知说明"
]
}
}
]
}
}

总结

  • 分析器主要有两种情况会被使用，一种是插入文档时，将text类型的字段做分词然后插入倒排索引，第二种就是在查询时，先对要查询的text类型的输入做分词，再去倒排索引搜索
  • 如果想要让 索引 和 查询 时使用不同的分词器，ElasticSearch也是能支持的，只需要在字段上加上search_analyzer参数
    1. 在索引时，只会去看字段有没有定义analyzer，有定义的话就用定义的，没定义就用ES预设的
    2. 在查询时，会先去看字段有没有定义search_analyzer，如果没有定义，就去看有没有analyzer，再没有定义，才会去使用ES预设的

发表于 更新于 分类于
记录下自己使用 docker 搭建 elasticsearch 集群环境的记录

.env

1
2
3
PRIVATE_REPO=34.0.7.183:5000
ES_VERSION=6.4.3
ELASTICSEARCH_CLUSTER_DIR=/Users/joylau/dev/idea-project/dev-app/es-doc-office/elasticsearch-cluster

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
version: '2.2'
services:
  node-0:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-0
    ports:
      - 9200:9200
      - 9300:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-0:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-0:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-0
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-1:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-1
    restart: always
    ports:
      - 9201:9200
      - 9301:9300
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-1:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-1:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-1
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=node-0"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-2:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-2
    ports:
      - 9202:9200
      - 9302:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-2:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-2:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-2
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=master,node-1"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-3:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-3
    ports:
      - 9203:9200
      - 9303:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-3:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-3:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-3
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=master,node-1,node-2"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-4:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-4
    ports:
      - 9204:9200
      - 9304:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-4:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-4:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-4
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=master,node-1,node-3"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
networks:
  esnet:

问题

  1. 挂载的日志和数据文件的权限
  2. vm.max_map_count 数目的设置
  3. mac 环境下注意配置 docker 的内存大小设置

env.init

1
2
3
4
5
6
7
8
9
#!/usr/bin/env bash
mkdir -p /home/liufa/es-data/data/{node-0,node-1,node-2,node-3,node-4} && echo es-data directory created success || echo es-data directory created failure && \
mkdir -p /home/liufa/es-data/logs/{node-0,node-1,node-2,node-3,node-4} && echo es-logs directory created success || echo es-logs directory created failure && \
groupadd elasticsearch && \
useradd elasticsearch -g elasticsearch && \
chown -R elasticsearch:elasticsearch /home/liufa/es-data/* && \
chmod -R 777 /home/liufa/es-data/* && \
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf && \
sysctl -p

发表于 更新于 分类于
springboot 使用 actuator 造成 elasticsearch health check failed

版本环境

  1. spring boot : 2.1.2.RELEASE
  2. spring-data-elasticsearch :3.1.4.RELEASE
  3. elasticsearch: 6.4.3

问题描述

使用 spring data elasticsearch 来连接使用 elasticsearch, 配置如下:

1
2
3
4
5
spring:
  data:
    elasticsearch:
      cluster-name: docker-cluster
      cluster-nodes: 192.168.10.68:9300

已经确认 elasticsearch 的 9300 和 9200 端口无任何问题,均可进行连接

可是在启动项目是报出如下错误:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
2019-01-16 17:17:35.376  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : no modules loaded
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019-01-16 17:17:36.045  INFO 36410 --- [           main] o.s.d.e.c.TransportClientFactoryBean     : Adding transport node : 192.168.10.68:9300
2019-01-16 17:17:36.740  INFO 36410 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-01-16 17:17:36.987  INFO 36410 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 15 endpoint(s) beneath base path '/actuator'
2019-01-16 17:17:37.041  INFO 36410 --- [           main] org.xnio                                 : XNIO version 3.3.8.Final
2019-01-16 17:17:37.049  INFO 36410 --- [           main] org.xnio.nio                             : XNIO NIO Implementation Version 3.3.8.Final
2019-01-16 17:17:37.091  INFO 36410 --- [           main] o.s.b.w.e.u.UndertowServletWebServer     : Undertow started on port(s) 8080 (http) with context path ''
2019-01-16 17:17:37.094  INFO 36410 --- [           main] cn.joylau.code.EsDocOfficeApplication    : Started EsDocOfficeApplication in 3.517 seconds (JVM running for 4.124)
2019-01-16 17:17:37.641  INFO 36410 --- [on(4)-127.0.0.1] io.undertow.servlet                      : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-01-16 17:17:37.641  INFO 36410 --- [on(4)-127.0.0.1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2019-01-16 17:17:37.660  INFO 36410 --- [on(4)-127.0.0.1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 19 ms
2019-01-16 17:17:37.704  WARN 36410 --- [on(5)-127.0.0.1] s.b.a.e.ElasticsearchRestHealthIndicator : Elasticsearch health check failed

java.net.ConnectException: Connection refused
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:943) ~[elasticsearch-rest-client-6.4.3.jar:6.4.3]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227) ~[elasticsearch-rest-client-6.4.3.jar:6.4.3]
	at org.springframework.boot.actuate.elasticsearch.ElasticsearchRestHealthIndicator.doHealthCheck(ElasticsearchRestHealthIndicator.java:61) ~[spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84) ~[spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:98) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:246) [spring-core-5.1.4.RELEASE.jar:5.1.4.RELEASE]
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:126) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:99) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [na:1.8.0_131]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) [na:1.8.0_131]
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) [na:1.8.0_131]
	at sun.rmi.transport.Transport$1.run(Transport.java:200) [na:1.8.0_131]
	at sun.rmi.transport.Transport$1.run(Transport.java:197) [na:1.8.0_131]
	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [na:1.8.0_131]
	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_131]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_131]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171) ~[httpcore-nio-4.4.10.jar:4.4.10]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145) ~[httpcore-nio-4.4.10.jar:4.4.10]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[httpcore-nio-4.4.10.jar:4.4.10]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar:4.1.4]
	... 1 common frames omitted


连接被拒绝???

发现无法进行 elasticsearch 的健康检查,于是想到我使用了 actuator 进行端点健康监控

经过调试发现如下代码为返回数据:
ElasticsearchRestHealthIndicator 类中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@Override
protected void doHealthCheck(Health.Builder builder) throws Exception {
	Response response = this.client
			.performRequest(new Request("GET", "/_cluster/health/"));
	StatusLine statusLine = response.getStatusLine();
	if (statusLine.getStatusCode() != HttpStatus.SC_OK) {
		builder.down();
		builder.withDetail("statusCode", statusLine.getStatusCode());
		builder.withDetail("reasonPhrase", statusLine.getReasonPhrase());
		return;
	}
	try (InputStream inputStream = response.getEntity().getContent()) {
		doHealthCheck(builder,
				StreamUtils.copyToString(inputStream, StandardCharsets.UTF_8));
	}
}

new Request("GET", "/_cluster/health/") 正是 elasticsearch 健康的请求,但是没有看到 host 和 port

于是用抓包工具发现其请求的是 127.0.0.1:9200

那这肯定是 springboot 的默认配置了

问题解决

查看 spring-boot-autoconfigure-2.1.2.RELEASE.jar
找到 elasticsearch 的配置 org.springframework.boot.autoconfigure.elasticsearch
在找到类 RestClientProperties
看到如下源码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
@ConfigurationProperties(prefix = "spring.elasticsearch.rest")
public class RestClientProperties {

	/**
	 * Comma-separated list of the Elasticsearch instances to use.
	 */
	private List<String> uris = new ArrayList<>(
			Collections.singletonList("http://localhost:9200"));

	/**
	 * Credentials username.
	 */
	private String username;

	/**
	 * Credentials password.
	 */
	private String password;

	public List<String> getUris() {
		return this.uris;
	}

	public void setUris(List<String> uris) {
		this.uris = uris;
	}

	public String getUsername() {
		return this.username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

	public String getPassword() {
		return this.password;
	}

	public void setPassword(String password) {
		this.password = password;
	}

}

Collections.singletonList("http://localhost:9200")); 没错了,这就是错误的起因

顺藤摸瓜, 根据 spring.elasticsearch.rest 的配置,配置好 uris 即可

于是进行如下配置:

1
2
3
4
5
6
7
8
spring:
  data:
    elasticsearch:
      cluster-name: docker-cluster
      cluster-nodes: 192.168.10.68:9300
  elasticsearch:
    rest:
      uris: ["http://192.168.10.68:9200"]

集群中的多个节点就写多个

启动,没有出现错误

还有一种方式也可以解决,但是并不是一种好的解决方式,那就是关闭 actuator 对 elasticsearch 的健康检查

1
2
3
4
management:
  health:
    elasticsearch:
      enabled: false

发表于 更新于 分类于
用 gradle 构建的基于 springboot 的 elastic-job 的项目发现始终连接不上 zookeeper, 一顿研究后,发现事情并不简单.....
  1. 按照官网的说法, gradle 的配置如下:
1
2
3
compile ('com.dangdang:elastic-job-lite-core:2.1.5')
    
compile ('com.dangdang:elastic-job-lite-spring:2.1.5')
  1. 这样配置后,写好示例代码,发现始终连接不上 zookeeper,抛出以下错误:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call the method org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable; but it does not exist. Its class, org.apache.curator.framework.api.CreateBuilder, is available from the following locations:

    jar:file:/Users/joylau/.gradle/caches/modules-2/files-2.1/org.apache.curator/curator-framework/4.0.1/3da85d2bda41cb43dc18c089820b67d12ba38826/curator-framework-4.0.1.jar!/org/apache/curator/framework/api/CreateBuilder.class

It was loaded from the following location:

    file:/Users/joylau/.gradle/caches/modules-2/files-2.1/org.apache.curator/curator-framework/4.0.1/3da85d2bda41cb43dc18c089820b67d12ba38826/curator-framework-4.0.1.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of org.apache.curator.framework.api.CreateBuilder

  1. 一开始我以为是搭建的 zookeeper 环境有问题,但是用其他工具可以连接的上

  2. 又怀疑是 zookeeper 的版本问题,查看了 com.dangdang:elastic-job-common-core:2.1.5 , 发现其依赖的 zookeeper 版本是 org.apache.zookeeper:zookeeper:3.5.3-beta

  3. 于是又用 docker 搭建了个 3.5.3-beta 的版本的 zookeeper 单机版

  4. 结果问题依旧…….

  5. 中间查找问题花费了很长的时间…..

  6. 后来把官方的 demo clone 到本地跑次看看,官方的 demo 仅仅依赖一个包 com.dangdang:elastic-job-lite-core:2.1.5

  7. 发现这个 demo 没有问题,可以连接的上 zookeeper

  8. 对比发现2个项目的依赖版本号不一致

对比图

  1. 看到 demo 里依赖的 org.apache.curator:curator-framework 和 org.apache.curator:curator-recipes 都是 2.10.0, 而我引入的版本却是gradle 上的最新版 4.0.1, 而且也能看到2者的 zookeeper 的版本也不一致,一个是 3.4.6,一个是 3.5.3-beta

  2. 问题所在找到了

  3. 解决问题

1
2
3
4
5
6
7
compile ('com.dangdang:elastic-job-lite-core:2.1.5')

compile ('com.dangdang:elastic-job-lite-spring:2.1.5')

compile ('org.apache.curator:curator-framework:2.10.0')

compile ('org.apache.curator:curator-recipes:2.10.0')

  1. 手动声明版本为 2.10.0

  2. 问题解决,但是为什么 gradle 会造成这样的问题? 为什么传递依赖时, gradle 会去找最新的依赖版本? 这些问题我还没搞清楚….

  3. 日后搞清楚了,或者有眉目了,再来更新这篇文章.

发表于 更新于 分类于
java 8 lamda 表达式的 stream 有很多实用的方法，这里记录下日常的使用记录

LocalDateTime 将 long 格式的时间转化本地时间字符串

1
2
3
LocalDateTime
        .ofEpochSecond(System.currentTimeMillis() / 1000, 0, ZoneOffset.ofHours(8))
        .format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"))

reduce 导致的源集合对象改变

例如下属代码导致 images 里的 DataImage 对象里的 stake 对象的数量改变

1
2
3
4
5
6
7
8
9
10
Map<String,List<HighwayStake>> roadStakeMap = images.stream()
               .filter(image -> !image.getStakes().isEmpty())
               .map(DataImage::getStakes())
               .reduce((highwayStakes, highwayStakes2) -> {
                   highwayStakes2.addAll(highwayStakes);
                   return highwayStakes2;
               })
               .orElse(new ArrayList<>())
               .stream()
               .collect(Collectors.groupingBy(HighwayStake::getDlmc));

因为对 dataImage 的 stakes 集合进行了合并,将 map 操作改为 复制一个新的 list , 而不是操作原来的 stakes

1
2
3
4
5
6
7
8
9
10
Map<String,List<HighwayStake>> roadStakeMap = images.stream()
               .filter(image -> !image.getStakes().isEmpty())
               .map(dataImage -> new ArrayList<>(dataImage.getStakes()))
               .reduce((highwayStakes, highwayStakes2) -> {
                   highwayStakes2.addAll(highwayStakes);
                   return highwayStakes2;
               })
               .orElse(new ArrayList<>())
               .stream()
               .collect(Collectors.groupingBy(HighwayStake::getDlmc));

List 的深度拷贝

上述的问题实际上是一个 list 的拷贝,而且是 浅度复制

new ArrayList<>(list) 和 Collections.copy(dest,src) 都是浅度复制

下面代码是一个靠谱的 深度拷贝, 需要 T 实现序列化接口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/**
 * list 深度复制
 */
public static <T> List<T> deepCopy(List<T> source) {
    ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
    List<T> dest = new ArrayList<>();
    try {
        ObjectOutputStream out = new ObjectOutputStream(byteOut);
        out.writeObject(source);

        ByteArrayInputStream byteIn = new ByteArrayInputStream(byteOut.toByteArray());
        ObjectInputStream in = new ObjectInputStream(byteIn);
        dest = (List<T>) in.readObject();
    } catch (IOException | ClassNotFoundException e) {
        e.printStackTrace();
    }
    return dest;
}

reduce() 使用记录

reduce 有三种方法可以使用:

  • Optional<T> reduce(BinaryOperator<T> accumulator)
  • T reduce(T identity, BinaryOperator<T> accumulator)
  • <U> U reduce(U identity,BiFunction<U, ? super T, U> accumulator,BinaryOperator<U> combiner)

第一种传入二元运算表达式,第二种是带初始值的二元运算表达式,这里着重记录下第三种的使用方式

第三种第一个参数方法的返回值类型,
第二个参数是一个二元运算表达式,这个表达式的第一个参数是方法的返回值,也就是方法的第一个参数,第二个参数是 Stream 里的值
第三个参数也是一个二元运算表达式,表达式的2个参数都是方法返回值的类型,用于对返回值类型的操作

第三个参数在非并发的情况下返回任何类型(甚至是 null)都没有影响,因为在非并发情况下,第三个二元表达式根本不会执行

那么第三个二元表达式用在并发的情况下,在并发的情况下,第二个二元表达式的第一个参数始终是方法的第一个类型,第三个三元表达式用于将不同线程操作的结果汇总起来

map() 和 flatMap()

区别在于, map() 返回自定义对象, 而 flatMap() 返回 Stream 流对象

distinct() 使用记录

最近在 lamda 的 stream 进行 list 去重复的时候，发现没有生效
代码如下：

1
2
3
4
Map<String, Map<String, List<FollowAnalysisPojo>>> maps = allList
            .parallelStream()
            .distinct()
            .collect(Collectors.groupingBy(FollowAnalysisPojo::getMainPlateNum,Collectors.groupingBy(FollowAnalysisPojo::getPlateNum)));

实体类：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
@Data
public class FollowAnalysisPojo {
    /*被跟车牌*/
    private String mainPlateNum;
    /*跟踪车牌*/
    private String plateNum;
    private String vehicleType;
    private String siteName;
    private String directionName;
    /*车主时间*/
    private String passTimeMain;
    /*伴随时间*/
    private String passTimeSub;
    /*跟踪次数*/
    private Integer trackCount;

    /*该条记录被跟踪车占据的行数，用于在前端合并单元格*/
    private Integer mainRowSpan = 0;

    /*该条记录跟踪车占据的行数，用于在前端合并单元格*/
    private Integer rowSpan;

    private String key = UUID.randomUUID().toString();
  }

上面的代码是想做 先对查询出来的数据进行去重复的操作，然后在按照被跟车牌和跟踪车牌进行分组操作
有点需要说明的是 parallelStream() 比我们常用的 stream() 是并行多管操作，速度上更快

然后发现的问题是并没有去重复，当时也在奇怪 distinct() 里并没有任何参数来指定如何使用规则来去重复

正解

重写List中实体类的 equals() 方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Data
public class FollowAnalysisPojo {
    ......
    /**
     * 当车主时间,伴随时间都相同时，则认为是一个对象
     * @param obj 对象
     * @return Boolean
     */
    @Override
    public boolean equals(Object obj) {
        if(!(obj instanceof FollowAnalysisPojo))return false;
        FollowAnalysisPojo followAnalysisPojo = (FollowAnalysisPojo)obj;
        return passTimeMain.equals(followAnalysisPojo.passTimeMain) && passTimeSub.equals(followAnalysisPojo.passTimeSub);
    }
}

这样我们就按照我自定义的规则进行去重复了
运行了一下，发现还是不起作用
debug了一下，发现根本没有执行重写的 equals 方法
原来还需要重写 hashCode() 方法
在 equals() 方法 执行前会先执行 hashCode() 方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@Data
public class FollowAnalysisPojo {
    ......
    /**
     * 重新 equals 方法必须重新 hashCode方法
     * @return int
     */
    @Override
    public int hashCode(){
        int result = passTimeMain.hashCode();
        result = 31 * result + passTimeMain.hashCode();
        return result;
    }
}

这样就可以了。

2018-9-13 更新

如果我们不重写方法，有没有办法按照List中bean的某个属性来去重复呢？答案是有的，利用的是 stream 的 reduce，用一个set 来存放 key,代码如下：

1
2
3
4
5
6
7
8
9
List<JSONObject> result = trails.stream()
            .filter(distinctByKey(VehicleTrail::getPlateNbr))
            .collect(Collectors.toList());


private  <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

2个集合的元素两两组合成一个 n * m 的集合 (笛卡尔积)

1
2
3
4
5
Integer[] xs = new Integer[]{3, 4};
Integer[] ys = new Integer[]{5, 6};

List<Image> images = Arrays.stream(xs).flatMap(x -> Arrays.stream(ys).map(y -> new Image(x,y))).collect(Collectors.toList());

集合合并

比如: List<List> list 将所有的 Demo 合并到一个集合;

  1. reduce
1
2
3
4
5
// 第一种
List<Demo> demos  = list.stream().reduce(new ArrayList<>(),(demo1,demo2) -> {demo1.addAll(demo2); return demo2;});

// 第二种
List<Demo> demos  = list.stream().reduce(new ArrayList<>(),(demo1,demo2) -> Stream.concat(demo1.stream(),demo2.stream()).collect(Collectors.toList()));
  1. flatMap
1
List<Demo> demos = list.stream().flatMap(Collection::stream).collect(Collectors.toList());

发表于 更新于 分类于
docker 进入容器失败,报错如下

无法进入容器

docker exec -it name /bin/sh 失败,
查看容器 inspect 报错信息如下:

1
2
3
pc error: code = 2 desc = oci runtime error: exec failed: 
container_linux.go:247: starting container process caused "process_linux.go:110: 
decoding init error from pipe caused \"read parent: connection reset by peer\""

问题分析

  1. docker 版本为: Docker version 1.13.1, build 07f3374/1.13.1
  2. centos 版本为: CentOS Linux release 7.3.1611 (Core)
  3. 错误原因: 似乎是 docker RPM 软件包的更新时引入的错误。一个临时的解决方法是将所有docker软件包降级到以前的版本（1.13.1-75似乎可以）

降级

1
yum downgrade docker docker-client docker-common

发表于 更新于 分类于
我们在 docker-compose 一条命令就启动我们的多个容器时,需要考虑到容器之间的启动顺序问题.....

背景

我们在 docker-compose 一条命令就启动我们的多个容器时,需要考虑到容器之间的启动顺序问题…..

比如有的服务依赖数据库的启动, service 依赖 eureka 的启动完成

docker compose 里有 depends_on 配置,但是他不能等上一个容器里的服务完全启动完成,才启动下一个容器,这仅仅定义了启动的顺序, 那么这就会导致很多问题的发生

比如应用正在等待数据库就绪,而此时数据库正在初始化数据, 导致无法连接退出等等

官方的做法

地址 : https://docs.docker.com/compose/startup-order/
官方的思路是使用一个脚本,轮询给定的主机和端口，直到它接受 TCP 连接
个人感觉这种方式不是很好

还有几个开源的工具解决方法, 这些是一些小型脚本,和上面的原理类似:

  1. wait-for-it : https://github.com/vishnubob/wait-for-it
  2. dockerize : https://github.com/jwilder/dockerize
  3. wait-for : https://github.com/Eficode/wait-for

这些工具也能解决问题,但有很大的局限性: 需要重新定义 command , 在执行完自己的脚本后在执行容器里的启动脚本

如果不知道容器的启动脚本或者容器的启动脚本很长,并且带有参数,那将非常头疼

查看容器的启动脚本:

1
docker ps --no-trunc --format="table {{.ID}}\t{{.Command}}:"

或者

1
docker inspect container

health 健康检查方法

比如下面的配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
server:
  image: 34.0.7.183:5000/joylau/traffic-service-server:1.2.0
  container_name: traffic-service-server
  ports:
    - 9368:9368
  restart: always
  volumes:
    - /Users/joylau/log/server:/home/liufa/app/server/logs
  environment:
    activeProfile: prod
  hostname: traffic-service-eureka
  healthcheck:
    test: "/bin/netstat -anp | grep 9367"
    interval: 10s
    timeout: 3s
    retries: 1
admin:
  image: 34.0.7.183:5000/joylau/traffic-service-admin:1.2.0
  container_name: traffic-service-admin
  ports:
    - 9335:9335
  restart: always
  volumes:
    - /Users/joylau/log/admin:/home/liufa/app/admin/logs
  environment:
    activeProfile: prod
  depends_on:
    server:
      condition: service_healthy
  hostname: traffic-service-admin
  links:
    - server:traffic-service-eureka

server 使用了健康检查 healthcheck

  • test : 命令,必须是字符串或列表，如果它是一个列表，第一项必须是 NONE，CMD 或 CMD-SHELL ；如果它是一个字符串，则相当于指定CMD-SHELL 后跟该字符串, 例如: test: ["CMD", "curl", "-f", "http://localhost"] 或者 test: ["CMD-SHELL", "curl -f http://localhost || exit 1"] 或者 test: curl -f https://localhost || exit 1
  • interval: 每次执行的时间间隔
  • timeout: 每次执行时的超时时间,超过这个时间,则认为不健康
  • retries: 重试次数,如果 retries 次后都是失败,则认为容器不健康
  • start_period: 启动后等待多次时间再做检查, version 2.3 版本才有

interval, timeout, start_period 格式如下:

1
2
3
4
5
2.5s
10s
1m30s
2h32m
5h34m56s

健康状态返回 0 (health) 1 (unhealth) 2(reserved)

test 命令的通用是 'xxxx && exit 0 || exit 1' , 2 一般不使用

admin depends_on server ,且条件是 service_healthy ,即容器为健康状态,即 9368 端口开启

最后

  1. depends_on 在 2.0 版本就有, healthcheck 在 2.1 版本上才添加,因此上述的写法至少在 docker-compose version: ‘2.1’ 版本中才生效
  2. docker-compose version 3 将不再支持 depends_on 中的 condition 条件
  3. depends_on 在 version 3 中以 docker swarm 模式部署时,将被忽略
0%