JoyLau's Blog

JoyLau 的技术学习与思考

发表于 更新于 分类于
记录下 docker swarm 集群搭建的过程

环境

  1. docker 18.09

说明

  1. 本篇文章中的搭建过程有多台物理机,如果说是自己测试使用的话,或者只有一台机器,可以使用 docker-machine 来创建多个 docker 主机
  2. 比如创建一个主机名为 work 的 docker 主机 : docker-machine create -d virtualbox worker
  3. 之后进入刚才创建的主机 : docker-machine ssh worker
  4. 然后就当成是一台独立机器来执行以下的操作

步骤

  1. 初始化 swarm 集群 : docker swarm init --advertise-addr 34.0.7.183
    1. 机器有多个网卡的指定 IP 地址 –advertise-addr
    2. 默认创建的是管理节点
  2. 加入刚才创建 swarm 集群
1
docker swarm join --token SWMTKN-1-1o1yfsquxasw7c7ah4t7lmd4i89i62u74tutzhtcbgb7wx6csc-1hf4tjv9oz9vpo937955mi0z2 34.0.7.183:2377

如果说忘了集群管理节点的 token, 可以使用 docker swarm join-token work/manage 来查看加入该集群的命令

  1. 查看集群节点: docker node list

服务部署

  1. 单服务部署 docker service create --name nginx -p 80:80 --replaces 4 containAddress
    上述命令部署了4个 nginx 服务,如果集群有2台主机的话,会在每台主机上部署 2 个服务

  2. 多服务部署, 使用 yml 配置文件,具体语法参看 https://docs.docker.com/compose/compose-file/

命令

docker swarm

docker swarm init 初始化集群
docker swarm join-token worker 查看工作节点的 token
docker swarm join-token manager 查看管理节点的 token
docker swarm join 加入集群中

docker stack

docker stack deploy 部署新的服务或更新现有服务
docker stack ls 列出现有服务
docker stack ps 列出服务中的任务
docker stack rm 删除服务
docker stack services 列出服务中的具体项
docker stack down 移除某个服务（不会删除数据）

docker node

docker node ls 查看所有集群节点
docker node rm 删除某个节点（-f强制删除）
docker node inspect 查看节点详情
docker node demote 节点降级，由管理节点降级为工作节点
docker node promote 节点升级，由工作节点升级为管理节点
docker node update 更新节点
docker node ps 查看节点中的 Task 任务

docker service

docker service create 部署服务
docker service inspect 查看服务详情
docker service logs 产看某个服务日志
docker service ls 查看所有服务详情
docker service rm 删除某个服务（-f强制删除）
docker service scale 设置某个服务个数
docker service update 更新某个服务

docker machine

docker-machine create 创建一个 Docker 主机（常用-d virtualbox）
docker-machine ls 查看所有的 Docker 主机
docker-machine ssh SSH 到主机上执行命令
docker-machine env 显示连接到某个主机需要的环境变量
docker-machine inspect 输出主机更多信息
docker-machine kill 停止某个主机
docker-machine restart 重启某台主机
docker-machine rm 删除某台主机
docker-machine scp 在主机之间复制文件
docker-machine start 启动一个主机
docker-machine status 查看主机状态
docker-machine stop 停止一个主机

swarm 集群节点可视化工具

portainer : 很强大的工具,可以监控本机和远程服务器或者集群环境,远程 docker 主机的话需要远程 docker 主机开启在 2375 端口的服务

https://www.portainer.io/installation/

1
2
3
4
5
6
7
8
9
10
version: '3'
services:
  portainer:
    image: 34.0.7.183:5000/joylau/portainer:latest
    container_name: portainer
    ports:
      - 80:9000
    restart: always
    volumes:
      - /home/liufa/portainer/data:/data

发表于 更新于 分类于
记录下 Spring Boot 中 @Validated 注解的 groups 使用记录

@Valid 和 @Validated

  1. @Valid 和 @Validated 注解都用于字段校验
  2. @Valid 所属包为：javax.validation.Valid ; @Validated 所属包为 org.springframework.validation.annotation.Validated
  3. @Validated 是 @Valid 的一次封装，是Spring提供的校验机制使用。@Valid 不提供分组功能

@Validated的特殊用法

当一个实体类需要多种验证方式时，例：对于一个实体类的id来说，新增的时候是不需要的，对于更新时是必须的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class Attachment {
    @Id
    @NotBlank(message = "id can not be blank!", groups = {All.class, Update.class})
    private String id;

    @NotBlank(message = "fileName can not be blank!", groups = {All.class})
    private String fileName;

    @NotBlank(message = "filePath can not be blank!", groups = {All.class})
    private String filePath;

    @Field
    private byte[] data;

    @NotBlank(message = "metaData can not be empty!", groups = {All.class})
    private String metaData;

    @NotBlank(message = "uploadTime can not be blank!", groups = {All.class})
    private String uploadTime;

    public Attachment(@NotBlank(message = "id can not be blank!", groups = {All.class, Update.class}) String id) {
        this.id = id;
    }

    public interface All {
    }

    public interface Update {
    }
}

单独对 groups 进行校验

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/**
 * 添加附件
 */
@PostMapping("addAttachment")
public MessageBody addAttachment(@RequestParam("file") final MultipartFile multipartFile,
                                 @Validated(Attachment.All.class) Attachment attachment,
                                 BindingResult results){
    return attachmentApiService.addAttachment(multipartFile,attachment,results);
}

/**
 * 更新单个附件
 */
@PostMapping("updateAttachment")
public MessageBody updateAttachment(@RequestParam(value = "file",required = false) final MultipartFile multipartFile,
                                    @Validated(Attachment.Update.class) Attachment attachment){
    return attachmentApiService.updateAttachment(multipartFile,attachment);
}

使用注意

  1. 校验的注解中不分配 groups，默认每次都要进行验证
  2. @Validated 没有添加 groups 属性时，默认验证没有分组的验证属性
  3. @Validated 添加特定 groups 属性时,只校验该注解中分配了该 groups 的属性
  4. 一个功能方法上处理多个模型对象时，需添加多个验证结果对象,如下所示
1
2
3
@RequestMapping("/addPeople")  
public @ResponseBody String addPeople(@Validated People p,BindingResult result,@Validated Person p2,BindingResult result2)  {
}

发表于 更新于 分类于
Docker 删除 状态为Dead 的容器

错误信息

因为一些不正确的操作,导致容器的状态变成了 dead

1
2
3
4
5
6
7
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                      PORTS                                                              NAMES
c21c993c5107        34.0.7.183:5000/joylau/traffic-service:2.1.7          "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service
dfbd1cdb31c2        34.0.7.183:5000/joylau/traffic-service-admin:1.2.1    "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service-admin
8778a28ab120        34.0.7.183:5000/joylau/traffic-service-data:2.0.4     "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service-data
65a3885e08b5        34.0.7.183:5000/joylau/traffic-service-node:1.2.3     "/bin/sh -c './nod..."   2 weeks ago         Dead                                                                                           traffic-service-node
90700440e1df        34.0.7.183:5000/joylau/traffic-service-server:1.2.1   "java -Djava.secur..."   2 weeks ago         Dead                                                                                           traffic-service-server

这类的容器删除时会报错

1
2
# docker rm c21c993c5107
Error response from daemon: Driver overlay2 failed to remove root filesystem c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64: remove /var/lib/docker/overlay2/099974dbeef827a3bbd932b7b36502763482ae8df25bd80f61a288b71b0ab810/merged: device or resource busy

解决方式

找到 filesystem 后面的字符串

1
# grep c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64 /proc/*/mountinfo

得到如下输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/proc/28032/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28033/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28034/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28035/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28036/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28037/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28038/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28039/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28040/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28041/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28042/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28043/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28044/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28045/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28046/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28047/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k
/proc/28048/mountinfo:973 957 0:164 / /var/lib/docker/containers/c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64/shm rw,nosuid,nodev,noexec,relatime shared:189 - tmpfs shm rw,size=65536k

proc 和 mountinfo 中间的数字将其 kill 掉即可

写一个批量处理的脚本列出所有的 pid

1
grep c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64 /proc/*/mountinfo | awk '{print substr($1,7,5)}'

再 kill 掉

1
grep c21c993c51073f41653aa7fd37dbfd232f8439ca79fd4315a410d0b41d8b0e64 /proc/*/mountinfo | awk '{print substr($1,7,5)}' | xargs kill -9

print 是awk打印指定内容的主要命令

$0 表示整个当前行
$1 每行第一个字段,每个字段以空格隔开
substr($1,7,5) 每行第一个字段,第7个字符开始,截取5个字符

然后在 docker rm container

完美解决.

发表于 更新于 分类于
记录下 elasticsearch analyzer 和 search_analyzer 的区别

环境

  • elasticsearch 6.4.3

示例

下面一段文字用 ik 进行分词

http://34.0.7.184:9200/_analyze/ POST

1
2
3
4
{
  "analyzer": "ik_smart",
  "text": "关于加快建设合肥地铁七号线的通知说明"
}

分词结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
{
"tokens": [
{
"token": "关于",
"start_offset": 0,
"end_offset": 2,
"type": "CN_WORD",
"position": 0
}
,
{
"token": "加快",
"start_offset": 2,
"end_offset": 4,
"type": "CN_WORD",
"position": 1
}
,
{
"token": "建设",
"start_offset": 4,
"end_offset": 6,
"type": "CN_WORD",
"position": 2
}
,
{
"token": "合肥",
"start_offset": 6,
"end_offset": 8,
"type": "CN_WORD",
"position": 3
}
,
{
"token": "地铁",
"start_offset": 8,
"end_offset": 10,
"type": "CN_WORD",
"position": 4
}
,
{
"token": "七号",
"start_offset": 10,
"end_offset": 12,
"type": "CN_WORD",
"position": 5
}
,
{
"token": "线",
"start_offset": 12,
"end_offset": 13,
"type": "CN_CHAR",
"position": 6
}
,
{
"token": "的",
"start_offset": 13,
"end_offset": 14,
"type": "CN_CHAR",
"position": 7
}
,
{
"token": "通知",
"start_offset": 14,
"end_offset": 16,
"type": "CN_WORD",
"position": 8
}
,
{
"token": "说明",
"start_offset": 16,
"end_offset": 18,
"type": "CN_WORD",
"position": 9
}
]
}

  • 这个时候如果配置的 analyzer 为 ik_smart 或者 analyzer 和 search_analyzer 都为 ik_smart, 则短语中每一个字都能搜到结果,还可以设置高亮信息来着重看一下

  • 如果配置的 analyzer 为 ik search_analyzer 为 standard ,则 通知,说明,七号 这样的词是搜不到的,而 线 和 的 这样的词可以搜到,理解一下

http://34.0.7.184:9200/attachment_libs/_search POST

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "query": {
    "multi_match": {
      "query": "关于",
      "fields": [
        "fileName^1.0"
      ],
      "type": "best_fields",
      "operator": "OR",
      "slop": 0,
      "prefix_length": 0,
      "max_expansions": 50,
      "zero_terms_query": "NONE",
      "auto_generate_synonyms_phrase_query": true,
      "fuzzy_transpositions": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
      "fileName"
    ],
    "excludes": [
      "data"
    ]
  },
  "highlight": {
    "pre_tags": [
      "<span style = 'color:red'>"
    ],
    "post_tags": [
      "</span>"
    ],
    "fields": {
      "*": {}
    }
  }
}

返回的结果为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": [ ]
}
}

而搜索 线 返回的结果为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.2876821,
"hits": [
{
"_index": "attachment_libs",
"_type": "attachment_info",
"_id": "fd45d5be-c314-488a-99d3-041acc015377",
"_score": 0.2876821,
"_source": {
"fileName": "关于加快建设合肥地铁七号线的通知说明"
},
"highlight": {
"fileName": [
"关于加快建设合肥地铁七号<span style = 'color:red'>线</span>的通知说明"
]
}
}
]
}
}

总结

  • 分析器主要有两种情况会被使用，一种是插入文档时，将text类型的字段做分词然后插入倒排索引，第二种就是在查询时，先对要查询的text类型的输入做分词，再去倒排索引搜索
  • 如果想要让 索引 和 查询 时使用不同的分词器，ElasticSearch也是能支持的，只需要在字段上加上search_analyzer参数
    1. 在索引时，只会去看字段有没有定义analyzer，有定义的话就用定义的，没定义就用ES预设的
    2. 在查询时，会先去看字段有没有定义search_analyzer，如果没有定义，就去看有没有analyzer，再没有定义，才会去使用ES预设的

发表于 更新于 分类于
记录下自己使用 docker 搭建 elasticsearch 集群环境的记录

.env

1
2
3
PRIVATE_REPO=34.0.7.183:5000
ES_VERSION=6.4.3
ELASTICSEARCH_CLUSTER_DIR=/Users/joylau/dev/idea-project/dev-app/es-doc-office/elasticsearch-cluster

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
version: '2.2'
services:
  node-0:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-0
    ports:
      - 9200:9200
      - 9300:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-0:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-0:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-0
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-1:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-1
    restart: always
    ports:
      - 9201:9200
      - 9301:9300
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-1:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-1:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-1
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=node-0"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-2:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-2
    ports:
      - 9202:9200
      - 9302:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-2:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-2:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-2
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=master,node-1"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-3:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-3
    ports:
      - 9203:9200
      - 9303:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-3:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-3:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-3
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=master,node-1,node-2"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
  node-4:
    image: ${PRIVATE_REPO}/joylau/es-doc:${ES_VERSION}
    container_name: node-4
    ports:
      - 9204:9200
      - 9304:9300
    restart: always
    volumes:
      - ${ELASTICSEARCH_CLUSTER_DIR}/data/node-4:/usr/share/elasticsearch/data
      - ${ELASTICSEARCH_CLUSTER_DIR}/logs/node-4:/usr/share/elasticsearch/logs
    environment:
      - bootstrap.memory_lock=true
      - cluster.name=es-doc-office
      - node.name=node-4
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
      - "discovery.zen.ping.unicast.hosts=master,node-1,node-3"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - esnet
networks:
  esnet:

问题

  1. 挂载的日志和数据文件的权限
  2. vm.max_map_count 数目的设置
  3. mac 环境下注意配置 docker 的内存大小设置

env.init

1
2
3
4
5
6
7
8
9
#!/usr/bin/env bash
mkdir -p /home/liufa/es-data/data/{node-0,node-1,node-2,node-3,node-4} && echo es-data directory created success || echo es-data directory created failure && \
mkdir -p /home/liufa/es-data/logs/{node-0,node-1,node-2,node-3,node-4} && echo es-logs directory created success || echo es-logs directory created failure && \
groupadd elasticsearch && \
useradd elasticsearch -g elasticsearch && \
chown -R elasticsearch:elasticsearch /home/liufa/es-data/* && \
chmod -R 777 /home/liufa/es-data/* && \
echo 'vm.max_map_count=262144' >> /etc/sysctl.conf && \
sysctl -p

发表于 更新于 分类于
springboot 使用 actuator 造成 elasticsearch health check failed

版本环境

  1. spring boot : 2.1.2.RELEASE
  2. spring-data-elasticsearch :3.1.4.RELEASE
  3. elasticsearch: 6.4.3

问题描述

使用 spring data elasticsearch 来连接使用 elasticsearch, 配置如下:

1
2
3
4
5
spring:
  data:
    elasticsearch:
      cluster-name: docker-cluster
      cluster-nodes: 192.168.10.68:9300

已经确认 elasticsearch 的 9300 和 9200 端口无任何问题,均可进行连接

可是在启动项目是报出如下错误:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
2019-01-16 17:17:35.376  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : no modules loaded
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2019-01-16 17:17:35.378  INFO 36410 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019-01-16 17:17:36.045  INFO 36410 --- [           main] o.s.d.e.c.TransportClientFactoryBean     : Adding transport node : 192.168.10.68:9300
2019-01-16 17:17:36.740  INFO 36410 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-01-16 17:17:36.987  INFO 36410 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 15 endpoint(s) beneath base path '/actuator'
2019-01-16 17:17:37.041  INFO 36410 --- [           main] org.xnio                                 : XNIO version 3.3.8.Final
2019-01-16 17:17:37.049  INFO 36410 --- [           main] org.xnio.nio                             : XNIO NIO Implementation Version 3.3.8.Final
2019-01-16 17:17:37.091  INFO 36410 --- [           main] o.s.b.w.e.u.UndertowServletWebServer     : Undertow started on port(s) 8080 (http) with context path ''
2019-01-16 17:17:37.094  INFO 36410 --- [           main] cn.joylau.code.EsDocOfficeApplication    : Started EsDocOfficeApplication in 3.517 seconds (JVM running for 4.124)
2019-01-16 17:17:37.641  INFO 36410 --- [on(4)-127.0.0.1] io.undertow.servlet                      : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-01-16 17:17:37.641  INFO 36410 --- [on(4)-127.0.0.1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2019-01-16 17:17:37.660  INFO 36410 --- [on(4)-127.0.0.1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 19 ms
2019-01-16 17:17:37.704  WARN 36410 --- [on(5)-127.0.0.1] s.b.a.e.ElasticsearchRestHealthIndicator : Elasticsearch health check failed

java.net.ConnectException: Connection refused
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:943) ~[elasticsearch-rest-client-6.4.3.jar:6.4.3]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227) ~[elasticsearch-rest-client-6.4.3.jar:6.4.3]
	at org.springframework.boot.actuate.elasticsearch.ElasticsearchRestHealthIndicator.doHealthCheck(ElasticsearchRestHealthIndicator.java:61) ~[spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84) ~[spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:98) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.health.HealthEndpoint.health(HealthEndpoint.java:50) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:246) [spring-core-5.1.4.RELEASE.jar:5.1.4.RELEASE]
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:126) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at org.springframework.boot.actuate.endpoint.jmx.EndpointMBean.invoke(EndpointMBean.java:99) [spring-boot-actuator-2.1.2.RELEASE.jar:2.1.2.RELEASE]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [na:1.8.0_131]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) [na:1.8.0_131]
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) [na:1.8.0_131]
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346) [na:1.8.0_131]
	at sun.rmi.transport.Transport$1.run(Transport.java:200) [na:1.8.0_131]
	at sun.rmi.transport.Transport$1.run(Transport.java:197) [na:1.8.0_131]
	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [na:1.8.0_131]
	at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_131]
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_131]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_131]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171) ~[httpcore-nio-4.4.10.jar:4.4.10]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145) ~[httpcore-nio-4.4.10.jar:4.4.10]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[httpcore-nio-4.4.10.jar:4.4.10]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar:4.1.4]
	... 1 common frames omitted


连接被拒绝???

发现无法进行 elasticsearch 的健康检查,于是想到我使用了 actuator 进行端点健康监控

经过调试发现如下代码为返回数据:
ElasticsearchRestHealthIndicator 类中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@Override
protected void doHealthCheck(Health.Builder builder) throws Exception {
	Response response = this.client
			.performRequest(new Request("GET", "/_cluster/health/"));
	StatusLine statusLine = response.getStatusLine();
	if (statusLine.getStatusCode() != HttpStatus.SC_OK) {
		builder.down();
		builder.withDetail("statusCode", statusLine.getStatusCode());
		builder.withDetail("reasonPhrase", statusLine.getReasonPhrase());
		return;
	}
	try (InputStream inputStream = response.getEntity().getContent()) {
		doHealthCheck(builder,
				StreamUtils.copyToString(inputStream, StandardCharsets.UTF_8));
	}
}

new Request("GET", "/_cluster/health/") 正是 elasticsearch 健康的请求,但是没有看到 host 和 port

于是用抓包工具发现其请求的是 127.0.0.1:9200

那这肯定是 springboot 的默认配置了

问题解决

查看 spring-boot-autoconfigure-2.1.2.RELEASE.jar
找到 elasticsearch 的配置 org.springframework.boot.autoconfigure.elasticsearch
在找到类 RestClientProperties
看到如下源码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
@ConfigurationProperties(prefix = "spring.elasticsearch.rest")
public class RestClientProperties {

	/**
	 * Comma-separated list of the Elasticsearch instances to use.
	 */
	private List<String> uris = new ArrayList<>(
			Collections.singletonList("http://localhost:9200"));

	/**
	 * Credentials username.
	 */
	private String username;

	/**
	 * Credentials password.
	 */
	private String password;

	public List<String> getUris() {
		return this.uris;
	}

	public void setUris(List<String> uris) {
		this.uris = uris;
	}

	public String getUsername() {
		return this.username;
	}

	public void setUsername(String username) {
		this.username = username;
	}

	public String getPassword() {
		return this.password;
	}

	public void setPassword(String password) {
		this.password = password;
	}

}

Collections.singletonList("http://localhost:9200")); 没错了,这就是错误的起因

顺藤摸瓜, 根据 spring.elasticsearch.rest 的配置,配置好 uris 即可

于是进行如下配置:

1
2
3
4
5
6
7
8
spring:
  data:
    elasticsearch:
      cluster-name: docker-cluster
      cluster-nodes: 192.168.10.68:9300
  elasticsearch:
    rest:
      uris: ["http://192.168.10.68:9200"]

集群中的多个节点就写多个

启动,没有出现错误

还有一种方式也可以解决,但是并不是一种好的解决方式,那就是关闭 actuator 对 elasticsearch 的健康检查

1
2
3
4
management:
  health:
    elasticsearch:
      enabled: false

发表于 更新于 分类于
用 gradle 构建的基于 springboot 的 elastic-job 的项目发现始终连接不上 zookeeper, 一顿研究后,发现事情并不简单.....
  1. 按照官网的说法, gradle 的配置如下:
1
2
3
compile ('com.dangdang:elastic-job-lite-core:2.1.5')
    
compile ('com.dangdang:elastic-job-lite-spring:2.1.5')
  1. 这样配置后,写好示例代码,发现始终连接不上 zookeeper,抛出以下错误:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call the method org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable; but it does not exist. Its class, org.apache.curator.framework.api.CreateBuilder, is available from the following locations:

    jar:file:/Users/joylau/.gradle/caches/modules-2/files-2.1/org.apache.curator/curator-framework/4.0.1/3da85d2bda41cb43dc18c089820b67d12ba38826/curator-framework-4.0.1.jar!/org/apache/curator/framework/api/CreateBuilder.class

It was loaded from the following location:

    file:/Users/joylau/.gradle/caches/modules-2/files-2.1/org.apache.curator/curator-framework/4.0.1/3da85d2bda41cb43dc18c089820b67d12ba38826/curator-framework-4.0.1.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of org.apache.curator.framework.api.CreateBuilder

  1. 一开始我以为是搭建的 zookeeper 环境有问题,但是用其他工具可以连接的上

  2. 又怀疑是 zookeeper 的版本问题,查看了 com.dangdang:elastic-job-common-core:2.1.5 , 发现其依赖的 zookeeper 版本是 org.apache.zookeeper:zookeeper:3.5.3-beta

  3. 于是又用 docker 搭建了个 3.5.3-beta 的版本的 zookeeper 单机版

  4. 结果问题依旧…….

  5. 中间查找问题花费了很长的时间…..

  6. 后来把官方的 demo clone 到本地跑次看看,官方的 demo 仅仅依赖一个包 com.dangdang:elastic-job-lite-core:2.1.5

  7. 发现这个 demo 没有问题,可以连接的上 zookeeper

  8. 对比发现2个项目的依赖版本号不一致

对比图

  1. 看到 demo 里依赖的 org.apache.curator:curator-framework 和 org.apache.curator:curator-recipes 都是 2.10.0, 而我引入的版本却是gradle 上的最新版 4.0.1, 而且也能看到2者的 zookeeper 的版本也不一致,一个是 3.4.6,一个是 3.5.3-beta

  2. 问题所在找到了

  3. 解决问题

1
2
3
4
5
6
7
compile ('com.dangdang:elastic-job-lite-core:2.1.5')

compile ('com.dangdang:elastic-job-lite-spring:2.1.5')

compile ('org.apache.curator:curator-framework:2.10.0')

compile ('org.apache.curator:curator-recipes:2.10.0')

  1. 手动声明版本为 2.10.0

  2. 问题解决,但是为什么 gradle 会造成这样的问题? 为什么传递依赖时, gradle 会去找最新的依赖版本? 这些问题我还没搞清楚….

  3. 日后搞清楚了,或者有眉目了,再来更新这篇文章.

发表于 更新于 分类于
java 8 lamda 表达式的 stream 有很多实用的方法，这里记录下日常的使用记录

LocalDateTime 将 long 格式的时间转化本地时间字符串

1
2
3
LocalDateTime
        .ofEpochSecond(System.currentTimeMillis() / 1000, 0, ZoneOffset.ofHours(8))
        .format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"))

reduce 导致的源集合对象改变

例如下属代码导致 images 里的 DataImage 对象里的 stake 对象的数量改变

1
2
3
4
5
6
7
8
9
10
Map<String,List<HighwayStake>> roadStakeMap = images.stream()
               .filter(image -> !image.getStakes().isEmpty())
               .map(DataImage::getStakes())
               .reduce((highwayStakes, highwayStakes2) -> {
                   highwayStakes2.addAll(highwayStakes);
                   return highwayStakes2;
               })
               .orElse(new ArrayList<>())
               .stream()
               .collect(Collectors.groupingBy(HighwayStake::getDlmc));

因为对 dataImage 的 stakes 集合进行了合并,将 map 操作改为 复制一个新的 list , 而不是操作原来的 stakes

1
2
3
4
5
6
7
8
9
10
Map<String,List<HighwayStake>> roadStakeMap = images.stream()
               .filter(image -> !image.getStakes().isEmpty())
               .map(dataImage -> new ArrayList<>(dataImage.getStakes()))
               .reduce((highwayStakes, highwayStakes2) -> {
                   highwayStakes2.addAll(highwayStakes);
                   return highwayStakes2;
               })
               .orElse(new ArrayList<>())
               .stream()
               .collect(Collectors.groupingBy(HighwayStake::getDlmc));

List 的深度拷贝

上述的问题实际上是一个 list 的拷贝,而且是 浅度复制

new ArrayList<>(list) 和 Collections.copy(dest,src) 都是浅度复制

下面代码是一个靠谱的 深度拷贝, 需要 T 实现序列化接口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/**
 * list 深度复制
 */
public static <T> List<T> deepCopy(List<T> source) {
    ByteArrayOutputStream byteOut = new ByteArrayOutputStream();
    List<T> dest = new ArrayList<>();
    try {
        ObjectOutputStream out = new ObjectOutputStream(byteOut);
        out.writeObject(source);

        ByteArrayInputStream byteIn = new ByteArrayInputStream(byteOut.toByteArray());
        ObjectInputStream in = new ObjectInputStream(byteIn);
        dest = (List<T>) in.readObject();
    } catch (IOException | ClassNotFoundException e) {
        e.printStackTrace();
    }
    return dest;
}

reduce() 使用记录

reduce 有三种方法可以使用:

  • Optional<T> reduce(BinaryOperator<T> accumulator)
  • T reduce(T identity, BinaryOperator<T> accumulator)
  • <U> U reduce(U identity,BiFunction<U, ? super T, U> accumulator,BinaryOperator<U> combiner)

第一种传入二元运算表达式,第二种是带初始值的二元运算表达式,这里着重记录下第三种的使用方式

第三种第一个参数方法的返回值类型,
第二个参数是一个二元运算表达式,这个表达式的第一个参数是方法的返回值,也就是方法的第一个参数,第二个参数是 Stream 里的值
第三个参数也是一个二元运算表达式,表达式的2个参数都是方法返回值的类型,用于对返回值类型的操作

第三个参数在非并发的情况下返回任何类型(甚至是 null)都没有影响,因为在非并发情况下,第三个二元表达式根本不会执行

那么第三个二元表达式用在并发的情况下,在并发的情况下,第二个二元表达式的第一个参数始终是方法的第一个类型,第三个三元表达式用于将不同线程操作的结果汇总起来

map() 和 flatMap()

区别在于, map() 返回自定义对象, 而 flatMap() 返回 Stream 流对象

distinct() 使用记录

最近在 lamda 的 stream 进行 list 去重复的时候，发现没有生效
代码如下：

1
2
3
4
Map<String, Map<String, List<FollowAnalysisPojo>>> maps = allList
            .parallelStream()
            .distinct()
            .collect(Collectors.groupingBy(FollowAnalysisPojo::getMainPlateNum,Collectors.groupingBy(FollowAnalysisPojo::getPlateNum)));

实体类：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
@Data
public class FollowAnalysisPojo {
    /*被跟车牌*/
    private String mainPlateNum;
    /*跟踪车牌*/
    private String plateNum;
    private String vehicleType;
    private String siteName;
    private String directionName;
    /*车主时间*/
    private String passTimeMain;
    /*伴随时间*/
    private String passTimeSub;
    /*跟踪次数*/
    private Integer trackCount;

    /*该条记录被跟踪车占据的行数，用于在前端合并单元格*/
    private Integer mainRowSpan = 0;

    /*该条记录跟踪车占据的行数，用于在前端合并单元格*/
    private Integer rowSpan;

    private String key = UUID.randomUUID().toString();
  }

上面的代码是想做 先对查询出来的数据进行去重复的操作，然后在按照被跟车牌和跟踪车牌进行分组操作
有点需要说明的是 parallelStream() 比我们常用的 stream() 是并行多管操作，速度上更快

然后发现的问题是并没有去重复，当时也在奇怪 distinct() 里并没有任何参数来指定如何使用规则来去重复

正解

重写List中实体类的 equals() 方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Data
public class FollowAnalysisPojo {
    ......
    /**
     * 当车主时间,伴随时间都相同时，则认为是一个对象
     * @param obj 对象
     * @return Boolean
     */
    @Override
    public boolean equals(Object obj) {
        if(!(obj instanceof FollowAnalysisPojo))return false;
        FollowAnalysisPojo followAnalysisPojo = (FollowAnalysisPojo)obj;
        return passTimeMain.equals(followAnalysisPojo.passTimeMain) && passTimeSub.equals(followAnalysisPojo.passTimeSub);
    }
}

这样我们就按照我自定义的规则进行去重复了
运行了一下，发现还是不起作用
debug了一下，发现根本没有执行重写的 equals 方法
原来还需要重写 hashCode() 方法
在 equals() 方法 执行前会先执行 hashCode() 方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@Data
public class FollowAnalysisPojo {
    ......
    /**
     * 重新 equals 方法必须重新 hashCode方法
     * @return int
     */
    @Override
    public int hashCode(){
        int result = passTimeMain.hashCode();
        result = 31 * result + passTimeMain.hashCode();
        return result;
    }
}

这样就可以了。

2018-9-13 更新

如果我们不重写方法，有没有办法按照List中bean的某个属性来去重复呢？答案是有的，利用的是 stream 的 reduce，用一个set 来存放 key,代码如下：

1
2
3
4
5
6
7
8
9
List<JSONObject> result = trails.stream()
            .filter(distinctByKey(VehicleTrail::getPlateNbr))
            .collect(Collectors.toList());


private  <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

2个集合的元素两两组合成一个 n * m 的集合 (笛卡尔积)

1
2
3
4
5
Integer[] xs = new Integer[]{3, 4};
Integer[] ys = new Integer[]{5, 6};

List<Image> images = Arrays.stream(xs).flatMap(x -> Arrays.stream(ys).map(y -> new Image(x,y))).collect(Collectors.toList());

集合合并

比如: List<List> list 将所有的 Demo 合并到一个集合;

  1. reduce
1
2
3
4
5
// 第一种
List<Demo> demos  = list.stream().reduce(new ArrayList<>(),(demo1,demo2) -> {demo1.addAll(demo2); return demo2;});

// 第二种
List<Demo> demos  = list.stream().reduce(new ArrayList<>(),(demo1,demo2) -> Stream.concat(demo1.stream(),demo2.stream()).collect(Collectors.toList()));
  1. flatMap
1
List<Demo> demos = list.stream().flatMap(Collection::stream).collect(Collectors.toList());
0%