JoyLau's Blog

JoyLau 的技术学习与思考

首先

首先记录,在这篇文章书写前,自己并不是刚刚上手 Hadoop, 其实学了有一段时间了
在这段时间内,由最开始的对 Hadoop 的懵懂无知到渐渐的熟悉 Hadoop 大致的开发流程
整个过程越来越清晰
于是就想着,把自己接下来在 Hadoop 上的学习计划记录下来

流程

  1. 了解 Hadoop 背景,开发作用
  2. 然后搭建Hadoop集群,先让它在自己电脑上运行。
  3. 学习分布式文件系统HDFS。
  4. 学习分布式计算框架MapReduce
  5. 学习流式计算Storm
  6. 学习分布式协作服务Zookeeper
  7. 学习Hive—数据仓库工具
  8. 学习Hbase—分布式存储系统
  9. 学习Spark
  10. 学习Scala
  11. 学习Spark开发技术

最后

这些技术在工作中是远远不够的,但也不是工作中每项都有用到了
就自己现在公司的大数据环境来说,还有像 impala,zookeeper,spark,kafka…等等
等有新的学习计划再补充吧

首先要说

java 操作 elasticsearch 有四种方式

  1. 调用 elasticsearch 的 restapis 接口
  2. 调用 java elasticsearch client 的接口
  3. 整合 spring data 使用 ElasticsearchTemplate 封装的方法
  4. 继承 ElasticsearchRepository 接口调用方法

测试准备

我们先来准备一些数据,写了一个之前的获取JoyMusic 的音乐数据的项目来说,项目的结构是这样的:
elasticsearch-test-project
获取数据的主要代码如下,只是为了增加数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
@RunWith(SpringJUnit4ClassRunner.class)
@SpringBootTest(classes = JoylauElasticsearchApplication.class,webEnvironment = SpringBootTest.WebEnvironment.MOCK)
public class JoylauElasticsearchApplicationTests {
@Autowired
private RestTemplate restTemplate;

@Autowired
private PlaylistDAO playlistDAO;

@Autowired
private SongDAO songDAO;

@Autowired
private CommentDAO commentDAO;
@Test
public void createData() {
String personalizeds = restTemplate.getForObject("http://localhost:3003/apis/v1"+"/personalized",String.class);
JSONObject perJSON = JSONObject.parseObject(personalizeds);
JSONArray perArr = perJSON.getJSONArray("result");
List<Playlist> list = new ArrayList<>();
List<Integer> playListIds = new ArrayList<>();
for (Object o : perArr) {
JSONObject playListJSON = JSONObject.parseObject(o.toString());
Playlist playlist = new Playlist();
playlist.setId(playListJSON.getIntValue("id"));
playListIds.add(playlist.getId());
playlist.setName(playListJSON.getString("name"));
playlist.setPicURL(playListJSON.getString("picUrl"));
playlist.setPlayCount(playListJSON.getIntValue("playCount"));
playlist.setBookCount(playListJSON.getIntValue("bookCount"));
playlist.setTrackCount(playListJSON.getIntValue("trackCount"));
list.add(playlist);
}
playlistDAO.saveAll(list);



/*存储歌曲*/
List<Integer> songIds = new ArrayList<>();
List<Song> songList = new ArrayList<>();
for (Integer playListId : playListIds) {
String res = restTemplate.getForObject("http://localhost:3003/apis/v1"+"/playlist/detail?id="+playListId,String.class);
JSONArray songJSONArr = JSONObject.parseObject(res).getJSONObject("playlist").getJSONArray("tracks");
for (Object o : songJSONArr) {
JSONObject songJSON = JSONObject.parseObject(o.toString());
Song song = new Song();
song.setId(songJSON.getIntValue("id"));
songIds.add(song.getId());
song.setName(songJSON.getString("name"));
song.setAuthor(getSongAuthor(songJSON.getJSONArray("ar")));
song.setTime(songJSON.getLong("dt"));
song.setPlaylistId(playListId);
song.setPicURL(songJSON.getJSONObject("al").getString("picUrl"));
song.setAlbum(songJSON.getJSONObject("al").getString("name"));
songList.add(song);
}
}
songDAO.saveAll(songList);



/*存储评论*/
List<Comment> commentList = new ArrayList<>();
for (Integer songId : songIds) {
String res = restTemplate.getForObject("http://localhost:3003/apis/v1"+"/comment/music?id="+songId+"&offset="+300,String.class);
JSONArray commentArr = JSONObject.parseObject(res).getJSONArray("comments");
for (Object o : commentArr) {
JSONObject commentJSON = JSONObject.parseObject(o.toString());
Comment comment = new Comment();
comment.setId(commentJSON.getIntValue("commentId"));
comment.setSongId(songId);
comment.setContent(commentJSON.getString("content"));
comment.setAuthor(commentJSON.getJSONObject("user").getString("nickname"));
comment.setPicUrl(commentJSON.getJSONObject("user").getString("avatarUrl"));
comment.setTime(commentJSON.getLong("time"));
comment.setSupport(commentJSON.getIntValue("likedCount"));
commentList.add(comment);
}

}

commentDAO.saveAll(commentList);
}

/**
* 获取歌曲作者名
* @param arr arr
* @return String
*/
private String getSongAuthor(JSONArray arr){
StringBuilder author = new StringBuilder();
for (Object o : arr) {
JSONObject json = JSONObject.parseObject(o.toString());
author.append(json.getString("name"));
if (arr.size() > 1){
author.append(",");
}
}
return author.toString();
}
}

跑了起来之后, elasticsearch 增加的数据如下:
elasticsearch-test-guide
elasticsearch-test-data

现在数据有了,接下来就是使用各种方法了

ElasticSearchTemplate 和 ElasticsearchRepository 的关系

ElasticSearchTemplate 是 spring date 对 elasticsearch 客户端 Java API 的封装,而 ElasticsearchRepository,是ElasticSearchTemplate更深层次的封装,可以使用注解,很类似以前 mybatis 的使用
ElasticSearchTemplate提供的方法更多,ElasticsearchRepository能用的方法其实全部都在而 ElasticSearchTemplate 都有实现
我们只要能熟悉调用的 ElasticSearchTemplate 里面的方法操作
ElasticsearchRepository都能够会操作

ElasticSearchTemplate

一些很底层的方法,我们最常用的就是elasticsearchTemplate.queryForList(searchQuery, class);
而这里面最主要的就是构建searchQuery,一下总结几个最常用的searchQuery以备忘
searchQuery能构建好,其他的就很简单了

queryStringQuery

单字符串全文查询

1
2
3
4
5
6
7
8
9
10
11
12
/**
* 单字符串模糊查询,默认排序。将从所有字段中查找包含传来的word分词后字符串的数据集
*/
@Test
public void queryStringQuerySong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(queryStringQuery("Time")).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

返回的结果如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
{
"query_string" : {
"query" : "Time",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
}
{"album":"Time","author":"Cat naps","id":459733590,"name":"Time","picURL":"http://p1.music.126.net/9DmApLeDwutb4HpuhD_E-Q==/18624627464667106.jpg","playlistId":900228548,"time":86465}
{"album":"Go Time","author":"Mark Petrie","id":29717271,"name":"Go Time","picURL":"http://p1.music.126.net/TJe468hZr_0ndQRfTAKdqA==/3233663697760186.jpg","playlistId":636015704,"time":136071}
{"album":"Out of Time","author":"R.E.M.","id":20282663,"name":"Losing My Religion","picURL":"http://p1.music.126.net/wYtpqN8Yu2jamQwdM6ugGg==/6638851209090428.jpg","playlistId":772430182,"time":269270}
{"album":"Time Flies... 1994-2009","author":"Oasis","id":17822660,"name":"Cigarettes & Alcohol","picURL":"http://p1.music.126.net/qDgXElJRtSsuqNwsTzW8lw==/667403558069001.jpg","playlistId":772430182,"time":291853}
{"album":"Electric Warrior","author":"T. Rex","id":29848501,"name":"There Was A Time","picURL":"http://p1.music.126.net/dn1MwEBfBcL4l6isrnEwDw==/3246857839528733.jpg","playlistId":772430182,"time":60577}
{"album":"Ride On Time","author":"山下達郎","id":22693846,"name":"DAYDREAM","picURL":"http://p1.music.126.net/GaQVveQiyTIqecs7hhoYpA==/749866930165154.jpg","playlistId":900228548,"time":273476}
{"album":"The Blossom Chronicles","author":"Philter","id":21375446,"name":"Adventure Time","picURL":"http://p1.music.126.net/YjMS5_kM3u9PCUU0lcRK8g==/6657542907248762.jpg","playlistId":636015704,"time":207412}
{"album":"Decimus","author":"Audio Machine","id":36586631,"name":"Ashes of Time","picURL":"http://p1.music.126.net/7InBepjNDGCzpzH8Feyw9A==/3395291908535260.jpg","playlistId":636015704,"time":190826}
{"album":"In Time: The Best Of R.E.M. 1988-2003","author":"R.E.M.","id":20283068,"name":"Bad Day","picURL":"http://p1.music.126.net/aZXu5ulRJvH4dnoWPjxb3A==/18277181789089107.jpg","playlistId":772430182,"time":248111}
{"album":"It's a Poppin' Time","author":"山下達郎","id":22693864,"name":"HEY THERE LONELY GIRL","picURL":"http://p1.music.126.net/PGZlyXk20_-5d6E3pDEKpg==/815837627833461.jpg","playlistId":900228548,"time":325956}
{"album":"Shire Music Annual Selection - Myth","author":"Shire Music,Songs To Your Eyes,","id":34916751,"name":"Between Space And Time","picURL":"http://p1.music.126.net/CCqLd2ly2XuuSPz0IW0u-g==/3284241233077333.jpg","playlistId":636015704,"time":222456}
{"album":"Double Live Doggie Style I","author":"X-Ray Dog","id":26246058,"name":"Time Will Tell","picURL":"http://p1.music.126.net/oYEIMWnAvpuRDTk4g_l-lg==/2503587976473913.jpg","playlistId":636015704,"time":202133}
{"album":"The Ghost Of Tom Joad","author":"Bruce Springsteen","id":16657852,"name":"Straight Time (Album Version)","picURL":"http://p1.music.126.net/yK0V-aD3Myh4xorvwUtCrw==/17889054184179160.jpg","playlistId":772430182,"time":210651}
{"album":"Epic Action & Adventure Vol. 6","author":"Epic Score","id":4054121,"name":"Time Will Remember Us","picURL":"http://p1.music.126.net/uN8AYI3sQEgoECuSYmi9Eg==/658607465082090.jpg","playlistId":636015704,"time":165000}

我们修改一下排序方式,按照id从大到小排序

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* 单字符串模糊查询,单字段排序。
*/
@Test
public void queryStringQueryWeightSong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(queryStringQuery("Time")).withPageable(of(0,100,new Sort(Sort.Direction.DESC,"id"))).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

也可以使用注解,这么写

1
2
3
4
5
6
7
8
public void queryStringQueryWeightSong(@PageableDefault(sort = "id", direction = Sort.Direction.DESC) Pageable pageable){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(queryStringQuery("Time")).withPageable(pageable).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

返回的结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
{
"query_string" : {
"query" : "Time",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
}
{"album":"Time","author":"Cat naps","id":459733590,"name":"Time","picURL":"http://p1.music.126.net/9DmApLeDwutb4HpuhD_E-Q==/18624627464667106.jpg","playlistId":900228548,"time":86465}
{"album":"Decimus","author":"Audio Machine","id":36586631,"name":"Ashes of Time","picURL":"http://p1.music.126.net/7InBepjNDGCzpzH8Feyw9A==/3395291908535260.jpg","playlistId":636015704,"time":190826}
{"album":"Shire Music Annual Selection - Myth","author":"Shire Music,Songs To Your Eyes,","id":34916751,"name":"Between Space And Time","picURL":"http://p1.music.126.net/CCqLd2ly2XuuSPz0IW0u-g==/3284241233077333.jpg","playlistId":636015704,"time":222456}
{"album":"Electric Warrior","author":"T. Rex","id":29848501,"name":"There Was A Time","picURL":"http://p1.music.126.net/dn1MwEBfBcL4l6isrnEwDw==/3246857839528733.jpg","playlistId":772430182,"time":60577}
{"album":"Go Time","author":"Mark Petrie","id":29717271,"name":"Go Time","picURL":"http://p1.music.126.net/TJe468hZr_0ndQRfTAKdqA==/3233663697760186.jpg","playlistId":636015704,"time":136071}
{"album":"Double Live Doggie Style I","author":"X-Ray Dog","id":26246058,"name":"Time Will Tell","picURL":"http://p1.music.126.net/oYEIMWnAvpuRDTk4g_l-lg==/2503587976473913.jpg","playlistId":636015704,"time":202133}
{"album":"It's a Poppin' Time","author":"山下達郎","id":22693864,"name":"HEY THERE LONELY GIRL","picURL":"http://p1.music.126.net/PGZlyXk20_-5d6E3pDEKpg==/815837627833461.jpg","playlistId":900228548,"time":325956}
{"album":"Ride On Time","author":"山下達郎","id":22693846,"name":"DAYDREAM","picURL":"http://p1.music.126.net/GaQVveQiyTIqecs7hhoYpA==/749866930165154.jpg","playlistId":900228548,"time":273476}
{"album":"The Blossom Chronicles","author":"Philter","id":21375446,"name":"Adventure Time","picURL":"http://p1.music.126.net/YjMS5_kM3u9PCUU0lcRK8g==/6657542907248762.jpg","playlistId":636015704,"time":207412}
{"album":"In Time: The Best Of R.E.M. 1988-2003","author":"R.E.M.","id":20283068,"name":"Bad Day","picURL":"http://p1.music.126.net/aZXu5ulRJvH4dnoWPjxb3A==/18277181789089107.jpg","playlistId":772430182,"time":248111}
{"album":"Out of Time","author":"R.E.M.","id":20282663,"name":"Losing My Religion","picURL":"http://p1.music.126.net/wYtpqN8Yu2jamQwdM6ugGg==/6638851209090428.jpg","playlistId":772430182,"time":269270}
{"album":"Time Flies... 1994-2009","author":"Oasis","id":17822660,"name":"Cigarettes & Alcohol","picURL":"http://p1.music.126.net/qDgXElJRtSsuqNwsTzW8lw==/667403558069001.jpg","playlistId":772430182,"time":291853}
{"album":"The Ghost Of Tom Joad","author":"Bruce Springsteen","id":16657852,"name":"Straight Time (Album Version)","picURL":"http://p1.music.126.net/yK0V-aD3Myh4xorvwUtCrw==/17889054184179160.jpg","playlistId":772430182,"time":210651}
{"album":"Epic Action & Adventure Vol. 6","author":"Epic Score","id":4054121,"name":"Time Will Remember Us","picURL":"http://p1.music.126.net/uN8AYI3sQEgoECuSYmi9Eg==/658607465082090.jpg","playlistId":636015704,"time":165000}

matchQuery

查询某个字段中模糊包含目标字符串,使用matchQuery

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* 单字段对某字符串模糊查询
*/
@Test
public void matchQuerySong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchQuery("name","Time")).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

返回结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
"match" : {
"name" : {
"query" : "Time",
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : true,
"lenient" : false,
"zero_terms_query" : "NONE",
"boost" : 1.0
}
}
}
{"album":"Time","author":"Cat naps","id":459733590,"name":"Time","picURL":"http://p1.music.126.net/9DmApLeDwutb4HpuhD_E-Q==/18624627464667106.jpg","playlistId":900228548,"time":86465}
{"album":"Go Time","author":"Mark Petrie","id":29717271,"name":"Go Time","picURL":"http://p1.music.126.net/TJe468hZr_0ndQRfTAKdqA==/3233663697760186.jpg","playlistId":636015704,"time":136071}
{"album":"The Blossom Chronicles","author":"Philter","id":21375446,"name":"Adventure Time","picURL":"http://p1.music.126.net/YjMS5_kM3u9PCUU0lcRK8g==/6657542907248762.jpg","playlistId":636015704,"time":207412}
{"album":"Shire Music Annual Selection - Myth","author":"Shire Music,Songs To Your Eyes,","id":34916751,"name":"Between Space And Time","picURL":"http://p1.music.126.net/CCqLd2ly2XuuSPz0IW0u-g==/3284241233077333.jpg","playlistId":636015704,"time":222456}
{"album":"Electric Warrior","author":"T. Rex","id":29848501,"name":"There Was A Time","picURL":"http://p1.music.126.net/dn1MwEBfBcL4l6isrnEwDw==/3246857839528733.jpg","playlistId":772430182,"time":60577}
{"album":"Double Live Doggie Style I","author":"X-Ray Dog","id":26246058,"name":"Time Will Tell","picURL":"http://p1.music.126.net/oYEIMWnAvpuRDTk4g_l-lg==/2503587976473913.jpg","playlistId":636015704,"time":202133}
{"album":"Decimus","author":"Audio Machine","id":36586631,"name":"Ashes of Time","picURL":"http://p1.music.126.net/7InBepjNDGCzpzH8Feyw9A==/3395291908535260.jpg","playlistId":636015704,"time":190826}
{"album":"The Ghost Of Tom Joad","author":"Bruce Springsteen","id":16657852,"name":"Straight Time (Album Version)","picURL":"http://p1.music.126.net/yK0V-aD3Myh4xorvwUtCrw==/17889054184179160.jpg","playlistId":772430182,"time":210651}
{"album":"Epic Action & Adventure Vol. 6","author":"Epic Score","id":4054121,"name":"Time Will Remember Us","picURL":"http://p1.music.126.net/uN8AYI3sQEgoECuSYmi9Eg==/658607465082090.jpg","playlistId":636015704,"time":165000}

matchPhraseQuery

PhraseMatch查询,短语匹配

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* 单字段对某短语进行匹配查询,短语分词的顺序会影响结果
*/
@Test
public void phraseMatchSong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchPhraseQuery("name","Time")).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

termQuery

这个是最严格的匹配,属于低级查询,不进行分词的,参考这篇文章 http://www.cnblogs.com/muniaofeiyu/p/5616316.html

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* term匹配,即不分词匹配,你传来什么值就会拿你传的值去做完全匹配
*/
@Test
public void termQuerySong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(termQuery("name","Time")).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

multiMatchQuery

多个字段匹配某字符串,如果我们希望name,author两个字段去匹配某个字符串,只要任何一个字段包括该字符串即可,就可以使用multiMatchQuery。

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* 多字段匹配
*/
@Test
public void multiMatchQuerySong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(multiMatchQuery("time","name","author")).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

完全包含查询

之前的查询中,当我们输入“我天”时,ES会把分词后所有包含“我”和“天”的都查询出来,如果我们希望必须是包含了两个字的才能被查询出来,那么我们就需要设置一下Operator。

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* 单字段包含所有输入
*/
@Test
public void matchQueryOperatorSong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchQuery("name","真的").operator(Operator.AND)).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

无论是matchQuery,multiMatchQuery,queryStringQuery等,都可以设置operator。默认为Or,设置为And后,就会把符合包含所有输入的才查出来。
如果是and的话,譬如用户输入了5个词,但包含了4个,也是显示不出来的。我们可以通过设置精度来控制。

1
2
3
4
5
6
7
8
9
10
11
12
/** 
* 单字段包含所有输入(按比例包含)
*/
@Test
public void matchQueryOperatorWithMinimumShouldMatchSong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchQuery("name","time").operator(Operator.AND).minimumShouldMatch("80%")).withPageable(of(0,100)).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

minimumShouldMatch可以用在match查询中,设置最少匹配了多少百分比的能查询出来。

合并查询

即boolQuery,可以设置多个条件的查询方式。它的作用是用来组合多个Query,有四种方式来组合,must,mustnot,filter,should。
must代表返回的文档必须满足must子句的条件,会参与计算分值;
filter代表返回的文档必须满足filter子句的条件,但不会参与计算分值;
should代表返回的文档可能满足should子句的条件,也可能不满足,有多个should时满足任何一个就可以,通过minimum_should_match设置至少满足几个。
mustnot代表必须不满足子句的条件。
譬如我想查询name包含“XXX”,且userId=“2345098”,且time最好小于165000的结果。那么就可以使用boolQuery来组合。

1
2
3
4
5
6
7
8
9
10
11
12
13
/** 
* 多字段合并查询
*/
@Test
public void boolQuerySong(){
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(boolQuery().must(termQuery("userId", "2345098"))
.should(rangeQuery("time").lt(165000)).must(matchQuery("name", "time"))).build();
System.out.println(searchQuery.getQuery().toString());
List<Song> songList = songDAO.search(searchQuery).getContent();
for (Song song : songList) {
System.out.println(JSONObject.toJSONString(song));
}
}

详细点的看这篇 http://blog.csdn.net/dm_vincent/article/details/41743955
boolQuery使用场景非常广泛,应该是主要学习的知识之一。

Query和Filter的区别

query和Filter都是QueryBuilder,也就是说在使用时,你把Filter的条件放到withQuery里也行,反过来也行。那么它们两个区别在哪?
查询在Query查询上下文和Filter过滤器上下文中,执行的操作是不一样的:

1、查询:是在使用query进行查询时的执行环境,比如使用search的时候。
在查询上下文中,查询会回答这个问题——“这个文档是否匹配这个查询,它的相关度高么?”
ES中索引的数据都会存储一个_score分值,分值越高就代表越匹配。即使lucene使用倒排索引,对于某个搜索的分值计算还是需要一定的时间消耗。

2、过滤器:在使用filter参数时候的执行环境,比如在bool查询中使用Must_not或者filter
在过滤器上下文中,查询会回答这个问题——“这个文档是否匹配?”
它不会去计算任何分值,也不会关心返回的排序问题,因此效率会高一点。
另外,经常使用过滤器,ES会自动的缓存过滤器的内容,这对于查询来说,会提高很多性能。

ElasticsearchRepository

ElasticsearchRepository接口的方法有

1
2
3
4
5
6
7
8
9
10
11
12
@NoRepositoryBean
public interface ElasticsearchRepository<T, ID extends Serializable> extends ElasticsearchCrudRepository<T, ID> {
<S extends T> S index(S var1);

Iterable<T> search(QueryBuilder var1);

FacetedPage<T> search(QueryBuilder var1, Pageable var2);

FacetedPage<T> search(SearchQuery var1);

Page<T> searchSimilar(T var1, String[] var2, Pageable var3);
}

执行复杂查询最常用的就是 FacetedPage search(SearchQuery var1); 这个方法了,需要的参数是 SearchQuery
主要是看QueryBuilder和SearchQuery两个参数,要完成一些特殊查询就主要看构建这两个参数。
我们先来看看它们之间的类关系
image

实际使用中,我们的主要任务就是构建NativeSearchQuery来完成一些复杂的查询的。

1
2
3
4
5
6
public NativeSearchQuery(QueryBuilder query, QueryBuilder filter, List<SortBuilder> sorts, Field[] highlightFields) {  
this.query = query;
this.filter = filter;
this.sorts = sorts;
this.highlightFields = highlightFields;
}

我们可以看到要构建NativeSearchQuery,主要是需要几个构造参数

当然了,我们没必要实现所有的参数。
可以看出来,大概是需要QueryBuilder,filter,和排序的SortBuilder,和高亮的字段。
一般情况下,我们不是直接是new NativeSearchQuery,而是使用NativeSearchQueryBuilder。
通过NativeSearchQueryBuilder.withQuery(QueryBuilder1).withFilter(QueryBuilder2).withSort(SortBuilder1).withXXXX().build();这样的方式来完成NativeSearchQuery的构建。
从名字就能看出来,QueryBuilder主要用来构建查询条件、过滤条件,SortBuilder主要是构建排序。

很幸运的 ElasticsearchRepository 里的 SearchQuery 也就是上述描述的 temple 的 SearchQuery,2 者可以共用

介绍

JSONPath。这是一个很强大的功能,可以在java框架中当作对象查询语言(OQL)来使用

语法说明

JSONPATH 描述
$ 根对象,例如$.name
[num] 数组访问,其中num是数字,可以是负数。例如$[0].leader.departments[-1].name
[num0,num1,num2…] 数组多个元素访问,其中num是数字,可以是负数,返回数组中的多个元素。例如$[0,3,-2,5]
[start:end] 数组范围访问,其中start和end是开始小表和结束下标,可以是负数,返回数组中的多个元素。例如$[0:5]
[start:end :step] 数组范围访问,其中start和end是开始小表和结束下标,可以是负数;step是步长,返回数组中的多个元素。例如$[0:5:2]
[?(key)] 对象属性非空过滤,例如$.departs[?(name)]
[key > 123] 数值类型对象属性比较过滤,例如$.departs[id >= 123],比较操作符支持=,!=,>,>=,<,<=
[key = ‘123’] 字符串类型对象属性比较过滤,例如$.departs[name = ‘123’],比较操作符支持=,!=,>,>=,<,<=
[key like ‘aa%’] 字符串类型like过滤,例如$.departs[name like ‘sz*’],通配符只支持% 支持not like
[key rlike ‘regexpr’] 字符串类型正则匹配过滤,例如departs[name like ‘aa(.)*’],正则语法为jdk的正则语法,支持not rlike
[key in (‘v0’, ‘v1’)] IN过滤, 支持字符串和数值类型 例如: $.departs[name in (‘wenshao’,’Yako’)] $.departs[id not in (101,102)]
[key between 234 and 456] BETWEEN过滤, 支持数值类型,支持not between 例如: $.departs[id between 101 and 201]$.departs[id not between 101 and 201]length() 或者 size() 数组长度。例如$.values.size() 支持类型java.util.Map和java.util.Collection和数组
. 属性访问,例如$.name
.. deepScan属性访问,例如$..name
* 对象的所有属性,例如$.leader.*
[‘key’] 属性访问。例如$[‘name’]
[‘key0’,’key1’] 多个属性访问。例如$[‘id’,’name’]

语法示例

JSONPath 语义
$ 根对象
$[-1] 最后元素
$[:-2] 第1个至倒数第2个
$[1:] 第2个之后所有元素
$[1,2,3] 集合中1,2,3个元素

java 示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{ "store": {
"book": [
{ "category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{ "category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99,
"isbn": "0-553-21311-3"
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
private static void jsonPathTest() {
JSONObject json = jsonTest();//调用自定义的jsonTest()方法获得json对象,生成上面的json

//输出book[0]的author值
String author = JsonPath.read(json, "$.store.book[0].author");

//输出全部author的值,使用Iterator迭代
List<String> authors = JsonPath.read(json, "$.store.book[*].author");

//输出book[*]中category == 'reference'的book
List<Object> books = JsonPath.read(json, "$.store.book[?(@.category == 'reference')]");

//输出book[*]中price>10的book
List<Object> books = JsonPath.read(json, "$.store.book[?(@.price>10)]");

//输出book[*]中含有isbn元素的book
List<Object> books = JsonPath.read(json, "$.store.book[?(@.isbn)]");

//输出该json中所有price的值
List<Double> prices = JsonPath.read(json, "$..price");

//可以提前编辑一个路径,并多次使用它
JsonPath path = JsonPath.compile("$.store.book[*]");
List<Object> books = path.read(json);
}

今天在安装完nodejs后执行 npm install 居然出错了

npm: relocation error: npm: symbol SSL_set_cert_cb, version libssl.so.10 not defined in file libssl.

npm: relocation error: npm: symbol SSL_set_cert_cb, version libssl.so.10 not defined in file libssl.so.10 with link time reference”, “rc”: 127, “stderr”: “npm: relocation error: npm: symbol SSL_set_cert_cb, version libssl.so.10 not defined in file libssl.so.10 with link time reference\n”, “stderr_lines”: [“npm: relocation error: npm: symbol SSL_set_cert_cb, version libssl.so.10 not defined in file libssl.so.10 with link time reference

解决办法:

yum -y install openssl

如果已经安装,就更新一下

yum -y update openssl

第一个坑

SpringBoot 在1.5版本后就有了 starter, 但是在依赖列表中却没有找到相应的依赖,原因是名字不叫starter,傻傻的我还用JavaConfig 配置了一遍
现在看下整合 starter 之后的是怎么样的吧!

1
2
3
4
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>

上面这个依赖其实就是starter, 不需要些版本,SpringBoot会自己选择版本

yml配置文件

1
2
3
4
5
6
7
8
spring
kafka:
bootstrap-servers: 192.168.10.192:9092
consumer:
group-id: secondary-identification
producer:
batch-size: 65536
buffer-memory: 524288

默认只需要 bootstrap-servers 和 group-id 即可

接下来 生产者 和 消费者

1
2
3
4
5
6
7
8
@Component
public class MsgProducer {
@Autowired
private KafkaTemplate kafkaTemplate;
public void sendMessage() {
kafkaTemplate.send("index-vehicle","key","hello,kafka" + LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS")));
}
}
1
2
3
4
5
6
7
@Component
public class MsgConsumer {
@KafkaListener(topics = {"index-vehicle"})
public void processMessage(String content) {
System.out.println(content);
}
}

第二个坑

可以发消息,但是SpringBoot始终收不到,我用Kafka自带的工具却可以收到,很气愤,搞了好长时间都没有解决
后来遍访Google和官方文档,终于找到原因了,只要修改下配置文件的一个配置即可:

1
2
3
4
5
6
7
# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://0.0.0.0:9092

上面的额这个 listeners,因为我的程序是加了@KafkaListener 来监听消息的,需要开启一个这样的配置项

这项配置项的含义在此也备注下:

监听列表(以逗号分隔 不同的协议(如plaintext,trace,ssl、不同的IP和端口)),hostname如果设置为0.0.0.0则绑定所有的网卡地址;如果hostname为空则绑定默认的网卡。如果没有配置则默认为java.net.InetAddress.getCanonicalHostName()

这2个坑在此记录下

一些常用命令在此记录下

zookeeper-server-start.bat ../../config/zookeeper.properties : 开启自带的zookeeper
kafka-server-start.bat ../../config.properties : 开启kafka
kafka-console-consumer.bat –bootstrap-server localhost:9092 –topic myTopic –from-beginning : 控制台接受指定topic消息
kafka-console-producer.bat –broker-list localhost:9092 –topic myTopic : 指定topic发送消息

注意的是用命令行创建的producer绑定的主题topic需要用命令行先创建topic,已经创建的就直接发送就好了

本文转自:http://blog.csdn.net/jsshaojinjie/article/details/64125458

maven dependencies增加

1
2
3
4
5
<dependency>  
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>

project增加

1
2
3
4
5
6
7
8
9
10
11
12
<build>  
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<!--fork : 如果没有该项配置,devtools不会起作用,即应用不会restart -->
<fork>true</fork>
</configuration>
</plugin>
</plugins>
</build>

idea设置

image

ctrl+shift+alt+/

image
image

重启项目即可。

通过注解读取文件

1
2
@Value("classpath:static/json/addTask.json")
Resource addTaskJson;

其他配置

前缀 例子 说明
classpath: classpath:com/myapp/config.xml 从classpath中加载
file: file:/data/config.xml 作为 URL 从文件系统中加载
http: http://myserver/logo.png 作为 URL 加载
(none) /data/config.xml 根据 ApplicationContext 进行判断

摘自于Spring Framework参考手册

转化为 字符串 转化为 JSON 对象

1
2
String jsonStr = new String(IOUtils.readFully(addTaskJson.getInputStream(), -1,true));
JSONObject json = JSONObject.parseObject(jsonStr);

注意: 该方法需要 jdk1.8的环境

SpringBoot 连接 Oracle

pom 文件配置

1
2
3
4
5
6
7
8
9
10
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>12.1.0.2</version>
</dependency>

注意: com.oracle.ojdbc6.12.1.0.2 在中央仓库没有,需要单独下载下来,再安装到本地仓库

yml文件配置

1
2
3
4
5
6
7
8
9
10
spring:
datasource:
driver-class-name: oracle.jdbc.OracleDriver
url: jdbc:oracle:thin:@192.168.10.240:1522:orcl12c
username: C##itmscz
password: itmscz
jpa:
hibernate:
ddl-auto: update
show-sql: true

接下来的套路都一样了,写好model实体类,注册个接口,然后就可以直接增删改查了

model :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@Entity(name = "t_samp_recog")
@Data
public class SampRecog {
@Id
@GeneratedValue
private int id; //主键
private String batch; // 批次
private String img_url; // 图片路径
private String plate_nbr; // 车辆号牌
private boolean plate_nbr_right; // 车辆号牌是否正确
private String brand; // 品牌
private boolean brand_right; // 品牌是否正确
private String veh_type; // 车辆类型
private boolean veh_type_right; // 车辆类型是否正确
private String veh_color; // 车身颜色
private boolean veh_color_right; // 车身颜色是否正确
private String sticker_pos; // 车标位置
private boolean sticker_pos_right; // 车标位置是否全部正确
private boolean is_right; // 是否全部正确
private int check_status; //核对状态 1.未核对,2,正在核对,3、已经核对
}

dao :

1
2
public interface SampRecogDAO extends JpaRepository<SampRecog,Integer> {
}

看来 SpringBoot 整合数据源的套路都一样,下次整合其他的猜都知道怎么做了

以前一直用的 Navicat Premiun,里面虽然支持 Oracle ,但是支持 Oracle 版本都比较老啦,新一点的根本连接不上去,今天在网上找到个绿色版的 Navicat for Oracle,赶紧记下来,mark一下

地址

百度云盘:链接: https://pan.baidu.com/s/1mhPS9wW 密码: gtq4

7z的是我替换操作后的

操作

下载 3个文件 :
Navicat for Oracle.zip

instantclient-basic-nt-12.1.0.2.0.zip

instantclient-sqlplus-nt-12.1.0.2.0.zip

直接把 instantclient-basic-nt-12.1.0.2.0.zip 解压到 Navicat for Oracle 的解压目录的instantclient_10_2目录下

然后这个目录下多了instantclient_12_1 这个目录

然后再把instantclient-sqlplus-nt-12.1.0.2.0.zip 解压到 instantclient_12_1下

完成

最后打开Navicat for Oracle 单击 工具->选项-> OCI

2个路径分别选:

\instantclient_12_1.oci.dll

\instantclient_12_1.sqlplus.exe

然后就可以连接使用了

0%