ES本地debug详解
所谓"工欲善其事,必先利其器"，同样要想成为es的顶级玩家,研究其源码肯定是必不可少的。研究源码的第一步就是搭建源码运行环境。
本文主要演示搭建elasticsearch的debug环境的具体步骤。目标是使读者能根据本文在本地搭建自己的es开发环境。
笔者使用的环境如下:
elasticsearch 7.6
Mac OS X 10.16 (x86_64)
jdk 12、13、11、8、9
gradle 6.3
idea 2020.3
笔者之前搭建开发环境也是在互联网找各种大牛的教程,但是经过几番尝试之后发现最后都失败了,究其原因不是因为大牛的方法不对,可能与不同的开发环境等都有关系。所以笔者就想既然是顶级开源项目肯定教我们如何做开源的相关文档,所以万事回归本源,最好的资料就是源码以及官方文档.
打开elasticsearch的github主页 https://github.com/elastic/elasticsearch 在readme.md中期待找有关如何编译项目的说明,很遗憾没有说明.
所以笔者想在源码中的某一些文档中肯定有说明,所以果断下载源码到本地,这里推荐先将源码fork一份都自己的github主页中，这样方便后期看源码时候可以添加一些注释,方便提交日后复习.
下载es源码,切换到7.6分支，因为笔者工作中使用es版本是7.6所以笔者切换到7.6分支，读者可以根据自己的需要切换到任意分支,方法都是一样的。
可以在elasticsearch的github主页选择自己的版本.
git clone https://github.com/elastic/elasticsearch.git
git checkout v7.6.0
打开elastisearch的源码,这一步可以用任意的工具打开,我这里用vscode打开,这一步主要是查找文档的说明。
到这一步我们就需要在源码中查找有关的说明文档,一般来说就是在源码的根目录中,所以经过一番艰难的英语阅读之后,笔者终于在CONTRIBUTING.md中找到答案了.真是源码面前无任何秘密可言.打开CONTRIBUTING.md中有这么一段说明.
Contributing to the Elasticsearch codebase ------------------------------------------ **Repository:** [https://github.com/elastic/elasticsearch](https://github.com/elastic/elasticsearch) JDK 13 is required to build Elasticsearch. You must have a JDK 13 installation with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 13 installation. By default, tests use the same runtime as `JAVA_HOME`. However, since Elasticsearch supports JDK 8, the build supports compiling with JDK 13 and testing on a JDK 8 runtime; to do this, set `RUNTIME_JAVA_HOME` pointing to the Java home of a JDK 8 installation. Note that this mechanism can be used to test against other JDKs as well, this is not only limited to JDK 8. > Note: It is also required to have `JAVA8_HOME`, `JAVA9_HOME`, `JAVA10_HOME` and `JAVA11_HOME`, and `JAVA12_HOME` available so that the tests can pass. Elasticsearch uses the Gradle wrapper for its build. You can execute Gradle using the wrapper via the `gradlew` script on Unix systems or `gradlew.bat` script on Windows in the root of the repository. The examples below show the usage on Unix. We support development in IntelliJ versions IntelliJ 2019.2 and onwards. We would like to support Eclipse, but few of us use it and has fallen into [disrepair][eclipse].
英语不好的可以借助有道词典或者google翻译,这句话的大致意思是如何给es贡献代码. 我们需要将JAVA_HOME设置为jdk13，并且需要设置RUNTIME_JAVA_HOME环境变量指向JDK8，而且由于es中的很多测试套件使用的是JDK8/JDK9/JDK10所以我们还需要设置JAVA8_HOME、JAVA10_HOME、JAVA11_HOME、JAVA12_HOME等环境变量。所以根据指示,笔者在mac电脑的本地环境变量文件.bash_profile中做了如下设置
这里我把JDK8开始到JDK14全部设置了一遍,那是因为es的不同版本要求的JDK不一样，所以干脆所有版本的JDK都下载下来,需要哪一个版本随时切换都可以了.
下载JDK不同版本地址信息 https://www.oracle.com/java/technologies/oracle-java-archive-downloads.html
设置玩环境变量之后,根据CONTRIBUTING.md的说明,我们就可以将es源码导入到idea中
注意笔者这里可以正常访问国外网站,而且网速通常也比较快,所以就没有单独设置es依赖的仓库信息,gradle版本信息也没有单独下载,因为在导入es时候idea会帮我们自动下载指定的gralde版本.
将源码导入到ES中之后，我们要将idea的JAVA版本正确设置。主要有两个地方
Gradle 依赖的JDK版本 ，笔者这里编译es7.6所以设置jdk13
源码的编译SDK
设置完这些之后.因为我导入的是idea中,idea会帮我们做一些自动配置. 紧接着就是一段漫长的等待过程,这种idea在中间会下载一些列的依赖等信息。
一切下载完成之后,如果一切顺利的话,那么我们就可以正常的打开代码了.
运行elasticsearch
打开es的启动类 Elasticsearch.java 执行main方法运行。发现报错如下
ERROR: the system property [es.path.conf] must be set
这是告诉我们需要指定一个包含配置文件的目录，我们可以直接下载一个发行版的ES，然后在idea的指定这个这个发行版ES的config目录，然后同时需要指定，配置如下：
-Des.path.conf=/Users/xxx/software/elasticsearch-7.6.0/config
-Des.path.home=/Users/xxx/software/elasticsearch-7.6.0/config
// 注意这里要下载对应的es发行版本例如我这里编译的是es 7.6那么我们就需要下载对应的release 7.6版本
然后运行，有报错：
Exception in thread "main" java.lang.NullPointerException
at org.elasticsearch.node.InternalSettingsPreparer.checkSettingsForTerminalDeprecation(InternalSettingsPreparer.java:119)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:91)
at org.elasticsearch.bootstrap.Bootstrap.createEnvironment(Bootstrap.java:267)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
debug跟到源码里，发现是这值是空导致的，那简单，在配置文件里加上就行了。修改，加上下面这个
node.name: node-1
然后继续运行，再次报错：
2020-12-20 10:20:53,679 main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:444)
at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanTrustPermission(DefaultMBeanServerInterceptor.java:1805)
at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:318)
at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.logging.log4j.core.jmx.Server.register(Server.java:393)
at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:168)
at org.apache.logging.log4j.core.jmx.Server.reregisterMBeansAfterReconfigure(Server.java:141)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:558)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
这是log4j JMX报错了。JMX是一个管理应用程序的接口，log4j提供了对JMX的支持，实现了远程动态修改配置等功能。
这个其实不是ES的核心功能，既然报错了，干脆就把它关闭吧
-Dlog4j2.disable.jmx=true
继续运行，还是报错：
[2020-12-20T10:22:36,426][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: org/elasticsearch/plugins/ExtendedPluginsClassLoader
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:545) ~[main/:?]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[main/:?]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[main/:?]
at org.elasticsearch.node.Node.(Node.java:313) ~[main/:?]
at org.elasticsearch.node.Node.(Node.java:257) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[main/:?]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[main/:?]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[main/:?]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[main/:?]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[main/:?]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[main/:?]
这个错误是跟idea的环境有关
idea > preferences > Build, Execution, Deployment > Build Tools > Gradle
把 改成 。
另一处修改是 ，找到，选中。
注意idea 2020.3这里的 非常难找，我也是参考一篇文章找到的。具体可以参考
https://youtrack.jetbrains.com/issue/IDEA-257364
运行，接着报错：
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createClassLoader")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.SecurityManager.checkCreateClassLoader(SecurityManager.java:470)
at java.base/java.lang.ClassLoader.checkCreateClassLoader(ClassLoader.java:369)
at java.base/java.lang.ClassLoader.checkCreateClassLoader(ClassLoader.java:359)
at java.base/java.lang.ClassLoader.(ClassLoader.java:456)
at org.elasticsearch.plugins.ExtendedPluginsClassLoader.(ExtendedPluginsClassLoader.java:36)
at org.elasticsearch.plugins.ExtendedPluginsClassLoader.lambda$create$0(ExtendedPluginsClassLoader.java:57)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
at org.elasticsearch.plugins.ExtendedPluginsClassLoader.create(ExtendedPluginsClassLoader.java:56)
at org.elasticsearch.plugins.PluginLoaderIndirection.createLoader(PluginLoaderIndirection.java:31)
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:545)
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471)
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163)
at org.elasticsearch.node.Node.(Node.java:313)
at org.elasticsearch.node.Node.(Node.java:257)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
这个问题查了一些资料，解决方案如下：
在发行版的config 目录下新建 java.policy 文件，填入下面内容:
grant {
permission java.lang.RuntimePermission "createClassLoader";
};
然后在 VM options 加入 java.security.policy 的设置，指向该文件即可
运行,继续报错
ElasticsearchException[Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].]
at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:587)
at org.elasticsearch.node.Node.lambda$new$9(Node.java:456)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1621)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.elasticsearch.node.Node.(Node.java:459)
at org.elasticsearch.node.Node.(Node.java:257)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
根据上面的说明提示在elasticsearch.yml中添加如下设置
运行之后，success，终于成功了.
浏览器输入 localhost:9200
{
"name" : "node-1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "tGwLMOdeRvye5ZaIF_68FQ",
"version" : {
"number" : "7.6.0",
"build_flavor" : "unknown",
"build_type" : "unknown",
"build_hash" : "unknown",
"build_date" : "unknown",
"build_snapshot" : true,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
为了更加方便的连接es,我们可以下载对应的kibana来管理es,在这里我下载kibana-7-6-0
https://www.elastic.co/cn/downloads/past-releases/kibana-7-6-0
启动kibana，如果有以下报错
{"type":"log","@timestamp":"2020-12-21T08:19:45Z","tags":["fatal","root"],"pid":4166,"message":"{ Error: getaddrinfo ENOTFOUND localhost\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)\n errno: 'ENOTFOUND',\n code: 'ENOTFOUND',\n syscall: 'getaddrinfo',\n hostname: 'localhost' }"}
FATAL Error: getaddrinfo ENOTFOUND localhost
解决办法是打开 kibana中的配置文件kibana.yml，修改一下两处
如图所示:
