俺も死んだ(ﾁｰﾝ
今頃なのか今更なのか知りませんが、Elasticstackに手を出し始めました。
柔軟性が高いのはいいけど、その柔らかさに殺されそうでまぢつらたんです(´；ω；｀)
ちなみに構成は、
CentOS6
elastic 6.x
です。
さて、今回問題のケースでは、ログをfilebeatで取得して、logstashに流し込み、elasticsearchに書き込む構成を取りました。
そして、複数サーバーのログを集約させたいので、各サーバーにfilebeatを設置する、という形にしました。
図（？）にするとこんな感じ？ずれてたらごめんなさい
[FB]―[LS]―[ES]
[FB]／
これで動かしたけれども、elasticsearchのほうにIndexがさっぱり作られません。おかしいと思って/var/log/logstash/logstash-plain.logを見ると、以下のエラーがいっぱい書かれてました。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 |
[2018-03-16T16:25:43,219][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::Beats host=>"XXX.XXX.XXX.XXX", port=>5044, id=>"7bc610a496f71704c2758c6d060cce6a7ec50abad3f67c2a20a0fd481ed8ed51", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_be37bfe4-9235-4fbc-a3ae-1656e87bbae0", enable_metric=>true, charset=>"UTF-8">, ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>24> Error: Cannot assign requested address Exception: Java::JavaNet::BindException Stack: sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:433) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:425) sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:223) io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:128) io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:558) io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1283) io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:501) io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:486) io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:989) io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:254) io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:364) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:163) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:403) io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:463) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:858) io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30) java.lang.Thread.run(java/lang/Thread.java:748) [2018-03-16T16:25:44,221][INFO ][org.logstash.beats.Server] Starting server on port: 5044 [2018-03-16T16:25:44,226][WARN ][io.netty.channel.AbstractChannel] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xce8dfa74] java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:479) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:260) [netty-all-4.1.18.Final.jar:4.1.18.Final] at org.logstash.beats.Server.listen(Server.java:65) [logstash-input-beats-5.0.6.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438) [jruby-complete-9.1.13.0.jar:?] at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302) [jruby-complete-9.1.13.0.jar:?] at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:129) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:246) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [jruby-complete-9.1.13.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] [2018-03-16T16:25:44,260][ERROR][io.netty.util.concurrent.DefaultPromise.rejectedExecution] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:490) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:282) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:278) [netty-all-4.1.18.Final.jar:4.1.18.Final] at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:260) [netty-all-4.1.18.Final.jar:4.1.18.Final] at org.logstash.beats.Server.listen(Server.java:65) [logstash-input-beats-5.0.6.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:438) [jruby-complete-9.1.13.0.jar:?] at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:302) [jruby-complete-9.1.13.0.jar:?] at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:129) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73) [jruby-complete-9.1.13.0.jar:?] at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:289) [jruby-complete-9.1.13.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:246) [jruby-complete-9.1.13.0.jar:?] at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104) [jruby-complete-9.1.13.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] |
長すぎ！
まぁ要点は「Error: Cannot assign requested address」だと思います。
この時、2つのfilebeatが設定していたlogstashの宛先ポートはどちらも5044でした。
冷静になって考えたら、1つのポートを奪い合う状態はまずいので、片方のfilebeatのポートを5055に変更してやり直してみたら、普通に動きました！(∩´∀｀)∩ﾜｰｲ
単純にポートの競合はやめましょうって話でした。よくよく考えると恥ずかしいｗ
ご質問させてください
当方同じような環境を考えています
logstashのポートを5055にされたと理解します。この場合logstash側にlistenポートは5055はどのように記載されたのか具体的に教えて欲しいです
おそらくlogstashのconfにlistenを記載されていると思いますが、並べて書かれたのでしょうか
またfilebeatで複数の種類のログをlogstashに送ることはできるのでしょうか
コメントありがとうございます！そして返信が大変遅れてしまいました……申し訳ありません。
もしかしてもう解決してしまったかもしれませんが、返信させていただきます。
> logstashのポートを5055にされたと理解します。この場合logstash側にlistenポートは5055はどのように記載されたのか具体的に教えて欲しいです
> おそらくlogstashのconfにlistenを記載されていると思いますが、並べて書かれたのでしょうか
logstashのポート設定は、confファイルに以下のように書いています。
input {
beats {
port => 55034
}
}
並べて書いてはおらず、ポートひとつだけ指定しています。
通りすがりのさんがやりたいことは、複数のfilebeatから一つのlogstashへ同時にログを流すこと、かとお見受けいたしますが、そういうときは、logstashのパイプラインという仕組みを使うとうまくいきます。
参考ページ – [Elasticsearch][Logstash] Multiple Pipelinesを試す。：
https://qiita.com/tsgkdt/items/3e7f07ca963676b37718
> またfilebeatで複数の種類のログをlogstashに送ることはできるのでしょうか
filebeat.yml内のfilebeat.inputs:のところで、監視するデータソースを複数指定できます。
参考ページ – Configure inputs：
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html
ちなみに自分はこんな感じでfilebeat.ymlに記述しています。
filebeat.inputs:
– type: log
paths:
– /var/log/httpd/hogehoge_access_log
tags: [“hogehoge”]
– type: log
paths:
– /var/log/httpd/hugahuga_access_log
tags: [“hugahuga”]
– type: log
paths:
– /var/www/html/acme_system/storage/log/hoge.log
tags: [“acme_system”]