Problema: graylog deja de funcionar y los nodos del cluster no van: estan en rojo y tus busquedas no funcionan….algo pasa :)

Te encuentras con estos logs en graylog

1
2
3
4
5
6
Caused by: java.lang.NumberFormatException: For input string: "INFO"
at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:1.8.0_131]
at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_131]
at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_131]
12:05
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [level]

Sacas un listado de los indices y hay algunos en rojo como _graylogdeflector

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<<< DEV >>> root@elasticsearch-3:/home/me# curl -XGET localhost:9200/_cat/indices/
green open infra_256 kGaMkaRFSC6U9Jmwy12rcg 1 0 865668 0 952.9mb 952.9mb
yellow open admin SKOJKDG9Su22Gaky88EYFg 5 1 0 0 955b 955b
green open graylog_1330 Dfg0gxIUQZWKk4befHPDgg 1 0 3246209 0 984mb 984mb
green open infra_268 hBWgkwWoQVq56sLGoutiZw 1 0 742742 0 934.6mb 934.6mb
green open graylog_1343 3hNgjtbMSo2vjLBFe9EwuQ 1 0 2718275 0 905.9mb 905.9mb
green open graylog_1341 prnmoaZ4TqWZvk5TwAku8w 1 0 2453011 0 770mb 770mb
green open infra_251 KamoFJo-S2S7LIIHcN2flw 1 0 739868 0 948.4mb 948.4mb
green open infra_248 ge77cCZBSZuTwmq0wr-Mcg 1 0 574679 0 747.8mb 747.8mb
green open graylog_1342 BzQtkgOkSf6Scp0ZPiqGfA 1 0 2446182 0 792.1mb 792.1mb
green open infra_263 yP1kpJFATS2uuh1v1mOR8Q 1 0 655589 0 927.1mb 927.1mb
green open infra_259 i1s6JlzNRTOmEB2Kq2GzTg 1 0 624826 0 852.1mb 852.1mb
green open snmp_0 _1Nqdwz_SxmObUSnNNdJgA 1 0 0 0 191b 191b
green open infra_241 OL0B4RJjRWyyEilxb51jcQ 1 0 806433 0 939mb 939mb
green open infra_254 0ru9BT5pTYeNYV4uAYC_cQ 1 0 628605 0 895mb 895mb
green open infra_240 Q_xMSj1tRgy5Iudm5i-jlw 1 0 809087 0 961.8mb 961.8mb
green open infra_255 q26q4o_YQQe5l6ISfXH2qQ 1 0 590433 0 699.6mb 699.6mb
green open infra_257 2oCAKjNnRKO4FyN-6qGcQA 1 0 618289 0 814.7mb 814.7mb
green open graylog_1339 QNbCiY5YSMSmwhIMEPeeug 1 0 1925438 0 598.2mb 598.2mb
green open infra_261 m2Jq0jpeQWybKw-4hoOHhg 1 0 567153 0 758.8mb 758.8mb
red open graylog_deflector IiS44uBOSYyBW5OLD5PmzA 5 1 33009570 799 13.3gb 13.3gb
green open graylog_1340 bea8-Bb4RHqPmEhub9hAcQ 1 0 2366439 0 732.5mb 732.5mb
green open infra_262 x-H5TBBlRDOn3d115ynlNg 1 0 706197 0 905.6mb 905.6mb

Y el graylog te dice que Deflector exists as an index and is not an alias

Solucion

Saca el estado de los indices, si hay en rojo: tienes posibles culpables

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<<< DEV >>> root@elasticsearch-3:/home/me# curl -XGET localhost:9200/_cat/indices/
green open infra_256 kGaMkaRFSC6U9Jmwy12rcg 1 0 865668 0 952.9mb 952.9mb
yellow open admin SKOJKDG9Su22Gaky88EYFg 5 1 0 0 955b 955b
green open graylog_1330 Dfg0gxIUQZWKk4befHPDgg 1 0 3246209 0 984mb 984mb
green open infra_268 hBWgkwWoQVq56sLGoutiZw 1 0 742742 0 934.6mb 934.6mb
green open graylog_1343 3hNgjtbMSo2vjLBFe9EwuQ 1 0 2718275 0 905.9mb 905.9mb
green open graylog_1341 prnmoaZ4TqWZvk5TwAku8w 1 0 2453011 0 770mb 770mb
green open infra_251 KamoFJo-S2S7LIIHcN2flw 1 0 739868 0 948.4mb 948.4mb
green open infra_248 ge77cCZBSZuTwmq0wr-Mcg 1 0 574679 0 747.8mb 747.8mb
green open graylog_1342 BzQtkgOkSf6Scp0ZPiqGfA 1 0 2446182 0 792.1mb 792.1mb
green open infra_263 yP1kpJFATS2uuh1v1mOR8Q 1 0 655589 0 927.1mb 927.1mb
green open infra_259 i1s6JlzNRTOmEB2Kq2GzTg 1 0 624826 0 852.1mb 852.1mb
green open snmp_0 _1Nqdwz_SxmObUSnNNdJgA 1 0 0 0 191b 191b
green open infra_241 OL0B4RJjRWyyEilxb51jcQ 1 0 806433 0 939mb 939mb
green open infra_254 0ru9BT5pTYeNYV4uAYC_cQ 1 0 628605 0 895mb 895mb
green open infra_240 Q_xMSj1tRgy5Iudm5i-jlw 1 0 809087 0 961.8mb 961.8mb
green open infra_255 q26q4o_YQQe5l6ISfXH2qQ 1 0 590433 0 699.6mb 699.6mb
green open infra_257 2oCAKjNnRKO4FyN-6qGcQA 1 0 618289 0 814.7mb 814.7mb
green open graylog_1339 QNbCiY5YSMSmwhIMEPeeug 1 0 1925438 0 598.2mb 598.2mb
green open infra_261 m2Jq0jpeQWybKw-4hoOHhg 1 0 567153 0 758.8mb 758.8mb
red open graylog_deflector IiS44uBOSYyBW5OLD5PmzA 5 1 33009570 799 13.3gb 13.3gb
green open graylog_1340 bea8-Bb4RHqPmEhub9hAcQ 1 0 2366439 0 732.5mb 732.5mb
green open infra_262 x-H5TBBlRDOn3d115ynlNg 1 0 706197 0 905.6mb 905.6mb

Saca el listado de los que tengas en estado UNASSIGNED

1
curl -s -XGET 10.25.152.30:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | sort | less

Mira a ver cuando datos tienes esos que te aparecen en rojo / UNASSIGNED
Para el Graylog
Elimina los indices corruptos

1
curl -X DELETE http://localhost:9200/[indices-podridos]

Haz otro cat de los indices

1
curl -X DELETE http://localhost:9200/.triggered_watches

Arranca graylog
Funciona? Enhorabuena!

Comentários

18-08-2018

⬆︎TOP