RabbitMQ: рукопожатие прекращено сервером (ОТКАЗ ОТ ДОСТУПА)
Я пытаюсь отправить сообщения RabbitMQ с моего хост-компьютера на экземпляр Minikube с развернутым кластером RabbitMQ.
При запуске моего сценария отправки я получаю эту ошибку:
В лог-файлах брокера я вижу такую строку:
Я уверен, что у меня есть правильные учетные данные, поскольку я получил их непосредственно из модуля RabbitMQ, следуя официальной документации (ссылка).
Мой сценарий отправки приведен ниже:
Я знаю, что код работает, поскольку я запустил тот же сценарий для моей настройки localhost, и он сработал. Единственное, что я изменил, это URL-адрес (для сервиса Minikube RabbitMQ).
Я видел несколько других сообщений, содержащих аналогичную проблему, но большинство решений связано с включением правильных учетных данных в URI, что я и сделал.
1 ответ
Вы можете использовать переадресацию порта для службы rabbitMQ на локальный компьютер и использовать вход в пользовательский интерфейс и проверять пароль с помощью пользовательского интерфейса, предоставленного самим RabbitMQ.
Должно быть достаточно
Для оформления вы можете проверить, можете ли вы войти в систему из самого контейнера. В случае сбоя входа в систему из контейнера вы также можете проверить файл yaml или схему управления, которую вы используете, на предмет методов входа и учетных данных. Простой вход может быть отключен.
Другая ситуация может быть с раздачей. При развертывании RabbitMQ я стараюсь использовать графики битнами. Я могу их предложить.
Если все это не помогает, вы можете использовать другой способ. Вы можете попробовать создать другого пользователя с правами администратора для подключения к RabbitMQ, а затем продолжать его использовать.
Для получения дополнительной информации вы можете опубликовать журналы контейнеров / подов, чтобы мы их увидели. Добрый день.
Обработка ошибок с помощью Spring AMQP
Узнайте о различных способах обработки ошибок с помощью Spring AMQP с помощью RabbitMQ.
1. введение
2. Настройка среды
Давайте запустим RabbitMQ как автономный сервер. Мы запустим его в контейнере Docker, выполнив следующую команду:
3. Сценарий Отказа
Обычно существует больше типов ошибок, которые могут возникнуть в системах обмена сообщениями по сравнению с монолитными или однопакетными приложениями из-за их распределенной природы.
Мы можем указать на некоторые типы исключений:
Конечно, этот список отказов не является исчерпывающим, но содержит наиболее распространенные типы ошибок.
В следующих разделах мы в основном сосредоточимся на специфичных для приложений и высокоуровневых ошибках, а затем рассмотрим глобальные стратегии обработки ошибок.
4. Настройка проекта
Теперь давайте определим простую конфигурацию очереди и обмена для начала:
Далее, давайте создадим простого производителя:
И, наконец, потребитель, который бросает исключение:
По умолчанию все неудачные сообщения будут немедленно запрашиваться в начале целевой очереди снова и снова.
Давайте запустим наш пример приложения, выполнив следующую команду Maven:
Теперь мы должны увидеть аналогичный результирующий результат:
Следовательно, по умолчанию мы будем видеть бесконечное количество таких сообщений в выходных данных.
Чтобы изменить это поведение у нас есть два варианта:
Теперь давайте узнаем, как обрабатывать неудачные сообщения более разумным способом.
5. Очередь Мертвых писем
Что еще более важно, это помогает предотвратить бесконечные циклы в очередях, которые постоянно обрабатывают плохие сообщения и снижают производительность системы.
Теперь давайте посмотрим, как обрабатывать исключения, применяя подход очереди мертвых писем.
5.1. Базовая конфигурация
Чтобы настроить DLQ, нам нужно указать дополнительные аргументы при определении нашей очереди:
Второй аргумент столь же важен, как и установка ключей маршрутизации для простых сообщений. Эта опция изменяет начальный ключ маршрутизации сообщения для дальнейшей маршрутизации по DLX.
5.2. Неудачная Маршрутизация Сообщений
Поэтому, когда сообщение не доставляется, оно направляется на обмен мертвыми письмами. Но, как мы уже отмечали, DLX-это нормальный обмен. Поэтому, если ключ маршрутизации неудачного сообщения не совпадает с обменом, он не будет доставлен в DLQ.
Таким образом, если мы опустим аргумент x-dead-letter-routing-key в нашем примере, неудачное сообщение застрянет в бесконечном цикле повторных попыток.
Кроме того, исходная метаинформация сообщения доступна в x-death heaven:
То приведенная выше информация доступна в консоли управления RabbitMQ обычно работает локально на порту 15672.
5.3. Обмен Мертвыми письмами
В предыдущем разделе мы видели, что ключ маршрутизации изменяется, когда сообщение направляется на обмен мертвыми буквами. Но такое поведение не всегда желательно. Мы можем изменить его, настроив DLX самостоятельно и определив его с помощью типа fanout :
Теперь, если мы запустим наш пример, неудачное сообщение должно быть доставлено в DLQ, но без изменения начального ключа маршрутизации:
5.4. Обработка Сообщений Очереди Мертвых Писем
Конечно, причина, по которой мы переместили их в очередь мертвых писем, заключается в том, что они могут быть переработаны в другое время.
Давайте определим слушателя для очереди мертвых писем:
Если мы запустим наш пример кода сейчас, то увидим вывод журнала:
Мы получили неудачное сообщение, но что нам делать дальше? Ответ зависит от конкретных системных требований, вида исключения или типа сообщения.
Например, мы можем просто запросить сообщение в исходное место назначения:
Но такая логика исключений не отличается от политики повторных попыток по умолчанию:
Общая стратегия может потребовать повторной обработки сообщения в течение n раз, а затем отклонить его. Давайте реализуем эту стратегию, используя заголовки сообщений:
5.5. Очередь на Парковку
С другой стороны, рассмотрим ситуацию, когда мы не можем просто отбросить сообщение, например, это может быть транзакция в банковской сфере. Кроме того, иногда сообщение может потребовать ручной обработки или нам просто нужно записать сообщения, которые потерпели неудачу более n раз.
Давайте теперь воплотим эту идею:
Во – вторых, давайте реорганизуем логику прослушивателя, чтобы отправить сообщение в очередь парковки:
В конце концов, нам также нужно обрабатывать сообщения, поступающие в очередь парковки:
Теперь мы можем сохранить неудачное сообщение в базе данных или, возможно, отправить уведомление по электронной почте.
Давайте проверим эту логику, запустив наше приложение:
Как видно из выходных данных, после нескольких неудачных попыток сообщение было отправлено в очередь парковки.
6. Пользовательская Обработка Ошибок
В предыдущем разделе мы видели, как обрабатывать сбои с помощью выделенных очередей и обменов. Однако иногда нам может потребоваться отловить все ошибки, например, для регистрации или сохранения их в базе данных.
6.1. Глобальный обработчик ошибок
Проще говоря, Условное отклонение ErrorHandler решает, отклонять ли конкретное сообщение или нет. Когда сообщение, вызвавшее исключение, отклоняется, оно не запрашивается.
6.2. Стратегия Фатальных Исключений
По умолчанию эти исключения фатальны:
Вместо реализации интерфейса Ierrorhandler мы можем просто предоставить нашу стратегию Фатальных исключений :
Наконец, нам нужно передать нашу пользовательскую стратегию конструктору Conditional Rejecting ErrorHandler :
7. Заключение
В этом уроке мы обсудили различные способы обработки ошибок при использовании Spring AMQP и, в частности, RabbitMQ.
Каждая система нуждается в определенной стратегии обработки ошибок. Мы рассмотрели наиболее распространенные способы обработки ошибок в архитектуре, управляемой событиями. Кроме того, мы убедились, что можем объединить несколько стратегий для создания более комплексного и надежного решения.
Error on amqp connection что это за ошибка
Development of a robust application, be it message publisher or message consumer, involves dealing with multiple kinds of failures: protocol exceptions, network failures, broker failures and so on. Correct error handling and recovery is not easy. This guide explains how the amqp gem helps you in dealing with issues like
This work is licensed under a Creative Commons Attribution 3.0 Unported License (including images and stylesheets). The source is available on Github.
Covered versions
This guide covers Ruby amqp gem 1.5.x.
Code examples
There are several examples in the git repository dedicated to the topic of error handling and recovery. Feel free to contribute new examples.
Initial broker connection failures
When applications connect to the broker, they need to handle connection failures. Networks are not 100% reliable, even with modern system configuration tools like Chef or Puppet misconfigurations happen and the broker might also be down. Error detection should happen as early as possible. There are two ways of detecting TCP connection failure, the first one is to catch an exception:
AMQP.connect (and AMQP.start ) will raise AMQP::TCPConnectionFailed if a connection fails. Code that catches it can write to a log about the issue or use retry to execute the begin block one more time. Because initial connection failures are due to misconfiguration or network outage, reconnection to the same endpoint (hostname, port, vhost combination) will result in the same issue over and over. TBD: failover, connection to the cluster.
An alternative way of handling connection failure is with an errback (a callback for specific kind of error):
If you connect to the broker from code in a class (as opposed to top-level scope in a script), Object#method can be used to pass object method as a handler instead of a Proc.
Authentication failures
Another reason why a connection may fail is authentication failure. Handling authentication failure is very similar to handling initial TCP connection failure:
Default handler
default handler raises AMQP::PossibleAuthenticationFailureError :
In case you are wondering why callback name has «possible” in it: AMQP 0.9.1 spec requires broker implementations to simply close TCP connection without sending any more data when an exception (such as authentication failure) occurs before AMQP connection is open. In practice, however, when broker closes TCP connection between successful TCP connection and before AMQP connection is open, it means that authentication has failed.
Handling network connection interruptions
Sometimes it is necessary for other entities in an application to react to network failures. amqp gem 0.8.0 and later provides a number of event handlers to make this task easier for developers. This set of features is known as the «shutdown protocol” (the word «protocol” here means «API interface” or «behavior”, not network protocol).
The Shutdown protocol revolves around two events:
In this section, we will concentrate on the former. When a network connection fails, the underlying networking library detects it and runs a piece of code on AMQP::Session to handle it. That, in turn, propagates this event to channels, channels propagate it to exchanges and queues, queues propagate it to their consumers (if any). Each of these entities in the object graph can react to network interruption by executing application-defined callbacks.
Shutdown Protocol methods on AMQP::Session
The difference between these methods is that AMQP::Session#on_tcp_connection_loss is used to define a callback that will be executed once when TCP connection fails. It is possible that reconnection attempts will not succeed immediately, so there will be subsequent failures. To react to those, AMQP::Session#on_connection_interruption method is used.
The first argument that both of these methods yield to the handler that your application defines is the connection itself. This is done to make sure that you can register Ruby objects as handlers, and they do not have to keep any state around (for example, connection instances):
Note that AMQP::Session#on_connection_interruption callback is called before this event is propagated to channels, queues and so on.
Different applications handle connection failures differently. It is very common to use AMQP::Session#reconnect method to schedule a reconnection to the same host, or use AMQP::Session#reconnect_to to connect to a different one. For some applications it is OK to simply exit and wait to be restarted at a later point in time, for example, by a process monitoring system like Nagios or Monit.
Shutdown Protocol methods on AMQP::Channel
Note that AMQP::Channel#on_connection_interruption callback is called after this event is propagated to exchanges, queues and so on. Right after that channel state is reset, except for error handling/recovery-related callbacks.
Many applications do not need per-channel network failure handling.
Shutdown Protocol methods on AMQP::Exchange
Many applications do not need per-exchange network failure handling.
Shutdown Protocol methods on AMQP::Queue
Note that AMQP::Queue#on_connection_interruption callback is called after this event is propagated to consumers.
Many applications do not need per-queue network failure handling.`
Shutdown Protocol methods on AMQP::Consumer
Many applications do not need per-consumer network failure handling.`
Recovering from network connection failures
Detecting network connections is nearly useless if an AMQP-based application cannot recover from them. Recovery is the hard part in «error handling and recovery”. Fortunately, the recovery process for many applications follows one simple scheme that the amqp gem can perform automatically for you.
The recovery process, both manual and automatic, always begins with re-opening an AMQP connection and then all the channels on that connection.`
Manual recovery
Similarly to the Shutdown Protocol, the amqp gem entities implement a Recovery Protocol. The Recovery Protocol consists of three methods that connections, channels, queues, consumers and exchanges all implement:
AMQP::Session#before_recovery lets application developers register a callback that will be executed after TCP connection is re-established but before AMQP connection is reopened. AMQP::Session#after_recovery is similar except that the callback is run after AMQP connection is reopened.
Recovery process for AMQP applications usually involves the following steps:
Automatic recovery
Many applications use the same recovery strategy that consists of the following steps:
The amqp gem provides a feature known as «automatic recovery” that is opt-in (not opt-out, not used by default) and lets application developers get the aforementioned recovery strategy by setting one additional attribute on AMQP::Channel instance:
A more verbose way to do the same thing:
Auto recovery mode can be turned on and off any number of times during channel life cycle, although a very small percentage of applications actually do this. Typically you decide what channels should be using automatic recovery during the application design stage.
Full example (run it, then shut down AMQP broker running on localhost, then bring it back up and use management tools such as rabbitmqctl to see that queues, bindings and consumers have all recovered):
Server-named queues, when recovered automatically, will get new server-generated names to guarantee there are no name collisions.
When in doubt, try using automatic recovery first. If it is not sufficient for your application, switch to manual recovery using events and callbacks introduced in the «Manual recovery” section.
Detecting broker failures
AMQP applications see broker failure as TCP connection loss. There is no reliable way to know whether there is a network problem or a network peer is down.
AMQP connection-level exceptions
Handling connection-level exceptions
Connection-level exceptions are rare and may indicate a serious issue with a client library or in-flight data corruption. The AMQP 0.9.1 specification mandates that a connection that has errored cannot be used any more and must be closed. In any case, your application should be prepared to handle this kind of error. To define a handler, use AMQP::Session#on_error method that takes a callback and yields two arguments to it when a connection-level exception happens:
Status codes are similar to those of HTTP. For the full list of error codes and their meaning, consult AMQP 0.9.1 constants reference.
Only one connection-level exception handler can be defined per connection instance (the one added last replaces previously added ones).
Handling graceful broker shutdown
When an AMQP broker is shut down, it properly closes connections first. To do so, it uses connection.close AMQP method. AMQP clients then need to check if the reply code is equal to 320 (CONNECTION_FORCED) to distinguish graceful shutdown. With RabbitMQ, when broker is stopped using
reply_text will be set to
Each application chooses how to handle graceful broker shutdowns individually, so amqp gem’s automatic reconnection does not cover graceful broker shutdowns. Applications that want to reconnect when broker is stopped can use AMQP::Session#periodically_reconnect like so:
Once AMQP connection is re-opened, channels in automatic recovery mode will recover just like they do after network outages.
Integrating channel-level exceptions handling with object-oriented Ruby code
Error handling can be easily integrated into object-oriented Ruby code (in fact, this is highly encouraged). A common technique is to combine Object#method and Method#to_proc and use object methods as error handlers:
Full example that uses objects:
AMQP channel-level exceptions
Handling channel-level exceptions
Channel-level exceptions are more common than connection-level ones. They are handled in a similar manner, by defining a callback with AMQP::Channel#on_error method that takes a callback and yields two arguments to it when a channel-level exception happens:
Status codes are similar to those of HTTP. For the full list of error codes and their meaning, consult AMQP 0.9.1 constants reference.
Only one channel-level exception handler can be defined per channel instance (the one added last replaces previously added ones).`
Integrating channel-level exceptions handling with object-oriented Ruby code
Error handling can be easily integrated into object-oriented Ruby code (in fact, this is highly encouraged).A common technique is to combine Object#method and Method#to_proc and use object methods as error handlers. For example of this, see section on connection-level exceptions above.
Because channel-level exceptions may be raised because of multiple unrelated reasons and often indicate misconfigurations, how they are handled isvery specific to particular applications. A common strategy is to log an error and then open and use another channel.
Common channel-level exceptions and what they mean
A few channel-level exceptions are common and deserve more attention.
406 Precondition Failed
405 Resource Locked
403 Access Refused
Conclusion
Distributed applications introduce a whole new class of failures developers need to be aware of. Many of them stem from unreliable networks. The famous Fallacies of Distributed Computing list common assumptions software engineers must not make:
Unfortunately, applications that use Ruby and AMQP are not immune to these problems and developers need to always keep that in mind. This list is just as relevant today as it was in the 90s.
Ruby amqp gem 0.8.x and later lets applications define handlers that handle connection-level exceptions, channel-level exceptions and TCP connection failures. Handling AMQP exceptions and network connection failures is relatively easy. Re-declaring AMQP instances that the application works with is where most of the complexity comes from. By using Ruby objects as error handlers, the declaration of AMQP entities can be done in one place, making code much easier to understand and maintain.
amqp gem error and interruption handling is not a copy of RabbitMQ Java client’s Shutdown Protocol, but they turn out to be similar with respect to network failures and connection-level exceptions.
Tell Us What You Think!
Please take a moment to tell us what you think about this guide on Twitter or the RabbitMQ mailing list
Let us know what was unclear or what has not been covered. Maybe you do not like the guide style or grammar or discover spelling mistakes. Reader feedback is key to making the documentation better.
This website was developed by the Ruby AMQP Team.
Comments
alex9434 commented Oct 27, 2019 •
report a bug?
What is the current behavior?
Fresh onlyoffice installation only starts with white screen with message «Document Server is running». No access to document server possible.
/var/log/onlyoffice/documentserver/docservice/out.log shows the following error:
RabbitMQ is installed and seems to be working:
DocumentServer version:
5.4
Operating System:
Debian Stretch
Browser version:
Firefox 70.0
The text was updated successfully, but these errors were encountered:
ShockwaveNN commented Nov 7, 2019
Hi, i checked installation on Debian 9 using this Docker Image
https://github.com/onlyoffice-testing-robot/documentserver-install-from-manual/blob/master/Debian-9/Dockerfile
And everything seems fine for me, RabbitMQ logs looks same, docservice log have no error.
Maybe it some permission or firewall issue?
alex9434 commented Nov 9, 2019
Probably it was. I managed to get it working after some try and error. Unfortunately I cannot replicate the error. I will therefore close this. Thank you for getting back.
brandan-schmitz commented Nov 19, 2019
I am having the same issue. I have disabled all firewalls and permissions for the networks and have verified the port it open and RabbitMQ is listening same as above. I am running this is a Debian 9 container LXC container on a Proxmox VE server. I am able to access the web page and it shows a green check saying that the service is running. Running supervisorctl status all produces the following, and nothing else is in the logs besides the error Error: connect ECONNREFUSED 127.0.0.1:5672 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
alex9434 commented Nov 21, 2019
Hi @Celestialdeath99, I also run this in an LXC container. I was orginally on Debian 10 when I had the error. I then re-installed on Debian 9 (or in my case Devuan 9) and everything is OK now. It might be the version of RabbitMQ, that is causing the issue.
voicenoise commented Mar 9, 2020
have a similar thing on ubuntu 18.04 by updating the system..
and now a endless loop since hours:
camr0 commented Mar 16, 2020
ShockwaveNN commented Mar 16, 2020
@voicenoise @camr0 Please create a new issue for your problem
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Cannot close connection #45
Comments
malinoff commented Jul 5, 2014
The text was updated successfully, but these errors were encountered:
malinoff commented Jul 5, 2014
Also, I can’t just create a channel:
malinoff commented Jul 5, 2014
vovkd commented Oct 4, 2014
same problem with amqp v2.0.0а1
here is a rabbitmq log
fake-name commented Jun 13, 2016 •
I’m seeing this too. Does anyone have a workaround?
RabbitMQ 3.6.2, Erlang 18.2
fake-name commented Jun 15, 2016 •
Ok, I’m seeing this a LOT, and it only started occurring after I upgraded from 1.4.8 to 2.0.2.
Additionally, I’m having a lot of issue keeping long-lived connections alive as well, though that’s not a completely new issue (it’s gotten worse, though). I regularly have a connection get wedged and stop processing messages at this point.
I can have an open connection, with heartbeats _and TCP keepalives forced on at the server end_ get wedged and never recover or throw any timeouts indefinitely. It’s really a problem.
fake-name commented Jun 29, 2016 •
I’ve been looking at fixing py-amqp for py3k as a possible alternative, since I’ve had /LOTS/ of issues with py-amqp, but the codebase for py-amqp is spectacularly bad. Apparently it generates C files by printing to stdout from python and piping that into temporary files, with loops and dynamically constructed C functions, and it’s basically impossible to follow.
Are there any actually maintained python AMQP libraries?
jakeczyz commented Jul 12, 2016 •
I’m also seeing these after upgrading from amqp 1.4.7 to 2.0.2; still on Python 3.4. I’m running erlang-18.3-1 and rabbitmq-server-3.6.2-1 on the server. This is a major issue for my system. Any help would be appreciated here. Barring quick help, I’d also like to know if this module is being maintained or if I need to switch to another (though neither pika nor haigha seem to have recent commits; which is troubling).
fake-name commented Jul 12, 2016 •
ask commented Jul 12, 2016 •
Sorry crowd! These emails are lost in a sea of mail and you were commenting on an issue that I thought were to only happen in the celery/py-amqp@callbacks branch. Sadly the Github Issue interface does not move most recently discussed issues to the top.
Celery itself is ignoring exceptions when the connection is closed, so was not seen by our integration tests. Celery itself does not depend on amqp 2.0 yet, we will first move to that version in Celery 4.0.
@fake-name I’m guessing you’re actually talking about librabbitmq here?
librabbitmq does not generate any C, but that’s maybe what rabbitmq-c, the underlying library used is doing. py-amqp is in general more stable than librabbitmq, not sure what issues you are having? Please do create new issues for anything you are seeing, but py-amqp 1.4.x is probably a better bet if 2.x is having issues.
py-amqp is a low level component, I don’t think there is any reason to use it directly as you are most likely to require the features defined in Kombu anyway (connection pools, serialization, failover, retrying operations, declaration cache, compression, etc, etc). The current stable version of Kombu is still depending on amqp 1.4.
ask commented Jul 12, 2016
amqp 2.0.3 is now available at PyPI and fixes the issue.
fake-name commented Jul 12, 2016 •
For this discussion, I was talking about having switched to rabbitpy: https://github.com/gmr/rabbitpy, which seems a lot more approachable (or at at least marginally maintained).
It also uses a completely independent low-level AMQP parsing library (pamqp) that they’re apparently looking to migrate pika to as well, so it has a decent amount of attention on it.
fake-name commented Jul 12, 2016 •
Anyways, basically implemented a bunch of stuff on top of celery initially, had horrible annoying issues everywhere, dumped celery entirely and doing so got rid of
95% of the problems. It was (I think) about two years ago, but basically after a few weeks hacking on it, I was unable to get it to be reliable, and it made lots of assumptions about my task structure that weren’t correct for my particular applcation
Fundamentally, I need a messaging library, not a task system-cum-entire-house-to-hold-a-kitchen-sink-and-as-a-precaution-another-spare-kitchen-sink system.
The system I have that executes tasks and handles marshaling the results provides retry logic and all the associated management stuff itself (because that is extremely closely bound to how the result data is stored), and from what I could see there isn’t (or at least wasn’t) any way to turn celery’s re-implementation of it off. I also needed much more fine control of per-remote-client session and persistence, so I wound up needing to basically use celery as a library, rather then a stand-alone client thing, which was (and may still be) entirely undocumented when I was looking at it, which certainly didn’t help.
fake-name commented Jul 12, 2016 •
Celery itself is ignoring exceptions when the connection is closed, so was not seen by our integration tests.
The lifetime of an application is not the lifetime of a connection, or if you are assuming that, you’re making a hell of an assumption about how everyone’s application works.
If at least some your unit tests aren’t testing the connection teardown, you’re not testing a significant and important facet of the library.
ask commented Jul 12, 2016 •
Yes. In the worker, the only reason for the connection to be closed is if
If you think that we should add a test to test pure py-amqp behavior, then I welcome you to contribute this test.
Thank you for the snarky description of my work, that so helpfully illustrates how useless the collective effort to collect these common patterns in a single library is. I was not suggesting you use Celery though, I was suggesting you use Kombu, which is a generic messaging library using the AMQP API directly (not even any abstraction over the Channel).
