This post was most recently updated on April 28th, 2022.
4 min read.Are you struggling with your Azure IoT Edge modules not receiving your Direct Method calls? Is EdgeHub reporting transient network issues? Are you seeing “Could not connect to your IoT hub: could not complete TLS handshake” errors? Are your modules experiencing constant 104 errors (connection reset by peer)?
Fear not! This article explains how to fix them. Kind of.
Problem
We were experiencing some major issues with our customer’s Edge devices. Direct Method calls would mostly not go through, message routing between Edge and the IoT Hub itself seemed to be down, restarting the modules did nothing, and even reading logs were extremely sluggish.
So, it was investigation time!
Pretty much the first thing that we did was to run a Sudo IoT edge check on the device, and we got something like this:
Connectivity checks (aziot-identity-service)
--------------------------------------------
× host can connect to and perform TLS handshake with iothub AMQP port - Error
Could not connect to your-iot-hub.azure-devices.net : could not complete TLS handshake
caused by: Connection reset by peer (os error 104)
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
× host can connect to and perform TLS handshake with iothub MQTT port - Error
Could not connect to your-iot-hub.azure-devices.net : could not complete TLS handshake
caused by: Connection reset by peer (os error 104)
But still, annoyingly, it might report successful connectivity checks:
Connectivity checks ------------------- √ container on the default network can connect to upstream AMQP port - OK √ container on the default network can connect to upstream HTTPS / WebSockets port - OK √ container on the default network can connect to upstream MQTT port - OK √ container on the IoT Edge module network can connect to upstream AMQP port - OK √ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK √ container on the IoT Edge module network can connect to upstream MQTT port - OK
What gives?
Reason
Laugh all you want, but this one was caused by a customer’s very independent, somewhat uncooperative, and a rather stubborn IT Department.
Whatever weird firewall/network configuration change they pushed through blocked the SSL/TLS negotiation for ports other than 443. That might also apply to your case.
You can try to investigate this by running OpenSSL s_client for your different protocols – you should get a successful connection for port 443 (HTTPS/WebSockets) but unsuccessful calls for ports 5671 and 8883.
In short, this call:
timeout 1 openssl s_client -connect your-iot-hub.azure-devices.net:443
Should return something like this:
CONNECTED(00000005) depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root verify return:1 depth=1 C = US, O = Microsoft Corporation, CN = Microsoft RSA TLS CA 02 verify return:1 depth=0 CN = *.azure-devices.net verify return:1 --- Certificate chain 0 s:CN = *.azure-devices.net i:C = US, O = Microsoft Corporation, CN = Microsoft RSA TLS CA 02 1 s:C = US, O = Microsoft Corporation, CN = Microsoft RSA TLS CA 02 i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root --- Server certificate -----BEGIN CERTIFICATE-----
Which is good! But both of these:
timeout 1 openssl s_client -connect your-iot-hub.azure-devices.net:5671 timeout 1 openssl s_client -connect your-iot-hub.azure-devices.net:8883
will return something like this:
CONNECTED(00000005) write:errno=104 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 340 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---
This is not quite as good, as it turns out for whatever reason, even though the port is open, you’re unable to access the SSL/TLS certificate and perform the handshake.
Solution
I’d have loved to actually solve the issue, but since that didn’t work out (at least at the time of writing this article), we had to find a workaround. And a workaround we found!
I guess that was obvious from the get-go because I wouldn’t be writing this article otherwise :) Or would I..?
Time needed: 15 minutes
How to configure Azure IoT Edge Hub to use AMQP or MQTT over WebSockets (using port 443)?
- Navigate to your Edge device
Easy to guess, sure, but it needs to be said. Azure Portal > IoT Hub -> Device management -> IoT Edge.
- Select “Set modules”
That’s in the “ribbon” for device management.
- Select “Runtime Settings”
This’ll let you access the edgeAgent and edgeHub’s settings!
- Add a new Environment variable
Variable name: UpstreamProtocol
Variable value: AMQPWS (or MQTTWS)
- Apply your changes
Okay, so now you need to apply your configuration change. That’s reasonably easy:
Click Apply -> “Review + create” -> “Create”
And that’s it! Our issue was resolved (well… worked around, I suppose), how about yours? Let me know in the comments -section below!
References
- This GitHub issue (more or less) explains the issue and the solution:
- Major props to my colleague, Juha, who knows most of the things one can know about Azure IoT Edge stuff – figured out the details with him:
- Port configuration and required firewall openings from Docs.Microsoft.com:
He's been a developer from 2004 (starting with PHP and Java), and he's been working on .NET projects, Azure, Office 365, SharePoint and a lot of other stuff. He's also Microsoft MVP for Azure.
This is his personal professional (e.g. professional, but definitely personal) blog.
- CSOM suddenly throwing exceptions when trying to access list contents in SharePoint? A weird fix. - April 2, 2024
- “Predefined type ‘System.Object’ is not defined or imported” and other System namespace stuff missing in your solution? - March 26, 2024
- How to import GraphQL schema to Postman in Windows? - March 19, 2024
