Troubleshooting client issues
This document offers practical tips and insights to help you debug various problems, ensuring a seamless user experience.
NetBird agent status
The netbird agent is a daemon service that runs in the background; it provides information about peers connected and about the NetBird control services. You can check the status of the agent with the following command:
netbird status --detail
This will output the following information:
Peers detail:
server-a.netbird.cloud:
NetBird IP: 100.75.232.118/32
Public key: kndklnsakldvnsld+XeRF4CLr/lcNF+DSdkd/t0nZHDqmE=
Status: Connected
-- detail --
Connection type: P2P
Direct: true
ICE candidate (Local/Remote): host/host
ICE candidate endpoints (Local/Remote): 10.128.0.35:51820/10.128.0.54:51820
Last connection update: 20 seconds ago
Last Wireguard handshake: 19 seconds ago
Transfer status (received/sent) 6.1 KiB/20.6 KiB
Quantum resistance: false
Routes: 10.0.0.0/24
Latency: 37.503682ms
server-b.netbird.cloud:
NetBird IP: 100.75.226.48/32
Public key: Mi6jtrK5Tokndklnsakldvnsld+XeRF4CLr/lcNF+DSdkd=
Status: Connected
-- detail --
Connection type: Relayed
Direct: false
ICE candidate (Local/Remote): relay/host
ICE candidate endpoints (Local/Remote): 108.54.10.33:60434/10.128.0.12:51820
Last connection update: 20 seconds ago
Last Wireguard handshake: 18 seconds ago
Transfer status (received/sent) 6.1 KiB/20.6 KiB
Quantum resistance: false
Routes: -
Latency: 37.503682ms
OS: darwin/amd64
Daemon version: 0.27.4
CLI version: 0.27.4
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays:
[stun:turn.netbird.io:5555] is Available
[turns:turn.netbird.ioo:443?transport=tcp] is Available
Nameservers:
[8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: maycons-mbp-2.netbird.cloud
NetBird IP: 100.75.143.239/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 2/2 Connected
As you can see, the output shows the peers connected, the NetBird IP address, the public key, the connection status, and the connection type. The status will also report if there is an issue connecting to the relay servers, the management server, or the signal server.
As for Peers, the status will show the following information:
Connection type
: P2P, Relayed, where relayed connections indicate a limitation in the network that prevents a direct connection between the peers.Direct
: true/false, where true indicates a direct connection between the peers without a local proxy. This case is common when the local peer is allocating the relay connection.ICE candidate (Local/Remote)
: relay/host, where relay indicates that the local peer is using a relay connection and host indicates that the remote peer is using a direct connection.Last Wireguard handshake
: Indicating the last time the Wireguard handshake was performed. Usually, this is performed every 2 minutes, and if you don't see an update here or if the value is empty, that indicates that the connection wasn't possible yet.Transfer status (received/sent)
: Indicating the amount of data received and sent by the peer. This is useful to check if the connection is being used.
See more details about the status command here.
Getting client logs
By default, client logs are located in the /var/log/netbird/client.log
file on macOS and Linux and in the
C:\ProgramData\netbird\client.log
file on Windows.
You can analyze the logs to identify the root cause of the problem. If you need help, open a github issue and attach the logs.
Debug bundle
A debug archive containing the recent logs and the status at the time of execution can be generated with the following command.
Adding the -A
flag will anonymize the logs, removing sensitive information such as public IP addresses and domain
names. Adding the -S
flag will add system information like network routes and interfaces
netbird debug bundle -AS
This will output the path of the generated file, which can be accessed with administrative privileges.
Debug for a specific time
To capture logs for a specific time period, you can use the debug for
command. This will generate a debug bundle after
the specified time has elapsed.
netbird debug for 5m -AS
To capture any issues arising during the up
and down
processes, this will set the log level to TRACE
and bring
netbird up
and down
up to a few times.
After 5 minutes the netbird status will be restored to the previous state and the debug bundle will be generated.
Enabling debug logs on agent
Logs can be temporarily set using the following command.
netbird debug log level debug
or
netbird debug log level trace
The next time the daemon is restarted, the log level will return to the configured level.
Using netbird down
and netbird up
will not reset the log level.
To permanently set the log level, see the following sections.
On Linux with systemd
The default systemd unit file reads a set of environment variables from the path /etc/sysconfig/netbird
.
You can add the following line to the file to enable debug logs:
sudo mkdir -p /etc/sysconfig
echo 'NB_LOG_LEVEL=debug' | sudo tee -a /etc/sysconfig/netbird
sudo systemctl restart netbird
On Other Linux and MacOS
sudo netbird service stop
sudo netbird service uninstall
sudo netbird service install --log-level debug # or trace
sudo netbird service start
On Windows
You need to run the following commands with an elevated PowerShell or cmd.exe
window.
netbird service stop
netbird service uninstall
netbird service install --log-level debug # or trace
netbird service start
On Docker
You can set the environment variable NB_LOG_LEVEL
to debug
to enable debug logs.
docker run --rm --name PEER_NAME --hostname PEER_NAME --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE -d \
-e NB_SETUP_KEY=<SETUP KEY> -e NB_LOG_LEVEL=debug -v netbird-client:/etc/netbird netbirdio/netbird:latest
On Android
Enable the ADB in the developer menu on the Android device. In the app set the the Trace log level setting - it is a checkbox in the advanced menu. With the ADB tool, you can get the logs from your device. The ADB is part of the SDK platform tools pack (zip file). You can download it from here. Please extract it and run the next command in the case of Linux:
sudo adb logcat -v time | grep GoLog
Running the agent in foreground mode
You can run the agent in foreground mode to see the logs in the terminal. This is useful to debugging issues with the agent.
Linux and MacOS
sudo netbird service stop
sudo netbird up -F
Windows
On Windows, the agent depends on the Wireguard's wintun.dll
and can only be executed as a system account.
To run the agent in foreground mode, you need to use a tool
called PSExec.
Once you have downloaded and extracted psexec
open an elevated Powershell window:
netbird service stop
.\PsExec64.exe -s cmd.exe /c "netbird up -F --log-level debug > c:\windows\temp\netbird.out.log 2>&1"
In case you need to configure environment variables, you need to add them as system variables so they get picked up by the agent on the next psexec run:
[Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine")
Enabling WireGuard in user space
Sometimes, you want to test NetBird running on userspace mode instead of a kernel module. That can be a check to see if there is a problem with NetBird's firewall management in kernel mode.
You must run the agent in foreground mode and set the environment variable NB_WG_KERNEL_DISABLED
to true
.
sudo netbird service stop
sudo bash -c 'NB_WG_KERNEL_DISABLED=true netbird up -F' > /tmp/netbird.log
Debugging GRPC
The NetBird agent communicates with the Management and Signal servers using the GRPC framework. With these parameters, you can set verbose logging for this service.
sudo netbird service stop
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F' > /tmp/netbird.log
Debugging ICE connections
The Netbird agent communicates with other peers through the Interactive Connectivity Establishment (ICE) protocol
described in the RFC 8445. To debug the connection procedure,
set verbose logging for the the Pion/ICE library with the PIONS_LOG_DEBUG
or
PIONS_LOG_TRACE
variable.
Environment variable
PIONS_LOG_DEBUG=all
NB_LOG_LEVEL=debug
sudo netbird service stop
sudo bash -c 'PIONS_LOG_DEBUG=all NB_LOG_LEVEL=debug netbird up -F' > /tmp/netbird.log
Debugging access to network resources
In this section we will be presenting methodology of troubleshooting access issues involving Netbird.
We will start by presenting a glossary of all machines and services involved. A sub-section will describe a specific use case. Each will start with a concise summary of usual troubleshooting steps then expand into more detailed step-by-step guides.
Glossary
We will be using the following names for resources outside the Netbird network:
int-net1
: an internal network10.123.45.0/24
,srv-c
: an internal HTTP server running at10.123.45.17
,int-dns1
: an internal DNS server running at10.123.45.6
,int-dns2
: an internal DNS server nunning at10.7.8.9
,cf-dns
: an Internet-accessible CloudFlare DNS server at1.1.1.1
and1.0.0.1
,
and following Netbird network resources:
peer-a
: end user's device running Netbird Client,peer-b
: a linux server inside the internal network running Netbird Client,- it has direct access to the whole
int-net1
IP range,
- it has direct access to the whole
users:employees
: a Netbird Group containingpeer-a
,routers:int-net1
: a Netbird Group containingpeer-b
,access:srv-c
: a Netbird Groups used as a target of ACL rules forsrv-c
only,access:int-net1
: a Netbird Groups used as a target of ACL rules for the whole subnet,net-a
: a Netbird Networknet-a:srv-c
: a Network Resource handling traffic to10.123.45.17/32
(srv-c
),net-a:int-net1
: a Network Resource handling traffic to10.123.45.0/24
(int-net1
),
route:int-net1
: a Netbird Network Route handling traffic to10.123.45.0/24
(int-net1
),route:srv-c
: a Netbird Network Route handling traffic to10.123.45.17/32
(srv-c
),
Access from peer-a
to srv-c
peer-a
to srv-c
In short:
- Does
peer-b
have direct access tosrv-c
's port80
? - Can a routing peer
peer-b
forward traffic tosrv-c
? - Are Netbird's network routing resources configured?
- Do Netbird's Access Control rules allow access from
peer-a
topeer-b
? - Do Netbird's Access Control rules allow access from
peer-a
to the target's ACL Group? - Is
peer-a
's operating system configured to use the route?
Does peer-b
have direct access to srv-c
's port 80
?
peer-b
have direct access to srv-c
's port 80
?After logging in to peer-b
you can confirm/troubleshoot the HTTP port 80
connection by issuing any of the following
commands:
curl -v "http://10.123.45.17"
curl --fail -v --max-time=2 "http://10.123.45.17:80"
wget -O - --timeout=2 "http://10.123.45.17:80"
nc -nvz -w 2 10.123.45.17 80
You can also try ping
(an ICMP packet), but the firewall might have a different configuration for ICMP and TCP ports:
ping --numeric --count=1 --timeout=2 10.123.45.17
Can a routing peer peer-b
forward traffic to srv-c
?
peer-b
forward traffic to srv-c
?Only Linux OS machines can be assigned as routing peers.
This is more complicated to test, but usually boils down to confirming net.ipv4.ip_forward
is set to 1
on peer-b
's
Linux operating system:
> sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
For setting up the value persistently (across reboots) please consult your operating system's documentation.
It is often handled by either /etc/sysctl.conf
or /etc/sysctl.d/*.conf
files.
Testing the functionality in practice involves:
- connecting to another machine with direct access to
peer-b
, - adding a routing table entry to route
int-net1
(10.123.45.0/24
) traffic through it, - trying to at least
ping 10.123.45.17
(srv-c
)
Are Netbird's network routing resources configured?
For Netbird network routing resources configurations you can use either (new) Networks or (old) Network Routes.
A Network net-a
should have at minimum:
- Network Resource:
net-a:srv-c
with either of:- an Address set to
10.123.45.17/32
to configure route tosrv-c
exclusively and nothing else, - Assigned Groups set to
access:srv-c
- an Address set to
- Routing Peer Group assigned to
routers:int-net1
A Network Route route:srv-c
should have at least:
- a Network Range set to
10.123.45.17/32
to configure route tosrv-c
exclusively and nothing else, - Routing Peer Group assigned to
routers:int-net1
, - Distribution Group assigned to
users:employees
, - (optional) Access Control Groups assigned to
access:srv-c
,
You can loosen the rules and replace following to grant access to the whole int-net1
network range:
- Address:
10.123.45.17/32
->10.123.45.0/24
, - Assigned Groups / Access Control Groups:
access:srv-c
->access:int-net1
Do Netbird's Access Control rules allow access from peer-a
to peer-b
?
peer-a
to peer-b
?There should be an Access Control Policy present allowing traffic from users:employees
Group to routers:int-net1
Group.
You can confirm the Policy is working by:
- logging in to
peer-a
, - issuing
netbird status -d
command, - finding
peer-b.netbird.cloud
underPeers detail
,
In the most specific setup it should have at:
- have
TCP
protocol selected, - a blue arrow should point from left to right and a second right-to-left arrow should be greyed out,
- a Source group set to
users:employees
, - a Destination group set to
routers:int-net1
, - have
80
in the Ports section,
You can loosen above example by:
- allowing
ALL
protocol, Ports will become greyed out because all traffic will be allowed, - creating a bidirectional rule (both arrows should be green), always true for the protocol
ALL
, - selecting a different source group from the pool assigned to
peer-a
,- it could be built-in
All
group, but it is discouraged,
- it could be built-in
- selecting a different destination group from the pool assigned to
peer-b
,- it could be built-in
All
group, but it is discouraged,
- it could be built-in
Do Netbird's Access Control rules allow access from peer-a
to the target's ACL Group?
peer-a
to the target's ACL Group?You can skip this check, when you are using (old) Network Route feature without filling in Access Control Groups ( optional) section.
Otherwise, there should be an Access Control Policy present allowing traffic from one of peer-a
's Groups to:
- Networks Resource's Assigned Groups:
access:srv-c
oraccess:int-net1
, - Network Route's Access Control Groups:
access:srv-c
oraccess:int-net1
,
You can confirm the Policy is working by:
- logging in to
peer-a
, - issuing
netbird status -d
command, - finding
peer-b.netbird.cloud
underPeers detail
, - finding
10.123.45.0/24
or10.123.45.17/32
underpeer-b.netbird.cloud
's Networks field,
In the most specific setup it should have at:
- have
TCP
protocol selected, - a blue arrow should point from left to right and a second right-to-left arrow should be greyed out,
- a Source group set to
users:employees
, - a Destination group set to
access:srv-c
, - have
80
in the Ports section,
Just like with the previous section you can loosen the above example by:
- replacing
access:srv-c
Group withaccess:int-net1
Group, - allowing
ALL
protocol, Ports will become greyed out because all traffic will be allowed, - creating a bidirectional rule (both arrows should be green), always true for the protocol
ALL
, - selecting a different source group from the pool assigned to
peer-a
,- it could be built-in
All
group, but it is discouraged,
- it could be built-in
- selecting a different destination group from the pool assigned to
peer-b
,- it could be built-in
All
group, but it is discouraged,
- it could be built-in
Is peer-a
's operating system configured to use the route?
peer-a
's operating system configured to use the route?After all resources are configured in the Netbird management you should check whether they are properly registered with your operating system.
You can start by checking Netbird client's configuration with netbird status -d
command:
% netbird status -d
Peers detail:
brys-vm-nbt-ubuntu-isolated-01.netbird.cloud:
...
Status: Connected
-- detail --
Connection type: P2P
...
Networks: 10.123.45.0/24
...
Peers count: 1/1 Connected
You should be primarily looking for Networks section under each Peers detail, but you can also check:
- Peer's name,
- Peer's Status: it should be
Connected
, - Peer's Connection type: it can be either
P2P
(direct) orRelayed
(over the Internet), - Peers count near the end of the output,
Verifying routing configuration on the Windows operating system
Below commands assume running a PowerShell prompt with administrator's privileges.
The easiest way is to read output of Get-NetRoute
command:
PS C:\Users\user> Get-NetRoute
ifIndex DestinationPrefix NextHop RouteMetric ifMetric PolicyStore
------- ----------------- ------- ----------- -------- -----------
...
17 10.123.45.255/32 0.0.0.0 256 5 ActiveStore
17 10.123.45.0/24 0.0.0.0 1 5 ActiveStore
...
17 100.83.255.255/32 0.0.0.0 256 5 ActiveStore
17 100.83.183.133/32 0.0.0.0 256 5 ActiveStore
17 100.83.0.0/16 0.0.0.0 256 5 ActiveStore
...
You should be looking for your specific subnet's IP ranges (10.123.45.0/24
in case of int-net1
) and anything from
100.*.0.0/16
range.
Some other alternatives are route print
& Get-NetIPConfiguration
.
Verifying routing configuration on the MacOS operating system
The easiest way to verify system configuration is netstat -nr
command:
% netstat -nr
Routing tables
Internet:
Destination Gateway Flags Netif Expire
...
100.83/16 utun100 USc utun100
100.83.19.63 100.83.19.63 UH utun100
...
10.123.45 utun100 USc utun100
...
Internet6:
Destination Gateway Flags Netif Expire
...
You should be looking for utun*
interface in 4th column and searching the rows for
your specific subnet's clamped IP ranges (10.123.45
in case of int-net1
) and anything from 100.*/16
range.
Verifying routing configuration on the Linux operating system
Depending on specifics of your Linux distribution (or even your configuration of it) you should be able to use either
iproute2
or net-tools
family of network commands.
Netbird client stores it's custom routes in the routing table 7120
(or 0x1BD0
) when it's available (through
iproute2
interface).
For iproute2
(ip
, ss
tools):
ip route
to find built-in100.*.0.0/16
route,ip route show table 7120
orip route show table all
to find the specific routed networks,
For net-tools
(ifconfig
, route
, netstat
tools):
route -n
to find built-in100.*.0.0/16
route,- neither
route
nornetstat
support viewing content of custom routing tables,
Public nameservers
When you configure a Nameserver accessible from the Internet without a VPN, the Netbird client acts as a proxy to the public nameserver.
There are really just two things you can check:
- Confirm Netbird client picked up the nameserver,
- Confirm the operating system is configured to use Netbird client's proxy nameserver,
You can check the first one in operating system independent manner by:
- running
netbird status -d
, - locating the Nameserver's IP address
- confirming it is Available (it could also be timed out or in other state)
...
Nameservers:
[1.1.1.1:53, 1.0.0.1:53] for [.] is Available
...
Verifying the DNS names resolve properly in practice
Here is a short summary of commands querying nameservers for name.at.example.com
in different operating systems.
The .
at the end makes sure you are querying a fully-qualified names independent of your local network's configuration
(specifically search domains):
# MacOS
dscacheutil -q host -a name name.at.example.com.
# Windows PowerShell
Resolve-DnsName -Name name.at.example.com.
# Linux/UNIX
dig name.at.example.com.
nslookup name.at.example.com.
# Linux with systemd-resolved
resolvectl query name.at.example.com.
Verifying the nameservers are properly registered in Windows operating system
To confirm the nameservers are properly registered in Windows operating system using PowerShell:
PS C:\Users\user> Get-DnsClientNrptRule
Name : NetBird-Match
Version : 2
Namespace : {.netbird.cloud, .83.100.in-addr.arpa}
...
NameServers : 100.83.255.254
...
PS C:\Users\kdn> Get-DnsClientNrptPolicy
Namespace : .83.100.in-addr.arpa
...
NameServers : 100.83.255.254
..
Namespace : .netbird.cloud
...
NameServers : 100.83.255.254
...
You should be searching for following in the outputs of above commands:
- the
100.XXX.255.254
under Nameservers (a local proxy address of the Netbird client) .netbird.cloud
and.XXX.100.in-addr.arpa
under matching Namespace for built-in entries,.your.custom.domain.example.com
under matching Namespace for your custom domains,
Verifying the nameservers are properly registered in MacOS operating system
To confirm the nameservers are properly registered in MacOS operating system using terminal:
> scutil --dns
...
resolver #2
domain : netbird.cloud
nameserver[0] : 100.83.255.254
port : 53
flags : Supplemental, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
order : 101200
...
resolver #8
domain : 83.100.in-addr.arpa
nameserver[0] : 100.83.255.254
port : 53
flags : Supplemental, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
order : 102402
...
You should be searching for following in the outputs of above commands:
- the
100.XXX.255.254
under nameserver[N] (a local proxy address of the Netbird client) netbird.cloud
and.XXX.100.in-addr.arpa
under matching domain for built-in entries,.your.custom.domain.example.com
under matching domain for your custom domains,Reachable
underreach
field,
Verifying the nameservers are properly registered in Linux operating system
Nameserver can be configured in different ways depending on your specific distribution's configuration:
For systemd-resolved
, you can see the config with resolvectl status
,
For other configuration backends, you should see additional entries in /etc/resolv.conf
:
127.0.0.1
- default address for Netbird DNS proxy listener127.0.0.153
- fallback address for Netbird DNS proxy listener- value of
$NB_DNS_RESOLVER_ADDRESS
- a custom override for the Netbird DNS proxy listener
You can find the address Netbird client is listening by issuing one of following commands:
sudo ss -nlptu 'sport = 53' | grep netbird
sudo netstat -ltnup | grep ':53' | grep netbird
Internal nameservers
When you configure an internal Nameserver, not accessible from the Internet in addition to steps described in the previous section Public nameservers you should make sure the Nameserver's IP addresses are properly routed and accessible.
Please refer to Access from peer-a
to srv-c
section above.
To configure int-dns1
, while following Access from peer-a
to srv-c
section you should:
- substitute port
80
for port53
- substitute ip address
10.123.45.17
for10.123.45.6
,
To configure int-dns2
, while following Access from peer-a
to srv-c
section you should:
- substitute port
80
for port53
- completely ignore the
10.123.45.0/24
network instructions, - substitute ip address
10.123.45.17
for10.7.8.9
, - create a respective Network (along with Resources and Routing Peers) or Network Route for the
10.7.8.9/32
IP address range,
To test the configuration in practice please refer to previous section Public nameservers. usb-storage,drive=raw-30F4989D-AE95B60F797D,serial=30F4989D-AE95B60F797D