RouterOS Troubleshooting Methodology

RouterOS Troubleshooting Methodology

This document provides a unified framework for troubleshooting RouterOS devices. Rather than focusing on specific features, it presents a systematic methodology applicable to any network issue you encounter.

The Systematic Approach

Effective troubleshooting follows a structured methodology:

1. Define the problem - What exactly is happening vs. what should happen
2. Gather information - Use diagnostic tools to collect data
3. Form hypotheses - Identify potential causes based on symptoms
4. Test systematically - Isolate variables to confirm or eliminate causes
5. Implement the fix - Apply the solution
6. Verify and document - Confirm the issue is resolved

Diagnostic Framework

Layer-by-Layer Analysis

Network issues often manifest at specific layers. Starting from the bottom and working up helps isolate problems efficiently:

Layer	What to Check	RouterOS Commands
Physical	Cable, link lights, power	/interface ethernet print, /system resource print
Data Link (L2)	MAC addresses, VLANs, switching	/bridge port print, /interface bridge vlan print
Network (L3)	IP addresses, routing, ARP	/ip address print, /ip route print, /ip arp print
Transport (L4)	Ports, connection states	/ip firewall connection print, /ip firewall nat print
Application (L7)	Application protocols, services	/ip service print, /tool torch

The Isolation Principle

When troubleshooting, isolate variables by changing one thing at a time:

• Test with minimal configuration
• Disable features that might interfere (FastTrack, firewall rules)
• Test with default settings before applying custom configurations
• Use “divide and conquer” - eliminate half the variables each iteration

Essential Diagnostic Tools

RouterOS provides built-in tools for every troubleshooting scenario:

Torch - Real-Time Traffic Monitor

Use torch to see active traffic flows through an interface in real-time:

/tool torch interface=ether1

Filter specific traffic:

/tool torch interface=ether1 src-address=10.0.0.50
/tool torch interface=ether1 protocol=tcp port=443

Caution

Torch shows traffic BEFORE firewall filtering. Packets visible in torch may be dropped by firewall rules.

Packet Sniffer - Detailed Capture

For deeper analysis, capture packets to a file for Wireshark analysis:

/tool sniffer start interface=ether1
# Wait for captures
/tool sniffer stop
/tool sniffer save file-name=capture.pcap

Quick capture without saving to file:

/tool sniffer quick ip-protocol=tcp

View capture statistics without saving:

/tool sniffer protocol print
/tool sniffer host print
/tool sniffer connection print

Note

Captured packets are retained in memory for 10 minutes. Save immediately if you need to keep them.

Bandwidth Test - Throughput Verification

Test actual throughput between routers or to a remote host:

# Test upload (this router transmits to the target)
/tool bandwidth-test address=10.0.0.2 direction=transmit

# Test download (this router receives from the target)
/tool bandwidth-test address=10.0.0.1 direction=receive

Bidirectional test:

/tool bandwidth-test address=10.0.0.1 direction=both

TCP vs UDP testing:

/tool bandwidth-test address=10.0.0.1 protocol=tcp
/tool bandwidth-test address=10.0.0.1 protocol=udp

Netwatch - Host Monitoring

Netwatch monitors host availability using ICMP probes:

/tool netwatch add host=8.8.8.8 interval=30s timeout=1s up-script="/log info host is up" down-script="/log warning host is down"

View netwatch status:

/tool netwatch print

Log Analysis

Logs contain critical diagnostic information:

/log print

Filter by topic:

/log print topic=firewall
/log print topic=error
/log print topic=warning

Enable specific logging topics:

/system logging add topics=firewall action=memory
/system logging add topics=interface action=memory
/system logging add topics=error action=memory

Connection Tracking

View active connections to understand what’s flowing through the router:

/ip firewall connection print

Filter to see specific traffic:

/ip firewall connection print where dst-address~"10.0.0."
/ip firewall connection print where protocol=tcp
/ip firewall connection print where connection-state=syn-sent

View NAT rules:

/ip firewall nat print

Common Troubleshooting Paths

No Connectivity - “Can’t Reach Destination”

Follow this systematic approach:

1. Verify local interface status:

   /interface print
   /interface ethernet print

2. Check if there’s a link:

   /interface ethernet monitor ether1

3. Test basic connectivity - ping the gateway:

   /ping 10.0.0.1

4. Check IP configuration:

   /ip address print
   /ip route print

5. Verify ARP resolution:

   /ip arp print

6. Check firewall rules:

   /ip firewall filter print
   /ip firewall connection print

7. Check NAT rules:

   /ip firewall nat print

8. Use traceroute to identify where packets stop:

   /tool traceroute 8.8.8.8

Slow Performance

Performance issues require comparison against a baseline:

1. Check CPU usage:

   /system resource print
   /system resource cpu print

2. Monitor interface statistics:

   /interface ethernet print
   /interface print

   Look for: rx-broadcast, tx-broadcast, rx-packet-drop, tx-queue-drop

3. Check for interface errors:

   /interface monitor-traffic ether1

4. Use torch to identify traffic patterns:

   /tool torch interface=ether1

5. Check queue usage:

   /queue simple print
   /queue tree print

6. Verify hardware offloading status:

   /interface bridge print

   Look for the H flag indicating hardware offloading

Intermittent Issues

Intermittent problems are challenging because the issue isn’t present when you look for it:

1. Enable logging:

   /system logging add topics=error action=memory
   /system logging add topics=warning action=memory

2. Use netwatch to monitor connectivity:

   /tool netwatch add host=8.8.8.8 interval=10s

3. Monitor resource usage over time:

   /system resource monitor interval=5

4. Check for traffic bursts with torch:

   /tool torch interface=ether1 duration=30

5. Review logs immediately after an incident:

   /log print

FastTrack Impact on Troubleshooting

FastTrack is a high-performance feature that significantly affects troubleshooting methodology.

What FastTrack Does

FastTrack marks connections for accelerated processing by bypassing:

• Firewall filter rules (for matched connections)
• Connection tracking (after initial marking)
• Queue processing
• IPsec processing (for matched connections)

Key Troubleshooting Implication

When troubleshooting connectivity or performance issues, always disable FastTrack first:

/ip firewall filter disable [find action=fasttrack-connection]

This ensures:

• All traffic goes through firewall rules
• Connection tracking data is complete
• Queue rules apply to all traffic
• Logs capture all connections

Re-enable FastTrack After Resolution

Once the issue is resolved, re-enable FastTrack for performance:

/ip firewall filter enable [find action=fasttrack-connection]

FastTrack and Hardware Offloading

When using L3 Hardware Offloading with FastTrack:

/interface bridge port print

Look for the H flag indicating hardware-offloaded connections.

Caution

FastTrack bypasses firewall rules. If you need to filter or control traffic, ensure those rules are placed BEFORE the FastTrack rule, or use connection-state-based rules.

FastTrack and VPN/IPsec

FastTrack can interfere with IPsec. If IPsec is not working:

1. Ensure IPsec firewall rules come BEFORE FastTrack:

   /ip firewall filter add action=accept chain=forward ipsec-policy=in,ipsec protocol=esp
   /ip firewall filter add action=accept chain=forward ipsec-policy=out,ipsec protocol=esp

2. Or exclude IPsec from FastTrack entirely by not including connection-state=established,related with FastTrack for IPsec traffic.

Quick Reference

Essential Commands Summary

Task	Command
Check interface status	/interface print
Monitor traffic	/tool torch interface=X
Capture packets	/tool sniffer start interface=X
Test connectivity	/ping X.X.X.X
Trace route	/tool traceroute X.X.X.X
Check connections	/ip firewall connection print
View logs	/log print
Monitor resources	/system resource monitor
Test bandwidth	/tool bandwidth-test address=X
Monitor hosts	/tool netwatch add host=X

Layer-by-Layer Checklist

• Physical: Cabling, power, link lights
• L2: MAC table, bridge ports, VLANs
• L3: IP addresses, routes, ARP
• L4: Ports, NAT, connection tracking
• L7: Application traffic, services

Related Documentation

• Diagnostics, Monitoring and Troubleshooting
• Packet Flow in RouterOS
• Firewall Filter Documentation
• L3 Hardware Offloading
• FastTrack Configuration