mmooresystems

Tag: iquery

  • F5 iQuery: The Silent Protocol That Makes GTM Actually Work

    If you’ve ever configured F5 GTM, set up an LTM HA pair, or joined BIG-IP devices into a Device Service Cluster, you’ve used iQuery—even if you didn’t realize it. iQuery is F5’s proprietary communication protocol that enables BIG-IP devices to discover each other, exchange configuration data, share health status, and synchronize state. It’s the invisible backbone of nearly every multi-device F5 deployment, yet it’s often overlooked until something breaks. Let’s explore what iQuery actually is, where it’s used, and why it matters.

    What Is iQuery?

    iQuery is F5’s proprietary protocol for BIG-IP device-to-device communication. It’s the universal language that allows BIG-IP systems to discover each other, establish trust, exchange data, and coordinate operations—regardless of whether they’re LTMs, GTMs, or any other BIG-IP module.

    Technical Details

    • Protocol: Encrypted TCP-based communication
    • Default Port: TCP 4353
    • Encryption: SSL/TLS with certificate-based mutual authentication
    • Scope: Device trust, config sync, health monitoring, state sharing
    • Firewall Requirements: Must allow TCP 4353 between all BIG-IP devices that need to communicate

    Think of iQuery as the nervous system connecting all your BIG-IP devices. It’s how they talk to each other, trust each other, and coordinate their actions.

    Where iQuery Is Used

    iQuery powers multiple critical F5 features across different deployment scenarios:

    1. LTM High Availability (Device Service Clustering)

    Use Case: Active-Standby or Active-Active LTM pairs

    When you set up an LTM HA pair, iQuery handles:

    • Device trust establishment: Initial pairing and certificate exchange
    • Configuration synchronization: Keeping both devices’ configs identical
    • Failover coordination: Detecting failures and triggering failover
    • Connection mirroring setup: Synchronizing connection tables for stateful failover

    Example Scenario:

    1. You create a virtual server on the active LTM
    2. iQuery synchronizes that configuration to the standby LTM
    3. Both devices now have identical configs
    4. If active fails, standby takes over seamlessly

    Without iQuery: Your HA pair can’t sync configs, coordinate failover, or mirror connections. You’d have to manually configure both devices and hope they stay in sync.

    2. GTM to LTM Communication

    Use Case: Global load balancing with GTM managing remote LTM pools

    This is where iQuery becomes highly visible and absolutely critical:

    The Scenario: GTM in New York making global load balancing decisions for LTM pools in:

    • New York data center (local LTM)
    • London data center (remote LTM)
    • Singapore data center (remote LTM)

    How iQuery enables this:

    1. GTM establishes iQuery connections to all three LTMs
    2. Each LTM reports pool member health status via iQuery
    3. LTMs share performance metrics (connections, throughput, response times)
    4. GTM uses this real-time data to make intelligent routing decisions

    Without iQuery: GTM has no idea if London’s web servers are down or Singapore is experiencing high latency. It would blindly send traffic to dead pools.

    3. GTM to GTM Synchronization

    Use Case: Redundant GTM pairs (active-active or active-standby)

    iQuery synchronizes between GTM devices:

    • Configuration changes: Wide IPs, pools, data centers
    • Wide IP states: Enabled/disabled status
    • Topology records: Geographic routing rules
    • Listener decisions: DNS query handling

    4. Device Trust and Discovery

    Use Case: Any multi-device BIG-IP deployment

    Before BIG-IP devices can work together, they must establish trust via iQuery:

    1. Administrator initiates device discovery
    2. Devices exchange SSL certificates via iQuery
    3. Mutual authentication validates both devices
    4. Trust relationship established
    5. Devices can now sync configs, share data, coordinate operations

    This certificate-based trust is the foundation for all other iQuery functionality.

    How iQuery Works: A Deep Dive

    Step 1: Certificate Exchange and Trust

    Every BIG-IP device has a unique SSL certificate. When you add a device to a trust domain or Device Service Cluster:

    1. Discovery: You specify the remote device’s IP address
    2. Connection: Device A connects to Device B on TCP 4353
    3. Certificate Exchange: Both devices share their SSL certificates
    4. Validation: Each device validates the other’s certificate
    5. Trust Established: Encrypted iQuery channel is now active

    This mutual authentication ensures only authorized BIG-IP devices can participate in the cluster.

    Step 2: Ongoing Communication

    Once trust is established, iQuery carries different types of data depending on the use case:

    For LTM HA:

    • Configuration changes (immediate sync)
    • Heartbeat signals (continuous)
    • Failover state (event-driven)
    • Connection mirror data (if enabled)

    For GTM → LTM:

    • Virtual server status (polling, typically every few seconds)
    • Pool member health (continuous monitoring)
    • Performance metrics (periodic updates)
    • System resources (CPU, memory, connections)

    Step 3: Encrypted Transport

    All iQuery traffic is encrypted with SSL/TLS, so:

    • Configuration data can’t be intercepted
    • Health status remains confidential
    • Performance metrics are protected
    • Only trusted devices can decrypt the data

    Configuration Examples

    Example 1: Setting Up LTM HA (Device Trust)

    On Device A (192.168.1.10):

    # Add Device B to the trust domain
tmsh modify cm device-group device_trust_group devices add { device-b.example.com }
tmsh run cm config-sync to-group device_trust_groupCode language: PHP (php)

    Behind the scenes:

    1. Device A initiates iQuery connection to Device B (192.168.1.11:4353)
    2. Certificates exchanged and validated
    3. Device trust established
    4. Configuration sync begins via iQuery

    Example 2: Adding LTM Servers to GTM

    On GTM:

    # Create datacenter
tmsh create gtm datacenter NYC address 10.1.1.1

# Add LTM server
tmsh create gtm server nyc-ltm1 {
    datacenter NYC
    addresses { 10.1.1.100 }
    product bigip
}

# GTM automatically discovers virtual servers via iQueryCode language: PHP (php)

    Behind the scenes:

    1. GTM connects to LTM at 10.1.1.100:4353
    2. Certificate exchange and validation
    3. GTM queries LTM for available virtual servers
    4. LTM begins reporting health/performance data via iQuery

    How Important Is iQuery?

    For Any Multi-Device F5 Deployment: Critical

    iQuery is not optional for multi-device F5 deployments. Here’s what breaks without it:

    LTM HA Failures:

    • Configuration sync stops working
    • HA pair can’t coordinate failover
    • Connection mirroring fails
    • Config drift between devices
    • Manual intervention required for every change

    GTM Failures:

    • GTM cannot determine pool member health
    • Load balancing decisions become stale and inaccurate
    • Traffic sent to failed data centers
    • Performance-based algorithms stop working
    • “Global” load balancing degrades to DNS round-robin

    Real-World Impact

    I’ve seen iQuery failures cause:

    • Split-brain HA pairs: Both devices think they’re active because they can’t communicate
    • Configuration drift: Changes on active LTM never sync to standby, then failover reveals completely different configs
    • GTM sending traffic to offline data centers: No iQuery = no health visibility
    • Unbalanced load distribution: One DC overwhelmed while others idle

    Common iQuery Problems and Solutions

    Problem 1: Firewall Blocking Port 4353

    Symptom: Devices show as “Unknown” or config sync fails with connection errors.

    Cause: Firewall between devices is blocking TCP 4353.

    Solution:

    # Test connectivity
telnet <remote-device-ip> 4353

# Check iQuery status
tmsh show cm device

# For GTM specifically
tmsh show gtm server <server-name>

# Verify device is listening
netstat -an | grep 4353</server-name></remote-device-ip>Code language: HTML, XML (xml)

    Work with your network team to allow bidirectional TCP 4353 between all BIG-IP devices that need to communicate.

    Problem 2: Certificate Mismatch or Expiration

    Symptom: iQuery connection fails with SSL/certificate errors in `/var/log/ltm`.

    Cause: Certificates were regenerated, expired, or trust relationship corrupted.

    Solution for LTM HA:

    # Remove device from trust
tmsh delete cm device <device-name>

# Re-establish trust
tmsh modify cm device-group device_trust_group devices add { <device-name> }

# Force config sync
tmsh run cm config-sync to-group device_trust_group</device-name></device-name>Code language: HTML, XML (xml)

    Solution for GTM:

    # Remove and re-add server to force certificate re-exchange
tmsh delete gtm server <server-name>
tmsh create gtm server <server-name> addresses { <ltm-ip> } datacenter <dc-name></dc-name></ltm-ip></server-name></server-name>Code language: HTML, XML (xml)

    Problem 3: Version Mismatch

    Symptom: Some features don’t work, partial data sync, or connection instability.

    Cause: Devices running significantly different TMOS versions with incompatible iQuery protocol changes.

    Solution: While iQuery is generally backward-compatible, F5 recommends keeping device versions within 2-3 major releases. Upgrade devices to align versions.

    Problem 4: Config Sync Failures

    Symptom: “Awaiting Initial Sync” or “Changes Pending” that never resolve.

    Cause: iQuery connection issues or sync-failover device group problems.

    Solution:

    # Check sync status
tmsh show cm sync-status

# Force sync from known-good device
tmsh run cm config-sync to-group <device-group-name>

# If all else fails, restart config sync service
tmsh restart sys service mcpd</device-group-name>Code language: PHP (php)

    Monitoring iQuery Health

    Proactive monitoring prevents iQuery failures from causing outages:

    Key Metrics to Monitor

    For LTM HA:

    • Device trust status: All devices should show as trusted
    • Config sync state: Should be “In Sync”
    • Failover status: Active/Standby as expected
    • Certificate expiration: Monitor device certs

    For GTM:

    • Server status: All GTM servers should show “Available (Enabled)”
    • Virtual server status: Monitor state of all VS objects
    • iQuery connection count: Should match expected number of LTMs
    • Last update timestamp: Data should be fresh (< 10 seconds)

    Monitoring via iControl REST API

    # Check LTM HA sync status
GET https://ltm-ip/mgmt/tm/cm/sync-status

# Check device trust
GET https://ltm-ip/mgmt/tm/cm/device

# Query GTM server status
GET https://gtm-ip/mgmt/tm/gtm/server

# Check GTM virtual server health
GET https://gtm-ip/mgmt/tm/gtm/server/~Common~ltm-server/virtual-servers/statsCode language: PHP (php)

    Integrate these checks into your monitoring platform (Prometheus, Zabbix, Nagios) to alert on iQuery failures before users are impacted.

    Security Considerations

    1. Mutual Certificate Authentication

    iQuery’s certificate-based mutual auth is strong, but:

    • Protect certificate private keys on all devices
    • Monitor for unauthorized devices attempting iQuery connections
    • Rotate certificates periodically (though F5 doesn’t make this easy)

    2. Network Segmentation

    Limit TCP 4353 access:

    • Only allow between trusted BIG-IP devices
    • Don’t expose port 4353 to the internet
    • Use management VLANs for iQuery traffic when possible
    • Implement firewall rules between data centers

    3. Encryption

    iQuery traffic is encrypted by default (SSL/TLS), so passive sniffing won’t reveal configuration or health data. Ensure you’re running modern TMOS versions with up-to-date cipher suites.

    The Bottom Line: iQuery’s Importance

    iQuery is the universal glue that holds multi-device F5 deployments together.

    • For LTM HA: iQuery enables config sync, failover coordination, and connection mirroring
    • For GTM: iQuery provides the health visibility that makes intelligent global load balancing possible
    • For any multi-device deployment: iQuery is how devices discover, trust, and communicate with each other

    Without iQuery, you don’t have high availability, you don’t have global load balancing, and you don’t have device clustering. You just have isolated BIG-IP boxes that happen to be on the same network.

    Key Takeaways

    1. iQuery is the universal BIG-IP device-to-device protocol, not just for GTM
    2. Runs on TCP port 4353 with SSL/TLS encryption
    3. Powers LTM HA: config sync, failover, connection mirroring
    4. Enables GTM intelligence: health monitoring and performance metrics from LTMs
    5. Requires device trust via certificate exchange before communication
    6. Firewall rules must permit TCP 4353 between all communicating devices
    7. Monitor iQuery health proactively to prevent deployment failures

    Conclusion

    iQuery is one of those foundational technologies that “just works” until it doesn’t—and when it breaks, entire F5 deployments fail. LTM HA pairs can’t sync. GTM sends traffic to dead pools. Failovers don’t happen. It’s catastrophic.

    Understanding iQuery, ensuring TCP 4353 connectivity, monitoring certificate health, and watching for sync failures will save you from 2 AM pages about your load balancers being in split-brain or your global traffic manager routing everyone to an offline data center.

    If you manage F5 infrastructure—whether LTM HA pairs or global GTM deployments—treat iQuery health as seriously as you treat power and network connectivity. It’s the invisible backbone holding everything together.

    Managing F5 infrastructure or troubleshooting iQuery? Let’s connect on LinkedIn.