mmooresystems

Tag: gtm

  • F5 iQuery: The Silent Protocol That Makes GTM Actually Work

    If you’ve ever configured F5 GTM, set up an LTM HA pair, or joined BIG-IP devices into a Device Service Cluster, you’ve used iQuery—even if you didn’t realize it. iQuery is F5’s proprietary communication protocol that enables BIG-IP devices to discover each other, exchange configuration data, share health status, and synchronize state. It’s the invisible backbone of nearly every multi-device F5 deployment, yet it’s often overlooked until something breaks. Let’s explore what iQuery actually is, where it’s used, and why it matters.

    What Is iQuery?

    iQuery is F5’s proprietary protocol for BIG-IP device-to-device communication. It’s the universal language that allows BIG-IP systems to discover each other, establish trust, exchange data, and coordinate operations—regardless of whether they’re LTMs, GTMs, or any other BIG-IP module.

    Technical Details

    • Protocol: Encrypted TCP-based communication
    • Default Port: TCP 4353
    • Encryption: SSL/TLS with certificate-based mutual authentication
    • Scope: Device trust, config sync, health monitoring, state sharing
    • Firewall Requirements: Must allow TCP 4353 between all BIG-IP devices that need to communicate

    Think of iQuery as the nervous system connecting all your BIG-IP devices. It’s how they talk to each other, trust each other, and coordinate their actions.

    Where iQuery Is Used

    iQuery powers multiple critical F5 features across different deployment scenarios:

    1. LTM High Availability (Device Service Clustering)

    Use Case: Active-Standby or Active-Active LTM pairs

    When you set up an LTM HA pair, iQuery handles:

    • Device trust establishment: Initial pairing and certificate exchange
    • Configuration synchronization: Keeping both devices’ configs identical
    • Failover coordination: Detecting failures and triggering failover
    • Connection mirroring setup: Synchronizing connection tables for stateful failover

    Example Scenario:

    1. You create a virtual server on the active LTM
    2. iQuery synchronizes that configuration to the standby LTM
    3. Both devices now have identical configs
    4. If active fails, standby takes over seamlessly

    Without iQuery: Your HA pair can’t sync configs, coordinate failover, or mirror connections. You’d have to manually configure both devices and hope they stay in sync.

    2. GTM to LTM Communication

    Use Case: Global load balancing with GTM managing remote LTM pools

    This is where iQuery becomes highly visible and absolutely critical:

    The Scenario: GTM in New York making global load balancing decisions for LTM pools in:

    • New York data center (local LTM)
    • London data center (remote LTM)
    • Singapore data center (remote LTM)

    How iQuery enables this:

    1. GTM establishes iQuery connections to all three LTMs
    2. Each LTM reports pool member health status via iQuery
    3. LTMs share performance metrics (connections, throughput, response times)
    4. GTM uses this real-time data to make intelligent routing decisions

    Without iQuery: GTM has no idea if London’s web servers are down or Singapore is experiencing high latency. It would blindly send traffic to dead pools.

    3. GTM to GTM Synchronization

    Use Case: Redundant GTM pairs (active-active or active-standby)

    iQuery synchronizes between GTM devices:

    • Configuration changes: Wide IPs, pools, data centers
    • Wide IP states: Enabled/disabled status
    • Topology records: Geographic routing rules
    • Listener decisions: DNS query handling

    4. Device Trust and Discovery

    Use Case: Any multi-device BIG-IP deployment

    Before BIG-IP devices can work together, they must establish trust via iQuery:

    1. Administrator initiates device discovery
    2. Devices exchange SSL certificates via iQuery
    3. Mutual authentication validates both devices
    4. Trust relationship established
    5. Devices can now sync configs, share data, coordinate operations

    This certificate-based trust is the foundation for all other iQuery functionality.

    How iQuery Works: A Deep Dive

    Step 1: Certificate Exchange and Trust

    Every BIG-IP device has a unique SSL certificate. When you add a device to a trust domain or Device Service Cluster:

    1. Discovery: You specify the remote device’s IP address
    2. Connection: Device A connects to Device B on TCP 4353
    3. Certificate Exchange: Both devices share their SSL certificates
    4. Validation: Each device validates the other’s certificate
    5. Trust Established: Encrypted iQuery channel is now active

    This mutual authentication ensures only authorized BIG-IP devices can participate in the cluster.

    Step 2: Ongoing Communication

    Once trust is established, iQuery carries different types of data depending on the use case:

    For LTM HA:

    • Configuration changes (immediate sync)
    • Heartbeat signals (continuous)
    • Failover state (event-driven)
    • Connection mirror data (if enabled)

    For GTM → LTM:

    • Virtual server status (polling, typically every few seconds)
    • Pool member health (continuous monitoring)
    • Performance metrics (periodic updates)
    • System resources (CPU, memory, connections)

    Step 3: Encrypted Transport

    All iQuery traffic is encrypted with SSL/TLS, so:

    • Configuration data can’t be intercepted
    • Health status remains confidential
    • Performance metrics are protected
    • Only trusted devices can decrypt the data

    Configuration Examples

    Example 1: Setting Up LTM HA (Device Trust)

    On Device A (192.168.1.10):

    # Add Device B to the trust domain
tmsh modify cm device-group device_trust_group devices add { device-b.example.com }
tmsh run cm config-sync to-group device_trust_groupCode language: PHP (php)

    Behind the scenes:

    1. Device A initiates iQuery connection to Device B (192.168.1.11:4353)
    2. Certificates exchanged and validated
    3. Device trust established
    4. Configuration sync begins via iQuery

    Example 2: Adding LTM Servers to GTM

    On GTM:

    # Create datacenter
tmsh create gtm datacenter NYC address 10.1.1.1

# Add LTM server
tmsh create gtm server nyc-ltm1 {
    datacenter NYC
    addresses { 10.1.1.100 }
    product bigip
}

# GTM automatically discovers virtual servers via iQueryCode language: PHP (php)

    Behind the scenes:

    1. GTM connects to LTM at 10.1.1.100:4353
    2. Certificate exchange and validation
    3. GTM queries LTM for available virtual servers
    4. LTM begins reporting health/performance data via iQuery

    How Important Is iQuery?

    For Any Multi-Device F5 Deployment: Critical

    iQuery is not optional for multi-device F5 deployments. Here’s what breaks without it:

    LTM HA Failures:

    • Configuration sync stops working
    • HA pair can’t coordinate failover
    • Connection mirroring fails
    • Config drift between devices
    • Manual intervention required for every change

    GTM Failures:

    • GTM cannot determine pool member health
    • Load balancing decisions become stale and inaccurate
    • Traffic sent to failed data centers
    • Performance-based algorithms stop working
    • “Global” load balancing degrades to DNS round-robin

    Real-World Impact

    I’ve seen iQuery failures cause:

    • Split-brain HA pairs: Both devices think they’re active because they can’t communicate
    • Configuration drift: Changes on active LTM never sync to standby, then failover reveals completely different configs
    • GTM sending traffic to offline data centers: No iQuery = no health visibility
    • Unbalanced load distribution: One DC overwhelmed while others idle

    Common iQuery Problems and Solutions

    Problem 1: Firewall Blocking Port 4353

    Symptom: Devices show as “Unknown” or config sync fails with connection errors.

    Cause: Firewall between devices is blocking TCP 4353.

    Solution:

    # Test connectivity
telnet <remote-device-ip> 4353

# Check iQuery status
tmsh show cm device

# For GTM specifically
tmsh show gtm server <server-name>

# Verify device is listening
netstat -an | grep 4353</server-name></remote-device-ip>Code language: HTML, XML (xml)

    Work with your network team to allow bidirectional TCP 4353 between all BIG-IP devices that need to communicate.

    Problem 2: Certificate Mismatch or Expiration

    Symptom: iQuery connection fails with SSL/certificate errors in `/var/log/ltm`.

    Cause: Certificates were regenerated, expired, or trust relationship corrupted.

    Solution for LTM HA:

    # Remove device from trust
tmsh delete cm device <device-name>

# Re-establish trust
tmsh modify cm device-group device_trust_group devices add { <device-name> }

# Force config sync
tmsh run cm config-sync to-group device_trust_group</device-name></device-name>Code language: HTML, XML (xml)

    Solution for GTM:

    # Remove and re-add server to force certificate re-exchange
tmsh delete gtm server <server-name>
tmsh create gtm server <server-name> addresses { <ltm-ip> } datacenter <dc-name></dc-name></ltm-ip></server-name></server-name>Code language: HTML, XML (xml)

    Problem 3: Version Mismatch

    Symptom: Some features don’t work, partial data sync, or connection instability.

    Cause: Devices running significantly different TMOS versions with incompatible iQuery protocol changes.

    Solution: While iQuery is generally backward-compatible, F5 recommends keeping device versions within 2-3 major releases. Upgrade devices to align versions.

    Problem 4: Config Sync Failures

    Symptom: “Awaiting Initial Sync” or “Changes Pending” that never resolve.

    Cause: iQuery connection issues or sync-failover device group problems.

    Solution:

    # Check sync status
tmsh show cm sync-status

# Force sync from known-good device
tmsh run cm config-sync to-group <device-group-name>

# If all else fails, restart config sync service
tmsh restart sys service mcpd</device-group-name>Code language: PHP (php)

    Monitoring iQuery Health

    Proactive monitoring prevents iQuery failures from causing outages:

    Key Metrics to Monitor

    For LTM HA:

    • Device trust status: All devices should show as trusted
    • Config sync state: Should be “In Sync”
    • Failover status: Active/Standby as expected
    • Certificate expiration: Monitor device certs

    For GTM:

    • Server status: All GTM servers should show “Available (Enabled)”
    • Virtual server status: Monitor state of all VS objects
    • iQuery connection count: Should match expected number of LTMs
    • Last update timestamp: Data should be fresh (< 10 seconds)

    Monitoring via iControl REST API

    # Check LTM HA sync status
GET https://ltm-ip/mgmt/tm/cm/sync-status

# Check device trust
GET https://ltm-ip/mgmt/tm/cm/device

# Query GTM server status
GET https://gtm-ip/mgmt/tm/gtm/server

# Check GTM virtual server health
GET https://gtm-ip/mgmt/tm/gtm/server/~Common~ltm-server/virtual-servers/statsCode language: PHP (php)

    Integrate these checks into your monitoring platform (Prometheus, Zabbix, Nagios) to alert on iQuery failures before users are impacted.

    Security Considerations

    1. Mutual Certificate Authentication

    iQuery’s certificate-based mutual auth is strong, but:

    • Protect certificate private keys on all devices
    • Monitor for unauthorized devices attempting iQuery connections
    • Rotate certificates periodically (though F5 doesn’t make this easy)

    2. Network Segmentation

    Limit TCP 4353 access:

    • Only allow between trusted BIG-IP devices
    • Don’t expose port 4353 to the internet
    • Use management VLANs for iQuery traffic when possible
    • Implement firewall rules between data centers

    3. Encryption

    iQuery traffic is encrypted by default (SSL/TLS), so passive sniffing won’t reveal configuration or health data. Ensure you’re running modern TMOS versions with up-to-date cipher suites.

    The Bottom Line: iQuery’s Importance

    iQuery is the universal glue that holds multi-device F5 deployments together.

    • For LTM HA: iQuery enables config sync, failover coordination, and connection mirroring
    • For GTM: iQuery provides the health visibility that makes intelligent global load balancing possible
    • For any multi-device deployment: iQuery is how devices discover, trust, and communicate with each other

    Without iQuery, you don’t have high availability, you don’t have global load balancing, and you don’t have device clustering. You just have isolated BIG-IP boxes that happen to be on the same network.

    Key Takeaways

    1. iQuery is the universal BIG-IP device-to-device protocol, not just for GTM
    2. Runs on TCP port 4353 with SSL/TLS encryption
    3. Powers LTM HA: config sync, failover, connection mirroring
    4. Enables GTM intelligence: health monitoring and performance metrics from LTMs
    5. Requires device trust via certificate exchange before communication
    6. Firewall rules must permit TCP 4353 between all communicating devices
    7. Monitor iQuery health proactively to prevent deployment failures

    Conclusion

    iQuery is one of those foundational technologies that “just works” until it doesn’t—and when it breaks, entire F5 deployments fail. LTM HA pairs can’t sync. GTM sends traffic to dead pools. Failovers don’t happen. It’s catastrophic.

    Understanding iQuery, ensuring TCP 4353 connectivity, monitoring certificate health, and watching for sync failures will save you from 2 AM pages about your load balancers being in split-brain or your global traffic manager routing everyone to an offline data center.

    If you manage F5 infrastructure—whether LTM HA pairs or global GTM deployments—treat iQuery health as seriously as you treat power and network connectivity. It’s the invisible backbone holding everything together.

    Managing F5 infrastructure or troubleshooting iQuery? Let’s connect on LinkedIn.

  • DNS Records on the F5 GTM

    In a standard environment, DNS is simple. But when you are managing ZoneRunner on an F5 BIG-IP, the stakes are higher. You aren’t just managing names; you’re managing entry points for global traffic. While there are dozens of record types, these are the ones that keep the enterprise running.

    The Essentials: A, AAAA, and CNAME

    These are the bread and butter of your zone files. If you get these wrong, nothing else matters.

    • A (Address): The classic. Maps a hostname to a 32-bit IPv4 address. In F5 terms, this is often the “LBP” (Load Balancing Protocol) target.
    • AAAA (IPv6 Address): The 128-bit counterpart. Essential for modern “Mobile First” deployments.
    • CNAME (Canonical Name): An alias. Pro-Tip: In GTM/DNS setups, we often use CNAMEs to point a user-friendly URL (www.mmooresystems.com) to a GTM Wide IP (www.gslb.mmooresystems.com).

    The “Infrastructure” Records: SOA and NS

    You cannot have a functional zone without these. They define the “Who’s in Charge” logic of your network.

    • SOA (Start of Authority): The first record in any zone file. It tells the world that this BIG-IP is the best source of truth for the domain. It contains your serial numbers and refresh timers.
    • NS (Name Server): Defines the actual servers responsible for the zone. Without an NS record pointing to your Listeners, your GTM will never receive a query.

    The Modern “Service” Stack: MX, SRV, and TXT

    Modern networking relies heavily on these for discovery and security.

    • MX (Mail Exchanger): Tells the world where to send your email.
    • SRV (Service): Used heavily in Active Directory and VoIP (SIP) environments. It doesn’t just point to an IP; it points to a specific Service and Port (e.g., pointing _sip._tcp to your load balancer).
    • TXT (Text): The “junk drawer” that became a security powerhouse. Today, TXT records are primarily used for SPF, DKIM, and DMARC to prevent email spoofing.

    Advanced & Specialized Records

    When things get complex, ZoneRunner supports the heavy hitters:

    Record Usage in BIG-IP DNS
    PTR The “Reverse Lookup.” Used to prove an IP belongs to a name (essential for SMTP).
    NAPTR Name Authority Pointer. Used for URN mapping, often in complex Telecom/IMS environments.
    DNAME Like a CNAME, but for an entire subtree of the DNS tree. Useful for IPv6 reverse lookups.
    HINFO Standard host info (Hardware/OS). Rarely used today for security reasons (don’t give attackers a map!).

    Closing Thought: ZoneRunner vs. Manual BIND

    The beauty of ZoneRunner is that it validates your syntax. If you try to create two SOA records or a CNAME that conflicts with an A-record, ZoneRunner will stop you before you reload the BIND configuration and break your production DNS. It’s the “safety rail” every network engineer needs.

  • F5 BIG-IP DNS: Demystifying ZoneRunner and the BIND Handshake

    If you’ve ever stepped into the F5 BIG-IP DNS (formerly GTM) world, you’ve likely encountered a service called ZoneRunner. To the uninitiated, it looks like a redundant layer of management. To the power user, it is the bridge between standard DNS and F5’s Intelligent Traffic Management. Here is how to understand the “magic” happening under the hood.

    1. The Foundation: What is ZoneRunner?

    At its core, ZoneRunner is a configuration daemon (zrd) that manages a local instance of ISC BIND running on the BIG-IP. F5 didn’t reinvent the wheel for DNS records; they simply packaged BIND and built a management layer to handle the zone files. When you create a record in the F5 GUI under DNS > Zones > ZoneRunner, the F5 is essentially writing a standard BIND zone file for you.

    When Should You Actually Use ZoneRunner?

    In many GSLB (Global Server Load Balancing) environments, the F5 is just a “smart proxy” for a few URLs. But you need ZoneRunner when:

    • The F5 is the Authoritative Master: If the BIG-IP is the “Start of Authority” (SOA) for a specific sub-domain (e.g., gslb.mmooresystems.com).
    • Defining “Glue” Records: When you need static A-records, MX records, or TXT records that don’t require intelligent load balancing.
    • Providing a Safety Net: ZoneRunner acts as the “fallback” answer if the GTM layer doesn’t have a dynamic answer ready.

    2. iQuery: The Nervous System of GTM

    If ZoneRunner is the “Database,” then iQuery is the nervous system. iQuery is a proprietary F5 protocol running over TCP port 4353. It is the “secret sauce” that allows a GTM in one data center to talk to an LTM in another.

    Without iQuery, your GTM is “blind.” It uses this connection to:

    • Monitor Health: Instead of the GTM pinging every server, it asks the local LTM via iQuery: “Are your Virtual Servers healthy?”
    • Exchange Metrics: It shares CPU and connection loads so the GTM can steer traffic to the least-burdened data center.
    • Sync Everything: It ensures that a configuration change on one GTM is instantly replicated to its peers in the Sync Group.

    3. The Handshake: How it All Flows

    The magic happens when a DNS query actually hits your Listener (the Virtual Server waiting on UDP/53). The BIG-IP performs a high-speed logic check:

    1. The GTM Intercept: If the query matches a Wide IP, the GTM layer takes over. It checks the iQuery data for health and path metrics and provides an “Intelligent” answer.
    2. The BIND Fallback: If the query doesn’t match a Wide IP, the F5 hands the request down to the ZoneRunner/BIND backend to see if a static record exists.
    3. The Silence: If neither layer has an answer, it returns NXDOMAIN.

    Pro-Tips for Greenfield Deployments

    Setting this up from scratch? Keep these two “gotchas” in mind:

    Watch Your Clocks: iQuery relies on SSL certificates for the bigip_add / gtm_add handshake. If your NTP isn’t synced, the certificates will be rejected, and your iQuery mesh will fail before it starts.

    The Listener is King: You can have the most perfect ZoneRunner records and iQuery health checks, but without a DNS Listener defined on a Self-IP or Virtual Server, the BIG-IP will never answer the phone.

    Have questions about your GTM mesh or general networking? Reach out!