certbot · 18 min read
How to Use Certbot with Custom DNS: Automating SSL Renewal via the DNS-01 Challenge
Walk through a production-ready Certbot DNS-01 setup — authenticator hooks, scoped API credentials, propagation checks, and renewal testing — so wildcard certificates renew themselves without a human in the loop.
You can automate TLS certificate issuance with Certbot against any DNS provider by writing custom authentication and cleanup hooks for the DNS-01 challenge. This guide explains how to use Certbot with custom DNS APIs to achieve fully unattended certificate renewal across edge routers, private infrastructure, and wildcard domains.
When you use Certbot out of the box, it relies on built-in plugins for a limited set of DNS providers. If your authoritative DNS platform lacks an official Certbot plugin, you do not have to fall back to manual verification or insecure webroot hacks. Instead, Certbot exposes hook points that execute your custom scripts, passing domain names and verification tokens through environment variables so your code can interact directly with your DNS provider's HTTP API.
Why DNS-01 beats HTTP-01 once you leave a single server
The standard ACME validation method, HTTP-01, requires port 80 to accept inbound traffic from the internet at the specific hostname requesting a certificate. According to the Let's Encrypt challenge types documentation, this challenge proves control by fetching an ephemeral token from a known path under /.well-known/acme-challenge/. That workflow operates smoothly on a standalone web server. It falls apart in complex network topologies.
Load-balanced fleets with multiple backend nodes struggle with HTTP-01 because the ACME server query might land on server A while Certbot placed the validation token on server B. Engineering teams often route around this by mounting shared NFS storage for the challenge directory or programming complex proxy bypass rules at ingress. Internal-only nodes, staging clusters, VPN gateways, and private database endpoints face an even stricter obstacle: they cannot expose port 80 to the public internet without violating firewall rules and zero-trust policies.
Wildcard certificates introduce an absolute boundary. The RFC 8555 ACME specification restricts wildcard issuance exclusively to the DNS-01 challenge type. Because a wildcard covers an arbitrary set of subdomains, the certificate authority must verify control over the entire DNS zone.
The DNS-01 challenge proves domain ownership by publishing an ephemeral TXT record under the subdomain _acme-challenge.<domain>. Because validation occurs entirely in DNS, your servers do not expose inbound network ports to the public internet. Firewalled bastions, internal microservices, and edge appliances can issue public certificates without public routing.
The operational tradeoff shifts where failures happen. HTTP-01 failures usually stem from blocked firewall ports, web server routing bugs, or proxy misconfigurations. The certbot dns-01 challenge shifts the failure domain to API credentials, rate limits, and record propagation timing. DNSCove serves standard authoritative records and does not offer GeoDNS, weighted, latency-based, or failover traffic steering in v1; the hook architecture described below operates reliably against any provider with a record-write API.
How Certbot's DNS plugin model works (and when to write your own hook)
Certbot isolates provider-specific DNS operations into standalone Python plugins such as certbot-dns-route53 or certbot-dns-cloudflare. When a team uses an authoritative provider without a bundled plugin, engineers often assume automated renewal is impossible. They resort to running Certbot interactively with --manual, which halts execution and prompts an engineer to paste a TXT record into a web console. Interactive manual runs cannot renew unattended. A certificate issued this way will expire after 90 days unless an engineer manually repeats the process.
Certbot solves this through hook integration. The Certbot manual hook documentation details the execution contract: when paired with --manual and --non-interactive, the flags --manual-auth-hook and --manual-cleanup-hook pass execution to custom binaries or shell scripts. Certbot populates specific environment variables before executing each script:
CERTBOT_DOMAIN: The domain name being authenticated, such asapi.example.comorexample.com.CERTBOT_VALIDATION: The cryptographic challenge string that must appear as the value of the TXT record.CERTBOT_TOKEN: The internal ACME token identifier (rarely needed for DNS validation).CERTBOT_REMAINING_CHALLENGES: An integer indicating how many challenge validations remain in the current certificate batch.CERTBOT_ALL_DOMAINS: A comma-separated list of all domains included in the certificate request.
The auth hook script reads these variables, issues an HTTP request to the DNS API to create the TXT record, verifies the record has propagated to the authoritative nameservers, and exits with return code 0. After the ACME server finishes validation, Certbot executes the cleanup hook, which calls the provider's API to purge the TXT record.
Use a native plugin if your provider maintains one. Write a custom hook when your provider exposes a JSON API. DNSCove does not expose a Route 53 wire-compatible API in v1; you manage DNS through DNSCove's own JSON API, console, and Terraform guides, and migrate off Route 53 with a one-step zone import. Because of this architectural choice, writing a dedicated hook script is the supported path for automated renewal. For a complete implementation reference, consult the DNSCove Let's Encrypt guide.
Step-by-step: how to use Certbot with custom DNS via auth and cleanup hooks
Building automated hooks requires reliable shell scripts, narrow API credentials, and dry-run issuance tests. Follow these steps to set up the authentication and cleanup lifecycle on a Linux host.
Step 1: install Certbot
Ensure Certbot is installed on your host and verify that it supports manual hooks.
certbot --version
Modern distributions provide Certbot via snap or distribution packages. Confirm that your binary version is 1.20 or newer to ensure standard hook environment variables operate as expected.
Step 2: provision a scoped API token
Do not place broad administrative credentials on an edge host. In your DNS provider console, create an API token restricted to record-level write permissions for the specific DNS zone requiring the certificate. The token requires permission to add and delete TXT records for the target zone, but should not have access to billing, user management, or other domains.
Step 3: secure the credentials file
Store your API credentials on the server in a file readable only by the root user. Do not hardcode API tokens directly inside shell scripts.
mkdir -p /etc/letsencrypt/credentials
chmod 700 /etc/letsencrypt/credentials
cat << 'EOF' > /etc/letsencrypt/credentials/dns-api-token.ini
DNS_API_KEY=your_scoped_token_here
DNS_ZONE_ID=zone_identifier_here
EOF
chmod 600 /etc/letsencrypt/credentials/dns-api-token.ini
Step 4: create the authentication hook
Save the following script to /usr/local/bin/certbot-dns-auth.sh and mark it executable with chmod 750 /usr/local/bin/certbot-dns-auth.sh. This script calculates the challenge subdomain, parses the variables, calls the JSON API to inject the TXT record, and waits for the change to appear on authoritative nameservers.
#!/usr/bin/env bash
set -euo pipefail
# Source credentials
CONFIG_FILE="/etc/letsencrypt/credentials/dns-api-token.ini"
if [[ ! -f "$CONFIG_FILE" ]]; then
echo "Configuration file missing: $CONFIG_FILE" >&2
exit 1
fi
source "$CONFIG_FILE"
# Determine the challenge record label
CHALLENGE_SUBDOMAIN="_acme-challenge.${CERTBOT_DOMAIN}"
# Payload for the record creation endpoint
JSON_PAYLOAD=$(cat <<EOF
{
"type": "TXT",
"name": "${CHALLENGE_SUBDOMAIN}",
"content": "${CERTBOT_VALIDATION}",
"ttl": 60
}
EOF
)
# Insert the TXT record via your provider API
RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \
-H "Authorization: Bearer ${DNS_API_KEY}" \
-H "Content-Type: application/json" \
-d "${JSON_PAYLOAD}" \
"https://api.dnscove.com/v1/zones/${DNS_ZONE_ID}/records")
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
BODY=$(echo "$RESPONSE" | sed '$d')
if [[ "$HTTP_CODE" -lt 200 || "$HTTP_CODE" -ge 300 ]]; then
echo "API request failed with status $HTTP_CODE: $BODY" >&2
exit 1
fi
# Query authoritative nameservers directly before exiting
AUTHORITATIVE_NS="ns1.dnscove.com"
MAX_ATTEMPTS=20
WAIT_SECONDS=5
for ((i=1; i<=MAX_ATTEMPTS; i++)); do
CURRENT_VALUE=$(dig +short TXT "@${AUTHORITATIVE_NS}" "${CHALLENGE_SUBDOMAIN}" | tr -d '"' || true)
if echo "$CURRENT_VALUE" | grep -Fq "${CERTBOT_VALIDATION}"; then
echo "TXT verification record propagated to ${AUTHORITATIVE_NS}."
exit 0
fi
sleep "$WAIT_SECONDS"
done
echo "Timed out waiting for DNS propagation on ${AUTHORITATIVE_NS}" >&2
exit 1
Step 5: create the cleanup hook
Save this script to /usr/local/bin/certbot-dns-cleanup.sh and grant execute permissions with chmod 750 /usr/local/bin/certbot-dns-cleanup.sh. The cleanup script locates the record by value and deletes it, preventing old validation records from cluttering the zone file.
#!/usr/bin/env bash
set -euo pipefail
CONFIG_FILE="/etc/letsencrypt/credentials/dns-api-token.ini"
if [[ ! -f "$CONFIG_FILE" ]]; then
exit 0
fi
source "$CONFIG_FILE"
CHALLENGE_SUBDOMAIN="_acme-challenge.${CERTBOT_DOMAIN}"
# Locate the record ID matching this domain and validation string
RECORD_ID=$(curl -s -H "Authorization: Bearer ${DNS_API_KEY}" \
"https://api.dnscove.com/v1/zones/${DNS_ZONE_ID}/records?type=TXT&name=${CHALLENGE_SUBDOMAIN}" \
| jq -r --arg VAL "$CERTBOT_VALIDATION" '.records[] | select(.content==$VAL) | .id')
if [[ -n "$RECORD_ID" && "$RECORD_ID" != "null" ]]; then
curl -s -X DELETE \
-H "Authorization: Bearer ${DNS_API_KEY}" \
"https://api.dnscove.com/v1/zones/${DNS_ZONE_ID}/records/${RECORD_ID}"
fi
exit 0
Step 6: test issuance using staging
Before issuing a production certificate, execute Certbot against the Let's Encrypt staging environment to avoid triggering production rate limits. This dry run tests the auth hook, token propagation, and cleanup script.
certbot certonly \
--manual \
--manual-auth-hook /usr/local/bin/certbot-dns-auth.sh \
--manual-cleanup-hook /usr/local/bin/certbot-dns-cleanup.sh \
--preferred-challenges dns \
--dry-run \
-d "test.example.com"
Step 7: verify manual DNS resolution
If you experience unexpected failures during the initial run, reproduce the record lookup manually. In a separate terminal, execute a dig query directly against the zone's authoritative nameserver to verify that the TXT response is valid and unquoted.
dig +short TXT @ns1.dnscove.com _acme-challenge.example.com
Handling DNS propagation without guessing
The single most fragile element of automated DNS-01 validation is propagation timing. Certbot provides a built-in flag named --dns-propagation-seconds. Many tutorials suggest setting this flag to an arbitrary number like 30, 60, or 120. That parameter is a blind sleep call. If your authoritative nameservers update in 4 seconds, sleeping 60 seconds wastes 56 seconds on every certificate run. If an infrastructure incident slows propagation to 65 seconds, a 60-second sleep causes the ACME server to query early, find no record, fail validation, and burn through account rate limits.
A resilient auth hook ignores static sleep flags and polls authoritative nameservers directly. As demonstrated in Step 4, your auth script should query the zone's designated primary nameserver with dig @ns1.dnscove.com inside a loop. The script exits as soon as the expected token appears. This design minimizes validation latency while creating a reliable safety margin during transient platform slowdowns.
Query authoritative nameservers directly. Public recursive resolvers such as 8.8.8.8 or 1.1.1.1 enforce negative caching. If an ACME node or your script queries _acme-challenge.example.com before the TXT record exists, the recursive resolver caches the NXDOMAIN response based on the zone SOA record's minimum TTL field. That negative cache entry can linger for 300 to 3600 seconds, causing subsequent queries to return NXDOMAIN even after your DNS API reports a successful write.
To reduce propagation latency across your DNS infrastructure:
- Set the challenge TXT record TTL to the lowest value supported by your provider, ideally 60 seconds or lower.
- Avoid pre-querying the challenge record hostname from recursive resolvers before the auth hook finishes writing the record.
- Query authoritative nameservers in your auth hook loop to ensure the record exists on the network edge before Certbot yields execution back to the ACME server.
Making automated SSL renewal with DNS survive real operations
Achieving initial certificate issuance in an interactive shell is easy. Making automated ssl renewal dns tasks run unattended for years requires defensive configuration across scheduler environments, daemon reloads, and file permissions.
Scheduler environment isolation
Certbot renewals run non-interactively from a systemd timer (certbot.timer) or a root cron job. Schedulers launch processes inside a stripped, minimal environment. Shell environment variables, custom PATH entries, and proxy settings present in your interactive SSH session do not exist in the systemd execution context. If your hook script calls tools like jq or curl without specifying full binary paths or explicitly setting PATH=/usr/local/bin:/usr/bin:/bin, the script fails under cron while working during manual testing.
Automating service reloads with deploy hooks
Obtaining a new certificate file on disk does not secure incoming connections. Web servers like Nginx and HAProxy load certificates into memory at startup. When Certbot saves updated pem files into /etc/letsencrypt/live/, running processes continue serving the old certificate until reloaded.
Do not reload web daemons from inside the DNS cleanup hook. The cleanup hook runs after challenge completion, even if issuance subsequently fails. Instead, use Certbot's dedicated --deploy-hook parameter. Deploy hooks execute only after Certbot successfully validates the domain, receives the certificate from the CA, and writes it to disk.
certbot certonly \
--manual \
--manual-auth-hook /usr/local/bin/certbot-dns-auth.sh \
--manual-cleanup-hook /usr/local/bin/certbot-dns-cleanup.sh \
--deploy-hook "systemctl reload nginx" \
--preferred-challenges dns \
-d "example.com" -d "*.example.com"
When Certbot creates its renewal configuration at /etc/letsencrypt/renewal/example.com.conf, it preserves these hook paths. Future invocations of certbot renew will run the auth hook, the cleanup hook, and the deploy hook automatically.
Concurrency and race conditions
If you configure separate certificates for multiple services within the same root domain (for example, mail.example.com and vpn.example.com), a system timer may invoke concurrent renewal processes. Both instances will attempt to read and write records under _acme-challenge.example.com simultaneously. Ensure your hook scripts identify and delete records using exact record IDs matched to the unique CERTBOT_VALIDATION token instead of deleting all records under the challenge subdomain.
Common failure modes and how to diagnose them
When DNS-01 challenges fail, Certbot usually prints a brief error stating that the ACME server could not find the expected TXT value. Diagnosing the underlying issue requires systematic checks across the DNS state.
Leftover TXT records from abandoned runs
If a previous script execution terminated abruptly before running its cleanup hook, stale TXT records remain in the zone. While the DNS specification permits multiple TXT records under a single label, some ACME validation nodes verify all published records. If an old, invalid record persists alongside the new token, validation may fail. Verify current records using the command below:
dig +short TXT @ns1.dnscove.com _acme-challenge.example.com
Purge stale validation records before re-running Certbot.
Combining root apex and wildcard domains
A frequent use case involves requesting a single certificate covering both an apex domain and a wildcard: -d example.com -d *.example.com. For both identifiers, Let's Encrypt requires a TXT record at the exact same DNS name: _acme-challenge.example.com.
Certbot calls the authentication hook twice in rapid succession, passing a different CERTBOT_VALIDATION value on each call. If your auth hook script issues a PUT request that overwrites existing TXT records at that name, the second call erases the token written by the first. The first validation then fails. Your API integration must append records or support multiple distinct TXT records under the same label.
CNAME delegation of the challenge label
Organizations often avoid granting web servers API write access to production DNS zones. You can delegate validation authority by placing a static CNAME record at _acme-challenge.example.com that points to a dedicated validation zone managed on an isolated DNS account:
_acme-challenge.example.com. IN CNAME _acme-challenge.auth.example.net.
ACME validation servers follow CNAME chains. When Let's Encrypt looks for the challenge TXT record at _acme-challenge.example.com, it resolves the CNAME and queries _acme-challenge.auth.example.net instead. Your hook script then writes TXT records strictly to the delegated zone. This restricts the blast radius of API credentials. If you implement this pattern, ensure the destination zone resolves reliably and does not dead-end into broken nameserver delegations.
Hook credential permission failures
Systemd units execute as isolated service users unless configured otherwise. If /etc/letsencrypt/credentials/dns-api-token.ini has permissions set to 0600 owned by root:root, but Certbot runs under an unprivileged automation account, the script will silently fail to load the token. The hook will exit with an error code, which Certbot presents as an ambiguous challenge failure. Check hook execution logs directly at /var/log/letsencrypt/letsencrypt.log to inspect the raw stdout and stderr output from your scripts.
Where the DNS provider choice actually matters
An ACME DNS hook script is only as reliable as the underlying DNS provider's control plane. If an authoritative API suffers from write latency, eventual consistency lags, or opaque error responses, automated certificate renewal becomes intermittent and fragile.
When selecting a DNS provider for production ACME automation, evaluate these operational criteria:
- API response latency: Changes must register immediately in the provider's database.
- Control plane to nameserver propagation speed: Updates pushed to the API should reach authoritative edge nameservers in under 10 seconds.
- Granular API scoping: The platform should support per-zone tokens restricted strictly to record modifications.
- Predictable error status codes: Clear JSON error payloads allow your shell scripts to distinguish invalid credentials from rate limits or validation errors.
Billing structures also influence your DNS architecture. Many cloud DNS providers bill on a hybrid model that charges per hosted zone plus a variable fee per million queries. Automated certificate renewals generate bursts of API writes, updates, and direct authoritative queries across staging and production fleets. DNSCove uses fixed-cost pricing rather than per-zone or per-query metering. This predictable model simplifies budgeting for infrastructure teams issuing certificates across large fleets of subdomains.
Production environments frequently host public web applications where the root apex must point to an elastic cloud load balancer. Standard DNS specifications forbid placing a CNAME record at the zone apex because the apex contains required SOA and NS records. DNSCove supports apex ALIAS records (CNAME-at-apex flattening, like Route53 Alias) with serve-stale protection. This allows engineering teams to point an apex domain directly to an external load balancer hostname while running Certbot DNS-01 challenges on the exact same zone.
Architects must plan their validation pathways around provider design constraints. DNSCove does not offer AXFR zone transfer or secondary-DNS operation in v1. Because secondary replication is not used, validation hooks must interact directly with the primary authoritative zone. For technical details on supported record schemas and API payloads, see the DNSCove control-plane API documentation.
Testing, monitoring, and the checklist before you walk away
Do not assume an automated renewal system works until you test it through the complete runtime environment. Follow this checklist before handing off certificate automation to production operations.
1. Test non-interactive renewal execution
Run the Certbot renewal workflow against the staging server using the exact command executed by your system timer:
certbot renew --dry-run
Inspect the output to confirm that Certbot executes the custom auth hook, verifies the challenge, runs the cleanup hook, and exits cleanly with zero errors.
2. Force a single real renewal cycle
After testing with --dry-run, verify production issuance by forcing a renewal against Let's Encrypt production. Monitor the process live:
certbot renew --force-renewal
Watch your DNS control plane to observe the TXT record creation, verify that the hook waits for propagation, and confirm the cleanup hook removes the record.
3. Confirm deployed certificate parameters
Query the live web listener to confirm that the service reloaded its process and is actively serving the issued certificate:
echo | openssl s_client -connect localhost:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates -subject
4. Deploy external expiry monitoring
Do not rely on Certbot's internal logs to alert you to failures. If a token expires, a network route breaks, or an API key is revoked, renewals will fail silently in the background while the existing certificate inches toward expiration. Configure an external synthetic monitoring check (using Prometheus Blackbox Exporter, Datadog, or an uptime monitor) to alert your engineering team if any production endpoint serves a TLS certificate with fewer than 21 days of remaining validity.
5. Operational production checklist
- Credentials file is owned by root and restricted to
chmod 600. - Hook scripts specify full paths for system binaries (
curl,jq,dig). - Deploy hooks are defined to reload edge web services idempotently.
- Systemd timer is active: check with
systemctl is-active certbot.timer. - Team runbook documents API token rotation procedures before keys expire.
Frequently asked questions
Can I use Certbot with a DNS provider that has no official plugin?
Yes. You can manage any DNS provider by running Certbot with the --manual flag alongside --manual-auth-hook and --manual-cleanup-hook. Certbot executes your custom scripts and exposes the target domain and challenge value through environment variables, enabling automated renewal against any provider offering an HTTP API.
How long should --dns-propagation-seconds be set to?
Do not rely on --dns-propagation-seconds for production automation. That parameter forces an arbitrary sleep period. Instead, configure your authentication hook script to poll your zone's authoritative nameservers directly using dig. This approach proceeds immediately once the TXT record appears on the edge, saving time while preventing false timeouts.
Does DNS-01 validation work for wildcard certificates?
Yes. DNS-01 is the only validation method accepted by Let's Encrypt and other ACME certificate authorities for wildcard certificates (such as *.example.com). HTTP-01 challenges cannot be used to issue wildcard certificates.
Why does Certbot say 'incorrect TXT record found' when my record looks right?
This error commonly occurs when a previous renewal failed and left behind an orphaned TXT record at _acme-challenge.<domain>. It also occurs when issuing a certificate covering both the apex and a wildcard (example.com and *.example.com) if your API script overwrites the first TXT record instead of appending the second one.
Can I delegate _acme-challenge to a different DNS zone with a CNAME?
Yes. You can configure a CNAME record at _acme-challenge.example.com pointing to a record in another zone, such as _acme-challenge.auth.example.net. ACME servers follow CNAME redirects to validate ownership, allowing you to restrict API write credentials to a dedicated validation zone rather than your primary domain.
If your renewal hook is the last thing standing between you and unattended certificates, start with the DNSCove Let's Encrypt guide for a working auth and cleanup hook against the JSON API, then run certbot renew --dry-run before you trust it in production.
- No AWS account required
- Zero-downtime Route 53 cutover
- Apex ALIAS / ANAME to any target
- DNS as code — Terraform, CloudFormation
Straight answer: DNSSEC signing isn't available yet — it's on the roadmap. Everything else here works today. Authoritative nameservers: ns1.dnscove.com, ns2.dnscove.org.