TODO

BUG FIX / VERIFICATION

• Fix and verify crawler receiver endpoint.
• Reconcile scheduler configuration and stale data paths.
• Verify Caddy imports of host, URI, method and status.

IMMEDIATE PRIORITY

Cloudflare

• Avoid unnecessary full-list retrievals.
• Record the date when an active block becomes removed-subsumed.

Scoring

• Complete scoring based on attack evidence and timing.
• Keep blocking, intelligence and abuse-report scores distinct.
• Use AbuseIPDB for enrichment and intelligence correlation, not as an automatic blocking decision.

ASN Intelligence

• Define ASN reputation policy: TRUSTED, MONITORED, NORMAL, HIGH-RISK, BLOCK/EXTREME-RISK.
• Apply ASN reputation before promotion.
• Add enrichment retry and ASN/ISP abuse-contact enrichment.
• Report hostile activity by ASN/ISP.
• Investigate ASN-wide blocking with safeguards.

Crawler Intelligence

• Complete crawler classification and automatic discovery.
• Separate crawler TYPE from STATUS.
• Default unknown crawlers to MONITORED.
• Apply host/URI policy and distinguish legitimate from illegitimate behaviour.
• Record unsafe behaviour by SAFE crawlers.
• Support historical URI exclusions and wildcards.
• Reprocess existing Caddy data.
• Investigate Jetpack's authoritative IP ranges.

BLOCK LIFECYCLE

• Define expiry rules for temporary /32, promoted CIDR and manual blocks.
• Preserve historical intelligence after expiry.
• Safely remove expired blocks from Cloudflare.
• Consider lower action for hits against honeypot hosts to collect more evidence.

REPORTING

• Complete ASN/ISP, active-block and Cloudflare synchronisation reports.
• Complete crawler and intelligence-score reports.
• Add historical trend analysis and HTML status dashboard.
• Add appropriate abuse-report submission for repeat offenders.

CONFIGURATION

• Validate missing keys, invalid paths and thresholds.
• Reconcile geo_tolerance.json and geo_policy.json.
• Establish one authoritative watcher configuration.
• Resolve remaining structural differences between sample and live scheduler configuration.

PERFORMANCE

• Measure pipeline latency and address any race conditions.

DEPLOYMENT / MAINTENANCE

• Finalise sanitised docker.sample/ files.
• Add app/startup.py to install dependencies, start and supervise the main, scheduler and worker processes, and handle failures and shutdown.
• Investigate database and runtime data maintenance requirements.

FUTURE / RESEARCH

• ASN-wide enforcement.
• Historical intelligence decay.
• Automatic rollback of false positives.
• Record aging and retention policies.