---

gau (Get All URLs) fetches known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, Common Crawl, and URLScan for a given domain. Great for discovering endpoints and parameters.

---

Basic Usage

• gau <domain> - Fetch URLs for domain

• echo "example.com" | gau - Pipe domain

• cat domains.txt | gau - Multiple domains

---

Options

• --blacklist - Comma-separated list of extensions to skip

• --fc - Filter by status code (comma-separated)

• --from - Fetch URLs from date (YYYYMM)

• --ft - Filter to status code (show only these)

• --fp - Remove query strings

• --json - Output as JSON

• --mc - Match status code (comma-separated)

• --mt - Match MIME type

• --o - Output file

• --providers - Comma-separated providers (wayback, otx, commoncrawl, urlscan)

• --retries - Number of retries

• --subs - Include subdomains

• --threads - Number of threads

• --to - Fetch URLs to date (YYYYMM)

• --verbose - Verbose output

---

Providers

• wayback - Wayback Machine (web.archive.org)
• otx - AlienVault Open Threat Exchange
• commoncrawl - Common Crawl
• urlscan - URLScan.io

---

Common Examples

Basic Fetch

gau example.com

Get all known URLs.

Include Subdomains

gau --subs example.com

Include all subdomain URLs.

Save to File

gau example.com --o urls.txt

Save results to file.

Exclude Extensions

gau --blacklist png,jpg,gif,css,js example.com

Skip static assets.

Filter Date Range

gau --from 202301 --to 202312 example.com

URLs from 2023 only.

Specific Providers

gau --providers wayback,otx example.com

Use only Wayback and OTX.

JSON Output

gau --json example.com

Output in JSON format.

Multiple Domains

cat domains.txt | gau --threads 5

Process multiple domains.

Remove Query Strings

gau --fp example.com | sort -u

Unique paths without parameters.

---

Pipeline Examples

Find Parameters

gau example.com | grep "=" | sort -u

Find URLs with parameters.

Find JS Files

gau example.com | grep "\.js$" | sort -u

Extract JavaScript files.

Find API Endpoints

gau example.com | grep -E "(api|v1|v2)" | sort -u

Find potential API endpoints.

With httpx

gau example.com | httpx -silent -status-code

Check which URLs are live.

Find Interesting Files

gau example.com | grep -E "\.(json|xml|config|sql|backup|bak)$"

Find potentially sensitive files.

---

Configuration

Config File Location

• ~/.gau.toml - Configuration file

Example Config

threads = 5
retries = 3
blacklist = ["png", "jpg", "gif"]
providers = ["wayback", "otx", "urlscan"]

---

Tips

• Use --blacklist to filter out static files

• Combine with grep to find interesting patterns

• Pipe to httpx to check live URLs

• Look for parameters for potential injection points

• Historical URLs may reveal removed functionality

• Great for bug bounty recon

• Combine with waybackurls for comprehensive coverage

• Check JS files for API keys and secrets

---