curl URL and Query Hacks That Beat Writing a Script

Most one-off API work dies at the query string: hand-typed & chains, jq pipelines four deep, a Python one-liner that should have been curl. These are the URL- and output-shaping tricks that keep it one line. Flag fundamentals are in the cheatsheet; this is the toolbox drawer.

-G + --data-urlencode: GETs that don't corrupt themselves

-d on a GET is wrong twice: it still sends POST, and it interprets your payload — a leading @ reads a file, ; and newlines cause surprises. -G fixes the first problem by moving the data into the URL; --data-urlencode fixes the second by percent-encoding each pair:

curl -G 'https://api.example.com/search' \
  --data-urlencode 'q=to be OR "not to be"' \
  --data-urlencode 'filter=a&b'

Repeat the flag for each parameter — order preserved, & inside values encoded as %26 instead of silently splitting your query in half. That's the whole bug class: -d 'q=a&b' without urlencoding sends q=a and a second empty param. If your search API "ignores" quotes or ampersands, it isn't the API.

--url-query (curl 7.87.0+) is the same encoder without touching the method — no -G needed, and it works when other options already fixed the method. It has one extension: --url-query '+q=%26' sends the rest unencoded, for when you want raw query syntax. Note the version gate: RHEL 9-era builds (curl 7.8x) don't have it; -G --data-urlencode is the everywhere-compatible form. See GET with a body for why -G exists at all.

Globs: one command, N URLs, no loop

Brace and range globs expand client-side (-g/--globoff disables if your URLs legitimately contain {}). The output flag #N is a placeholder for the Nth glob variable — this is the part people miss:

# three hosts x five shards, distinct filenames
curl -s 'https://host[1-3].example.com/shard[0-9].json' -o '#1_#2.json'

# sequential pages, named outputs
curl 'https://api.example.com/items[000-099]' -o 'item_#1.json'

{a,b} picks from a list, [0-9]/[00-99] runs a zero-padded range. [0-9] is ten URLs, [00-99] is a hundred. Multiple groups cross-multiply, and each group is one # slot. Without -o '#1_#2' you get one file overwriting another and a very convincing "the API is flaky."

For query strings there is no --query-glob — that flag does not exist in curl, however many blog posts swear it does. Globbing is a URL-parser feature, so put the range in the URL itself, query included — curl cross-products the bracket groups and #N counts them in order:

# range in the query: 7 requests, r_1.json .. r_7.json
curl 'https://api.example.com/metrics?range=[1-7]d' -o 'r_#1.json'

What does not work: globs inside --data-urlencode/-d values. The encoder sees [ and dutifully emits %5b1-7%5dd — verified against a local server; the brackets arrive percent-encoded and nothing fans out. Query globbing lives in the URL string, not in the encoder flags.

Bulk fetches: -Z, --parallel-immediate, and xargs -P

-Z/--parallel (7.66.0+) runs multiple URLs concurrently over one connection when the server speaks HTTP/2+. Two knobs:

curl -sZ 'https://api.example.com/a' 'https://api.example.com/b' -o 'out_#1'
curl -sZ --parallel-max 10 --parallel-immediate ...

--parallel-max caps concurrency (default 50). --parallel-immediate stops curl from waiting for multiplexing potential — instead of holding transfers back hoping they'll share one connection, it starts them right away. With unrelated hosts, waiting is pure latency: use --parallel-immediate. With ten paths on one HTTP/2 API host, don't.

When the URL list is a file, xargs beats flag juggling:

xargs -a urls.txt -P 8 -n 1 -I{} curl -sO {}

-P 8 is eight workers; curl's own queue has no per-URL retry, so for flaky upstreams the xargs version (one curl per URL) plus --retry 2 inside the template is the sturdier shape.

-w as an instrumentation layer

-w isn't just status codes. A two-field template turns curl into a latency histogram feeder:

for i in $(seq 20); do
  curl -so /dev/null -w '%{time_total} %{time_appconnect} %{time_starttransfer}\n' \
    https://api.example.com/health
done | sort -n | awk '{a[NR]=$1} END{print "p50", a[int(NR*0.5)], "p95", a[int(NR*0.95)]}'

%{time_total} is full transfer seconds; time_appconnect is TLS-completed, so total - appconnect is server+network service time — the number that tells you whether the cert chain is your latency tax. %{url_effective} logs where redirects actually left you (add %{num_redirects} to count hops), and %{http_version} pairs with the h3 checks in HTTP/3 and QUIC status.

The oldest trick in the file, now with percentiles:

curl -o /dev/null -s -w '%{time_total}s %{http_code}\n' https://example.com

A TCP+TLS+HTTP round trip is a fuzzier ping, but it also answers the question ping never does: did the application answer, and how fast. Put -w in your shell alias, not your bash profile's graveyard.

FAQ

Why does -d '{"q":"a&b"}' -G still mangle my JSON? -d does no encoding — the & splits the query. Use --data-urlencode '{"q":"a&b"}', and remember urlencoded JSON in a query string is still a string the server must choose to parse.

Does --parallel work with globs? Yes — a glob expands to multiple transfers, and -Z runs them concurrently. Check -o '#1' numbering with -v once; the counter order is expansion order.

-o /dev/null with -w — why does the exit code still say 0 on a 404? -w is reporting, not asserting. Add --fail-with-body (or -f) to make 4xx/5xx non-zero; see patterns and gotchas for the exit-code trap it prevents.