gosearch.v0.2.0

synced from github releases · refreshes hourly

Release notes

v0.2.0LATEST

v0.2.0 — Bing engine + quality package

Fully backward-compatible minor release.

New engine

  • Bing provider — parses li.b_algo results and unwraps /ck/a click-tracker links (u=a1<base64url> param, else best-effort reconstruction from the visible cite). Bing served clean organic results even to a flagged datacenter IP, making it the most automation-tolerant engine after DuckDuckGo and a strong fallback candidate.

Quality package

  • Canonical URL dedup across all providers — percent-encoding, parameter order, host case, fragments, default ports, and Unicode simple case fold into one comparison key; Result.URL keeps the original spelling. Fixes live-observed duplicates like ?il=Istanbul vs ?il=%C4%B0stanbul.
  • WithMarkdown() — Fetch renders extracted content as GitHub-flavored Markdown (headings, lists, fenced code, links, emphasis preserved). Built for LLM context; default plain-text output unchanged.
  • WithDates() — opt-in Result.Date freshness stamps from engine metadata.
  • WithBlockedDomains() / WithAllowedDomains() — caller-side result filtering by host or subdomain.

Fixed

  • readability no longer removes <html>/<body> when feature-flag classes contain noise-marker substrings (found live on en.wikipedia.org; previously erased page content entirely).

Docs

  • Professional README overhaul with LLM-agent pipeline guidance and expanded recipes.

view on GitHub ↗

v0.1.0

v0.1.0 — first release

First tagged release of gosearch.

Zero-API-key web search for local-first Go programs: Search() queries Google, Yandex, or DuckDuckGo by fetching and parsing each engine's public HTML result page; Fetch() extracts the readable main content of any URL. The only third-party dependency is golang.org/x/net/html.

go get github.com/BugraAkdemir/gosearch@v0.1.0

Highlights

  • Three engines, one interface — ordered fallback via WithFallback(...); advances only on ErrBlocked/ErrChallenge, never on a valid empty result.
  • Honest anti-bot handling — realistic browser headers, cookie jar, per-host rate limiting, and block detection (ErrBlocked/ErrChallenge); no CAPTCHA solving, JS-challenge execution, or proxy rotation — ever.
  • Retries where they help — transient failures (network errors, HTTP 408/5xx) retried with exponential backoff (WithRetries(n), default 2).
  • Readable content extraction — Fetch() returns title + main text with nav/ads/footer stripped.

Reliability, honestly

  • DuckDuckGo — validated against a real captured success page; most reliable of the three.
  • Google / Yandex — best-effort heuristics against documented markup, tested offline on synthetic + real block fixtures; these engines serve result pages only to clients they trust, so reliability is IP-reputation dependent.

Notes

  • Tests are offline and deterministic; go test ./... never contacts the live engines.
  • CI runs build, vet, gofmt, golangci-lint, and go test -race on every push.

view on GitHub ↗