Go Rod Master

1---
2name: go-rod-master
3description: "Comprehensive guide for browser automation and web scraping with go-rod (Chrome DevTools Protocol) including stealth anti-bot-detection patterns."
4risk: safe
5source: https://github.com/go-rod/rod
6---
7 
8# Go-Rod Browser Automation Master
9 
10## Overview
11 
12[Rod](https://github.com/go-rod/rod) is a high-level Go driver built directly on the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) for browser automation and web scraping. Unlike wrappers around other tools, Rod communicates with the browser natively via CDP, providing thread-safe operations, chained context design for timeouts/cancellation, auto-wait for elements, correct iframe/shadow DOM handling, and zero zombie browser processes.
13 
14The companion library [go-rod/stealth](https://github.com/go-rod/stealth) injects anti-bot-detection evasions based on [puppeteer-extra stealth](https://github.com/nichochar/puppeteer-extra/tree/master/packages/extract-stealth-evasions), hiding headless browser fingerprints from detection systems.
15 
16## When to Use This Skill
17 
18- Use when the user asks to **scrape**, **automate**, or **test** a website using Go.
19- Use when the user needs a **headless browser** for dynamic/SPA content (React, Vue, Angular).
20- Use when the user mentions **stealth**, **anti-bot**, **avoiding detection**, **Cloudflare**, or **bot detection bypass**.
21- Use when the user wants to work with the **Chrome DevTools Protocol (CDP)** directly from Go.
22- Use when the user needs to **intercept** or **hijack** network requests in a browser context.
23- Use when the user asks about **concurrent browser scraping** or **page pooling** in Go.
24- Use when the user is migrating from **chromedp** or **Playwright Go** and wants a simpler API.
25 
26## Safety & Risk
27 
28**Risk Level: 🔵 Safe**
29 
30- **Read-Only by Default:** Default behavior is navigating and reading page content (scraping/testing).
31- **Isolated Contexts:** Browser contexts are sandboxed; cookies and storage do not persist unless explicitly saved.
32- **Resource Cleanup:** Designed around Go's `defer` pattern — browsers and pages close automatically.
33- **No External Mutations:** Does not modify external state unless the script explicitly submits forms or POSTs data.
34 
35## Installation
36 
37```bash
38# Core rod library
39go get github.com/go-rod/rod@latest
40 
41# Stealth anti-detection plugin (ALWAYS include for production scraping)
42go get github.com/go-rod/stealth@latest
43```
44 
45Rod auto-downloads a compatible Chromium binary on first run. To pre-download:
46 
47```bash
48go run github.com/nichochar/go-rod.github.io/cmd/launcher@latest
49```
50 
51## Core Concepts
52 
53### Browser Lifecycle
54 
55Rod manages three layers: **Browser → Page → Element**.
56 
57```go
58// Launch and connect to a browser
59browser := rod.New().MustConnect()
60defer browser.MustClose()
61 
62// Create a page (tab)
63page := browser.MustPage("https://example.com")
64 
65// Find an element
66el := page.MustElement("h1")
67fmt.Println(el.MustText())
68```
69 
70### Must vs Error Patterns
71 
72Rod provides two API styles for every operation:
73 
74| Style | Method | Use Case |
75|:------|:-------|:---------|
76| **Must** | `MustElement()`, `MustClick()`, `MustText()` | Scripting, debugging, prototyping. Panics on error. |
77| **Error** | `Element()`, `Click()`, `Text()` | Production code. Returns `error` for explicit handling. |
78 
79**Production pattern:**
80 
81```go
82el, err := page.Element("#login-btn")
83if err != nil {
84    return fmt.Errorf("login button not found: %w", err)
85}
86if err := el.Click(proto.InputMouseButtonLeft, 1); err != nil {
87    return fmt.Errorf("click failed: %w", err)
88}
89```
90 
91**Scripting pattern with Try:**
92 
93```go
94err := rod.Try(func() {
95    page.MustElement("#login-btn").MustClick()
96})
97if errors.Is(err, context.DeadlineExceeded) {
98    log.Println("timeout finding login button")
99}
100```
101 
102### Context & Timeout
103 
104Rod uses Go's `context.Context` for cancellation and timeouts. Context propagates recursively to all child operations.
105 
106```go
107// Set a 5-second timeout for the entire operation chain
108page.Timeout(5 * time.Second).
109    MustWaitLoad().
110    MustElement("title").
111    CancelTimeout(). // subsequent calls are not bound by the 5s timeout
112    Timeout(30 * time.Second).
113    MustText()
114```
115 
116### Element Selectors
117 
118Rod supports multiple selector strategies:
119 
120```go
121// CSS selector (most common)
122page.MustElement("div.content > p.intro")
123 
124// CSS selector with text regex matching
125page.MustElementR("button", "Submit|Send")
126 
127// XPath
128page.MustElementX("//div[@class='content']//p")
129 
130// Search across iframes and shadow DOM (like DevTools Ctrl+F)
131page.MustSearch(".deeply-nested-element")
132```
133 
134### Auto-Wait
135 
136Rod automatically retries element queries until the element appears or the context times out. You do not need manual sleeps:
137 
138```go
139// This will automatically wait until the element exists
140el := page.MustElement("#dynamic-content")
141 
142// Wait until the element is stable (position/size not changing)
143el.MustWaitStable().MustClick()
144 
145// Wait until page has no pending network requests
146wait := page.MustWaitRequestIdle()
147page.MustElement("#search").MustInput("query")
148wait()
149```
150 
151---
152 
153## Stealth & Anti-Bot Detection (go-rod/stealth)
154 
155> **IMPORTANT:** For any production scraping or automation against real websites, ALWAYS use `stealth.MustPage()` instead of `browser.MustPage()`. This is the single most important step for avoiding bot detection.
156 
157### How Stealth Works
158 
159The `go-rod/stealth` package injects JavaScript evasions into every new page that:
160 
161- **Remove `navigator.webdriver`** — the primary headless detection signal.
162- **Spoof WebGL vendor/renderer** — presents real GPU info (e.g., "Intel Inc." / "Intel Iris OpenGL Engine") instead of headless markers like "Google SwiftShader".
163- **Fix Chrome plugin array** — reports proper `PluginArray` type with realistic plugin count.
164- **Patch permissions API** — returns `"prompt"` instead of bot-revealing values.
165- **Set realistic languages** — reports `en-US,en` instead of empty arrays.
166- **Fix broken image dimensions** — headless browsers report 0x0; stealth fixes this to 16x16.
167 
168### Usage
169 
170**Creating a stealth page (recommended for all production use):**
171 
172```go
173import (
174    "github.com/go-rod/rod"
175    "github.com/go-rod/stealth"
176)
177 
178browser := rod.New().MustConnect()
179defer browser.MustClose()
180 
181// Use stealth.MustPage instead of browser.MustPage
182page := stealth.MustPage(browser)
183page.MustNavigate("https://bot.sannysoft.com")
184```
185 
186**With error handling:**
187 
188```go
189page, err := stealth.Page(browser)
190if err != nil {
191    return fmt.Errorf("failed to create stealth page: %w", err)
192}
193page.MustNavigate("https://example.com")
194```
195 
196**Using stealth.JS directly (advanced — for custom page creation):**
197 
198```go
199// If you need to create the page yourself (e.g., with specific options),
200// inject stealth.JS manually via EvalOnNewDocument
201page := browser.MustPage()
202page.MustEvalOnNewDocument(stealth.JS)
203page.MustNavigate("https://example.com")
204```
205 
206### Verifying Stealth
207 
208Navigate to a bot detection test page to verify evasions:
209 
210```go
211page := stealth.MustPage(browser)
212page.MustNavigate("https://bot.sannysoft.com")
213page.MustScreenshot("stealth_test.png")
214```
215 
216Expected results for a properly stealth-configured browser:
217- **WebDriver**: `missing (passed)`
218- **Chrome**: `present (passed)`
219- **Plugins Length**: `3` (not `0`)
220- **Languages**: `en-US,en`
221 
222---
223 
224## Implementation Guidelines
225 
226### 1. Launcher Configuration
227 
228Use the `launcher` package to customize browser launch flags:
229 
230```go
231import "github.com/go-rod/rod/lib/launcher"
232 
233url := launcher.New().
234    Headless(true).             // false for debugging
235    Proxy("127.0.0.1:8080").    // upstream proxy
236    Set("disable-gpu", "").     // custom Chrome flag
237    Delete("use-mock-keychain"). // remove a default flag
238    MustLaunch()
239 
240browser := rod.New().ControlURL(url).MustConnect()
241defer browser.MustClose()
242```
243 
244**Debugging mode (visible browser + slow motion):**
245 
246```go
247l := launcher.New().
248    Headless(false).
249    Devtools(true)
250defer l.Cleanup()
251 
252browser := rod.New().
253    ControlURL(l.MustLaunch()).
254    Trace(true).
255    SlowMotion(2 * time.Second).
256    MustConnect()
257```
258 
259### 2. Proxy Support
260 
261```go
262// Set proxy at launch
263url := launcher.New().
264    Proxy("socks5://127.0.0.1:1080").
265    MustLaunch()
266 
267browser := rod.New().ControlURL(url).MustConnect()
268 
269// Handle proxy authentication
270go browser.MustHandleAuth("username", "password")()
271 
272// Ignore SSL certificate errors (for MITM proxies)
273browser.MustIgnoreCertErrors(true)
274```
275 
276### 3. Input Simulation
277 
278```go
279import "github.com/go-rod/rod/lib/input"
280 
281// Type into an input field (replaces existing value)
282page.MustElement("#email").MustInput("user@example.com")
283 
284// Simulate keyboard keys
285page.Keyboard.MustType(input.Enter)
286 
287// Press key combinations
288page.Keyboard.MustPress(input.ControlLeft)
289page.Keyboard.MustType(input.KeyA)
290page.Keyboard.MustRelease(input.ControlLeft)
291 
292// Mouse click at coordinates
293page.Mouse.MustClick(input.MouseLeft)
294page.Mouse.MustMoveTo(100, 200)
295```
296 
297### 4. Network Request Interception (Hijacking)
298 
299```go
300router := browser.HijackRequests()
301defer router.MustStop()
302 
303// Block all image requests
304router.MustAdd("*.png", func(ctx *rod.Hijack) {
305    ctx.Response.Fail(proto.NetworkErrorReasonBlockedByClient)
306})
307 
308// Modify request headers
309router.MustAdd("*api.example.com*", func(ctx *rod.Hijack) {
310    ctx.Request.Req().Header.Set("Authorization", "Bearer token123")
311    ctx.MustLoadResponse()
312})
313 
314// Modify response body
315router.MustAdd("*.js", func(ctx *rod.Hijack) {
316    ctx.MustLoadResponse()
317    ctx.Response.SetBody(ctx.Response.Body() + "\n// injected")
318})
319 
320go router.Run()
321```
322 
323### 5. Waiting Strategies
324 
325```go
326// Wait for page load event
327page.MustWaitLoad()
328 
329// Wait for no pending network requests (AJAX idle)
330wait := page.MustWaitRequestIdle()
331page.MustElement("#search").MustInput("query")
332wait()
333 
334// Wait for element to be stable (not animating)
335page.MustElement(".modal").MustWaitStable().MustClick()
336 
337// Wait for element to become invisible
338page.MustElement(".loading").MustWaitInvisible()
339 
340// Wait for JavaScript condition
341page.MustWait(`() => document.title === 'Ready'`)
342 
343// Wait for specific navigation/event
344wait := page.WaitEvent(&proto.PageLoadEventFired{})
345page.MustNavigate("https://example.com")
346wait()
347```
348 
349### 6. Race Selectors (Multiple Outcomes)
350 
351Handle pages where the result can be one of several outcomes (e.g., login success vs error):
352 
353```go
354page.MustElement("#username").MustInput("user")
355page.MustElement("#password").MustInput("pass").MustType(input.Enter)
356 
357// Race between success and error selectors
358elm := page.Race().
359    Element(".dashboard").MustHandle(func(e *rod.Element) {
360        fmt.Println("Login successful:", e.MustText())
361    }).
362    Element(".error-message").MustDo()
363 
364if elm.MustMatches(".error-message") {
365    log.Fatal("Login failed:", elm.MustText())
366}
367```
368 
369### 7. Screenshots & PDF
370 
371```go
372// Full-page screenshot
373page.MustScreenshot("page.png")
374 
375// Custom screenshot (JPEG, specific region)
376img, _ := page.Screenshot(true, &proto.PageCaptureScreenshot{
377    Format:  proto.PageCaptureScreenshotFormatJpeg,
378    Quality: gson.Int(90),
379    Clip: &proto.PageViewport{
380        X: 0, Y: 0, Width: 1280, Height: 800, Scale: 1,
381    },
382})
383utils.OutputFile("screenshot.jpg", img)
384 
385// Scroll screenshot (captures full scrollable page)
386img, _ := page.MustWaitStable().ScrollScreenshot(nil)
387utils.OutputFile("full_page.jpg", img)
388 
389// PDF export
390page.MustPDF("output.pdf")
391```
392 
393### 8. Concurrent Page Pool
394 
395```go
396pool := rod.NewPagePool(5) // max 5 concurrent pages
397 
398create := func() *rod.Page {
399    return browser.MustIncognito().MustPage()
400}
401 
402var wg sync.WaitGroup
403for _, url := range urls {
404    wg.Add(1)
405    go func(u string) {
406        defer wg.Done()
407 
408        page := pool.MustGet(create)
409        defer pool.Put(page)
410 
411        page.MustNavigate(u).MustWaitLoad()
412        fmt.Println(page.MustInfo().Title)
413    }(url)
414}
415wg.Wait()
416 
417pool.Cleanup(func(p *rod.Page) { p.MustClose() })
418```
419 
420### 9. Event Handling
421 
422```go
423// Listen for console.log output
424go page.EachEvent(func(e *proto.RuntimeConsoleAPICalled) {
425    if e.Type == proto.RuntimeConsoleAPICalledTypeLog {
426        fmt.Println(page.MustObjectsToJSON(e.Args))
427    }
428})()
429 
430// Wait for a specific event before proceeding
431wait := page.WaitEvent(&proto.PageLoadEventFired{})
432page.MustNavigate("https://example.com")
433wait()
434```
435 
436### 10. File Download
437 
438```go
439wait := browser.MustWaitDownload()
440 
441page.MustElementR("a", "Download PDF").MustClick()
442 
443data := wait()
444utils.OutputFile("downloaded.pdf", data)
445```
446 
447### 11. JavaScript Evaluation
448 
449```go
450// Execute JS on the page
451page.MustEval(`() => console.log("hello")`)
452 
453// Pass parameters and get return value
454result := page.MustEval(`(a, b) => a + b`, 1, 2)
455fmt.Println(result.Int()) // 3
456 
457// Eval on a specific element ("this" = the DOM element)
458title := page.MustElement("title").MustEval(`() => this.innerText`).String()
459 
460// Direct CDP calls for features Rod doesn't wrap
461proto.PageSetAdBlockingEnabled{Enabled: true}.Call(page)
462```
463 
464### 12. Loading Chrome Extensions
465 
466```go
467extPath, _ := filepath.Abs("./my-extension")
468 
469u := launcher.New().
470    Set("load-extension", extPath).
471    Headless(false). // extensions require headed mode
472    MustLaunch()
473 
474browser := rod.New().ControlURL(u).MustConnect()
475```
476 
477---
478 
479## Examples
480 
481See the `examples/` directory for complete, runnable Go files:
482- `examples/basic_scrape.go` — Minimal scraping example
483- `examples/stealth_page.go` — Anti-detection with go-rod/stealth
484- `examples/request_hijacking.go` — Intercepting and modifying network requests
485- `examples/concurrent_pages.go` — Page pool for concurrent scraping
486 
487---
488 
489## Best Practices
490 
491- ✅ **ALWAYS use `stealth.MustPage(browser)`** instead of `browser.MustPage()` for real-world sites.
492- ✅ **ALWAYS `defer browser.MustClose()`** immediately after connecting.
493- ✅ Use the error-returning API (not `Must*`) in production code.
494- ✅ Set explicit timeouts with `.Timeout()` — never rely on defaults for production.
495- ✅ Use `browser.MustIncognito().MustPage()` for isolated sessions.
496- ✅ Use `PagePool` for concurrent scraping instead of spawning unlimited pages.
497- ✅ Use `MustWaitStable()` before clicking elements that might be animating.
498- ✅ Use `MustWaitRequestIdle()` after actions that trigger AJAX calls.
499- ✅ Use `launcher.New().Headless(false).Devtools(true)` for debugging.
500- ❌ **NEVER** use `time.Sleep()` for waiting — use Rod's built-in wait methods.
501- ❌ **NEVER** create a new `Browser` per task — create one Browser, use multiple `Page` instances.
502- ❌ **NEVER** use `browser.MustPage()` for production scraping — use `stealth.MustPage()`.
503- ❌ **NEVER** ignore errors in production — always handle them explicitly.
504- ❌ **NEVER** forget to defer-close browsers, pages, and hijack routers.
505 
506## Common Pitfalls
507 
508- **Problem:** Element not found even though it exists on the page.
509  **Solution:** The element may be inside an iframe or shadow DOM. Use `page.MustSearch()` instead of `page.MustElement()` — it searches across all iframes and shadow DOMs.
510 
511- **Problem:** Click doesn't work because the element is animating.
512  **Solution:** Call `el.MustWaitStable()` before `el.MustClick()`.
513 
514- **Problem:** Bot detection despite using stealth.
515  **Solution:** Combine `stealth.MustPage()` with: randomized viewport sizes, realistic User-Agent strings, human-like input delays between keystrokes, and random idle behaviors (scroll, hover).
516 
517- **Problem:** Browser process leaks (zombie processes).
518  **Solution:** Always `defer browser.MustClose()`. Rod uses [leakless](https://github.com/ysmood/leakless) to kill zombies after main process crash, but explicit cleanup is preferred.
519 
520- **Problem:** Timeout errors on slow pages.
521  **Solution:** Use chained context: `page.Timeout(30 * time.Second).MustWaitLoad()`. For AJAX-heavy pages, use `MustWaitRequestIdle()` instead of `MustWaitLoad()`.
522 
523- **Problem:** HijackRequests router not intercepting requests.
524  **Solution:** You must call `go router.Run()` after setting up routes, and `defer router.MustStop()` for cleanup.
525 
526## Limitations
527 
528- **CAPTCHAs:** Rod does not include CAPTCHA solving. External services (2captcha, etc.) must be integrated separately.
529- **Extreme Anti-Bot:** While `go-rod/stealth` handles common detection (WebDriver, plugin fingerprints, WebGL), extremely strict systems (some Cloudflare configurations, Akamai Bot Manager) may still detect automation. Additional measures (residential proxies, human-like behavioral patterns) may be needed.
530- **DRM Content:** Cannot interact with DRM-protected media (e.g., Widevine).
531- **Resource Usage:** Each browser instance consumes significant RAM (~100-300MB+). Use `PagePool` and limit concurrency on memory-constrained systems.
532- **Extensions in Headless:** Chrome extensions do not work in headless mode. Use `Headless(false)` with XVFB for server environments.
533- **Platform:** Requires a Chromium-compatible browser. Does not support Firefox or Safari.
534 
535## Documentation References
536 
537- [Official Documentation](https://go-rod.github.io/) — Guides, tutorials, FAQ
538- [Go API Reference](https://pkg.go.dev/github.com/go-rod/rod) — Complete type and method documentation
539- [go-rod/stealth](https://github.com/go-rod/stealth) — Anti-bot detection plugin
540- [Examples (source)](https://github.com/go-rod/rod/blob/main/examples_test.go) — Official example tests
541- [Rod vs Chromedp Comparison](https://github.com/nichochar/go-rod.github.io/blob/main/lib/examples/compare-chromedp) — Migration reference
542- [Chrome DevTools Protocol Docs](https://chromedevtools.github.io/devtools-protocol/) — Underlying protocol reference
543- [Chrome CLI Flags Reference](https://peter.sh/experiments/chromium-command-line-switches) — Launcher flag documentation
544- `references/api-reference.md` — Quick-reference cheat sheet
545