Acquisition & Extraction
Goal
Turn JS-heavy faculty pages into structured outreach context.
Design
BeautifulSoup plus Playwright for pages that defeat static HTML. Extraction quality varies; the pipeline records failures instead of guessing emails.
Challenges
- JS-rendered directories break naive scrapers.
- Email formats and robots policies vary by department.
Iterations
- Static scrape
- Playwright path
- Per-site failure logging
Final implementation
Hybrid extraction with stored context for the drafting stage.