Scraper

Why this exists

Many niche sites don't have an API, even though you wish they'd serve their information more easily. I've always wanted to plug these sites into real, usable programs like Shortcuts and Scriptable.

This setup does that, or tries to, depending on the site. It uses Playwright, a Python library that runs Chromium (the open-source engine behind Chrome) as an invisible browser. It clicks, types and logs in just like a person would, only automatically. We are basically the robots that CAPTCHA tests are trying to catch.

Two ways to use it

A config file covers most simple sites. You describe the page in a small JSON file (where to go, how to log in, which bits to grab) and scraper.py does the rest. No code to write.

A custom script is for sites a config can't handle. McDonald's myRestaurant logs in through Microsoft SSO and buries shifts in messy HTML, so it has its own script, scrape_roster.py. That one is the real-world example further down this page.

Files

Setup

1. Download the code

   Clone or download the repo, then open a terminal (like Command Prompt) in the folder by pasting the commands below.

   git clone https://github.com/Walter-CK/WalterCK.git
   cd WalterCK/scraper

2. Install Python

   You need Python 3.10 or newer. If running python --version in a terminal doesn't show that, download it from python.org. On Windows, tick "Add Python to PATH" during install. You'll need it for a lot of other stuff anyway.

3. Install the dependencies

   pip install -r requirements.txt
   playwright install chromium

   The second command downloads the Chromium browser that Playwright runs its robot on.

4. Add your login

   Copy .env.example to a new file called .env and fill in whatever the site needs. Some sites have no login at all. The demo accepts any values:

   QUOTES_USERNAME=any-name-works
   QUOTES_PASSWORD=any-password-works

   This file never leaves your machine. It's listed in .gitignore, so a normal git push won't include it. Don't paste your real credentials anywhere else, for security ofc. Only automate logins you're allowed to: some sites ban bots in their terms and can lock your account.

5. Run it

   python scraper.py sites/quotes-demo.json --show

   The --show flag pops up a visible browser window so you can watch the login happen and confirm it works. Once you've seen it succeed, drop the flag and it runs invisibly.

   A successful run ends with a line like Wrote 10 items to output/quotes-demo.json. If it fails partway through, it saves a screenshot and the page's HTML into output/ so you can see exactly what it was looking at when it broke.

6. Point it at your own site

   Copy sites/quotes-demo.json, change the URLs, and swap in your own selectors (right-click a page, Inspect, and copy the CSS selector). The keys:

   Key	What it does
   url	The page to read once you're logged in.
   wait_for	A selector that only exists once the page has loaded its data.
   login	Optional. Login page URL, the username, password and submit selectors, and the .env names holding your credentials. Add next_selector for sites that ask for the username first.
   extract.rows	The selector for one repeating item (a quote, a notice, a shift).
   extract.fields	What to pull out of each row. "span.text" gives the text, "a@href" gives an attribute.

Real-world example: McDonald's roster

This is the one I built the whole thing for. myRestaurant has no API, no calendar export and no notification when a new roster drops, so scrape_roster.py logs in through Microsoft SSO, reads the roster page and turns every shift into JSON. The rest of this page walks through that setup.

What comes out

Every run produces a fresh roster.json, shifts grouped by the roster's own weekly periods:

{
  "last_updated": "2026-09-25T09:25:00+10:00",
  "weeks": [
    {
      "week_start": "21/09/26",
      "week_end": "27/09/26",
      "shifts": [
        {
          "date": "23/09/26",
          "start": "1:00 PM",
          "end": "4:00 PM",
          "location": "BERRINBA QLD",
          "hours": "3:00"
        }
      ]
    }
  ]
}

hours is a duration, not a decimal ("3:15" means 3 hours 15 minutes, not 3.15 hours.)

Getting it onto your phone

The script runs on a computer, but the point is having your shifts on your phone without opening anything. The way I've got it wired up:

GitHub Actions Setup

GitHub Actions automates the scraping process entirely in the background.

1. Repo Secrets

   Store your myRestaurant credentials as GitHub Repository Secrets (MYJOB_USERNAME and MYJOB_PASSWORD). The workflow securely injects these during execution.

2. The Workflow File

   The scrape (1).yml file defines the environment. It installs Python, sets up the Playwright browser, executes the script, and commits the updated roster.json back to the repository.

3. The Schedule

   The workflow utilizes cron to run automatically every day at 12:00 AM UTC (0 0 * * *). It also supports repository_dispatch, allowing the iOS Shortcut to trigger a fresh scrape on demand.

4. Token Setup

   To allow the iOS Shortcut to read the private repository and trigger dispatches, generate a GitHub Personal Access Token (PAT). Scope this token strictly to read/write access for this specific repository and provide it to the Shortcut during setup.

What the Shortcut actually does

The Shortcut is the other half of this system. It reads roster.json and turns it into something you actually look at, without opening the myRestaurant app.

1. Pulls your latest roster

   Fetches roster.json straight from GitHub and groups your shifts into This Week and Next Week, showing the day, start time and end time for each one.

2. Sets alarms ahead of each shift

   Choose Create/Change Alarms, tell it how many minutes before a shift you want waking up, and it adds one alarm per upcoming shift automatically.

3. Keeps your alarms in sync

   Run it again after a new roster's published and it clears out the old shift alarms before creating the new ones, so you're never left with a stale alarm for a shift that got moved or dropped.

4. Triggers a fresh scrape on demand

   Choose Re-scan shifts and the Shortcut asks GitHub to run the scraper again right away, instead of waiting for the next scheduled run so you can check the moment a new roster gets published.

Built by Walter MacLean. Source and issues on GitHub.