Advanced Topics
***************


Adding URLs from the command line
=================================

Quickly adding new URLs to the job list from the command line:

   urlwatch --add url=http://example.org,name=Example


Using word-based differences
============================

You can also specify an external "diff"-style tool (a tool that takes
two filenames (old, new) as parameter and returns on its standard
output the difference of the files), for example to use *wdiff(1)* to
get word-based differences instead of line-based difference, or
pandiff to get markdown differences:

   url: https://example.com/
   diff_tool: wdiff

Note that "diff_tool" specifies an external command-line tool, so that
tool must be installed separately (e.g. "apt install wdiff" on Debian
or "brew install wdiff" on macOS). Syntax highlighting is supported
for "wdiff"-style output, but potentially not for other diff tools.


Ignoring whitespace changes
===========================

If you would like to ignore whitespace changes so that you don’t
receive notifications for trivial differences, you can use "diff_tool"
for this. For example:

   diff_tool: "diff --ignore-all-space --unified"

When using another external "diff"-like tool, make sure it returns
unified output format to retain syntax highlighting.


Only show added or removed lines
================================

The "diff_filter" feature can be used to filter the diff output text
with the same tools (see Filters) used for filtering web pages.

In order to show only diff lines with added lines, use:

   url: http://example.com/things-get-added.html
   diff_filter:
     - grep: '^[@+]'

This will only keep diff lines starting with "@" or "+". Similarly, to
only keep removed lines:

   url: http://example.com/things-get-removed.html
   diff_filter:
     - grep: '^[@-]'

More sophisticated diff filtering is possibly by combining existing
filters, writing a new filter or using "shellpipe" to delegate the
filtering/processing of the diff output to an external tool.

Read the next section if you want to disable empty notifications.


Disable empty notifications
===========================

As an extension to the previous example, let’s say you want to only
get notified with all lines added, but receive no notifications at all
if lines are removed.

A diff usually looks like this:

   --- @       Fri, 04 Mar 2022 19:58:14 +0100
   +++ @       Fri, 04 Mar 2022 19:58:22 +0100
   @@ -1,3 +1,3 @@
    someline
   -someotherlines
   +someotherline
    anotherline

We want to filter all lines starting with “+” only, but because of the
headers we also want to filter lines that start with “+++”, which can
be accomplished like so:

   url: http://example.com/only-added.html
   diff_filter:
     - grep: '^[+]'      # Include all lines starting with "+"
     - grepi: '^[+]{3}'  # Exclude the line starting with "+++"

This deals with all diff lines now, but since urlwatch reports
“changed” pages even when the "diff_filter" returns an empty string
(which might be useful in some cases), you have to explicitly opt out
by using "urlwatch --edit-config" and setting the "empty-diff" option
to "false" in the "display" category:

   display:
     empty-diff: false


Pass diff output to a custom script
===================================

In some situations, it might be useful to run a script with the diff
as input when changes were detected (e.g. to start an update or
process something). This can be done by combining "diff_filter" with
the "shellpipe" filter, which can be any custom script.

The output of the custom script will then be the diff result as
reported by urlwatch, so if it outputs any status, the "CHANGED"
notification that urlwatch does will contain the output of the custom
script, not the original diff. This can even have a “normal” filter
attached to only watch links (the "css: a" part of the filter
definitions):

   url: http://example.org/downloadlist.html
   filter:
     - css: a
   diff_filter:
     - shellpipe: /usr/local/bin/process_new_links.sh


Comparing web pages visually
============================

To compare the visual contents of web pages, Nicolai has written
pyvisualcompare as a frontend (with GUI) to "urlwatch". The tool can
be used to select a region of a web page. It then generates a
configuration for "urlwatch" to run "pyvisualcompare" and generate a
hash for the screen contents.


Ignoring connection errors
==========================

In some cases, it might be useful to ignore (temporary) network errors
to avoid notifications being sent. While there is a "display.error"
config option (defaulting to "true") to control reporting of errors
globally, to ignore network errors for specific jobs only, you can use
the "ignore_connection_errors" key in the job list configuration file:

   url: https://example.com/
   ignore_connection_errors: true

Similarly, you might want to ignore some (temporary) HTTP errors on
the server side:

   url: https://example.com/
   ignore_http_error_codes: 408, 429, 500, 502, 503, 504

or ignore all HTTP errors if you like:

   url: https://example.com/
   ignore_http_error_codes: 4xx, 5xx

You can also ignore incomplete reads:

   url: "https://example.com/"
   ignore_incomplete_reads: true


Overriding the content encoding
===============================

For web pages with misconfigured HTTP headers or rare encodings, it
may be useful to explicitly specify an encoding from Python’s Standard
Encodings.

   url: https://example.com/
   encoding: utf-8


Changing the default timeout
============================

By default, url jobs timeout after 60 seconds. If you want a different
timeout period, use the "timeout" key to specify it in number of
seconds, or set it to 0 to never timeout.

   url: https://example.com/
   timeout: 300


Supplying cookie data
=====================

It is possible to add cookies to HTTP requests for pages that need it,
the YAML syntax for this is:

   url: http://example.com/
   cookies:
       Key: ValueForKey
       OtherKey: OtherValue


Comparing with several latest snapshots
=======================================

If a webpage frequently changes between several known stable states,
it may be desirable to have changes reported only if the webpage
changes into a new unknown state. You can use "compared_versions" to
do this.

   url: https://example.com/
   compared_versions: 3

In this example, changes are only reported if the webpage becomes
different from the latest three distinct states. The differences are
shown relative to the closest match.


Receiving a report every time urlwatch runs
===========================================

If you are watching pages that change seldomly, but you still want to
be notified daily if "urlwatch" still works, you can watch the output
of the "date" command, for example:

   name: "urlwatch watchdog"
   command: "date"

Since the output of "date" changes every second, this job should
produce a report every time urlwatch is run.


Using Redis as a cache backend
==============================

If you want to use Redis as a cache backend over the default SQLite3
file:

   urlwatch --cache=redis://localhost:6379/

There is no migration path from the SQLite3 format, the cache will be
empty the first time Redis is used.


Watching changes on .onion (Tor) pages
======================================

Since pages on the Tor Network are not accessible via public DNS and
TCP, you need to either configure a Tor client as HTTP/HTTPS proxy or
use the *torify(1)* tool from the "tor" package ("apt install tor" on
Debian, "brew install tor" on macOS). Setting up Tor is out of scope
for this document. On a properly set up Tor installation, one can just
prefix the "urlwatch" command with the "torify" wrapper to access
.onion pages:

   torify urlwatch


Watching Facebook Page Events
=============================

If you want to be notified of new events on a public Facebook page,
you can use the following job pattern, replace "PAGE" with the name of
the page (can be found by navigating to the events page on your
browser):

   url: http://m.facebook.com/PAGE/pages/permalink/?view_type=tab_events
   filter:
     - css:
         selector: div#objects_container
         exclude: 'div.x, #m_more_friends_who_like_this, img'
     - re.sub:
         pattern: '(/events/\d*)[^"]*'
         repl: '\1'
     - html2text: pyhtml2text


Setting the content width for "html2text" ("lynx" method)
=========================================================

When using the "lynx" method in the "html2text" filter, it uses a
default width that will cause additional line breaks to be inserted.

To set the "lynx" output width to 400 characters, use this filter
setup:

   url: http://example.com/longlines.html
   filter:
     - html2text:
         method: lynx
         width: 400


Configuring how long browser jobs wait for pages to load
========================================================

For browser jobs, you can configure how long the headless browser will
wait before a page is considered loaded by using the "wait_until"
option.

It can take one of four values (see wait_until docs of Playwright):

   * "load" - consider operation to be finished when the load event is
     fired

   * "domcontentloaded" - consider operation to be finished when the
     DOMContentLoaded event is fired

   * "networkidle" - **discouraged** consider operation to be finished
     when there are no network connections for at least 500 ms. Don’t
     use this method for testing, rely on web assertions to assess
     readiness instead

   * "commit" - consider operation to be finished when network
     response is received and the document started loading


Treating "NEW" jobs as "CHANGED"
================================

In some cases (e.g. when the "diff_tool" or "diff_filter" executes
some external command as a side effect that should also run for the
initial page state), you can set the "treat_new_as_changed" to "true",
which will make the job report as "CHANGED" instead of "NEW" the first
time it is retrieved (and the diff will be reported, too).

   url: http://example.com/initialpage.html
   treat_new_as_changed: true

This option will also change the behavior of "--test-diff-filter", and
allow testing the diff filter if only a single version of the page has
been retrieved.


Monitoring the same URL in multiple jobs
========================================

Because urlwatch uses the "url"/"navigate" (for URL/Browser jobs)
and/or the "command" (for Shell jobs) key as unique identifier, each
URL can only appear in a single job. If you want to monitor the same
URL multiple times, you can append "#1", "#2", … (or anything that
makes them unique) to the URLs, like this:

   name: "Looking for Thing A"
   url: http://example.com/#1
   filter:
     - grep: "Thing A"
   ---
   name: "Looking for Thing B"
   url: http://example.com/#2
   filter:
     - grep: "Thing B"


Updating a URL and keeping past history
=======================================

Job history is stored based on the value of the "url" parameter, so
updating a job’s URL in the configuration file "urls.yaml" will create
a new job with no history.  Retain history by using "--change-
location":

   urlwatch --change-location http://example.org#old http://example.org#new

The command also works with Browser and Shell jobs, changing
"navigate" and "command" respectively.


Running a subset of jobs
========================

To run one or more specific jobs instead of all known jobs, provide
the job index numbers to the urlwatch command. For example, to run
jobs with index 2, 4, and 7:

   urlwatch 2 4 7


Sending HTML form data using POST
=================================

To simulate submitting a HTML form using the POST method, you can pass
the form fields in the "data" field of the job description:

   name: "My POST Job"
   url: http://example.com/foo
   data:
     username: "foo"
     password: "bar"
     submit: "Send query"

By default, the request will use the HTTP "POST" method, and the
"Content-type" will be set to "application/x-www-form-urlencoded".


Sending arbitrary data using HTTP PUT
=====================================

It is possible to customize the HTTP method and "Content-type" header,
allowing you to send arbitrary requests to the server:

   name: "My PUT Request"
   url: http://example.com/item/new
   method: PUT
   headers:
     Content-type: application/json
   data: '{"foo": true}'


UTF-8 support on Windows
========================

On Windows, the default file encoding might be locale-specific and not
work correctly if files are saved using the (recommended) UTF-8
encoding.

If you are having problems loading UTF-8-encoded files on Windows, you
might see an issue like the following when "urlwatch" parses your
config files:

   UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 214: character maps to <undefined>

To work around this issue, Python 3.7 and newer have a new UTF-8 Mode
that can be enabled by setting the environment variable "PYTHONUTF8"
to "1":

   set PYTHONUTF8=1
   urlwatch

You can also add this environment variable to your user environment or
system environment to apply the UTF-8 Mode to all Python programs on
your machine.
