API documentation

The following documentation is based on the source code of version 7.3 of the apt-mirror-updater package. The following modules are available:

apt_mirror_updater

Automated, robust apt-get mirror selection for Debian and Ubuntu.

The main entry point for this module is the AptMirrorUpdater class, so if you don’t know where to start that would be a good place :-). You can also take a look at the source code of the apt_mirror_updater.cli module for an example that uses the AptMirrorUpdater class.

apt_mirror_updater.MAIN_SOURCES_LIST = '/etc/apt/sources.list'

The absolute pathname of the list of configured APT data sources (a string).

apt_mirror_updater.SOURCES_LIST_ENCODING = 'UTF-8'

The text encoding of MAIN_SOURCES_LIST (a string).

apt_mirror_updater.MAX_MIRRORS = 50

A sane default value for AptMirrorUpdater.max_mirrors.

apt_mirror_updater.LAST_UPDATED_DEFAULT = 2419200

A default, pessimistic last_updated value (a number).

class apt_mirror_updater.AptMirrorUpdater(**kw)[source]

Python API for the apt-mirror-updater program.

repr_properties = ('architecture', 'backend', 'blacklist', 'concurrency', 'context', 'distribution_codename', 'distributor_id', 'max_mirrors', 'old_releases_url', 'security_url')

Override the list of properties included in repr() output (a tuple of strings).

The PropertyManager superclass defines a __repr__() method that includes the values of computed properties in its output.

In the case of apt-mirror-updater this behavior would trigger external command execution and (lots of) HTTP calls, sometimes with unintended side effects, namely infinite recursion.

By setting repr_properties to a list of “safe” properties this problematic behavior can be avoided.

architecture[source]

The name of the Debian package architecture (a string like ‘i386’ or ‘amd64’).

The package architecture is used to detect whether Debian LTS status applies to the given system (the Debian LTS team supports a specific subset of package architectures).

Note

The architecture property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

available_mirrors[source]

A list of CandidateMirror objects (ordered from best to worst).

On Ubuntu the mirrors will be ordered by the time since they were most recently updated. On Debian this information isn’t available and the ordering of the list should be considered arbitrary.

Note

The available_mirrors property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

backend[source]

The backend module whose name matches distributor_id.

Raises:EnvironmentError when no matching backend module is available.

Note

The backend property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

best_mirror[source]

The URL of the first mirror in ranked_mirrors (a string).

This is a shortcut for using ranked_mirrors to select the best mirror from available_mirrors, falling back to the old releases URL when release_is_archived is True.

Note

The best_mirror property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

blacklist[source]

A set of strings with fnmatch patterns (defaults to an empty set).

When available_mirrors encounters a mirror whose URL matches one of the patterns in blacklist the mirror will be ignored.

Note

The blacklist property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

concurrency[source]

The number of concurrent HTTP connections allowed while ranking mirrors (a number).

The value of this property defaults to the value computed by get_default_concurrency().

Note

The concurrency property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

context[source]

An execution context created using executor.contexts.

The value of this property defaults to a LocalContext object.

Note

The context property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

current_mirror[source]

The URL of the main mirror in use in MAIN_SOURCES_LIST (a string).

The current_mirror property’s value is computed using find_current_mirror().

Note

The current_mirror property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

distribution_codename[source]

The distribution codename (a lowercase string like ‘trusty’ or ‘xenial’).

The value of this property defaults to the value of the executor.contexts.AbstractContext.distribution_codename property which is the right choice 99% of the time.

Note

The distribution_codename property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

distributor_id[source]

The distributor ID (a lowercase string like ‘debian’ or ‘ubuntu’).

The default value of this property is based on the distributor_id property of release (which in turn is based on distribution_codename).

Because Debian and Ubuntu code names are unambiguous this means that in practice you can provide a value for distribution_codename and omit distributor_id and everything should be fine.

Note

The distributor_id property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

max_mirrors[source]

Limits the number of mirrors to rank (a number, defaults to MAX_MIRRORS).

Note

The max_mirrors property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

old_releases_url[source]

The URL of the mirror that serves old releases for this backend (a string).

Note

The old_releases_url property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ranked_mirrors[source]

A list of CandidateMirror objects (ordered from best to worst).

The value of this property is computed by concurrently testing the mirrors in available_mirrors for the following details:

The number of mirrors to test is limited to max_mirrors and you can change the number of simultaneous HTTP connections allowed by setting concurrency.

Note

The ranked_mirrors property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

release[source]

A Release object corresponding to distributor_id and distribution_codename.

Note

The release property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

release_is_eol[source]

True if the release is EOL (end of life), False otherwise.

There are three ways in which the value of this property can be computed:

  • When available, the first of the following EOL dates will be compared against the current date to determine whether the release is EOL:

    • If the backend module contains a get_eol_date() function (only the debian module does at the time of writing) then it is called and if it returns a number, this number is the EOL date for the release.

      This function was added to enable apt-mirror-updater backend modules to override the default EOL dates, more specifically to respect the Debian LTS release schedule (see also issue #5).

    • Otherwise the eol_date of release is used.

  • As a fall back validate_mirror() is used to check whether security_url results in MirrorStatus.MAYBE_EOL.

See also

The release_is_archived property

Note

The release_is_eol property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

release_is_archived[source]

True if the release has been archived, False otherwise.

Archived releases are no longer available on regular package mirrors, instead they’re served from a dedicated old-releases environment.

This property was added to acknowledge the discrepancy between when a release hits its EOL date and when it’s actually removed from mirrors:

  • The EOL date of Ubuntu 14.04 (Trusty Tahr) is 2019-04-25.
  • At the time of writing it is 2020-04-17 and this release still hasn’t been archived! (a year later)

For more information please refer to issue #9.

See also

The release_is_eol property

Note

The release_is_archived property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

security_url[source]

The URL of the mirror that serves security updates for this backend (a string).

Note

The security_url property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

stable_mirror[source]

A mirror URL that is stable for the given execution context (a string).

The value of this property defaults to the value of current_mirror, however if the current mirror can’t be determined or is deemed inappropriate by validate_mirror() then best_mirror will be used instead.

This provides a stable mirror selection algorithm which is useful because switching mirrors causes apt-get update to unconditionally download all package lists and this takes a lot of time so should it be avoided when unnecessary.

Note

The stable_mirror property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

validated_mirrors[source]

Dictionary of validated mirrors (used by validate_mirror()).

Note

The validated_mirrors property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

change_mirror(new_mirror=None, update=True)[source]

Change the main mirror in use in MAIN_SOURCES_LIST.

Parameters:
  • new_mirror – The URL of the new mirror (a string, defaults to best_mirror).
  • update – Whether an apt-get update should be run after changing the mirror (a boolean, defaults to True).
clear_package_lists()[source]

Clear the package list cache by removing the files under /var/lib/apt/lists.

create_chroot(directory, arch=None)[source]

Bootstrap a basic Debian or Ubuntu system using debootstrap.

Parameters:
  • directory – The pathname of the target directory (a string).
  • arch – The target architecture (a string or None).
Returns:

A ChangeRootContext object.

If directory already exists and isn’t empty then it is assumed that the chroot has already been created and debootstrap won’t be run. Before this method returns it changes context to the chroot.

dumb_update(*args)[source]

Update the system’s package lists (by running apt-get update).

Parameters:args – Command line arguments to apt-get update (zero or more strings).

The dumb_update() method doesn’t do any error handling or retrying, if that’s what you’re looking for then you need smart_update() instead.

generate_sources_list(**options)[source]

Generate the contents of /etc/apt/sources.list.

If no mirror_url keyword argument is given then stable_mirror is used as a default.

Please refer to the documentation of the Debian (apt_mirror_updater.backends.debian.generate_sources_list()) and Ubuntu (apt_mirror_updater.backends.ubuntu.generate_sources_list()) backend implementations of this method for details on argument handling and the return value.

get_sources_list()[source]

Get the contents of MAIN_SOURCES_LIST.

Returns:A Unicode string.

This code currently assumes that the sources.list file is encoded using SOURCES_LIST_ENCODING. I’m not actually sure if this is correct because I haven’t been able to find a formal specification! Feedback is welcome :-).

ignore_mirror(pattern)[source]

Add a pattern to the mirror discovery blacklist.

Parameters:pattern – A shell pattern (containing wild cards like ? and *) that is matched against the full URL of each mirror.

When a pattern is added to the blacklist any previously cached values of available_mirrors, best_mirror, ranked_mirrors and stable_mirror are cleared. This makes sure that mirrors blacklisted after mirror discovery has already run are ignored.

install_sources_list(contents)[source]

Install a new /etc/apt/sources.list file.

Parameters:contents – The new contents of the sources list (a Unicode string). You can generate a suitable value using the generate_sources_list() method.
smart_update(*args, **kw)[source]

Update the system’s package lists (switching mirrors if necessary).

Parameters:
  • args – Command line arguments to apt-get update (zero or more strings).
  • max_attempts – The maximum number of attempts at successfully updating the system’s package lists (an integer, defaults to 10).
  • switch_mirrors – True if we’re allowed to switch mirrors on ‘hash sum mismatch’ errors, False otherwise.
Raises:

If updating of the package lists fails 10 consecutive times (max_attempts) an exception is raised.

While dumb_update() simply runs apt-get update the smart_update() function works quite differently:

  • First the system’s package lists are updated using dumb_update(). If this is successful we’re done.
  • If the update fails we check the command’s output for the phrase ‘hash sum mismatch’. If we find this phrase we assume that the current mirror is faulty and switch to another one.
  • Failing apt-get update runs are retried up to max_attempts.
validate_mirror(mirror_url)[source]

Make sure a mirror serves distribution_codename.

Parameters:mirror_url – The base URL of the mirror (a string).
Returns:One of the values in the MirrorStatus enumeration.

This method assumes that old_releases_url is always valid.

class apt_mirror_updater.CandidateMirror(**kw)[source]

A candidate mirror groups a mirror URL with its availability and performance metrics.

bandwidth[source]

The bytes per second achieved while fetching release_gpg_url (a number or None).

The value of this property is computed based on the values of release_gpg_contents and release_gpg_latency.

Note

The bandwidth property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

archive_update_in_progress_url[source]

The URL of the file whose existence indicates that the mirror is being updated (a string).

The value of this property is computed based on the value of mirror_url.

Note

The archive_update_in_progress_url property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

mirror_url[source]

The base URL of the mirror (a string).

Note

The mirror_url property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named mirror_url (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

is_available[source]

True if release_gpg_contents contains the expected header, False otherwise.

The value of this property is computed by checking whether release_gpg_contents contains the expected BEGIN PGP SIGNATURE header. This may seem like a rather obscure way of validating a mirror, but it was specifically chosen to detect all sorts of ways in which mirrors can be broken:

  • Webservers with a broken configuration that return an error page for all URLs.
  • Mirrors whose domain name registration has expired, where the domain is now being squatted and returns HTTP 200 OK responses for all URLs (whether they “exist” or not).

Note

The is_available property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_updating[source]

True if the mirror is being updated, False otherwise.

Note

The is_updating property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

last_updated[source]

The time in seconds since the most recent mirror update (a number or None).

Note

The last_updated property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

release_gpg_contents[source]

The contents downloaded from release_gpg_url (a string or None).

By downloading the file available at release_gpg_url and setting release_gpg_contents and release_gpg_latency you enable the bandwidth and is_available properties to be computed.

Note

The release_gpg_contents property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

release_gpg_latency[source]

The time it took to download release_gpg_url (a number or None).

By downloading the file available at release_gpg_url and setting release_gpg_contents and release_gpg_latency you enable the bandwidth and is_available properties to be computed.

Note

The release_gpg_latency property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

release_gpg_url[source]

The URL of the Release.gpg file that will be used to test the mirror (a string or None).

The value of this property is based on mirror_url and the distribution_codename property of the updater object.

Note

The release_gpg_url property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

sort_key[source]

A tuple that can be used to sort the mirror by its availability/performance metrics.

The tuple created by this property contains four numbers in the following order:

  1. The number 1 when is_available is True or the number 0 when is_available is False (because most importantly a mirror must be available).
  2. The number 0 when is_updating is True or the number 1 when is_updating is False (because being updated at this very moment is bad).
  3. The negated value of last_updated (because the lower last_updated is, the better). If last_updated is None then LAST_UPDATED_DEFAULT is used instead.
  4. The value of bandwidth (because the higher bandwidth is, the better).

By sorting CandidateMirror objects on these tuples in ascending order, the last mirror in the sorted results will be the “most suitable mirror” (given the available information).

Note

The sort_key property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

updater[source]

A reference to the AptMirrorUpdater object that created the candidate.

Note

The updater property is a custom_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class apt_mirror_updater.MirrorStatus[source]

Enumeration for mirror statuses determined by AptMirrorUpdater.validate_mirror().

AVAILABLE = <EnumValue: MirrorStatus.AVAILABLE [value=1]>

The mirror is accepting connections and serving the expected content.

MAYBE_EOL = <EnumValue: MirrorStatus.MAYBE_EOL [value=2]>

The mirror is serving HTTP 404 “Not Found” responses instead of the expected content.

UNAVAILABLE = <EnumValue: MirrorStatus.UNAVAILABLE [value=3]>

The mirror is not accepting connections or not serving the expected content.

apt_mirror_updater.find_current_mirror(sources_list)[source]

Find the URL of the main mirror that is currently in use by apt-get.

Parameters:sources_list – The contents of apt’s package resource list, e.g. the contents of MAIN_SOURCES_LIST (a string).
Returns:The URL of the main mirror in use (a string).
Raises:If the main mirror can’t be determined EnvironmentError is raised.

The main mirror is determined by looking for the first deb or deb-src directive in apt’s package resource list whose URL uses the HTTP or FTP scheme and whose components contain main.

apt_mirror_updater.mirrors_are_equal(a, b)[source]

Check whether two mirror URLS are equal.

Parameters:
  • a – The first mirror URL (a string).
  • b – The second mirror URL (a string).
Returns:

True if the mirror URLs are equal, False otherwise.
apt_mirror_updater.normalize_mirror_url(url)[source]

Normalize a mirror URL so it can be compared using string equality comparison.

Parameters:url – The mirror URL to normalize (a string).
Returns:The normalized mirror URL (a string).

apt_mirror_updater.backends.debian

Discovery of Debian package archive mirrors.

Here are references to some of the material that I’ve needed to consult while working on this module:

apt_mirror_updater.backends.debian.LTS_ARCHITECTURES = ('i386', 'amd64', 'armel', 'armhf')

The names of the architectures supported by the Debian LTS team (a tuple of strings).

apt_mirror_updater.backends.debian.LTS_RELEASES = {'jessie': 1593468000, 'stretch': 1656540000}

A dictionary with Debian LTS releases and their EOL dates.

This is needed because distro-info-data doesn’t contain information about Debian LTS releases but nevertheless archive.debian.org doesn’t adopt a release until the LTS status expires (this was originally reported in issue #5).

apt_mirror_updater.backends.debian.MIRRORS_URL = 'https://www.debian.org/mirror/list'

The URL of the HTML page listing all primary Debian mirrors (a string).

apt_mirror_updater.backends.debian.SECURITY_URL = 'http://security.debian.org/'

The base URL of the Debian mirror with security updates (a string).

apt_mirror_updater.backends.debian.OLD_RELEASES_URL = 'http://archive.debian.org/debian-archive/debian/'

The URL where EOL (end of life) Debian releases are hosted (a string).

apt_mirror_updater.backends.debian.DEFAULT_SUITES = ('release', 'security', 'updates')

A tuple of strings with the Debian suites that are enabled by default.

apt_mirror_updater.backends.debian.VALID_COMPONENTS = ('main', 'contrib', 'non-free')

A tuple of strings with the names of the components available in the Debian package repositories.

apt_mirror_updater.backends.debian.VALID_SUITES = ('release', 'security', 'updates', 'backports', 'proposed-updates')

A tuple of strings with the names of the suites available in the Debian package repositories.

apt_mirror_updater.backends.debian.discover_mirrors()[source]

Discover available Debian mirrors by querying MIRRORS_URL.

Returns:A set of CandidateMirror objects that have their mirror_url property set.
Raises:If no mirrors are discovered an exception is raised.

An example run:

>>> from apt_mirror_updater.backends.debian import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
set([CandidateMirror(mirror_url='http://ftp.at.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.au.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.be.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.bg.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.br.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.by.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.ca.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.ch.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.cn.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.cz.debian.org/debian/'),
     ...])
apt_mirror_updater.backends.debian.generate_sources_list(mirror_url, codename, suites=('release', 'security', 'updates'), components=('main', 'contrib', 'non-free'), enable_sources=False)[source]

Generate the contents of /etc/apt/sources.list for a Debian system.

Parameters:
  • mirror_url – The base URL of the mirror (a string).
  • codename – The codename of a Debian release (a string like ‘wheezy’ or ‘jessie’) or a Debian release class (a string like ‘stable’, ‘testing’, etc).
  • suites – An iterable of strings (defaults to DEFAULT_SUITES, refer to VALID_SUITES for details).
  • components – An iterable of strings (refer to VALID_COMPONENTS for details).
  • enable_sources – True to include deb-src entries, False to omit them.
Returns:

The suggested contents of /etc/apt/sources.list (a string).
apt_mirror_updater.backends.debian.get_eol_date(updater)[source]

Override the EOL date for Debian LTS releases.

Parameters:updater – The AptMirrorUpdater object.
Returns:The overridden EOL date (a number) or None.

apt_mirror_updater.backends.elementary

Support for Elementary OS package archive mirror selection.

Elementary OS is based on Ubuntu LTS releases and as such this module is a very thin wrapper for the apt_mirror_updater.backends.ubuntu module.

apt_mirror_updater.backends.elementary.OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/'

Alias for apt_mirror_updater.backends.ubuntu.OLD_RELEASES_URL.

apt_mirror_updater.backends.elementary.SECURITY_URL = 'http://security.ubuntu.com/ubuntu'

Alias for apt_mirror_updater.backends.ubuntu.SECURITY_URL.

apt_mirror_updater.backends.elementary.discover_mirrors()[source]

Alias for apt_mirror_updater.backends.ubuntu.discover_mirrors().

apt_mirror_updater.backends.elementary.generate_sources_list(mirror_url, codename, suites=('release', 'updates', 'backports', 'security'), components=('main', 'restricted', 'universe', 'multiverse'), enable_sources=False)[source]

Alias for apt_mirror_updater.backends.ubuntu.generate_sources_list().

apt_mirror_updater.backends.elementary.KNOWN_RELEASES = [Release(codename='Jupiter', distributor_id='elementary', series='jupiter'), Release(codename='Luna', distributor_id='elementary', series='luna'), Release(codename='Freya', distributor_id='elementary', series='freya'), Release(codename='Loki', distributor_id='elementary', series='loki'), Release(codename='Juno', distributor_id='elementary', series='juno'), Release(codename='Hera', distributor_id='elementary', series='hera')]

List of Release objects corresponding to known elementary OS releases based on the summary table on the following web page: https://en.wikipedia.org/wiki/Elementary_OS#Summary_table

apt_mirror_updater.backends.ubuntu

Discovery of Ubuntu package archive mirrors.

apt_mirror_updater.backends.ubuntu.MIRRORS_URL = 'https://launchpad.net/ubuntu/+archivemirrors'

The URL of the HTML page listing official Ubuntu mirrors (a string).

apt_mirror_updater.backends.ubuntu.MIRROR_SELECTION_URL = 'http://mirrors.ubuntu.com/mirrors.txt'

The URL of a plain text listing of “geographically suitable” mirror URLs (a string).

apt_mirror_updater.backends.ubuntu.OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/'

The URL where EOL (end of life) Ubuntu releases are hosted (a string).

apt_mirror_updater.backends.ubuntu.SECURITY_URL = 'http://security.ubuntu.com/ubuntu'

The URL where Ubuntu security updates are hosted (a string).

apt_mirror_updater.backends.ubuntu.DEFAULT_SUITES = ('release', 'updates', 'backports', 'security')

A tuple of strings with the Ubuntu suites that are enabled by default.

apt_mirror_updater.backends.ubuntu.VALID_COMPONENTS = ('main', 'restricted', 'universe', 'multiverse')

A tuple of strings with the names of the components available in the Ubuntu package repositories.

apt_mirror_updater.backends.ubuntu.VALID_SUITES = ('release', 'security', 'updates', 'backports', 'proposed')

A tuple of strings with the names of the suites available in the Ubuntu package repositories.

The actual name of the ‘release’ suite is the codename of the relevant Ubuntu release, while the names of the other suites are formed by concatenating the codename with the suite name (separated by a dash).

As an example to make things more concrete, Ubuntu 16.04 has the following five suites available: xenial (this is the release suite), xenial-security, xenial-updates, xenial-backports and xenial-proposed.

apt_mirror_updater.backends.ubuntu.MIRROR_STATUSES = (('Up to date', 0), ('One hour behind', 3600), ('Two hours behind', 7200), ('Four hours behind', 14400), ('Six hours behind', 21600), ('One day behind', 86400), ('Two days behind', 172800), ('One week behind', 604800), ('Unknown', None))

A tuple of tuples with Launchpad mirror statuses. Each tuple consists of two values:

  1. The human readable mirror latency (a string) as used on MIRRORS_URL.
  2. The mirror latency expressed in seconds (a number).

The ‘known statuses’ used by Launchpad were checked as follows:

$ curl -s https://launchpad.net/+icing/rev18391/combo.css | tr '{},.' '\n' | grep distromirrorstatus
distromirrorstatusUP
distromirrorstatusONEHOURBEHIND
distromirrorstatusTWOHOURSBEHIND
distromirrorstatusFOURHOURSBEHIND
distromirrorstatusSIXHOURSBEHIND
distromirrorstatusONEDAYBEHIND
distromirrorstatusTWODAYSBEHIND
distromirrorstatusONEWEEKBEHIND
distromirrorstatusUNKNOWN
apt_mirror_updater.backends.ubuntu.discover_mirrors()[source]

Discover available Ubuntu mirrors.

Returns:A set of CandidateMirror objects that have their mirror_url property set and may have the last_updated property set.
Raises:If no mirrors are discovered an exception is raised.

This queries MIRRORS_URL and MIRROR_SELECTION_URL to discover available Ubuntu mirrors. Here’s an example run:

>>> from apt_mirror_updater.backends.ubuntu import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
set([CandidateMirror(mirror_url='http://archive.ubuntu.com/ubuntu/'),
     CandidateMirror(mirror_url='http://ftp.nluug.nl/os/Linux/distr/ubuntu/'),
     CandidateMirror(mirror_url='http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/'),
     CandidateMirror(mirror_url='http://ftp.tudelft.nl/archive.ubuntu.com/'),
     CandidateMirror(mirror_url='http://mirror.1000mbps.com/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.amsiohosting.net/archive.ubuntu.com/'),
     CandidateMirror(mirror_url='http://mirror.i3d.net/pub/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.nforce.com/pub/linux/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.nl.leaseweb.net/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.transip.net/ubuntu/ubuntu/'),
     ...])
apt_mirror_updater.backends.ubuntu.discover_mirror_selection()[source]

Discover “geographically suitable” Ubuntu mirrors.

apt_mirror_updater.backends.ubuntu.generate_sources_list(mirror_url, codename, suites=('release', 'updates', 'backports', 'security'), components=('main', 'restricted', 'universe', 'multiverse'), enable_sources=False)[source]

Generate the contents of /etc/apt/sources.list for an Ubuntu system.

Parameters:
  • mirror_url – The base URL of the mirror (a string).
  • codename – The codename of the Ubuntu release (a string like ‘trusty’ or ‘xenial’).
  • suites – An iterable of strings (defaults to DEFAULT_SUITES, refer to VALID_SUITES for details).
  • components – An iterable of strings (refer to VALID_COMPONENTS for details).
  • enable_sources – True to include deb-src entries, False to omit them.
Returns:

The suggested contents of /etc/apt/sources.list (a string).

apt_mirror_updater.cli

Usage: apt-mirror-updater [OPTIONS]

The apt-mirror-updater program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating.

Supported options:

Option Description
-r, --remote-host=SSH_ALIAS Operate on a remote system instead of the local system. The SSH_ALIAS argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access.
-f, --find-current-mirror Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output.
-b, --find-best-mirror Discover available mirrors, rank them, select the best one and report its URL on standard output.
-l, --list-mirrors List available (ranked) mirrors on the terminal in a human readable format.
-c, --change-mirror=MIRROR_URL Update /etc/apt/sources.list to use the given MIRROR_URL.
-a, --auto-change-mirror Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror.
-u, --update, --update-package-lists Update the package lists using “apt-get update”, retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated.
-x, --exclude=PATTERN Add a pattern to the mirror selection blacklist. PATTERN is expected to be a shell pattern (containing wild cards like “?” and “*”) that is matched against the full URL of each mirror.
-m, --max=COUNT

Don’t query more than COUNT mirrors for their connection status (defaults to 50). If you give the number 0 no limit will be applied.

Because Ubuntu mirror discovery can report more than 300 mirrors it’s useful to limit the number of mirrors that are queried, otherwise the ranking of mirrors will take a long time (because over 300 connections need to be established).
-v, --verbose Increase logging verbosity (can be repeated).
-q, --quiet Decrease logging verbosity (can be repeated).
-h, --help Show this message and exit.
apt_mirror_updater.cli.main()[source]

Command line interface for the apt-mirror-updater program.

apt_mirror_updater.cli.report_current_mirror(updater)[source]

Print the URL of the currently configured apt-get mirror.

apt_mirror_updater.cli.report_best_mirror(updater)[source]

Print the URL of the “best” mirror.

apt_mirror_updater.cli.report_available_mirrors(updater)[source]

Print the available mirrors to the terminal (in a human friendly format).

apt_mirror_updater.http

Simple, robust and concurrent HTTP requests (designed for one very narrow use case).

apt_mirror_updater.http.fetch_url(url, timeout=10, retry=False, max_attempts=3)[source]

Fetch a URL, optionally retrying on failure.

Parameters:
  • url – The URL to fetch (a string).
  • timeout – The maximum time in seconds that’s allowed to pass before the request is aborted (a number, defaults to 10 seconds).
  • retry – Whether to retry on failure (defaults to False).
  • max_attempts – The maximum number of attempts when retrying is enabled (an integer, defaults to three).
Returns:

The response body (a byte string).
Raises:

Any of the following exceptions can be raised:

  • NotFoundError when the URL returns a 404 status code.
  • InvalidResponseError when the URL returns a status code that isn’t 200.
  • stopit.TimeoutException when the request takes longer than timeout seconds (refer to the linked documentation for details).
  • Any exception raised by Python’s standard library in the last attempt (assuming all attempts raise an exception).
apt_mirror_updater.http.fetch_concurrent(urls, concurrency=None)[source]

Fetch the given URLs concurrently using multiprocessing.

Parameters:
  • urls – An iterable of URLs (strings).
  • concurrency – Override the concurrency (an integer, defaults to the value computed by get_default_concurrency()).
Returns:

A list of tuples like those returned by fetch_worker().
apt_mirror_updater.http.get_default_concurrency()[source]

Get the default concurrency for fetch_concurrent().

Returns:A positive integer number.
apt_mirror_updater.http.fetch_worker(url)[source]

Fetch the given URL for fetch_concurrent().

Parameters:url – The URL to fetch (a string).
Returns:A tuple of three values:
  1. The URL that was fetched (a string).
  2. The data that was fetched (a string or None).
  3. The number of seconds it took to fetch the URL (a number).
exception apt_mirror_updater.http.InvalidResponseError[source]

Raised by fetch_url() when a URL returns a status code that isn’t 200.

exception apt_mirror_updater.http.NotFoundError[source]

Raised by fetch_url() when a URL returns a 404 status code.

apt_mirror_updater.releases

Easy to use metadata on Debian and Ubuntu releases.

This module started out with the purpose of reliable end of life (EOL) detection for Debian and Ubuntu releases based on data provided by the distro-info-data package. Since then the need arose to access more of the available metadata and so the eol module became the releases module.

Debian and Ubuntu releases have an EOL date that marks the end of support for each release. At that date the release stops receiving further (security) updates and some time after package mirrors stop serving the release.

The distro-info-data package contains CSV files with metadata about Debian and Ubuntu releases. This module parses those CSV files to make this metadata available in Python. This enables apt-mirror-updater to make an informed decision about the following questions:

  1. Is a given Debian or Ubuntu release expected to be available on mirrors or will it only be available in the archive of old releases?
  2. Is the signing key of a given Ubuntu release expected to be included in the main keyring (UBUNTU_KEYRING_CURRENT) or should the keyring with removed keys (UBUNTU_KEYRING_REMOVED) be used?

To make it possible to run apt-mirror-updater without direct access to the CSV files, a copy of the relevant information has been embedded in the source code.

apt_mirror_updater.releases.DISTRO_INFO_DIRECTORY = '/usr/share/distro-info'

The pathname of the directory with CSV files containing release metadata (a string).

apt_mirror_updater.releases.DEBIAN_KEYRING_CURRENT = '/usr/share/keyrings/debian-keyring.gpg'

The pathname of the main Debian keyring file (a string).

apt_mirror_updater.releases.UBUNTU_KEYRING_CURRENT = '/usr/share/keyrings/ubuntu-archive-keyring.gpg'

The pathname of the main Ubuntu keyring file (a string).

apt_mirror_updater.releases.UBUNTU_KEYRING_REMOVED = '/usr/share/keyrings/ubuntu-archive-removed-keys.gpg'

The pathname of the Ubuntu keyring file with removed keys (a string).

apt_mirror_updater.releases.coerce_release(value)[source]

Try to coerce the given value to a Debian or Ubuntu release.

Parameters:value – The value to coerce (a number, a string or a Release object).
Returns:A Release object.
Raises:ValueError when the given value cannot be coerced to a known release.

The following values can be coerced:

Warning

Don’t use floating point numbers like 10.04 because their actual value will be something like 10.039999999999999147 which won’t match the intended release.

apt_mirror_updater.releases.discover_releases()[source]

Discover known Debian, Elementary OS and Ubuntu releases.

Returns:A list of discovered Release objects sorted by distributor_id and version.

The first time this function is called it will try to parse the CSV files in /usr/share/distro-info using parse_csv_file() and merge any releases it finds with the releases embedded into the source code of this module and the releases defined by apt_mirror_updater.backends.elementary.KNOWN_RELEASES. The result is cached and returned each time the function is called. It’s not a problem if the /usr/share/distro-info directory doesn’t exist or doesn’t contain any *.csv files (it won’t cause a warning or error). Of course in this case only the embedded releases will be returned.

apt_mirror_updater.releases.is_version_string(value)[source]

Check whether the given value is a string containing a positive number.

apt_mirror_updater.releases.parse_csv_file(filename)[source]

Parse a CSV file in the format of the /usr/share/distro-info/*.csv files.

Parameters:filename – The pathname of the CSV file (a string).
Returns:A generator of Release objects.
apt_mirror_updater.releases.parse_date(value)[source]

Convert a YYYY-MM-DD string to a datetime.date object.

apt_mirror_updater.releases.parse_version(value)[source]

Convert a version string to a Decimal number.

apt_mirror_updater.releases.ubuntu_keyring_updated()[source]

Detect update #1363482 to the ubuntu-keyring package.

Returns:True when version 2016.10.27 or newer is installed, False when an older version is installed.

This function checks if the changes discussed in Launchpad bug #1363482 apply to the current system using the dpkg-query --show and dpkg --compare-versions commands. For more details refer to issue #8.

class apt_mirror_updater.releases.Release(**kw)[source]

Data class for metadata on Debian, Elementary OS and Ubuntu releases.

codename[source]

The long version of series (a string).

Note

The codename property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named codename (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

created_date[source]

The date on which the release was created (a date object).

Note

The created_date property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named created_date (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

distributor_id[source]

The name of the distributor (one of the strings debian, elementary or ubuntu).

Note

The distributor_id property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named distributor_id (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

eol_date[source]

The date on which the desktop release stops being supported (a date object).

Note

The eol_date property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

extended_eol_date[source]

The date on which the server release stops being supported (a date object).

Note

The extended_eol_date property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

is_eol[source]

Whether the release has reached its end-of-life date (a boolean or None).

Note

The is_eol property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

is_lts[source]

Whether a release is a long term support release (a boolean).

Note

The is_lts property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

release_date[source]

The date on which the release was published (a date object).

Note

The release_date property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

series[source]

The short version of codename (a string).

Note

The series property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named series (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

upstream_distributor_id[source]

The upstream distributor ID (a string, defaults to distributor_id).

Note

The upstream_distributor_id property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

upstream_series[source]

The upstream series (a string, defaults to series).

Note

The upstream_series property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

upstream_version[source]

The upstream version (a string, defaults to version).

Note

The upstream_version property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

version[source]

The version number of the release (a Decimal number).

This property has a Decimal value to enable proper sorting based on numeric comparison.

Note

The version property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

keyring_file[source]

The pathname of the keyring with signing keys for this release (a string).

This property exists to work around a bug in debootstrap which may use the wrong keyring to create Ubuntu chroots, for more details refer to ubuntu_keyring_updated().

Note

The keyring_file property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

__str__()[source]

Render a human friendly representation of a Release object.

The result will be something like this:

  • Debian 9 (stretch)
  • Ubuntu 18.04 (bionic)