apt-mirror-updater: Automated Debian/Ubuntu mirror selection

Welcome to the documentation of apt-mirror-updater version 7.3! The following sections are available:

User documentation

The readme is the best place to start reading, it’s targeted at all users and documents the command line interface:

apt-mirror-updater: Automated Debian/Ubuntu mirror selection

The apt-mirror-updater package automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating (see features). It’s currently tested on Python 2.7, 3.5+ and PyPy (although test coverage is still rather low, see status).

Features

Discovery of available mirrors

Debian and Ubuntu mirrors are discovered automatically by querying the Debian mirror list or the Ubuntu mirror list (the applicable mirror list is automatically selected based on the current platform).

Ranking of available mirrors

Discovered mirrors are ranked by bandwidth (to pick the fastest mirror) and excluded if they’re being updated (see issues with mirror updates).

Automatic switching between mirrors

The main mirror configured in /etc/apt/sources.list can be changed with a single command. The new (to be configured) mirror can be selected automatically or configured explicitly by the user.

Robust package list updating

Several apt-get subcommands can fail if the current mirror is being updated (see issues with mirror updates) and apt-mirror-updater tries to work around this by wrapping apt-get update to retry on failures and automatically switch to a different mirror when it looks like the current mirror is being updated (because I’ve seen such updates take more than 15 minutes and it’s not always acceptable to wait for so long, especially in automated solutions).

Status

On the one hand the apt-mirror-updater package was developed based on quite a few years of experience in using apt-get on Debian and Ubuntu systems and large scale automation of apt-get (working on 150+ remote systems). On the other hand the Python package itself is relatively new: it was developed and published in March 2016. As such:

Warning

Until apt-mirror-updater has been rigorously tested I consider it a proof of concept (beta software) so if it corrupts your system you can’t complain that you weren’t warned! I’ve already tested it on a variety of Ubuntu systems but haven’t found the time to set up a Debian virtual machine for testing. Most of the logic is exactly the same though. The worst that can happen (assuming you trust my judgement ;-) is that /etc/apt/sources.list is corrupted however a backup copy is made before any changes are applied, so I don’t see how this can result in irreversible corruption.

I’m working on an automated test suite but at the moment I’m still a bit fuzzy on how to create representative tests for the error handling code paths (also, writing a decent test suite requires a significant chunk of time :-).

Installation

The apt-mirror-updater package is available on PyPI which means installation should be as simple as:

$ pip install apt-mirror-updater

There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).

Usage

There are two ways to use the apt-mirror-updater package: As the command line program apt-mirror-updater and as a Python API. For details about the Python API please refer to the API documentation available on Read the Docs. The command line interface is described below.

Usage: apt-mirror-updater [OPTIONS]

The apt-mirror-updater program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating.

Supported options:

Option	Description
-r, --remote-host=SSH_ALIAS	Operate on a remote system instead of the local system. The SSH_ALIAS argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access.
-f, --find-current-mirror	Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output.
-b, --find-best-mirror	Discover available mirrors, rank them, select the best one and report its URL on standard output.
-l, --list-mirrors	List available (ranked) mirrors on the terminal in a human readable format.
-c, --change-mirror=MIRROR_URL	Update /etc/apt/sources.list to use the given MIRROR_URL.
-a, --auto-change-mirror	Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror.
-u, --update, --update-package-lists	Update the package lists using “apt-get update”, retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated.
-x, --exclude=PATTERN	Add a pattern to the mirror selection blacklist. PATTERN is expected to be a shell pattern (containing wild cards like “?” and “*”) that is matched against the full URL of each mirror.
-m, --max=COUNT	Don’t query more than COUNT mirrors for their connection status (defaults to 50). If you give the number 0 no limit will be applied. Because Ubuntu mirror discovery can report more than 300 mirrors it’s useful to limit the number of mirrors that are queried, otherwise the ranking of mirrors will take a long time (because over 300 connections need to be established).
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

Issues with mirror updates

Over the past five years my team (at work) and I have been managing a cluster of 150+ Ubuntu servers, initially using manual system administration but over time automating apt-get for a variety of use cases (provisioning, security updates, deployments, etc.). As we increased our automation we started running into various transient failure modes of apt-get, primarily with apt-get update but incidentally also with other subcommands.

The most frequent failure that we run into is apt-get update crapping out with ‘hash sum mismatch’ errors (see also Debian bug #624122). When this happens a file called Archive-Update-in-Progress-* can sometimes be found on the index page of the mirror that is being used (see also Debian bug #110837). I’ve seen these situations last for more than 15 minutes.

My working theory about these ‘hash sum mismatch’ errors is that they are caused by the fact that mirror updates aren’t atomic, apparently causing apt-get update to download a package list whose datafiles aren’t consistent with each other. If this assumption proves to be correct (and also assuming that different mirrors are updated at different times :-) then the command apt-mirror-updater --update-package-lists should work around this annoying failure mode (by automatically switching to a different mirror when ‘hash sum mismatch’ errors are encountered).

Publishing apt-mirror-updater to the world is my attempt to contribute to this situation instead of complaining in bug trackers (see above) where no robust and automated solution is emerging (at the time of writing). Who knows, maybe some day these issues will be resolved by moving logic similar to what I’ve implemented here into apt-get itself. Of course it would also help if mirror updates were atomic…

Contact

The latest version of apt-mirror-updater is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.

License

This software is licensed under the MIT license.

© 2020 Peter Odding.

API documentation

The following API documentation is automatically generated from the source code:

API documentation

The following documentation is based on the source code of version 7.3 of the apt-mirror-updater package. The following modules are available:

apt_mirror_updater

Automated, robust apt-get mirror selection for Debian and Ubuntu.

The main entry point for this module is the AptMirrorUpdater class, so if you don’t know where to start that would be a good place :-). You can also take a look at the source code of the apt_mirror_updater.cli module for an example that uses the AptMirrorUpdater class.

apt_mirror_updater.MAIN_SOURCES_LIST = '/etc/apt/sources.list'

The absolute pathname of the list of configured APT data sources (a string).

apt_mirror_updater.SOURCES_LIST_ENCODING = 'UTF-8'

The text encoding of MAIN_SOURCES_LIST (a string).

apt_mirror_updater.MAX_MIRRORS = 50

A sane default value for AptMirrorUpdater.max_mirrors.

apt_mirror_updater.LAST_UPDATED_DEFAULT = 2419200

A default, pessimistic last_updated value (a number).

class apt_mirror_updater.AptMirrorUpdater(**kw)

Python API for the apt-mirror-updater program.

repr_properties = ('architecture', 'backend', 'blacklist', 'concurrency', 'context', 'distribution_codename', 'distributor_id', 'max_mirrors', 'old_releases_url', 'security_url')

Override the list of properties included in repr() output (a tuple of strings).

The PropertyManager superclass defines a __repr__() method that includes the values of computed properties in its output.

In the case of apt-mirror-updater this behavior would trigger external command execution and (lots of) HTTP calls, sometimes with unintended side effects, namely infinite recursion.

By setting repr_properties to a list of “safe” properties this problematic behavior can be avoided.

architecture

The name of the Debian package architecture (a string like ‘i386’ or ‘amd64’).

The package architecture is used to detect whether Debian LTS status applies to the given system (the Debian LTS team supports a specific subset of package architectures).

Note

The architecture property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

available_mirrors

A list of CandidateMirror objects (ordered from best to worst).

On Ubuntu the mirrors will be ordered by the time since they were most recently updated. On Debian this information isn’t available and the ordering of the list should be considered arbitrary.

Note

The available_mirrors property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

backend

The backend module whose name matches distributor_id.

Raises:EnvironmentError when no matching backend module is available.

Note

The backend property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

best_mirror

The URL of the first mirror in ranked_mirrors (a string).

This is a shortcut for using ranked_mirrors to select the best mirror from available_mirrors, falling back to the old releases URL when release_is_archived is True.

Note

The best_mirror property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

blacklist

A set of strings with fnmatch patterns (defaults to an empty set).

When available_mirrors encounters a mirror whose URL matches one of the patterns in blacklist the mirror will be ignored.

Note

The blacklist property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

concurrency

The number of concurrent HTTP connections allowed while ranking mirrors (a number).

The value of this property defaults to the value computed by get_default_concurrency().

Note

The concurrency property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

context

An execution context created using executor.contexts.

The value of this property defaults to a LocalContext object.

Note

The context property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

current_mirror

The URL of the main mirror in use in MAIN_SOURCES_LIST (a string).

The current_mirror property’s value is computed using find_current_mirror().

Note

The current_mirror property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

distribution_codename

The distribution codename (a lowercase string like ‘trusty’ or ‘xenial’).

The value of this property defaults to the value of the executor.contexts.AbstractContext.distribution_codename property which is the right choice 99% of the time.

Note

The distribution_codename property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

distributor_id

The distributor ID (a lowercase string like ‘debian’ or ‘ubuntu’).

The default value of this property is based on the distributor_id property of release (which in turn is based on distribution_codename).

Because Debian and Ubuntu code names are unambiguous this means that in practice you can provide a value for distribution_codename and omit distributor_id and everything should be fine.

Note

The distributor_id property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

max_mirrors

Limits the number of mirrors to rank (a number, defaults to MAX_MIRRORS).

Note

The max_mirrors property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

old_releases_url

The URL of the mirror that serves old releases for this backend (a string).

Note

The old_releases_url property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ranked_mirrors

A list of CandidateMirror objects (ordered from best to worst).

The value of this property is computed by concurrently testing the mirrors in available_mirrors for the following details:

• availability (is_available)
• connection speed (bandwidth)
• update status (is_updating)

The number of mirrors to test is limited to max_mirrors and you can change the number of simultaneous HTTP connections allowed by setting concurrency.

Note

The ranked_mirrors property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

release

A Release object corresponding to distributor_id and distribution_codename.

Note

The release property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

release_is_eol

True if the release is EOL (end of life), False otherwise.

There are three ways in which the value of this property can be computed:

• When available, the first of the following EOL dates will be compared against the current date to determine whether the release is EOL:

  • If the backend module contains a get_eol_date() function (only the debian module does at the time of writing) then it is called and if it returns a number, this number is the EOL date for the release.

    This function was added to enable apt-mirror-updater backend modules to override the default EOL dates, more specifically to respect the Debian LTS release schedule (see also issue #5).

  • Otherwise the eol_date of release is used.

• As a fall back validate_mirror() is used to check whether security_url results in MirrorStatus.MAYBE_EOL.

See also

The release_is_archived property

Note

The release_is_eol property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

release_is_archived

True if the release has been archived, False otherwise.

Archived releases are no longer available on regular package mirrors, instead they’re served from a dedicated old-releases environment.

This property was added to acknowledge the discrepancy between when a release hits its EOL date and when it’s actually removed from mirrors:

• The EOL date of Ubuntu 14.04 (Trusty Tahr) is 2019-04-25.
• At the time of writing it is 2020-04-17 and this release still hasn’t been archived! (a year later)

For more information please refer to issue #9.

See also

The release_is_eol property

Note

The release_is_archived property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

security_url

The URL of the mirror that serves security updates for this backend (a string).

Note

The security_url property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

stable_mirror

A mirror URL that is stable for the given execution context (a string).

The value of this property defaults to the value of current_mirror, however if the current mirror can’t be determined or is deemed inappropriate by validate_mirror() then best_mirror will be used instead.

This provides a stable mirror selection algorithm which is useful because switching mirrors causes apt-get update to unconditionally download all package lists and this takes a lot of time so should it be avoided when unnecessary.

Note

The stable_mirror property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

validated_mirrors

Dictionary of validated mirrors (used by validate_mirror()).

Note

The validated_mirrors property is a cached_property. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

change_mirror(new_mirror=None, update=True)

Change the main mirror in use in MAIN_SOURCES_LIST.

Parameters:

• new_mirror – The URL of the new mirror (a string, defaults to best_mirror).
• update – Whether an apt-get update should be run after changing the mirror (a boolean, defaults to True).

clear_package_lists()

Clear the package list cache by removing the files under /var/lib/apt/lists.

create_chroot(directory, arch=None)

Bootstrap a basic Debian or Ubuntu system using debootstrap.

Parameters:

• directory – The pathname of the target directory (a string).
• arch – The target architecture (a string or None).

Returns:

A ChangeRootContext object.

If directory already exists and isn’t empty then it is assumed that the chroot has already been created and debootstrap won’t be run. Before this method returns it changes context to the chroot.

dumb_update(*args)

Update the system’s package lists (by running apt-get update).

Parameters:args – Command line arguments to apt-get update (zero or more strings).

The dumb_update() method doesn’t do any error handling or retrying, if that’s what you’re looking for then you need smart_update() instead.

generate_sources_list(**options)

Generate the contents of /etc/apt/sources.list.

If no mirror_url keyword argument is given then stable_mirror is used as a default.

Please refer to the documentation of the Debian (apt_mirror_updater.backends.debian.generate_sources_list()) and Ubuntu (apt_mirror_updater.backends.ubuntu.generate_sources_list()) backend implementations of this method for details on argument handling and the return value.

get_sources_list()

Get the contents of MAIN_SOURCES_LIST.

Returns:A Unicode string.

This code currently assumes that the sources.list file is encoded using SOURCES_LIST_ENCODING. I’m not actually sure if this is correct because I haven’t been able to find a formal specification! Feedback is welcome :-).

ignore_mirror(pattern)

Add a pattern to the mirror discovery blacklist.

Parameters:pattern – A shell pattern (containing wild cards like ? and *) that is matched against the full URL of each mirror.

When a pattern is added to the blacklist any previously cached values of available_mirrors, best_mirror, ranked_mirrors and stable_mirror are cleared. This makes sure that mirrors blacklisted after mirror discovery has already run are ignored.

install_sources_list(contents)

Install a new /etc/apt/sources.list file.

Parameters:contents – The new contents of the sources list (a Unicode string). You can generate a suitable value using the generate_sources_list() method.

smart_update(*args, **kw)

Update the system’s package lists (switching mirrors if necessary).

Parameters:

• args – Command line arguments to apt-get update (zero or more strings).
• max_attempts – The maximum number of attempts at successfully updating the system’s package lists (an integer, defaults to 10).
• switch_mirrors – True if we’re allowed to switch mirrors on ‘hash sum mismatch’ errors, False otherwise.

Raises:

If updating of the package lists fails 10 consecutive times (max_attempts) an exception is raised.

While dumb_update() simply runs apt-get update the smart_update() function works quite differently:

• First the system’s package lists are updated using dumb_update(). If this is successful we’re done.
• If the update fails we check the command’s output for the phrase ‘hash sum mismatch’. If we find this phrase we assume that the current mirror is faulty and switch to another one.
• Failing apt-get update runs are retried up to max_attempts.

validate_mirror(mirror_url)

Make sure a mirror serves distribution_codename.

Parameters:mirror_url – The base URL of the mirror (a string). Returns:One of the values in the MirrorStatus enumeration.

This method assumes that old_releases_url is always valid.

class apt_mirror_updater.CandidateMirror(**kw)

A candidate mirror groups a mirror URL with its availability and performance metrics.

bandwidth

The bytes per second achieved while fetching release_gpg_url (a number or None).

The value of this property is computed based on the values of release_gpg_contents and release_gpg_latency.

Note

The bandwidth property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

archive_update_in_progress_url

The URL of the file whose existence indicates that the mirror is being updated (a string).

The value of this property is computed based on the value of mirror_url.

Note

The archive_update_in_progress_url property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

mirror_url

The base URL of the mirror (a string).

Note

The mirror_url property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named mirror_url (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

is_available

True if release_gpg_contents contains the expected header, False otherwise.

The value of this property is computed by checking whether release_gpg_contents contains the expected BEGIN PGP SIGNATURE header. This may seem like a rather obscure way of validating a mirror, but it was specifically chosen to detect all sorts of ways in which mirrors can be broken:

• Webservers with a broken configuration that return an error page for all URLs.
• Mirrors whose domain name registration has expired, where the domain is now being squatted and returns HTTP 200 OK responses for all URLs (whether they “exist” or not).

Note

The is_available property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_updating

True if the mirror is being updated, False otherwise.

Note

The is_updating property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

last_updated

The time in seconds since the most recent mirror update (a number or None).

Note

The last_updated property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

release_gpg_contents

The contents downloaded from release_gpg_url (a string or None).

By downloading the file available at release_gpg_url and setting release_gpg_contents and release_gpg_latency you enable the bandwidth and is_available properties to be computed.

Note

The release_gpg_contents property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

release_gpg_latency

The time it took to download release_gpg_url (a number or None).

By downloading the file available at release_gpg_url and setting release_gpg_contents and release_gpg_latency you enable the bandwidth and is_available properties to be computed.

Note

The release_gpg_latency property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

release_gpg_url

The URL of the Release.gpg file that will be used to test the mirror (a string or None).

The value of this property is based on mirror_url and the distribution_codename property of the updater object.

Note

The release_gpg_url property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

sort_key

A tuple that can be used to sort the mirror by its availability/performance metrics.

The tuple created by this property contains four numbers in the following order:

1. The number 1 when is_available is True or the number 0 when is_available is False (because most importantly a mirror must be available).
2. The number 0 when is_updating is True or the number 1 when is_updating is False (because being updated at this very moment is bad).
3. The negated value of last_updated (because the lower last_updated is, the better). If last_updated is None then LAST_UPDATED_DEFAULT is used instead.
4. The value of bandwidth (because the higher bandwidth is, the better).

By sorting CandidateMirror objects on these tuples in ascending order, the last mirror in the sorted results will be the “most suitable mirror” (given the available information).

Note

The sort_key property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

updater

A reference to the AptMirrorUpdater object that created the candidate.

Note

The updater property is a custom_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class apt_mirror_updater.MirrorStatus

Enumeration for mirror statuses determined by AptMirrorUpdater.validate_mirror().

AVAILABLE = <EnumValue: MirrorStatus.AVAILABLE [value=1]>

The mirror is accepting connections and serving the expected content.

MAYBE_EOL = <EnumValue: MirrorStatus.MAYBE_EOL [value=2]>

The mirror is serving HTTP 404 “Not Found” responses instead of the expected content.

UNAVAILABLE = <EnumValue: MirrorStatus.UNAVAILABLE [value=3]>

The mirror is not accepting connections or not serving the expected content.

apt_mirror_updater.find_current_mirror(sources_list)

Find the URL of the main mirror that is currently in use by apt-get.

Parameters:sources_list – The contents of apt’s package resource list, e.g. the contents of MAIN_SOURCES_LIST (a string). Returns:The URL of the main mirror in use (a string). Raises:If the main mirror can’t be determined EnvironmentError is raised.

The main mirror is determined by looking for the first deb or deb-src directive in apt’s package resource list whose URL uses the HTTP or FTP scheme and whose components contain main.

apt_mirror_updater.mirrors_are_equal(a, b)

Check whether two mirror URLS are equal.

Parameters:

• a – The first mirror URL (a string).
• b – The second mirror URL (a string).

Returns:

True if the mirror URLs are equal, False otherwise.

apt_mirror_updater.normalize_mirror_url(url)

Normalize a mirror URL so it can be compared using string equality comparison.

Parameters:url – The mirror URL to normalize (a string). Returns:The normalized mirror URL (a string).

apt_mirror_updater.backends.debian

Discovery of Debian package archive mirrors.

Here are references to some of the material that I’ve needed to consult while working on this module:

• Notes about sources.list on the Debian wiki
• The Debian backports webpages
• Documentation about the “proposed-updates” mechanism

apt_mirror_updater.backends.debian.LTS_ARCHITECTURES = ('i386', 'amd64', 'armel', 'armhf')

The names of the architectures supported by the Debian LTS team (a tuple of strings).

apt_mirror_updater.backends.debian.LTS_RELEASES = {'jessie': 1593468000, 'stretch': 1656540000}

A dictionary with Debian LTS releases and their EOL dates.

This is needed because distro-info-data doesn’t contain information about Debian LTS releases but nevertheless archive.debian.org doesn’t adopt a release until the LTS status expires (this was originally reported in issue #5).

apt_mirror_updater.backends.debian.MIRRORS_URL = 'https://www.debian.org/mirror/list'

The URL of the HTML page listing all primary Debian mirrors (a string).

apt_mirror_updater.backends.debian.SECURITY_URL = 'http://security.debian.org/'

The base URL of the Debian mirror with security updates (a string).

apt_mirror_updater.backends.debian.OLD_RELEASES_URL = 'http://archive.debian.org/debian-archive/debian/'

The URL where EOL (end of life) Debian releases are hosted (a string).

apt_mirror_updater.backends.debian.DEFAULT_SUITES = ('release', 'security', 'updates')

A tuple of strings with the Debian suites that are enabled by default.

apt_mirror_updater.backends.debian.VALID_COMPONENTS = ('main', 'contrib', 'non-free')

A tuple of strings with the names of the components available in the Debian package repositories.

apt_mirror_updater.backends.debian.VALID_SUITES = ('release', 'security', 'updates', 'backports', 'proposed-updates')

A tuple of strings with the names of the suites available in the Debian package repositories.

apt_mirror_updater.backends.debian.discover_mirrors()

Discover available Debian mirrors by querying MIRRORS_URL.

Returns:A set of CandidateMirror objects that have their mirror_url property set. Raises:If no mirrors are discovered an exception is raised.

An example run:

>>> from apt_mirror_updater.backends.debian import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
set([CandidateMirror(mirror_url='http://ftp.at.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.au.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.be.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.bg.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.br.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.by.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.ca.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.ch.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.cn.debian.org/debian/'),
     CandidateMirror(mirror_url='http://ftp.cz.debian.org/debian/'),
     ...])

apt_mirror_updater.backends.debian.generate_sources_list(mirror_url, codename, suites=('release', 'security', 'updates'), components=('main', 'contrib', 'non-free'), enable_sources=False)

Generate the contents of /etc/apt/sources.list for a Debian system.

Parameters:

• mirror_url – The base URL of the mirror (a string).
• codename – The codename of a Debian release (a string like ‘wheezy’ or ‘jessie’) or a Debian release class (a string like ‘stable’, ‘testing’, etc).
• suites – An iterable of strings (defaults to DEFAULT_SUITES, refer to VALID_SUITES for details).
• components – An iterable of strings (refer to VALID_COMPONENTS for details).
• enable_sources – True to include deb-src entries, False to omit them.

Returns:

The suggested contents of /etc/apt/sources.list (a string).

apt_mirror_updater.backends.debian.get_eol_date(updater)

Override the EOL date for Debian LTS releases.

Parameters:updater – The AptMirrorUpdater object. Returns:The overridden EOL date (a number) or None.

apt_mirror_updater.backends.elementary

Support for Elementary OS package archive mirror selection.

Elementary OS is based on Ubuntu LTS releases and as such this module is a very thin wrapper for the apt_mirror_updater.backends.ubuntu module.

apt_mirror_updater.backends.elementary.OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/'

Alias for apt_mirror_updater.backends.ubuntu.OLD_RELEASES_URL.

apt_mirror_updater.backends.elementary.SECURITY_URL = 'http://security.ubuntu.com/ubuntu'

Alias for apt_mirror_updater.backends.ubuntu.SECURITY_URL.

apt_mirror_updater.backends.elementary.discover_mirrors()

Alias for apt_mirror_updater.backends.ubuntu.discover_mirrors().

apt_mirror_updater.backends.elementary.generate_sources_list(mirror_url, codename, suites=('release', 'updates', 'backports', 'security'), components=('main', 'restricted', 'universe', 'multiverse'), enable_sources=False)

Alias for apt_mirror_updater.backends.ubuntu.generate_sources_list().

apt_mirror_updater.backends.elementary.KNOWN_RELEASES = [Release(codename='Jupiter', distributor_id='elementary', series='jupiter'), Release(codename='Luna', distributor_id='elementary', series='luna'), Release(codename='Freya', distributor_id='elementary', series='freya'), Release(codename='Loki', distributor_id='elementary', series='loki'), Release(codename='Juno', distributor_id='elementary', series='juno'), Release(codename='Hera', distributor_id='elementary', series='hera')]

List of Release objects corresponding to known elementary OS releases based on the summary table on the following web page: https://en.wikipedia.org/wiki/Elementary_OS#Summary_table

apt_mirror_updater.backends.ubuntu

Discovery of Ubuntu package archive mirrors.

apt_mirror_updater.backends.ubuntu.MIRRORS_URL = 'https://launchpad.net/ubuntu/+archivemirrors'

The URL of the HTML page listing official Ubuntu mirrors (a string).

apt_mirror_updater.backends.ubuntu.MIRROR_SELECTION_URL = 'http://mirrors.ubuntu.com/mirrors.txt'

The URL of a plain text listing of “geographically suitable” mirror URLs (a string).

apt_mirror_updater.backends.ubuntu.OLD_RELEASES_URL = 'http://old-releases.ubuntu.com/ubuntu/'

The URL where EOL (end of life) Ubuntu releases are hosted (a string).

apt_mirror_updater.backends.ubuntu.SECURITY_URL = 'http://security.ubuntu.com/ubuntu'

The URL where Ubuntu security updates are hosted (a string).

apt_mirror_updater.backends.ubuntu.DEFAULT_SUITES = ('release', 'updates', 'backports', 'security')

A tuple of strings with the Ubuntu suites that are enabled by default.

apt_mirror_updater.backends.ubuntu.VALID_COMPONENTS = ('main', 'restricted', 'universe', 'multiverse')

A tuple of strings with the names of the components available in the Ubuntu package repositories.

apt_mirror_updater.backends.ubuntu.VALID_SUITES = ('release', 'security', 'updates', 'backports', 'proposed')

A tuple of strings with the names of the suites available in the Ubuntu package repositories.

The actual name of the ‘release’ suite is the codename of the relevant Ubuntu release, while the names of the other suites are formed by concatenating the codename with the suite name (separated by a dash).

As an example to make things more concrete, Ubuntu 16.04 has the following five suites available: xenial (this is the release suite), xenial-security, xenial-updates, xenial-backports and xenial-proposed.

apt_mirror_updater.backends.ubuntu.MIRROR_STATUSES = (('Up to date', 0), ('One hour behind', 3600), ('Two hours behind', 7200), ('Four hours behind', 14400), ('Six hours behind', 21600), ('One day behind', 86400), ('Two days behind', 172800), ('One week behind', 604800), ('Unknown', None))

A tuple of tuples with Launchpad mirror statuses. Each tuple consists of two values:

1. The human readable mirror latency (a string) as used on MIRRORS_URL.
2. The mirror latency expressed in seconds (a number).

The ‘known statuses’ used by Launchpad were checked as follows:

$ curl -s https://launchpad.net/+icing/rev18391/combo.css | tr '{},.' '\n' | grep distromirrorstatus
distromirrorstatusUP
distromirrorstatusONEHOURBEHIND
distromirrorstatusTWOHOURSBEHIND
distromirrorstatusFOURHOURSBEHIND
distromirrorstatusSIXHOURSBEHIND
distromirrorstatusONEDAYBEHIND
distromirrorstatusTWODAYSBEHIND
distromirrorstatusONEWEEKBEHIND
distromirrorstatusUNKNOWN

apt_mirror_updater.backends.ubuntu.discover_mirrors()

Discover available Ubuntu mirrors.

Returns:A set of CandidateMirror objects that have their mirror_url property set and may have the last_updated property set. Raises:If no mirrors are discovered an exception is raised.

This queries MIRRORS_URL and MIRROR_SELECTION_URL to discover available Ubuntu mirrors. Here’s an example run:

>>> from apt_mirror_updater.backends.ubuntu import discover_mirrors
>>> from pprint import pprint
>>> pprint(discover_mirrors())
set([CandidateMirror(mirror_url='http://archive.ubuntu.com/ubuntu/'),
     CandidateMirror(mirror_url='http://ftp.nluug.nl/os/Linux/distr/ubuntu/'),
     CandidateMirror(mirror_url='http://ftp.snt.utwente.nl/pub/os/linux/ubuntu/'),
     CandidateMirror(mirror_url='http://ftp.tudelft.nl/archive.ubuntu.com/'),
     CandidateMirror(mirror_url='http://mirror.1000mbps.com/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.amsiohosting.net/archive.ubuntu.com/'),
     CandidateMirror(mirror_url='http://mirror.i3d.net/pub/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.nforce.com/pub/linux/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.nl.leaseweb.net/ubuntu/'),
     CandidateMirror(mirror_url='http://mirror.transip.net/ubuntu/ubuntu/'),
     ...])

apt_mirror_updater.backends.ubuntu.discover_mirror_selection()

Discover “geographically suitable” Ubuntu mirrors.

apt_mirror_updater.backends.ubuntu.generate_sources_list(mirror_url, codename, suites=('release', 'updates', 'backports', 'security'), components=('main', 'restricted', 'universe', 'multiverse'), enable_sources=False)

Generate the contents of /etc/apt/sources.list for an Ubuntu system.

Parameters:

• mirror_url – The base URL of the mirror (a string).
• codename – The codename of the Ubuntu release (a string like ‘trusty’ or ‘xenial’).
• suites – An iterable of strings (defaults to DEFAULT_SUITES, refer to VALID_SUITES for details).
• components – An iterable of strings (refer to VALID_COMPONENTS for details).
• enable_sources – True to include deb-src entries, False to omit them.

Returns:

The suggested contents of /etc/apt/sources.list (a string).

apt_mirror_updater.cli

Usage: apt-mirror-updater [OPTIONS]

The apt-mirror-updater program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating.

Supported options:

Option	Description
-r, --remote-host=SSH_ALIAS	Operate on a remote system instead of the local system. The SSH_ALIAS argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access.
-f, --find-current-mirror	Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output.
-b, --find-best-mirror	Discover available mirrors, rank them, select the best one and report its URL on standard output.
-l, --list-mirrors	List available (ranked) mirrors on the terminal in a human readable format.
-c, --change-mirror=MIRROR_URL	Update /etc/apt/sources.list to use the given MIRROR_URL.
-a, --auto-change-mirror	Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror.
-u, --update, --update-package-lists	Update the package lists using “apt-get update”, retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated.
-x, --exclude=PATTERN	Add a pattern to the mirror selection blacklist. PATTERN is expected to be a shell pattern (containing wild cards like “?” and “*”) that is matched against the full URL of each mirror.
-m, --max=COUNT	Don’t query more than COUNT mirrors for their connection status (defaults to 50). If you give the number 0 no limit will be applied. Because Ubuntu mirror discovery can report more than 300 mirrors it’s useful to limit the number of mirrors that are queried, otherwise the ranking of mirrors will take a long time (because over 300 connections need to be established).
-v, --verbose	Increase logging verbosity (can be repeated).
-q, --quiet	Decrease logging verbosity (can be repeated).
-h, --help	Show this message and exit.

apt_mirror_updater.cli.main()

Command line interface for the apt-mirror-updater program.

apt_mirror_updater.cli.report_current_mirror(updater)

Print the URL of the currently configured apt-get mirror.

apt_mirror_updater.cli.report_best_mirror(updater)

Print the URL of the “best” mirror.

apt_mirror_updater.cli.report_available_mirrors(updater)

Print the available mirrors to the terminal (in a human friendly format).

apt_mirror_updater.http

Simple, robust and concurrent HTTP requests (designed for one very narrow use case).

apt_mirror_updater.http.fetch_url(url, timeout=10, retry=False, max_attempts=3)

Fetch a URL, optionally retrying on failure.

Parameters:

• url – The URL to fetch (a string).
• timeout – The maximum time in seconds that’s allowed to pass before the request is aborted (a number, defaults to 10 seconds).
• retry – Whether to retry on failure (defaults to False).
• max_attempts – The maximum number of attempts when retrying is enabled (an integer, defaults to three).

Returns:

The response body (a byte string).

Raises:

Any of the following exceptions can be raised:

• NotFoundError when the URL returns a 404 status code.
• InvalidResponseError when the URL returns a status code that isn’t 200.
• stopit.TimeoutException when the request takes longer than timeout seconds (refer to the linked documentation for details).
• Any exception raised by Python’s standard library in the last attempt (assuming all attempts raise an exception).

apt_mirror_updater.http.fetch_concurrent(urls, concurrency=None)

Fetch the given URLs concurrently using multiprocessing.

Parameters:

• urls – An iterable of URLs (strings).
• concurrency – Override the concurrency (an integer, defaults to the value computed by get_default_concurrency()).

Returns:

A list of tuples like those returned by fetch_worker().

apt_mirror_updater.http.get_default_concurrency()

Get the default concurrency for fetch_concurrent().

Returns:A positive integer number.

apt_mirror_updater.http.fetch_worker(url)

Fetch the given URL for fetch_concurrent().

Parameters:url – The URL to fetch (a string). Returns:A tuple of three values:

1. The URL that was fetched (a string).
2. The data that was fetched (a string or None).
3. The number of seconds it took to fetch the URL (a number).

exception apt_mirror_updater.http.InvalidResponseError

Raised by fetch_url() when a URL returns a status code that isn’t 200.

exception apt_mirror_updater.http.NotFoundError

Raised by fetch_url() when a URL returns a 404 status code.

apt_mirror_updater.releases

Easy to use metadata on Debian and Ubuntu releases.

This module started out with the purpose of reliable end of life (EOL) detection for Debian and Ubuntu releases based on data provided by the distro-info-data package. Since then the need arose to access more of the available metadata and so the eol module became the releases module.

Debian and Ubuntu releases have an EOL date that marks the end of support for each release. At that date the release stops receiving further (security) updates and some time after package mirrors stop serving the release.

The distro-info-data package contains CSV files with metadata about Debian and Ubuntu releases. This module parses those CSV files to make this metadata available in Python. This enables apt-mirror-updater to make an informed decision about the following questions:

1. Is a given Debian or Ubuntu release expected to be available on mirrors or will it only be available in the archive of old releases?
2. Is the signing key of a given Ubuntu release expected to be included in the main keyring (UBUNTU_KEYRING_CURRENT) or should the keyring with removed keys (UBUNTU_KEYRING_REMOVED) be used?

To make it possible to run apt-mirror-updater without direct access to the CSV files, a copy of the relevant information has been embedded in the source code.

apt_mirror_updater.releases.DISTRO_INFO_DIRECTORY = '/usr/share/distro-info'

The pathname of the directory with CSV files containing release metadata (a string).

apt_mirror_updater.releases.DEBIAN_KEYRING_CURRENT = '/usr/share/keyrings/debian-keyring.gpg'

The pathname of the main Debian keyring file (a string).

apt_mirror_updater.releases.UBUNTU_KEYRING_CURRENT = '/usr/share/keyrings/ubuntu-archive-keyring.gpg'

The pathname of the main Ubuntu keyring file (a string).

apt_mirror_updater.releases.UBUNTU_KEYRING_REMOVED = '/usr/share/keyrings/ubuntu-archive-removed-keys.gpg'

The pathname of the Ubuntu keyring file with removed keys (a string).

apt_mirror_updater.releases.coerce_release(value)

Try to coerce the given value to a Debian or Ubuntu release.

Parameters:value – The value to coerce (a number, a string or a Release object). Returns:A Release object. Raises:ValueError when the given value cannot be coerced to a known release.

The following values can be coerced:

• Numbers and numbers formatted as strings match Release.version.
• Strings match Release.codename (case insensitive).

Warning

Don’t use floating point numbers like 10.04 because their actual value will be something like 10.039999999999999147 which won’t match the intended release.

apt_mirror_updater.releases.discover_releases()

Discover known Debian, Elementary OS and Ubuntu releases.

Returns:A list of discovered Release objects sorted by distributor_id and version.

The first time this function is called it will try to parse the CSV files in /usr/share/distro-info using parse_csv_file() and merge any releases it finds with the releases embedded into the source code of this module and the releases defined by apt_mirror_updater.backends.elementary.KNOWN_RELEASES. The result is cached and returned each time the function is called. It’s not a problem if the /usr/share/distro-info directory doesn’t exist or doesn’t contain any *.csv files (it won’t cause a warning or error). Of course in this case only the embedded releases will be returned.

apt_mirror_updater.releases.is_version_string(value)

Check whether the given value is a string containing a positive number.

apt_mirror_updater.releases.parse_csv_file(filename)

Parse a CSV file in the format of the /usr/share/distro-info/*.csv files.

Parameters:filename – The pathname of the CSV file (a string). Returns:A generator of Release objects.

apt_mirror_updater.releases.parse_date(value)

Convert a YYYY-MM-DD string to a datetime.date object.

apt_mirror_updater.releases.parse_version(value)

Convert a version string to a Decimal number.

apt_mirror_updater.releases.ubuntu_keyring_updated()

Detect update #1363482 to the ubuntu-keyring package.

Returns:True when version 2016.10.27 or newer is installed, False when an older version is installed.

This function checks if the changes discussed in Launchpad bug #1363482 apply to the current system using the dpkg-query --show and dpkg --compare-versions commands. For more details refer to issue #8.

class apt_mirror_updater.releases.Release(**kw)

Data class for metadata on Debian, Elementary OS and Ubuntu releases.

codename

The long version of series (a string).

Note

The codename property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named codename (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

created_date

The date on which the release was created (a date object).

Note

The created_date property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named created_date (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

distributor_id

The name of the distributor (one of the strings debian, elementary or ubuntu).

Note

The distributor_id property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named distributor_id (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

eol_date

The date on which the desktop release stops being supported (a date object).

Note

The eol_date property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

extended_eol_date

The date on which the server release stops being supported (a date object).

Note

The extended_eol_date property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

is_eol

Whether the release has reached its end-of-life date (a boolean or None).

Note

The is_eol property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

is_lts

Whether a release is a long term support release (a boolean).

Note

The is_lts property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

release_date

The date on which the release was published (a date object).

Note

The release_date property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

series

The short version of codename (a string).

Note

The series property is a key_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named series (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). Once this property has been assigned a value you are not allowed to assign a new value to the property.

upstream_distributor_id

The upstream distributor ID (a string, defaults to distributor_id).

Note

The upstream_distributor_id property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

upstream_series

The upstream series (a string, defaults to series).

Note

The upstream_series property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

upstream_version

The upstream version (a string, defaults to version).

Note

The upstream_version property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

version

The version number of the release (a Decimal number).

This property has a Decimal value to enable proper sorting based on numeric comparison.

Note

The version property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

keyring_file

The pathname of the keyring with signing keys for this release (a string).

This property exists to work around a bug in debootstrap which may use the wrong keyring to create Ubuntu chroots, for more details refer to ubuntu_keyring_updated().

Note

The keyring_file property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

__str__()

Render a human friendly representation of a Release object.

The result will be something like this:

• Debian 9 (stretch)
• Ubuntu 18.04 (bionic)

Change log

The change log lists notable changes to the project:

Changelog

The purpose of this document is to list all of the notable changes to this project. The format was inspired by Keep a Changelog. This project adheres to semantic versioning.

Release 7.3 (2021-09-15)

Significant changes:

• Updated the releases bundled in the releases module to include the following:
  • Ubuntu 20.10 (Groovy Gorilla)
  • Ubuntu 21.04 (Hirsute Hippo)
  • Ubuntu 21.10 (Impish Indri)
• Relaxed the beautifulsoup4 requirement to facilitate Python 2 compatibility (release 4.9.3 was the last to support Python 2 whereas our requirements insisted on >= 4.4.1).

Miscellaenous changes:

• Updated the parse_version() documentation to refer to decimal.Decimal instead of “a floating point number” (to avoid any confusion).

Release 7.2 (2020-04-18)

Add support for Elementary OS (suggested in issue #10).

Because Elementary OS is based on Ubuntu the new backend is nothing more than a thin wrapper for the Ubuntu backend. Its most significant content is a data structure with known Elementary OS releases and how they map to Ubuntu.

Release 7.1 (2020-04-17)

After Ubuntu 14.04 went EOL it became clear that when a release goes EOL and when it is archived (moved from the package mirrors to the old-releases environment) can differ by a year…

This release of apt-mirror-updater acknowledges this distinction in the code and properly handles the situation where a release has already gone EOL but has not yet been archived.

This was also reported in issue #9 by a colleague and friend of mine.

Release 7.0 (2020-04-16)

This is a maintenance release that (primarily) updates Python compatibility.

Backwards incompatible changes:

• Python 3.7 and 3.8 are now officially supported.
• Python 2.6 and 3.4 are no longer supported.
• Added python_requires to setup.py to aid pip in version selection given these compatibility changes.

Significant changes:

• Updated the releases bundled in the apt_mirror_updater.releases module to include the following:
  • Ubuntu 19.04 (Disco Dingo)
  • Ubuntu 19.10 (Eoan Ermine)
  • Ubuntu 20.04 (Focal Fossa).
• Bug fix for “accidental tuple” in apt_mirror_updater.releases.parse_csv_file().

Miscellaenous changes:

• Spent some time stabilizing the test suite on Travis CI (tests were passing for me locally but not on Travis CI because the mirror selection differed). As a result the test suite got a bit slower, but it’s not too bad.
• Move caching decorator to humanfriendly.
• Fixed deprecation warnings emitted by recent humanfriendly releases and bumped requirements I authored that went through similar changes.
• Made multiprocessing usage compatible with coverage collection. Note that I don’t expect this to increase coverage, I just wanted to get rid of the warnings 😇 (because warnings about harmless things are just as distracting as more pertinent warnings).
• Default to Python 3 for local development (required by Sphinx among other things).
• Fixed existing Sphinx reference warnings in the documentation and changed the sphinx-build invocation to promote warnings to errors (to aid me in the discipline of not introducing broken references from now on).

Release 6.1 (2018-10-19)

• Bug fix for Ubuntu keyring selection that prevented ubuntu-archive-removed-keys.gpg from being used.
• Bug fix for coerce_release() when given a release number.
• Moved pathnames of Debian and Ubuntu keyring files to constants.
• Added logging to enable debugging of keyring selection process.
• Added proper tests for keyring selection and release coercion.

Release 6.0 (2018-10-14)

Enable the creation of Ubuntu <= 12.04 chroots on Ubuntu >= 17.04 hosts by working around (what I am convinced is) a bug in debootstrap which picks the wrong keyring when setting up chroots of old releases. For more information refer to issue #8.

I’ve bumped the major version number for this release because the highly specific apt_mirror_updater.eol module changed into the much more generic apt_mirror_updater.releases module. Also the release_label property was removed.

Release 5.2 (2018-10-08)

Use mirrors.ubuntu.com/mirrors.txt without placing our full trust in it like older versions of apt-mirror-updater did 😇.

Feedback in issue #6 suggested that mirrors.ubuntu.com/mirrors.txt is working properly (again) and should be preferred over scraping Launchpad. However I prefer for apt-mirror-updater to be a reliable “do what I mean” program and mirrors.ubuntu.com/mirrors.txt has proven to be unreliable in the past (see the discussion in #6). As a compromise I’ve changed the Ubuntu mirror discovery as follows:

1. Discover Ubuntu mirrors on Launchpad.
2. Try to discover mirrors using mirrors.ubuntu.com/mirrors.txt and iff successful, narrow down the list produced in step 1 based on the URLs reported in step 2.
3. Rank the discovered / narrowed down mirrors and pick the best one.

The reason why I’ve decided to add this additional complexity is because it has bothered me in the past that Ubuntu mirror discovery was slow and this does help a lot. Also, why not use a service provided by Ubuntu to speed things up?

Unrelated to the use of mirrors.ubuntu.com/mirrors.txt I’ve also bumped the executor requirement (twice) in order to pull in upstream improvements discussed in executor issue #10 and executor issue #15.

Release 5.1 (2018-06-22)

Work on release 5.1 started with the intention of publishing a 5.0.2 bug fix release for the EOL detection of Debian LTS releases reported in #5, however unrelated changes were required to stabilize the test suite. This explains how 5.0.2 became 5.1 😇.

When I started working on resolving the issue reported in #5 it had been quite a while since the previous release (233 days) and so some technical debt had accumulated in the project, causing the test suite to break. Most significantly, Travis CI switched their workers from Ubuntu 12.04 to 14.04.

Here’s a detailed overview of changes:

• Bug fix for EOL detection of Debian LTS releases (reported in #5).
• Bug fix for trivial string matching issue in test suite (caused by a naively written test).
• Bug fix for recursive repr() calls potentially causing infinite recursion, depending on logging level (see e.g. build 395421319).
• Updated bundled EOL dates based on distro-info-data available in Ubuntu 18.04.
• Added this changelog to the documentation, including a link in the readme.
• Make sure the test_gather_eol_dates test method runs on Travis CI (by installing the distro-info-data package). This exposed a Python 3 incompatibility (in build 395410569) that has since been resolved.
• Include documentation in source distributions (MANIFEST.in).
• Silence flake8 complaining about bogus D402 issues.
• Add license=’MIT’ key to setup.py script.
• Bumped copyright to 2018.

Release 5.0.1 (2017-11-01)

Bug fix release for invalid enumeration value (oops).

Release 5.0 (2017-11-01)

Reliable end of life (EOL) detection.

Recently I ran into the issue that the logic to check whether a release is EOL (that works by checking if the security mirror serves a Release.gpg file for the release) failed on me. More specifically the following URL existed at the time of writing (2017-11-01) even though Ubuntu 12.04 went EOL back in April:

http://security.ubuntu.com/ubuntu/dists/precise/Release.gpg

At the same time issue #1 and pull request #2 were also indications that the EOL detection was fragile and error prone. This potential fragility had bugged me ever since publishing apt-mirror-updater and this week I finally finished a more robust and deterministic EOL detection scheme.

This release includes pull requests #2 and #4, fixing issues #1 and #3. Here’s a detailed overview of changes:

• Addition: Allow optional arguments to apt-get update (#3, #4).
  • I simplified and improved the feature requested in issue #3 and implemented in pull request #4 by switching from an optional list argument to ‘star-args’ and applying the same calling convention to smart_update() as well.
  • This is backwards incompatible with the implementation in pull request #4 (which I merged into the dev branch but never published to PyPI) and it’s also technically backwards incompatible in the sense that keyword arguments could previously be given to smart_update() as positional arguments. This explains why I’m bumping the major version number.
• Bug fix for incorrect marking of EOL when HTTP connections fail (#2).
• Refactoring: Apply timeout handling to HTTP response bodies.
• Refactoring: Distinguish 404 from other HTTP errors:
  • This change enhances validate_mirror() by making a distinction between a confirmed HTTP 404 response versus other error conditions which may be of a more transient nature.
  • The goal of this change is to preserve the semantics requested in issue #1 and implemented in pull request #2 without needing the additional HTTP request performed by can_connect_to_mirror().
  • Because validate_mirror() previously returned a boolean but now returns an enumeration member this change is technically backwards incompatible, then again validate_mirror() isn’t specifically intended for callers because it concerns internal logic of apt-mirror-updater. I’m nevertheless bumping the major version number.
• Refactoring: Improve HTTP request exception handling:
  • 404 responses and timeouts are no longer subject to retrying.
  • The exception apt_mirror_updater.http.NotFoundError is now raised on HTTP 404 responses. Other unexpected HTTP response codes raise apt_mirror_updater.http.InvalidResponseError.
  • The specific distinction between 404 and !200 was made because the 404 response has become significant in checking for EOL status.

Release 4.0 (2017-06-14)

Robust validation of available mirrors (backwards incompatible).

Release 3.1 (2017-06-13)

Made mirror comparison more robust.

Release 3.0 (2017-06-13)

Added Debian archive support (with old releases):

• Addition: Added Debian archive support (old releases).
• Improvement: Don’t bother validating archive / old-releases mirror.
• Refactoring: Moved URLs to backend specific modules.

Release 2.1 (2017-06-12)

Restored Python 3 compatibility, improved robustness:

• Improvement: Make the is_available and is_updating properties of the CandidateMirror class more robust.
• Bug fix: I suck at Unicode in Python (most people do :-p).
• Cleanup: Remove unused import from test suite.

Release 2.0 (2017-06-11)

Generation of sources.list files and chroot creation.

Detailed overview of changes:

• Addition: Added a simple debootstrap wrapper.
• Addition: Programmatic /etc/apt/sources.list generation
• Bug fix for check_suite_available().
• Bug fix: Never apply Ubuntu’s old release handling to Debian.
• Bug fix: Never remove /var/lib/apt/lists/lock file.
• Improvement: Enable stable mirror selection
• Improvement: Make it possible to override distributor ID and codename
• Improvement: Render interactive spinner during mirror ranking.
• Refactoring: Generalize AptMirrorUpdater initializer (backwards incompatible!)
• Refactoring: Generalize backend module loading
• Refactoring: Modularize /etc/apt/sources.list writing.

Release 1.0 (2017-06-08)

Improved Ubuntu mirror discovery, added an automated test suite, and more.

The bump to version 1.0 isn’t so much intended to communicate that this is now mature software, it’s just that I made several backwards incompatible changes in order to improve the modularity of the code base, make it easier to develop automated tests, maintain platform support, etc :-).

A more detailed overview of (significant) changes:

• Improved Ubuntu mirror discovery (by scraping Launchpad instead).
• Extracted mirror discovery to separate (backend specific) modules.
• Extracted HTTP handling to a separate module.
• Enable Control-C to interrupt concurrent connection tests.
• Expose limit in Python API and command line interface and make limit optional by passing 0.
• Bug fix for Python 3 incompatibility: Stop using sys.maxint :-).

Release 0.3.1 (2016-06-29)

Avoid ‘nested’ smart updates (the old code worked fine but gave confusing output and performed more work than necessary, which bothered me :-).

Release 0.3 (2016-06-29)

Make smart update understand EOL suites.

Release 0.2 (2016-06-29)

Bug fix: Replace security.ubuntu.com as well.

Release 0.1.2 (2016-06-29)

Bug fix: Explicitly terminate multiprocessing pool.

Release 0.1.1 (2016-03-10)

Initial release (added MANIFEST.in).

Release 0.1 (2016-03-10)

Initial commit.