Site Tools

Site menu


Translations of this page:
en:tech:dev:debmirror

debmirror (tips and tricks)

debmirror is a program who aims to download the content of a debian repository in order to set a local mirror up and maintain it up to date.

The parameters are numerous and depend on each repository, so it is sometimes hard to find right settings on the web. That's why I gather here the parameters that were good for me + a touch of methodology.

Note: As I actually do not use debmirror, some informations on this page may be quite outdated.
Médéric 02/12/2018 23:35

Getting the public GPG keys

debmirror runs in userspace and tries to get each public GnuPG key repository from the curent user trusted keyring ~/.gnupg/trustedkeys.gpg.

How to retrieve repository keys ?

Each GPG repository key must be requested by its ID to key servers which return the public key content if available:

  1. Search for the public GPG key ID on the official repository web site
  2. Send a query to a key server for that ID

The official repository procedures tell the root user to import the public key into the APT keyring, by providing the ID of the public GPG Key either like this:
apt-key adv --keyserver a-server.tld --recv-keys ID

or like this:
gpg --keyserver a-server.tld --recv-keys ID && gpg --export -a ID | apt-key add -

With the key ID and the server name ("a-server.tld"), the user can download the public repository GPG key into its own trusted keyring ~/.gnupg/trustedkeys.gpg either by importing first to its personal public keyring ~/.gnupg/pubring.gpg:

gpg --keyserver a-server.tld --recv-keys ID
gpg --export -a ID | gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --import -

or directly to its trusted keyring ~/.gnupg/trustedkeys.gpg:

gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver a-server.tld --recv-keys ID

In most cases, the key server keys.gnupg.net will be sufficient to retrieve keys so that the only variable parameter will be the ID:
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys ID

GPG keys may change so it's important to perform an update from time to time. A failsafe strategy is to request the public repository key before each debmirror session.

Rely on the APT trusted keyring

Export the APT trusted keyring to the user's one

If all the repositories that one want to mirror are already configured within the system (/etc/apt/sources.list and /etc/apt/sources.list.d/…), then the respectives public GPG keys should be up to date. In this case, it's convenient to update the user GPG trusted keyring by importing from the APT's one as follow:

gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg  --export | gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --import

Aternate way: using debmirror with APT keyring

An alternate way is to tell debmirror to look directly into the APT keyring:
debmirror --keyring /usr/share/keyrings/debian-archive-keyring.gpg ...

Check the repository key validation

Looking at standard output, debmirror shows if a GPG key is missing in ~/.gnupg/trustedkeys.gpg.

Example:

[GNUPG:] ERRSIG 8B48AD6246925553 1 8 00 1441453660 9
[GNUPG:] NO_PUBKEY 8B48AD6246925553
[GNUPG:] ERRSIG 7638D0442B90D010 1 8 00 1441453660 9
[GNUPG:] NO_PUBKEY 7638D0442B90D010
[GNUPG:] ERRSIG 6FB2A1C265FFB764 1 8 00 1441453675 9
[GNUPG:] NO_PUBKEY 6FB2A1C265FFB764
gpgv: keyblock resource `/home/demo/.gnupg/trustedkeys.gpg': file open error
gpgv: Signature made Sat Sep  5 13:47:40 2015 CEST using RSA key ID 46925553
gpgv: Can't check signature: public key not found
gpgv: Signature made Sat Sep  5 13:47:40 2015 CEST using RSA key ID 2B90D010
gpgv: Can't check signature: public key not found
gpgv: Signature made Sat Sep  5 13:47:55 2015 CEST using RSA key ID 65FFB764
gpgv: Can't check signature: public key not found
gpgv: keyblock resource `/home/demo/.gnupg/trustedkeys.gpg': file open error
gpgv: Signature made Sat Sep  5 13:47:40 2015 CEST using RSA key ID 46925553
gpgv: Can't check signature: public key not found
:Failed to download some Release or Release.gpg files!

In this case, one have to retrieve the public GPG keys whose IDs are:

  • 46925553
  • 2B90D010
  • 65FFB764

Repository timeout

Warning!!
Some repositories have a validation timeout. In this case, if one exceeds that limit before updating the repository, all datas will be regarded as outdated and will be re-downloaded and overwritten!! This can lead an overload bandwidth!!

In this case, I advise to adjust the cron task that launches your debmirror script in order to save resources.

In the following examples, the base directory of the local mirrored repositories is stored in the shell variable TARGET_DIR.

If one want to know the repositories that have a timeout and their respectives values the following command do the trick:

find $TARGET_DIR -name Release -exec grep -HE '^Valid-Until:' "{}" \; | \
grep -v '/.temp/' | gawk -F '/' '{date=gensub(/^.*Valid-Until: */,"",$0); print $4": "date}'

Example:

find $TARGET_DIR -name Release -exec grep -HE '^Valid-Until:' "{}" \; | grep -v '/.temp/' | gawk -F '/' '{date=gensub(/^.*Valid-Until: */,"",$0); print $4" : "date}'
debian.org: Tue, 22 Sep 2015 02:42:42 UTC
debian.org: Tue, 22 Sep 2015 02:42:42 UTC
security.debian.org : Wed, 23 Sep 2015 20:01:59 UTC

Each day, the Debian repositories are refreshed and their timeout is set to 7 days up.

Debian

If you're concerned with Debian, see also the Devuan project from the "Veteran Unix Admins" (VUA) team who aims to revert to an UNIX authentic distribution : http://devuan.org/ and newsletter_22dec.html.

debian.org

Debian 7 (wheezy):

## Debian Archive Automatic Signing Key (Comment: 7.0/wheezy)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 46925553

## Debian Archive Automatic Signing Key (Comment: 8/jessie)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 2B90D010

## Wheezy Stable Release Key (Comment: none)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 65FFB764

debmirror -v --verbose --arch=i386,amd64 --dist=wheezy,wheezy-updates,wheezy-backports --source --method=rsync --host=ftp.fr.debian.org --root=:debian --section=main,contrib,non-free --i18n --postcleanup --diff=mirror $TARGET_DIR/debian.org/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/debian.org/debian/ wheezy main contrib non-free >>/etc/apt/sources.list.local
echo deb file:$TARGET_DIR/debian.org/debian/ wheezy-updates main contrib >>/etc/apt/sources.list.local

Debian 8 (jessie):

## Debian Archive Automatic Signing Key (Comment: 7.0/wheezy)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 46925553

## Debian Archive Automatic Signing Key (Comment: 8/jessie)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 2B90D010

## Jessie Stable Release Key (Comment: none)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 518E17E1

debmirror -v --verbose --arch=i386,amd64 --dist=jessie,jessie-updates,jessie-backports --source --method=rsync --host=ftp.fr.debian.org --root=:debian --section=main,contrib,non-free --i18n --postcleanup --diff=mirror $TARGET_DIR/debian.org/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/debian.org/debian/ jessie main contrib non-free >>/etc/apt/sources.list.local
echo deb file:$TARGET_DIR/debian.org/debian/ jessie-updates main contrib >>/etc/apt/sources.list.local

security.debian.org

Debian 7 (wheezy):

## Debian Security Archive Automatic Signing Key (Comment: 8/jessie)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys C857C906

## Debian Archive Automatic Signing Key (Comment: 7.0/wheezy)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 46925553 

debmirror -v --verbose --arch=i386,amd64 --dist=wheezy/updates --source --method=rsync --host=security.debian.org --root=:debian-security/ --section=main,contrib,non-free --i18n --postcleanup --diff=mirror $TARGET_DIR/debian.org/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/security.debian.org/ wheezy/updates main contrib non-free >>/etc/apt/sources.list.local

Debian 8 (jessie):

## Debian Security Archive Automatic Signing Key (Comment: 8/jessie)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys C857C906

## Debian Archive Automatic Signing Key (Comment: 7.0/wheezy)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 46925553 

debmirror -v --verbose --arch=i386,amd64 --dist=jessie/updates --source --method=rsync --host=security.debian.org --root=:debian-security/ --section=main,contrib,non-free --i18n --postcleanup --diff=mirror $TARGET_DIR/debian.org/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/security.debian.org/ jessie/updates main contrib non-free >>/etc/apt/sources.list.local

mozilla.debian.net

The Debian Mozilla Team maintains the Mozilla Foundation softwares rebranded by the Debian project.

## Debian Mozilla team APT archive (Comment: none)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 06C4AE2A

debmirror -v --verbose --arch=i386,amd64 --host=mozilla.debian.net --root=:. --dist=wheezy-backports --section=iceweasel-aurora,iceweasel-beta,iceweasel-release,iceweasel-esr,icedove-esr --i18n --postcleanup --source --method=http --diff=mirror $TARGET_DIR/mozilla.debian.net

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/mozilla.debian.net/ wheezy-backports iceweasel-release >>/etc/apt/sources.list.local

ignorantguru.github.com

Ignorant Guru is well know to be the systemd-free creator of SpaceFM. His main web pages are:

## IgnorantGuru (Comment: none) (Expiration: 01/02/2015)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 01937621

debmirror -v --verbose --arch=i386,amd64 --host=ignorantguru.github.com --root=:debian --dist=unstable --section=main --i18n --postcleanup --source --method=http --diff=mirror $TARGET_DIR/ignorantguru.github.com/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/ignorantguru.github.com/debian/ unstable main >>/etc/apt/sources.list.local

deb-multimedia.org

Some usefull multimedia softwares are banned from debian repository for ethic and legitimate reasons: not enough free to the Debian Social Contract.

But some of this softwares are sometimes necessary – like the famous FFmpeg from the famous Fabrice Bellard –. Fortunatly, Christian Marillat and Thibaut Varene maintain theses softwares up to date for debian users from the site deb-multimedia.org:

## Christian Marillat (Comment: none)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 65558117

debmirror -v --verbose --arch=i386,amd64 --host=deb-multimedia.org --root=/ --dist=wheezy --section=main,non-free --i18n --postcleanup --source --method=http --diff=mirror $TARGET_DIR/www.deb-multimedia.org

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/www.deb-multimedia.org/ wheezy main non-free >>/etc/apt/sources.list.local

Trinity Desktop

Trinity Desktop is a fork of KDE 3.5 maintained for debian based systems.

The following code downloads official Trinity R14.0.x packages from the repository of the University of Kent http://mirrorservice.org/sites/trinitydesktop.org/trinity/:

## QuickBuild Trinity Desktop Environment (Comment: none)
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keyserver.quickbuild.pearsoncomputing.net --recv-keys F5CFC95C

debmirror -v --verbose --arch=i386,amd64 --host=mirrorservice.org --root=:sites/trinitydesktop.org/trinity/trinity-r14.0.0/debian --dist=wheezy --section=main --i18n --postcleanup --source --method=http --diff=mirror $TARGET_DIR/trinity-r14.0.0/debian

debmirror -v --verbose --arch=i386,amd64 --host=mirrorservice.org --root=:sites/trinitydesktop.org/trinity/trinity-builddeps-r14.0.0/debian --dist=wheezy --section=main --i18n --postcleanup --source --method=http --diff=mirror $TARGET_DIR/trinity-builddeps-r14.0.0/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/trinity-r14.0.0/debian wheezy main >>/etc/apt/sources.list.local
echo deb file:$TARGET_DIR/trinity-builddeps-r14.0.0/debian wheezy main >>/etc/apt/sources.list.local

See also:

virtualbox.org

VirtualBox is a multiplateform software to host various PC's operating systems.

## Oracle Corporation (comment: VirtualBox archive signing key)
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --import -
## or:
#gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 54422A4B98AB5139

debmirror -v --verbose --arch=i386,amd64 --host=download.virtualbox.org --root=:virtualbox/debian --dist=wheezy --section=non-free,contrib --postcleanup --source --method=http --diff=mirror $TARGET_DIR/download.virtualbox.org/virtualbox/debian

One can add that local repository by adding it to a future sources.list file under construction:

echo deb file:$TARGET_DIR/download.virtualbox.org/virtualbox/debian wheezy non-free contrib >>/etc/apt/sources.list.local

References

See also

  • Aptly : (Debian repository management tool)
    aptly is a swiss army knife for Debian repository management: it allows you to mirror remote repositories, manage local package repositories, take snapshots, pull new versions of packages along with dependencies, publish as Debian repository. :

Comments

Jami, 02/12/2018 22:11
I come in peace! hello from the dark side 'windows world' after 25 years of study. :) Just wanted to say, thanks to whoever acquired, studied and wrote the information written at this website. After spending few days here, reading information (mostly about debmirror) got to say as "almost total newbie" to linux, I still understood what was written amd how to implement (although, went through trouble for gpg keys, but figured that directly using debian-archive-keyring at '/usr/share/keyrings/' works as well directly input to debmirror with --keyrings= argument than acquiring new ones). Anyway, thanks for all the hard work!
Mederic, 02/12/2018 22:39
Thanks for your comment. I'm happy this page was helpful to you, althougt some informations maybe outdated.
Enter your comment:
If you can't read the letters on the image, download this .wav file to get them read to you.
 
en/tech/dev/debmirror.txt · Last modified: 05/01/2019 00:46 (external edit)