Man Linux: Main Page and Category List

NAME

       Emdebian::Grip - internal Emdebian module for Emdebian Grip
       repositories

Description

       It is worthwhile being familiar with "reprepro" (1) when working with
       this module.

       Emdebian::Grip provides repository management support for the
       em_autogrip scripting to manage a three-way repository hierarchy for
       Emdebian Grip, including support for migrating packages from unstable
       into testing.

       Considering just "unstable" initially, the hierarchy is as follows:

       Debian unstable is filtered using ${base}${filter_name}/conf/pkglist
       which in turn is based on the output of dpkg --get-selections. Only
       packages that are actually installed on machines running Emdebian Grip
       get added to the pkglist. "reprepro" then updates ${filter_name} from
       the chosen Debian mirror, downloading only the binary and source
       packages specified in pkglist, to create a partial local mirror for the
       list of supported architectures.

       Grip unstable is built from the filtered mirror by a combined process
       of converting the source package and then identifying the missing
       binary packages from the pkglist. During the "emgrip" process, Emdebian
       TDebs are created which get included into the locale repository:

       Locale is a very specialised repository that is designed for Debian
       source packages and Emdebian TDebs only, organised by locale root.
       e.g. en_GB support is in the en component. Whereas Debian has three
       components (main, contrib and non-free), the locale repository has
       ninety seven components (including main for the source packages). In
       the locale pool, packages are organised by component so whereas in the
       filter or grip repository, "apt" packages are under pool/main/a/apt in
       the locale repository the en TDebs for apt are in pool/en/a/apt/ and
       the French under pool/fr/a/apt/ etc. The list of locale roots supported
       by the locale repository is returned by &get_locale_roots, part of the
       "Debian::Packages::Compare" module.  For more information on locale
       roots as components see:
       <http://www.emdebian.org/emdebian/langupdate.html>

       As far as testing is concerned, the hierarchy works in a similar way to
       Debian, but with extra support for repositories that are started after
       packages have migrated into Debian testing and a newer version exists
       in Debian unstable.

       Note that the testing repository data needs to be setup manually in
       many cases - editing conf/distributions for each repository. The
       current Emdebian conf files will find their way onto the Debian Wiki in
       due course.

       With a fully populated Debian unstable, Grip unstable and locale
       unstable, &grip_britney can then migrate packages into Grip testing
       using the criteria implemented in &get_britney_list from
       "Debian::Packages::Compare" and the "reprepro copysrc" command. As
       most, if not all, Grip repositories will begin at a time when some
       packages in Debian have already migrated into testing and had a newer
       version uploaded to unstable, "Emdebian::Grip" also includes catch-up
       support via &migrate_missing which includes the relevant packages from
       Debian testing as if that version had been uploaded directly to Grip
       testing (which, in effect, is what is happening).

       The emgrip process can take a significant amount of time - think many
       hours, not minutes - principally due to the overhead of generating the
       TDebs at one per source per locale per architecture.  To create a full
       unstable and testing set is likely to take a day or more but is mostly
       automated.

       The problems that remain are related to dependency issues. The &edos
       function in "Emdebian::Grip" does have problems trying to calculate the
       solution for dependency problems identified by "edos-debcheck" and the
       problems become particularly acute if the repository itself is not in
       good shape at the start. Therefore, &edos is not run automatically by
       the current tools. The repository must be fully up to date and all
       relevant packages need to exist in the filter and in Grip before using
       &edos support. Initially, use "edos-debcheck" directly to identify
       major problems:

        edos-debcheck -explain -failures < ${base}${grip_name}/dists/sid/main/binary-i386/Packages

       Packages that are NOT AVAILABLE need to be gripped as source or binary
       packages as appropriate but care is needed to correctly interpret the
       "edos-debcheck" output to identify the correct link in the chain.

       Two major problems exist with automating this process, see
       "em_autogrip" (1) for specific information, but suffice it to say here
       that there exist in Debian unstable, packages that will appear in the
       list of NOT AVAILABLE packages where the correct action is to REMOVE
       the package that depends on the unavailable package. These issues are
       being discussed within Debian.

Copyright and Licence

        Copyright (C) 2007-2009  Neil Williams <codehelp@debian.org>

        This package is free software; you can redistribute it and/or modify
        it under the terms of the GNU General Public License as published by
        the Free Software Foundation; either version 3 of the License, or
        (at your option) any later version.

        This program is distributed in the hope that it will be useful,
        but WITHOUT ANY WARRANTY; without even the implied warranty of
        MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
        GNU General Public License for more details.

        You should have received a copy of the GNU General Public License
        along with this program.  If not, see <http://www.gnu.org/licenses/>.

Example

        use strict;
        use warnings;
        use Emdebian::Grip;
        use Debian::Packages::Compare;

        use vars qw/ $filter_name $grip_name $suite $base $verbose
         $noskip @archlist @locroots @lines $line %pkg @filter $have
         %debianunstable %gripunstable %tdebunstable /;

        my $mirror='http://ftp.uk.debian.org/debian'; # default
        $filter_name = 'filter';
        $grip_name = 'grip';
        $suite = "unstable"; # at first
        $base = '/opt/reprepro/';
        $verbose = 0;

        &set_base($base);
        &set_repo_names ($filter_name, $grip_name);
        my $a = &get_archlist ($suite, $filter_name);
        @archlist = (not defined $a or not @$a) ?
          qw/i386 amd64 arm armel powerpc mips mipsel/ : @$a;
        my $l = &get_locale_roots ($suite, 'locale');
        @locroots = (not defined $l or not @$l) ? qw/ af am ang ar as ast az be bg
         bn br bs ca cs cy da de dz el en eo es et eu fa fi fr ga gl gu he hi hr
         hu hy ia id io is it ja ka kn km ko ku ky lg li lt lv mai mg mi mk ml mn mr
         ms nb ne nl nn no ns nso oc or pa pl ps pt rm ro ru rw si sk sl sq sr sv
         ta te th tk tl tr tt ug uk ur uz vi wa wo xh yi zh zu / : @$l;

        &setup_repos if ( not -f "${base}${filter_name}/conf/pkglist" );

        my $debu  = &read_packages ('unstable', $filter_name);
        my $gripu = &read_packages ('unstable', $grip_name);
        my $tdebu = &read_locale   ('unstable', 'locale');
        %debianunstable = %$debu   if (defined $debu);
        %gripunstable   = %$gripu  if (defined $gripu);
        %tdebunstable   = %$tdebu  if (defined $tdebu);
        &update_filter;
        &update_repo($verbose);

        # begin the work.

set_repo_names

       Copies the default or user-specified filter and grip repository names
       into the internal module.

set_noskip

       Copies the user-specified --noskipold option into the internal module
       for use when calling reprepro.

set_dry_run

       Configures a dry-run where external commands are printed instead of
       executed.

print_missing

       Outputs the list returned by
       "Debian::Package::Compare::get_missing_sources" in a simple format

print_build_deps

       Outputs the list returned by
       "Debian::Package::Compare::get_missing_builddeps" in a simple format

print_testing_status

       Provides a detailed output of the status of the testing repository for
       Grip, comparing Debian unstable (filter) against Debian testing
       (filter) and identifying which source packages in Emdebian Grip testing
       are behind Debian.

       This list is further split into those where the source version in
       Debian unstable is precisely the same as the source version in Debian
       testing (packages that need to be migrated into Grip testing) and those
       where the version in testing differs.

grip_britney

       Grip version of the Britney script that migrates packages into Debian
       testing. See "Debian::Packages::Compare" (3) for details on
       get_britney_list criteria.

       In addition, in order to allow new repositories to catch up with
       Debian, packages in Grip unstable that are already ahead of Debian
       testing need to be gripped directly from Debian testing.

       This can also happen when packages are put directly into testing using
       testing-proposed-updates during a release freeze in Debian.

       Note that grip_britney only migrates packages where the version in
       Debian unstable is the same as the version in Debian testing - i.e.
       where a migration has already taken place in Debian. Packages that
       arrive in testing via testing-proposed-updates or migrated into testing
       before the Grip repository was started and which have a later version
       now in unstable, can be migrated using migrate_missing.

       If you are running grip_britney and migrate_missing together, run
       grip_britney first and reload all the data using read_packages before
       attempting to run migrate_missing. See migrate_missing for more
       information.

migrate_missing

       Corollary of grip_britney that completes the process by including into
       testing, packages that have a different version in Debian unstable.

       Missing packages in testing are included using grip_source and
       grip_binary as if uploaded directly to testing (which, in effect, is
       what is happening). This means that migrate_missing will take just as
       long to process the package in testing as it did to process the later
       version that went into unstable. Depending on the number of packages
       affected, migrate_missing can take a lot longer than grip_britney but
       it should be needed less often once the repository is complete.

edos

       Only a basic framework at this stage, this function needs to eventually
       collate the results by architecture and repository and try to present a
       solution.

       See "em_autogrip" (1) for problems with edos recursion.

       In $mode eq ’edos’, simply outputs the check data for each of the
       filter and grip repositories.

clean_incoming

       Call intermittently during long runs.

cleanup

       Only call before exit.

update_filter

       Fill in the FilterList so that automatic updates can proceed - without
       a filter, the entire archive is added. However, updating takes an
       appreciable amount of time, so call only when necessary.

extend_filter

       Adds package(s) to the filter list, a space separated list is
       acceptable.

update_repo

       Once the pkglist filter file is correct, call reprepro to get updated
       packages.

incoming_locale

       Checks the incoming directory for Emdebian TDebs and includes them into
       reprepro.

       Until the .tdeb extension is supported, reprepro needs to be persuaded
       to allow such files using the "--ignore=extension" option.

       Amongst all the expected error messages, some useful output is sent to
       "STDERR" by reprepro when handling TDebs so errors are not redirected
       to /dev/null.

check_dirs

       Check that critical directories in the three-way repository hierarchy
       exist and create any that are missing.

setup_repos

       Note: When setting up your own repository, remember that including the
       Contents: specifier in reprepro costs a lot of time during each and
       every operation on the repository. Contents is not enabled by
       "setup_repos" and it is strongly recommended that it is not re-enabled
       in later changes.

       Need to decide whether this will cope with multiple suites or just
       enforce suite=unstable.

       locale is shared with Crush. It’s not particularly easy to add a new
       locale root as the scripts and the repos need to be changed at the same
       time.

       necessary reprepro data (needs config support): these files need to be
       created for reprepro final repos will also need "SignWith: 0x97BB3B58"
       except stable which should be signed manually to bridge the security
       gap between Grip and Debian.

convert_prefix

       Debian repositories use the index character of the source package name
       in the path to the package directory beneath the pool in order to
       reduce the number of listings per directory to a manageable level. The
       exception is source packages beginning with ’"lib"’, because there are
       so many that the ’"l"’ section would be too large. Whilst ’"l"’
       remains, any source package that begins with ’"lib"’ is put under a
       separate set of directories using the first four letters instead of
       just ’"l"’, e.g. "libaa" will be found under "pool/main/liba/libaa/"
       and "libfoo" under "pool/main/libf/libfoo/" but "limpet" would be under
       "pool/main/l/limpet/"

       convert_prefix is a quick function to determine the correct prefix
       under these rules.

grip_source

       Wrapper for emgrip that takes a single Debian source package, grips the
       binary package where the source package name matches the binary package
       name (due to a hitch in how "reprepro" works), generates any Emdebian
       TDebs, includes the unchanged source package into Grip, includes any
       gripped binary packages into Grip and includes any generated Emdebian
       TDebs into locale.

       Once the source package has been processed, runs &clean_incoming.

grip_binary

       Wrapper for emgrip that takes a single Debian binary package, and grips
       it, generates any Emdebian TDebs, includes the gripped binary package
       into Grip and includes any generated Emdebian TDebs into locale.

       Once the binary package has been processed, runs &clean_incoming.

   Components and Sections
       Each binary package belongs to a "Section" that broadly describes the
       type of functionality in the package. If a component exists in the
       repository with the same name as the "Section", the package will be put
       into that component. Additionally, if the package name ends in "-dev"
       or the "Section" is devel, the "dev" component is selected. If the
       package name ends in "-doc" or the "Section" is "doc", the "doc"
       component is selected. If there is no matching component configured for
       the Grip repository, "main" is used. Section names themselves are read
       from the filter repository - i.e. set by the Debian ftp-master
       overrides.

switch_component

       Single routine to work out which components should be used for which
       sections.

AUTHOR

       Neil Williams, "<codehelp@debian.org>"

BUGS

       Please report any bugs or feature requests to the Debian Bug Tracking
       System using "reportbug emdebian-tools".

Known bugs

       (Otherwise known as a ToDo list.)

       The status of TDebs in the locale repository needs to be checked before
       generating more but this means changes in emgrip too so that tdebs can
       be skipped with a command line option. However, in normal operation,
       the package is only processed because it is out of date so the TDebs
       and other files need to be replaced anyway.

       "Architecture: all" are packages being continually reprocessed for
       several architectures. Originally this was because the package could
       contain translations which would need to be architecture-dependent in
       Emdebian. However, TDebs should be being generated only from source
       packages and this step probably needs to resolved. "reprepro" correctly
       handles such duplication, it is just a bit noisy when handling it and
       it adds to the time required by the process as a whole.

       Also need a general overview function that summaries all three
       repositories.