waptpackage module

waptpackage.control_to_dict(control, int_params=('size', 'installed_size'))[source]

Convert a control file like object key1: value1 key2: value2 … list of lines into a dict

Multilines strings begins with a space

Breaks when an empty line is reached (limit between 2 package in Packages indexes)

Parameters:
  • control (file) – file like object to read control from (until an empty line is reached)
  • int_params (list) – attributes which must be converted to int
Returns:

dict

waptpackage.md5_for_file(fname, block_size=1048576)[source]

Calculate the md5 hash of file.

Returns:md5 hash as hexadecimal string.
Return type:str
waptpackage.parse_major_minor_patch_build(version)[source]

Parse version to major, minor, patch, pre-release, build parts.

waptpackage.make_version(major_minor_patch_build)[source]
class waptpackage.PackageRequest(package_request)[source]

Bases: waptutils.BaseObjectClass

Package and version request / condition

>>> PackageRequest('7-zip( >= 7.2)')
PackageRequest('7-zip (>=7.2)')
class waptpackage.PackageEntry(package='', version='0', repo='', waptfile=None, section='base', _default_md='sha256', **kwargs)[source]

Bases: waptutils.BaseObjectClass

Manage package attributes coming from either control files in WAPT package, local DB, or developement dir.

Methods to build, unzip, sign or check a package. Methods to sign the control attributes and check them.

>>> pe = PackageEntry('testgroup','0')
>>> pe.depends = 'tis-7zip'
>>> pe.section = 'group'
>>> pe.description = 'A test package'
>>> print(pe.ascontrol())
package           : testgroup
version           : 0
architecture      : all
section           : group
priority          : optional
maintainer        :
description       : A test package
depends           : tis-7zip
conflicts         :
maturity          :
locale            :
min_os_version    :
max_os_version    :
min_wapt_version  :
sources           :
installed_size    :
signer            :
signer_fingerprint:
signature_date    :
signed_attributes :
>>>
all_attributes
as_dict()[source]
as_zipfile(mode='r')[source]

Return a Zipfile for this package for read only operations

ascontrol(with_non_control_attributes=False, with_empty_attributes=False)[source]

Return control attributes and values as stored in control packages file

Each attribute on a line with key : value If value is multiline, new line begin with a space.

Parameters:
  • with_non_control_attributes (bool) – weither to include all attributes or only those relevant for final package content.
  • with_empty_attributes (bool) – weither to include attribute with empty value too or only non empty and/or signed attributes
Returns:

lines of attr: value

Return type:

str

asrequirement()[source]

resturn package and version for designing this package in depends or install actions

Returns:“packagename (=version)”
Return type:str
build_management_package(target_directory=None)[source]
Build the WAPT package from attributes only, without setup.py

stores the result in target_directory. self.sourcespath must be None. Package will contain only a control file.

Args:
target_directory (str): where to create Zip wapt file.
if None, temp dir will be used.
Returns:
str: path to zipped Wapt file. It is unsigned.
>>> pe = PackageEntry('testgroup','0',description='Test package',maintainer='Hubert',sources='https://dev/svn/testgroup',architecture='x86')
>>> waptfn = pe.build_management_package()
>>> key = SSLPrivateKey('c:/private/htouvet.pem',password='monmotdepasse')
>>> crt = SSLCertificate('c:/private/htouvet.crt')
>>> pe.sign_package(crt,key)
'qrUUQeNJ3RSSeXQERrP9vD7H/Hfvw8kmBXZvczq0b2PVRKPdjMCElYKzryAbQ+2nYVDWAGSGrXxs

y2WzhOhrdMfGfcy6YLaY5muApoArBn3CjKP5G6HypOGD5agznLEKkcUg5/Y3aIR8bL55Ylmp3RaS WKnezUcuA2yuNuKwHsXr9CdihK5pRyYrm5KhCNy8S7+kAJvayrUj5q8ur6z0nNMQCHEnWGN+V3MI 84PymR1eXsuauKeYNqIESWCyyD/lFZv0JEYfrfml8rirC6yd6iTJW0OqH7gKwCEl03JpRaF91vWB OXN65S1j2oV8Jgjq43oa7lyywKC01a/ehQF5Jw== ‘

>>> pe.unzip_package()
'c:\users\htouvet\appdata\local\temp\waptob4gcd'
>>> ca = SSLCABundle('c:/wapt/ssl')
>>> pe.check_control_signature(ca)
<SSLCertificate cn=u'htouvet' issuer=u'tranquilit-ca-test' validity=2017-06-28 - 2027-06-26 Code-Signing=True CA=True>
build_manifest(exclude_filenames=None, block_size=1048576, forbidden_files=[], md='sha256')[source]

Calc the manifest of an already built (zipped) wapt package

Returns:{filepath:shasum,}
Return type:dict
build_package(excludes=['.svn', '.git', '.gitignore', 'setup.pyc'], target_directory=None)[source]

Build the WAPT package, stores the result in target_directory Zip the content of self.sourcespath directory into a zipfile named with default package filename based on control attributes.

Update filename attribute. Update localpath attribute with result filepath.

Parameters:
  • excludes (list) – list of patterns for source files to exclude from built package.
  • target_directory (str) – target directory where to store built package. If None, use parent directory of package sources dircetory.
Returns:

full filepath to built wapt package

Return type:

str

change_prefix(new_prefix)[source]

Change prefix of package name to new_prefix and return True if it was really changed.

check_control_signature(trusted_bundle, signers_bundle=None)[source]

Check in memory control signature against a list of public certificates

Parameters:
  • trusted_bundle (SSLCABundle) – Trusted certificates. : packages certificates must be signed by a one of this bundle.
  • signers_bundle – Optional. List of potential packages signers certificates chains. When checking Packages index, actual packages are not available, only certificates embedded in Packages index. Package signature are checked againt these certificates looking here for potential intermediate CA too. and matching certificate is checked against trusted_bundle.
Returns:

matching trusted package’s signers SSLCertificate

Return type:

SSLCertificate

>>> from waptpackage import *
>>> from common import SSLPrivateKey,SSLCertificate
>>> k = SSLPrivateKey('c:/private/test.pem')
>>> c = SSLCertificate('c:/private/test.crt')
>>> p = PackageEntry('test',version='1.0-0')
>>> p.depends = 'test'
>>> p._sign_control(k,c)
>>> p.check_control_signature(c)
>>> p.check_control_signature(SSLCABundle('c:/wapt/ssl'))
check_package_signature(trusted_bundle)[source]

Check - hash of files in unzipped package_dir with list in package’s manifest file - try to decrypt manifest signature with package’s certificate - check that the package certificate is issued by a know CA or the same as one the authorized certitificates.

Parameters:trusted_bundle (SSLCABundle) – list of authorized certificates / ca filepaths
Returns:matching certificate
Return type:SSLCertificate
Raises:Exception if no certificate match is found.
delete_localsources()[source]

Remove the unzipped local directory

download_url

Calculate the download URL for this entry

get(name, default=None)[source]

Get PackageEntry property.

Parameters:
  • name (str) – property to get. name is forced to lowercase.
  • default (any) – value to return in case the property doesn’t not exist.
Returns:

property value

Return type:

any

get_default_signed_attributes()[source]
get_manifest_filename(md=None)[source]
get_signature_filename(md=None)[source]
has_file(fname)[source]

Return None if fname is not in package, else return file datetime

Parameters:fname (unicode) – file path like WAPT/signature
Returns:last modification datetime of file in Wapt archive if zipped or local sources if unzipped
Return type:datetime
inc_build()[source]

Increment last number part of version in memory

invalidate_signature()[source]

Remove all signature informations from control and unzipped package directory Package must be in unzipped state.

list_corrupted_files()[source]

Check hexdigest sha for the files in manifest. Package must be already unzipped.

Returns:non matching files (corrupted files)
Return type:list
load_control_from_dict(adict)[source]

Fill in members of entry with keys from supplied dict

adict members which are not a registered control attribute are set too and attribute name is put in list of “calculated” attributes.

Parameters:adict (dict) – key,value to put in this entry
Returns:self
Return type:PackageEntry
load_control_from_wapt(fname=None, calc_md5=True)[source]

Load package attributes from the control file (utf8 encoded) included in WAPT zipfile fname

Parameters:
  • fname (str or list) – If None, try to load entry attributes from self.sourcespath or self.localpath If fname is a list, consider fname is the lines of control file If fname is file, it must be Wapt zipped file, and try to load control data from it If fname is a directory, it must be root dir of unzipped package file and load control from <fname>/WAPT/control
  • calc_md5 (boolean) – if True and fname is a zipped file, initialize md5sum attribute with md5 sum of Zipped file/
Returns:

self

Return type:

PackageEntry

make_package_edit_directory()[source]

Return the standard package directory to edit the package based on current attributes

Returns:
standard package filename
  • packagename_arch_maturity_locale-wapt for softwares and groups
  • packagename-wapt for host.
Return type:str
make_package_filename()[source]

Return the standard package filename based on current attributes parts of control which are either ‘all’ or empty are not included in filename

Returns:
standard package filename
  • packagename.wapt for host
  • packagename_arch_maturity_locale.wapt for group
  • packagename_version_arch_maturity_locale.wapt for others
Return type:str
manifest_filename_excludes = ['WAPT/signature', 'WAPT/signature.sha256', 'WAPT/manifest.sha256', 'WAPT/manifest.sha1']
match(match_expr)[source]

Return True if package entry match a package string like ‘tis-package (>=1.0.1-00)

Check if entry match search words

Parameters:search (str) – words to search for separated by spaces
Returns:True if entry contains the words in search in correct order and at word boundaries
Return type:boolean
match_version(match_expr)[source]

Return True if package entry match a version string condition like ‘>=1.0.1-00’

non_control_attributes = ['localpath', 'sourcespath', 'filename', 'size', 'repo_url', 'md5sum', 'repo']
not_duplicated_attributes = ['signer', 'signer_fingerprint', 'signature', 'signature_date', 'signed_attributes']
optional_attributes = ['maintainer', 'description', 'depends', 'conflicts', 'maturity', 'locale', 'min_os_version', 'max_os_version', 'min_wapt_version', 'sources', 'installed_size']
package_certificate()[source]

Return certificates from package. If package is built, take it from Zip else take the certificates from unzipped directory

Returns:
list of embedded certificates when package was signed or None if not provided or signed.
First one of the list is the signer, the others are optional intermediate CA
Return type:list
parse_version()[source]

Parse version to major, minor, patch, pre-release, build parts.

required_attributes = ['package', 'version', 'architecture', 'section', 'priority']
save_control_to_wapt(fname=None, force=True)[source]

Save package attributes to the control file (utf8 encoded)

Update self.locapath or self.sourcespath if not already set.

Parameters:
  • fname (str) – base directoy of waptpackage or filepath of Zipped Packges. If None, use self.sourcespath if exists, or self.localpath if exists
  • force (bool) – write control in wapt zip file even if it already exist
Returns:

None if nothing written, or previous PackageEntry if new data written

Return type:

PackageEntry

Raises:
  • Exception – if fname is None and no sourcespath and no localpath
  • Exception – if control exists and force is False
sign_package(certificate, private_key=None, password_callback=None, private_key_password=None, mds=['sha256'])[source]

Sign an already built package. Should follow immediately the build_package step.

Append signed control, manifest.sha256 and signature to zip wapt package If these files are already in the package, they are first removed.

Use the self.localpath attribute to get location of waptfile build file.

Parameters:
  • certificate (SSLCertificate or list) – signer certificate chain
  • private_key (SSLPrivateKey) – signer private key
  • password_callback (func) – function to call to get key password if encrypted.
  • private_key_password (str) – password to use if key is encrypted. Use eithe this or password_callback
  • mds (list) – list of message digest manifest and signature methods to include. For backward compatibility.
Returns:

signature

Return type:

str

signature_attributes = ['signer', 'signer_fingerprint', 'signature', 'signature_date', 'signed_attributes']
unzip_package(target_dir=None, cabundle=None)[source]

Unzip package and optionnally check content

Parameters:
  • target_dir (str) – where to unzip package content. If None, a temp dir is created
  • cabundle (list) – list of Certificates to check content. If None, no check is done
Returns:

path to unzipped packages files

Return type:

str

Raises:
  • EWaptNotAPackage, EWaptBadSignature,EWaptCorruptedFiles
  • if check is not successful, unzipped files are deleted.
class waptpackage.WaptBaseRepo(name='abstract', cabundle=None, config=None)[source]

Bases: waptutils.BaseObjectClass

Base abstract class for a Wapt Packages repository

get(packagename, default=None)[source]
get_certificates(packages_zipfile=None)[source]

Download signers certificates and crl from Package index on remote repository.

These certificates and CRL are appended to Packages index when scanning packages.
Parameters:packages_zipfile (zipfile) – if None, donwload it from repo
Returns :
SSLCABundle
get_package_entries(packages_names)[source]

Return most up to date packages entries for packages_names packages_names is either a list or a string :returns: a dictionnary with {‘packages’:[],’missing’:[]} :rtype: dict

>>> r = WaptRemoteRepo()
>>> r.load_config_from_file('c:/wapt/wapt-get.ini')
>>> res = r.get_package_entries(['tis-firefox','tis-putty'])
>>> isinstance(res['missing'],list) and isinstance(res['packages'][0],PackageEntry)
True
is_available()[source]

Return isodate of last updates of the repo is available else None

load_config(config=None, section=None)[source]
Load configuration from inifile section.

Use name of repo as section name if section is not provided. Use ‘global’ if no section named section in ini file Value not defined in ini file are taken from class _default_config

load_config is called at __init__, eventually with config = None. In this case, all parameters are initialized from defaults

Parameters:
  • config (RawConfigParser) – ini configuration
  • section (str) – section where to loads parameters defaults to name of repository
Returns:

return itself to chain calls.

Return type:

self

load_config_from_file(config_filename, section=None)[source]

Load repository configuration from an inifile located at config_filename

Parameters:
  • config_filename (str) – path to wapt inifile
  • section (str) – ini section from which to get parameters. default to repo name
Returns:

self

Return type:

WaptBaseRepo

need_update(last_modified=None)[source]

Check if packages index has changed on repo and local index needs an update

Compare date on local package index DB with the Packages file on remote
repository with a HEAD http request.
Parameters:last_modified (str) – iso datetime of last known update of packages.
Returns
bool: True if either Packages was never read or remote date of Packages is
more recent than the provided last_modifed date.
>>> repo = WaptRemoteRepo(name='main',url='http://wapt/wapt',timeout=4)
>>> waptdb = WaptDB('c:/wapt/db/waptdb.sqlite')
>>> res = repo.need_update(waptdb.read_param('last-%s'% repo.url))
>>> isinstance(res,bool)
True
packages

Local list of packages

packages_date

Date of last known packages index

packages_matching(package_cond)[source]
Return an ordered list of available packages entries which match
the condition “packagename[([=<>]version)]?” version ascending
Args;
package_cond (str): package name with optional version sepcifier.
Returns:list of PackageEntry
>>> from waptpackage import *
>>> r = WaptRemoteRepo('http://wapt.tranquil.it/wapt')
>>> r.packages_matching('tis-firefox (>=20)')
[PackageEntry('tis-firefox','20.0.1-02'),
 PackageEntry('tis-firefox','21.0.0-00'),
 ...]
search(searchwords=[], sections=[], newest_only=False)[source]
Return list of package entries
with description or name matching all the searchwords and section in provided sections list
>>> r = WaptRemoteRepo(name='test',url='http://wapt.tranquil.it/wapt')
>>> r.search('test')
update()[source]

Update local index of packages from source index

class waptpackage.WaptLocalRepo(localpath='/var/www/wapt', name='waptlocal', cabundle=None, config=None)[source]

Bases: waptpackage.WaptBaseRepo

Index of Wapt local repository.
Index of packages is located in a Packages zip file, having one

Packages file, containing the concatenated content of “control” files of the packages.

A blank line means new package.

>>> localrepo = WaptLocalRepo('c:/wapt/cache')
>>> localrepo.update()
load_config(config=None, section=None)[source]

Load waptrepo configuration from inifile section.

Use name of repo as section name if section is not provided. Use ‘global’ if no section named section in ini file
Parameters:
  • config (RawConfigParser) – ini configuration
  • section (str) – section where to loads parameters defaults to name of repository
Returns:

return itself to chain calls.

Return type:

WaptRemoteRepo

packages_path
update_packages_index(force_all=False)[source]

Scan self.localpath directory for WAPT packages and build a Packages (utf8) zip file with control data and MD5 hash

Extract icons from packages (WAPT/icon.png) and stores them in <repo path>/icons/<package name>.png Extract certificate and add it to Packages zip file in ssl/<fingerprint.crt> Append CRL for certificates.

class waptpackage.WaptRemoteRepo(url=None, name='', verify_cert=None, http_proxy=None, timeout=2, cabundle=None, config=None)[source]

Bases: waptpackage.WaptBaseRepo

Gives access to a remote http repository, with a zipped Packages packages index

>>> repo = WaptRemoteRepo(name='main',url='http://wapt/wapt',timeout=4)
>>> last_modified = repo.is_available()
>>> isinstance(last_modified,str)
True
as_dict()[source]
client_auth()[source]

Return SSL pair (cert,key) filenames for client side SSL auth

download_packages(package_requests, target_dir=None, usecache=True, printhook=None)[source]
Download a list of packages (requests are of the form packagename (>version) )
returns a dict of {“downloaded,”skipped”,”errors”}

If package_requests is a list of PackageEntry, update localpath of entry to match downloaded file.

Parameters:package_requests (list) – list of PackageEntry to download or list of package with optional version
Returns:‘downloaded’, ‘skipped’, ‘errors’
Return type:dict
>>> repo = WaptRemoteRepo(url='http://wapt.tranquil.it/wapt')
>>> wapt.download_packages(['tis-firefox','tis-waptdev'],printhook=nullhook)
{'downloaded': [u'c:/wapt\\cache\\tis-firefox_37.0.2-9_all.wapt', u'c:/wapt\\cache\\tis-waptdev.wapt'], 'skipped': [], 'errors': []}
is_available()[source]

Check if repo is reachable an return createion date of Packages.

Try to access the repo and return last modified date of repo index or None if not accessible

Returns:Iso creation date of remote Package file as returned in http headers
Return type:str
>>> repo = WaptRemoteRepo(name='main',url='https://wapt/wapt',timeout=1)
>>> repo.is_available() <= datetime2isodate()
True
>>> repo = WaptRemoteRepo(name='main',url='https://badwapt/wapt',timeout=1)
>>> repo.is_available() is None
True
load_config(config=None, section=None)[source]

Load waptrepo configuration from inifile section.

Use name of repo as section name if section is not provided. Use ‘global’ if no section named section in ini file
Parameters:
  • config (RawConfigParser) – ini configuration
  • section (str) – section where to loads parameters defaults to name of repository
Returns:

return itself to chain calls.

Return type:

WaptRemoteRepo

packages
packages_url

return url of Packages index file

>>> repo = WaptRemoteRepo(name='main',url='http://wapt/wapt',timeout=4)
>>> repo.packages_url
'http://wapt/wapt/Packages'

hardcoded path to the Packages index.

proxies
repo_url
waptpackage.update_packages(adir, force=False)[source]

Helper function to update a local packages index

This function is used on repositories to rescan all packages and
update the Packages index.
>>> if os.path.isdir('c:\wapt\cache'):
...     repopath = 'c:\wapt\cache'
... else:
...     repopath = '/var/www/wapt'
>>> p = PackageEntry()
>>> p.package = 'test'
>>> p.version = '10'
>>> new_package_fn = os.path.join(repopath,p.make_package_filename())
>>> if os.path.isfile(new_package_fn):
...     os.unlink(new_package_fn)
>>> res = update_packages(repopath)
>>> os.path.isfile(res['packages_filename'])
True
>>> r = WaptLocalRepo(localpath=repopath)
>>> l1 = r.packages
>>> res = r.update_packages_index()
>>> l2 = r.packages
>>> [p for p in l2 if p not in l1]
["test (=10)"]
exception waptpackage.EWaptException[source]

Bases: exceptions.Exception

exception waptpackage.EWaptBadSignature[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptCorruptedFiles[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptNotSigned[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptBadControl[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptBadSetup[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptNeedsNewerAgent[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptDiskSpace[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptBadTargetOS[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptNotAPackage[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptDownloadError[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptMissingLocalWaptFile[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptNeedsNewerAgent[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptConfigurationError[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptUnavailablePackage(msg, install_status='ERROR', retry_count=None)[source]

Bases: waptpackage.EWaptInstallError

exception waptpackage.EWaptNotSourcesDirPackage[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptPackageSignError[source]

Bases: waptpackage.EWaptException

exception waptpackage.EWaptConflictingPackage(msg, install_status='ERROR', retry_count=None)[source]

Bases: waptpackage.EWaptInstallError

exception waptpackage.EWaptInstallPostponed(msg, install_status='POSTPONED', retry_count=5, grace_delay=3600)[source]

Bases: waptpackage.EWaptInstallError

exception waptpackage.EWaptInstallError(msg, install_status='ERROR', retry_count=None)[source]

Bases: waptpackage.EWaptException

Exception raised during installation of package msg is logged in local install database if retry_count is None, install will be retried indefinitely until success else install is retried at most retry_count times.