./sysutils/py-borgbackup, Deduplicating backup program with compression and encryption

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]

Branch: CURRENT, Version: 1.1.4nb1, Package name: py36-borgbackup-1.1.4nb1, Maintainer: bsiegert

BorgBackup (short: Borg) is a deduplicating backup program. Optionally,
it supports compression and authenticated encryption.

The main goal of Borg is to provide an efficient and secure way to
backup data. The data deduplication technique used makes Borg suitable
for daily backups since only changes are stored. The authenticated
encryption technique makes it suitable for backups to not fully trusted

Required to run:
[devel/py-setuptools] [devel/py-cython] [devel/py-msgpack] [archivers/lz4] [lang/python36]

Required to build:
[pkgtools/cwrappers] [devel/py-setuptools_scm]

Master sites:

SHA1: e3c26df1dc7f18a1798b1286a17c637bba6b9d9a
RMD160: b8375cd66fb3f6567bb9c68721c75eb3093cbaa0
Filesize: 3310.592 KB

Version history: (Expand)

CVS history: (Expand)

   2018-01-23 10:50:18 by Thomas Klausner | Files touched by this commit (3) | Package updated
Log message:
py-borgbackup: fix runtime problem

msgpack distribution changed name from 'msgpack-python' to 'msgpack'.

   2018-01-07 15:42:08 by Benny Siegert | Files touched by this commit (2) | Package updated
Log message:
Update borgbackup to 1.1.4.


- check: data corruption fix: fix for borg check --repair malfunction, #3444.
  See the more detailled notes close to the top of this document.
- delete: also delete security dir when deleting a repo, #3427
- prune: fix building the "borg prune" man page, #3398
- init: use given --storage-quota for local repo, #3470
- init: properly quote repo path in output
- fix startup delay with dns-only own fqdn resolving, #3471

New features:

- added zstd compression. try it!
- added placeholder {reverse-fqdn} for fqdn in reverse notation
- added BORG_BASE_DIR environment variable, #3338

Other changes:

- list help topics when invalid topic is requested
- fix lz4 deprecation warning, requires lz4 >= 1.7.0 (r129)
- add parens for C preprocessor macro argument usages (did not cause malfunction)
- exclude broken pytest 3.3.0 release
- updated fish/bash completions
- init: more clear exception messages for borg create, #3465
- docs:

  - add auto-generated docs for borg config
  - don't generate HTML docs page for borgfs, #3404
  - docs update for lz4 b2 zstd changes
  - add zstd to compression help, readme, docs
  - update requirements and install docs about bundled lz4 and zstd
- refactored build of the compress and crypto.low_level extensions, #3415:

  - move some lib/build related code to setup_{zstd,lz4,b2}.py
  - bundle lz4 1.8.0 (requirement: >= 1.7.0 / r129)
  - bundle zstd 1.3.2 (requirement: >= 1.3.0)
  - blake2 was already bundled
  - rename BORG_LZ4_PREFIX env var to BORG_LIBLZ4_PREFIX for better consistency:
    we also have BORG_LIBB2_PREFIX and BORG_LIBZSTD_PREFIX now.
  - add prefer_system_lib* = True settings to setup.py - by default the build
    will prefer a shared library over the bundled code, if library and headers
    can be found and meet the minimum requirements.

Pre-1.1.4 potential data corruption issue

A data corruption bug was discovered in borg check --repair, see issue #3444.

This is a 1.1.x regression, releases < 1.1 (e.g. 1.0.x) are not affected.

To avoid data loss, you must not run borg check --repair using an unfixed version
of borg 1.1.x. The first official release that has the fix is 1.1.4.

Package maintainers may have applied the fix to updated packages of 1.1.x (x<4)
though, see the package maintainer's package changelog to make sure.

If you never had missing item metadata chunks, the bug has not affected you
even if you did run borg check --repair with an unfixed version.

When borg check --repair tried to repair corrupt archives that miss item metadata
chunks, the resync to valid metadata in still present item metadata chunks
malfunctioned. This was due to a broken validator that considered all (even valid)
item metadata as invalid. As they were considered invalid, borg discarded them.
Practically, that means the affected files, directories or other fs objects were
discarded from the archive.

Due to the malfunction, the process was extremely slow, but if you let it
complete, borg would have created a "repaired" archive that has lost a \ 
lot of items.
If you interrupted borg check --repair because it was so strangely slow (killing
borg somehow, e.g. Ctrl-C) the transaction was rolled back and no corruption \ 

The log message indicating the precondition for the bug triggering looks like:

    item metadata chunk missing [chunk: 001056_bdee87d...a3e50d]

If you never had that in your borg check --repair runs, you're not affected.

But if you're unsure or you actually have seen that, better check your archives.
By just using "borg list repo::archive" you can see if all expected \ 
items are listed.
   2017-12-10 09:30:01 by Adam Ciarcinski | Files touched by this commit (1)
Log message:
Added python to CATEGORIES
   2017-12-08 22:12:05 by Benny Siegert | Files touched by this commit (1)
Log message:
Add PLIST.Linux
   2017-12-02 09:53:10 by Benny Siegert | Files touched by this commit (3) | Package updated
Log message:
Update py-borgbackup to 1.1.3 (security release).

Version 1.1.3 (2017-11-27)

Compatibility notes:

- When upgrading from borg 1.0.x to 1.1.x, please note:

  - read all the compatibility notes for 1.1.0*, starting from 1.1.0b1.
  - borg upgrade: you do not need to and you also should not run it.
  - borg might ask some security-related questions once after upgrading.
    You can answer them either manually or via environment variable.
    One known case is if you use unencrypted repositories, then it will ask
    about a unknown unencrypted repository one time.
  - your first backup with 1.1.x might be significantly slower (it might
    completely read, chunk, hash a lot files) - this is due to the
    --files-cache mode change (and happens every time you change mode).
    You can avoid the one-time slowdown by using the pre-1.1.0rc4-compatible
    mode (but that is less safe for detecting changed files than the default).
    See the --files-cache docs for details.


- Security Fix for CVE-2017-15914: Incorrect implementation of access controls
  allows remote users to override repository restrictions in Borg servers.
  A user able to access a remote Borg SSH server is able to circumvent access
  controls post-authentication.
  Affected releases: 1.1.0, 1.1.1, 1.1.2. Releases 1.0.x are NOT affected.
- crc32: deal with unaligned buffer, add tests - this broke borg on older ARM
  CPUs that can not deal with unaligned 32bit memory accesses and raise a bus
  error in such cases. the fix might also improve performance on some CPUs as
  all 32bit memory accesses by the crc32 code are properly aligned now. #3317
- mount: fixed support of --consider-part-files and do not show .borg_part_N
  files by default in the mounted FUSE filesystem. #3347
- fixed cache/repo timestamp inconsistency message, highlight that information
  is obtained from security dir (deleting the cache will not bypass this error
  in case the user knows this is a legitimate repo).
- borgfs: don't show sub-command in borgfs help, #3287
- create: show an error when --dry-run and --stats are used together, #3298

New features:

- mount: added exclusion group options and paths, #2138

  Reused some code to support similar options/paths as borg extract offers -
  making good use of these to only mount a smaller subset of dirs/files can
  speed up mounting a lot and also will consume way less memory.

  borg mount [options] repo_or_archive mountpoint path [paths...]

  paths: you can just give some "root paths" (like for borg extract) to
  only partially populate the FUSE filesystem.

  new options: --exclude[-from], --pattern[s-from], --strip-components
- create/extract: support st_birthtime on platforms supporting it, #3272
- add "borg config" command for querying/setting/deleting config \ 
values, #3304

Other changes:

- clean up and simplify packaging (only package committed files, do not install
  .c/.h/.pyx files)
- docs:

  - point out tuning options for borg create, #3239
  - add instructions for using ntfsclone, zerofree, #81
  - move image backup-related FAQ entries to a new page
  - clarify key aliases for borg list --format, #3111
  - mention break-lock in checkpointing FAQ entry, #3328
  - document sshfs rename workaround, #3315
  - add FAQ about removing files from existing archives
  - add FAQ about different prune policies
  - usage and man page for borgfs, #3216
  - clarify create --stats duration vs. wall time, #3301
  - clarify encrypted key format for borg key export, #3296
  - update release checklist about security fixes
  - document good and problematic option placements, fix examples, #3356
  - add note about using --nobsdflags to avoid speed penalty related to
    bsdflags, #3239
  - move most of support section to www.borgbackup.org
   2017-11-12 11:52:42 by Benny Siegert | Files touched by this commit (6) | Package updated
Log message:
Update borgbackup to 1.1.2.

Full changelog (long) at
https://github.com/borgbackup/borg/blob … anges.rst.

When upgrading from borg 1.0.x to 1.1.x, please note:

- borg might ask some security-related questions once after upgrading. You can
  answer them either manually or via environment variable. One known case is if
you use unencrypted repositories, then it will ask about a unknown unencrypted
repository one time.
- your first backup with 1.1.x might be significantly slower (it might
  completely read, chunk, hash a lot files) - this is due to the --files-cache
mode change (and happens every time you change mode). You can avoid the
one-time slowdown by using the pre-1.1.0rc4-compatible mode (but that is less
safe for detecting changed files than the default). See the --files-cache docs
for details.
   2017-08-06 10:34:02 by Thomas Klausner | Files touched by this commit (1)
Log message:
   2017-08-05 12:25:46 by Benny Siegert | Files touched by this commit (3) | Package updated
Log message:
Update borgbackup to 1.0.11.
(I am not 100% sure about the PLIST changes, I'd appreciate for someone
to verify.)

Version 1.0.11 (2017-07-21)

Bug fixes:

- use limited unpacker for outer key (security precaution), #2174
- fix paperkey import bug

Other changes:

- change --checkpoint-interval default from 600s to 1800s, #2841.
  this improves efficiency for big repositories a lot.
- docs: fix OpenSUSE command and add OpenSUSE section
- tests: add tests for split_lstring and paperkey
- vagrant:

  - fix openbsd shell
  - backport cpu/ram setup from master
  - add stretch64 VM

Version 1.0.11rc1 (2017-06-27)

Bug fixes:

- performance: rebuild hashtable if we have too few empty buckets, #2246.
  this fixes some sporadic, but severe performance breakdowns.
- Archive: allocate zeros when needed, #2308
  fixes huge memory usage of mount (8 MiB × number of archives)
- IPv6 address support
  also: Location: more informative exception when parsing fails
- borg single-file binary: use pyinstaller v3.2.1, #2396
  this fixes that the prelink cronjob on some distros kills the
  borg binary by stripping away parts of it.
- extract:

  - warning for unextracted big extended attributes, #2258
  - also create parent dir for device files, if needed.
  - don't write to disk with --stdout, #2645
- archive check: detect and fix missing all-zero replacement chunks, #2180
- fix (de)compression exceptions, #2224 #2221
- files cache: update inode number, #2226
- borg rpc: use limited msgpack.Unpacker (security precaution), #2139
- Manifest: use limited msgpack.Unpacker (security precaution), #2175
- Location: accept //servername/share/path
- fix ChunkIndex.__contains__ assertion  for big-endian archs (harmless)
- create: handle BackupOSError on a per-path level in one spot
- fix error msg, there is no --keep-last in borg 1.0.x, #2282
- clamp (nano)second values to unproblematic range, #2304
- fuse / borg mount:

  - fix st_blocks to be an integer (not float) value
  - fix negative uid/gid crash (they could come into archives e.g. when
    backing up external drives under cygwin), #2674
  - fix crash if empty (None) xattr is read
  - do pre-mount checks before opening repository
  - check llfuse is installed before asking for passphrase
- borg rename: expand placeholders, #2386
- borg serve: fix forced command lines containing BORG_ env vars
- fix error msg, it is --keep-within, not --within
- fix borg key/debug/benchmark crashing without subcommand, #2240
- chunker: fix invalid use of types, don't do uint32_t >> 32
- document follow_symlinks requirements, check libc, #2507

New features:

- added BORG_PASSCOMMAND environment variable, #2573
- add minimal version of in repository mandatory feature flags, #2134

  This should allow us to make sure older borg versions can be cleanly
  prevented from doing operations that are no longer safe because of
  repository format evolution. This allows more fine grained control than
  just incrementing the manifest version. So for example a change that
  still allows new archives to be created but would corrupt the repository
  when an old version tries to delete an archive or check the repository
  would add the new feature to the check and delete set but leave it out
  of the write set.
- borg delete --force --force to delete severely corrupted archives, #1975

Other changes:

- embrace y2038 issue to support 32bit platforms
- be more clear that this is a "beyond repair" case, #2427
- key file names: limit to 100 characters and remove colons from host name
- upgrade FUSE for macOS to 3.5.8, #2346
- split up parsing and filtering for --keep-within, better error message, #2610
- docs:

  - fix caskroom link, #2299
  - address SSH batch mode, #2202 #2270
  - improve remote-path description
  - document snapshot usage, #2178
  - document relative path usage, #1868
  - one link per distro in the installation page
  - development: new branching model in git repository
  - kill api page
  - added FAQ section about backing up root partition
  - add bountysource badge, #2558
  - create empty docs.txt reequirements, #2694
  - README: how to help the project
  - note -v/--verbose requirement on affected options, #2542
  - document borg init behaviour via append-only borg serve, #2440
  - be clear about what buzhash is used for (chunking) and want it is not
    used for (deduplication)- also say already in the readme that we use a
    cryptohash for dedupe, so people don't worry, #2390
  - add hint about chunker params to borg upgrade docs, #2421
  - clarify borg upgrade docs, #2436
  - quickstart: delete problematic BORG_PASSPRHASE use, #2623
  - faq: specify "using inline shell scripts"
  - document pattern denial of service, #2624
- tests:

  - remove attic dependency of the tests, #2505
  - travis:

    - enhance travis setuptools_scm situation
    - install fakeroot for Linux
  - add test for borg delete --force
  - enable remote tests on cygwin (the cygwin issue that caused these tests
    to break was fixed in cygwin at least since cygwin 2.8, maybe even since
  - remove skipping the noatime tests on GNU/Hurd, #2710
  - fix borg import issue, add comment, #2718
  - include attic.tar.gz when installing the package
    also: add include_package_data=True

Version 1.0.10 (2017-02-13)

Bug fixes:

- Manifest timestamps are now monotonically increasing,
  this fixes issues when the system clock jumps backwards
  or is set inconsistently across computers accessing the same repository, #2115
- Fixed testing regression in 1.0.10rc1 that lead to a hard dependency on
  py.test >= 3.0, #2112

New features:

- "key export" can now generate a printable HTML page with both a QR \ 
code and
  a human-readable "paperkey" representation (and custom text) through the
  ``--qr-html`` option.

  The same functionality is also available through `paperkey.html \ 
  which is the same HTML page generated by ``--qr-html``. It works with existing
  "key export" files and key files.

Other changes:

- docs:

  - language clarification - "borg create --one-file-system" option \ 
does not respect
    mount points, but considers different file systems instead, #2141
- setup.py: build_api: sort file list for determinism

Version 1.0.10rc1 (2017-01-29)

Bug fixes:

- borg serve: fix transmission data loss of pipe writes, #1268
  This affects only the cygwin platform (not Linux, BSD, OS X).
- Avoid triggering an ObjectiveFS bug in xattr retrieval, #1992
- When running out of buffer memory when reading xattrs, only skip the
  current file, #1993
- Fixed "borg upgrade --tam" crashing with unencrypted repositories. Since
  :ref:`the issue <tam_vuln>` is not relevant for unencrypted repositories,
  it now does nothing and prints an error, #1981.
- Fixed change-passphrase crashing with unencrypted repositories, #1978
- Fixed "borg check repo::archive" indicating success if \ 
"archive" does not exist, #1997
- borg check: print non-exit-code warning if --last or --prefix aren't fulfilled
- fix bad parsing of wrong repo location syntax
- create: don't create hard link refs to failed files,
  mount: handle invalid hard link refs, #2092
- detect mingw byte order, #2073
- creating a new segment: use "xb" mode, #2099
- mount: umount on SIGINT/^C when in foreground, #2082

Other changes:

- binary: use fixed AND freshly compiled pyinstaller bootloader, #2002
- xattr: ignore empty names returned by llistxattr(2) et al
- Enable the fault handler: install handlers for the SIGSEGV, SIGFPE, SIGABRT,
  SIGBUS and SIGILL signals to dump the Python traceback.
- Also print a traceback on SIGUSR2.
- borg change-passphrase: print key location (simplify making a backup of it)
- officially support Python 3.6 (setup.py: add Python 3.6 qualifier)
- tests:

  - vagrant / travis / tox: add Python 3.6 based testing
  - vagrant: fix openbsd repo, #2042
  - vagrant: fix the freebsd64 machine, #2037 #2067
  - vagrant: use python 3.5.3 to build binaries, #2078
  - vagrant: use osxfuse 3.5.4 for tests / to build binaries
    vagrant: improve darwin64 VM settings
  - travis: fix osxfuse install (fixes OS X testing on Travis CI)
  - travis: require succeeding OS X tests, #2028
  - travis: use latest pythons for OS X based testing
  - use pytest-xdist to parallelize testing
  - fix xattr test race condition, #2047
  - setup.cfg: fix pytest deprecation warning, #2050
- docs:

  - language clarification - VM backup FAQ
  - borg create: document how to backup stdin, #2013
  - borg upgrade: fix incorrect title levels
  - add CVE numbers for issues fixed in 1.0.9, #2106
- fix typos (taken from Debian package patch)
- remote: include data hexdump in "unexpected RPC data" error message
- remote: log SSH command line at debug level
- API_VERSION: use numberspaces, #2023
- remove .github from pypi package, #2051
- add pip and setuptools to requirements file, #2030
- SyncFile: fix use of fd object after close (cosmetic)
- Manifest.in: simplify, exclude \*.{so,dll,orig}, #2066
- ignore posix_fadvise errors in repository.py, #2095
  (works around issues with docker on ARM)
- make LoggedIO.close_segment reentrant, avoid reentrance