Subject: CVS commit: pkgsrc/net/gallery-dl
From: Leonardo Taccari
Date: 2023-03-19 12:56:01
Message id: 20230319115601.B9641FA90@cvs.NetBSD.org

Log Message:
gallery-dl: Update to 1.25.0

Changes:
## 1.25.0 - 2023-03-11
### Changes
- [e621] split `e621` extractors from `danbooru` module
- [deviantart] remove mature scraps warning
- [deviantart] use `/collections/all` endpoint for favorites
- [newgrounds] update default image and audio archive IDs to prevent ID overlap
- rename `--ignore-config` to `--config-ignore`

### Extractors
- [catbox] add `file` extractor
- [deviantart] add `search` extractor
- [deviantart] add `gallery-search` extractor
- [deviantart] support `fxdeviantart.com` URLs (##3740)
- [e621] implement `notes` and `pools` metadata extraction
- [gelbooru] add `favorite` extractor
- [imagetwist] support `phun.imagetwist.com` and `imagehaha.com` domains
- [instagram] add `user` metadata field
- [manganelo] update and fix metadata extraction
- [manganelo] support mobile-only chapters
- [mangasee] extract `author` and `genre` metadata
- [misskey] add `misskey` extractors
- [pornpics] add `gallery` and `search` extractors
- [redgifs] support v3 URLs
- [redgifs] add `collection` extractors
- [shopify] support ohpolly.com
- [szurubooru] add `tag` and `post` extractors
- [twitter] add `transform` option

### Options
- [postprocessor:metadata] add `sort` and `separators` options
- [postprocessor:exec] implement archive options
- add `--config-create` command-line option
- add `--config-toml` command-line option to load config files in TOML format
- add `output.stdout`, `output.stdin`, and `output.stderr` options
- add `hash_md5` and `hash_sha1` functions
- implement `globals` option to enable defining custom functions for `eval` \ 
statements
- implement `archive-pragma` option to use SQLite PRAGMA statements
- implement `actions` to trigger events on logging messages
- implement ability to load external extractor classes
  - `-X/--extractors` command-line options
  - `extractor.modules-sources` config option

### Fixes
- [bunkr] fix extraction
- [danbooru] send gallery-dl User-Agent
- [deviantart] fix crash when handling deleted deviations in status updates
- [fanbox] fix crash with missing images
- [imagefap] update `gallery` URLs
- [imagefap] fix infinite pagination loop
- [imagefap] fix metadata extraction
- [oauth] use default name for browsers without `name` attribute
- [pinterest] unescape search terms
- [pixiv] fix `--write-tags` for `"tags": "original"`
- [poipiku] warn about incorrect passwords
- [reddit] update `videos` option
- [soundgasm] rewrite
- [telegraph] fix extraction when images are not in `<figure>` elements
- [tumblr] raise more detailed errors for dashboard-only blogs
- [twitter] fix some `original` retweets not downloading
- [ytdl] fix `--parse-metadata`
- [downloader:ytdl] prevent exception on empty results

### Improvements
- [downloader:http] use `time.monotonic()`
- [downloader:http] update `_http_retry` to accept a Python function
- [postprocessor:metadata] speed up JSON encoding
- replace `json.loads/dumps` with direct calls to \ 
`JSONDecoder.decode/JSONEncoder.encode`
- improve `option.Formatter` performance

### Removals
- [nitter] remove `nitter.pussthecat.org`

## 1.24.5 - 2023-01-28
### Additions
- [booru] add `url` option
- [danbooru] extend `metadata` option
- [deviantart] add extractor for status updates
- [deviantart] add support for `/deviation/` and `fav.me` URLs
- [kemonoparty] extract `hash` metadata for discord files
- [lexica] add `search` extractor
- [mastodon] add `num` and `count` metadata fields
- [nudecollect] add `image` and `album` extractors
- [wikifeet] add `gallery` extractor
- [downloader:http] add signature checks for `.blend`, `.obj`, and `.clip` files
- add `extractor.retry-codes` option
- add `-O/--postprocessor-option` command-line option
- improve `write-pages` output

### Fixes
- [bunkr] fix downloading `.mkv` and `.ts` files
- [fantia] send `X-CSRF-Token` headers
- [generic] fix regex for non-src image URLs
- [hiperdex] update domain
- [hotleak] fix video URLs
- [instagram] always show `cursor` value after errors
- [instagram] update API domain, headers, and csrf token handling
- [oauth] show `client-id`/`api-key` values
- [philomena] match URLs with www subdomain
- [sankaku] update URL pattern
- [twitter] refresh guest tokens
- [twitter] fix search pagination
- [twitter] use `"browser": "firefox"` by default

## 1.24.4 - 2023-01-11
### Additions
- [downloader:http] add `validate` option

### Fixes
- [kemonoparty] fix regression from commit 473bd380

## 1.24.3 - 2023-01-10
### Additions
- [danbooru] extract `uploader` metadata
- [deviantart] initial implementation of username & password login for `scraps`
- [fanleaks] add `post` and `model` extractors
- [imagefap] add `folder` extractor
- [lynxchan] support `bbw-chan.nl`
- [pinterest] support `All Pins` boards
- [pinterest] add `domain` option
- [pixiv] implement `metadata-bookmark` option
- [tcbscans] add `chapter` and `manga` extractors
- [twitter] implement `syndication=extended`
- implement slice notation for `range` options
- allow `filter` options to be a list of expressions

### Fixes
- [behance] use delay between requests
- [bunkr] fix URLs returned by API
- [fanbox] return `imageMap` files in order
- [imagefap] use delay between requests
- [imagefap] warn about redirects to `/human-verification`
- [kemonoparty] reject invalid/empty files
- [myhentaigallery] handle whitespace before title tag
- [poipiku] fix extraction for a different warning button style
- [poipiku] warn about login requirements
- [telegraph] fix file URLs
- [twitter] fix crash when using `expand` and `syndication`
- [twitter] apply tweet type checks before uniqueness check
- [twitter] force `https://` for TwitPic URLs
- [ytdl] adapt to yt-dlp changes
- update and improve documentation

## 1.24.2 - 2022-12-18
### Additions
- [2chen] support `.club` URLs
- [deviantart] extract sta.sh URLs from `text_content`
- [deviantart] add `/view` URL support
- [e621] implement `threshold` option to control pagination
- [fapello] add `post`, `user` and `path` extractors
- [imgur] add support for imgur.io URLs
- [lynxchan] add generic extractors for lynxchan imageboards
- [mangafox] extract more metadata
- [pixiv] extract `date_url` metadata
- [soundgasm] add `audio` and `user` extractors
- [webmshare] add `video` extractor
- support Firefox containers for `--cookies-from-browser`

### Fixes
- [2chen] fix file URLs
- [bunkr] update domain
- [exhentai] fix pagination
- [imagetwist] fix extraction
- [imgth] rewrite
- [instagram] prevent post `date` overwriting file `date`
- [khinsider] fix metadata extraction
- [komikcast] update domain and fix extraction
- [reddit] increase `id-max` default value
- [seiga] raise error when redirected to login page
- [sexcom] fix video URLs
- [twitter] update `search` pagination
- [warosu] fix and update
- [zerochan] update for layout v3
- restore paths for archived files
- use `util.NONE` as `keyword-default` default value

### Removals
- [foolslide] remove `kireicake`
- [kissgoddess] remove module

## 1.24.1 - 2022-12-04
### Additions
- [artstation] add `pro-first` option
- [artstation] add `max-posts` option
- [fapachi] add `post` and `user` extractors
- [inkbunny] provide additional metadata
- [nitter] add `retweets` option
- [nitter] add `videos` option
- [nitter] support `/i/web/` and `/i/user/` URLs
- [pixhost] add `gallery` support
- [weibo] add `count` metadata field
- [downloader:http] add `retry-codes` option
- [formatter] implement `S` format specifier to sort lists
- implement `version-metadata` option

### Fixes
- [2chen] fix extraction
- [bcy] fix JSONDecodeError
- [bunkr] fix video downloads
- [bunkr] use `media-files` servers for more file types
- [itaku] remove `Extreme` rating
- [hitomi] apply format check for every image
- [hotleak] fix UnboundLocalError
- [nitter] sanitize filenames
- [nitter] retry downloads on 404
- [nitter] set `hlsPlayback` cookie
- [patreon] fix `403 Forbidden` errors
- [patreon] improve `campaign_id` extraction
- [patreon] update API query parameters
- [pixiv] preserve `tags` order
- [reddit] use `dash_url` for videos
- [twitter] fix error when using user IDs for suspended accounts
- [weibo] fix bug with empty `playback_list`
- [downloader:http] fix potential `ZeroDivisionError`

### Removals
- [lolisafe] remove `zz.ht`

## 1.24.0 - 2022-11-20
### Additions
- [exhentai] add metadata to search results
- [gelbooru_v02] implement `notes` extraction
- [instagram] add `guide` extractor
- [lolisafe] add support for xbunkr
- [mastodon] add `instance_remote` metadata field
- [nitter] add extractors for Nitter instances
- [pixiv] add support for new daily AI rankings category
- [twitter] add `avatar` and `background` extractors
- [uploadir] add support for `uploadir.com`
- [wallhaven] add `user` extractor
- [downloader:http] add `chunk-size` option
- [downloader:http] add file signature check for `.mp4` files
- [downloader:http] add file signature check and MIME type for `.avif` files
- [postprocessor] implement `post-after` event
- [postprocessor:metadata] implement `"mode": "jsonl"`
- [postprocessor:metadata] add `open`, `encoding`, and `private` options
- add `--chunk-size` command-line option
- add `--user-agent` command-line option
- implement `http-metadata` option
- implement `"user-agent": "browser"`

### Changes
- [deviantart] restore cookies warning for mature scraps
- [instagram] use REST API for unauthenticated users by default
- [downloader:http] increase default `chunk-size` to 32768 bytes
- build Windows executables using py2exe's new `freeze()` API
- build executables on GitHub Actions with Python 3.11
- reword error text for unsupported URLs

### Fixes
- [exhentai] fix pagination
- [khinsider] fix extraction
- [realbooru] fix download URLs
- [realbooru] fix `tags` extraction
- [tumblr] fall back to `gifv` when possible
- [twitter] fix login
- [twitter] update URL for syndication API
- [weibo] send `Referer` headers
- [ytdl] update `parse_bytes` location

### Improvements
- [imxto] extract additional metadata
- [instagram] allow downloading avatars for private profiles
- [pixiv] raise error for invalid search/ranking parameters
- [twitter] update `bookmarks` pagination
- [downloader:http] refactor file signature checks
- [downloader:http] improve `-r/--limit-rate` accuracy
- add loaded config files to debug output
- improve `-K` output for lists

### Removals
- [instagram] remove login support
- [instagram] remove `channel` extractor
- [ngomik] remove module

## 1.23.5 - 2022-10-30
### Fixes
- [instagram] fix AttributeError on user stories extraction

## 1.23.4 - 2022-10-29
### Additions
- [aibooru] add support for aibooru.online
- [instagram] add 'avatar' extractor
- [instagram] support 'instagram.com/s/' highlight URLs
- [instagram] extract 'coauthors' metadata
- [mangasee] add support for 'mangalife'
- [mastodon] add 'bookmark' extractor
- [mastodon] support cross-instance user references and '/web/' URLs
- [moebooru] implement 'notes' extraction
- [pixiv] extend 'metadata' option
- [reactor] match 'best', 'new', 'all' URLs
- [smugloli] add 'smugloli' extractors
- [tumblr] add 'fallback-delay' and 'fallback-retries' options
- [vichan] add generic extractors for vichan imageboards
### Fixes
- [bcy] fix extraction
- [gelbooru] support alternate parameter order in post URLs
- [hentai2read] support minor versions in chapter URLs
- [hentaihere] support minor versions in chapter URLs
- [kemonoparty] fix 'dms' extraction
- [kemonoparty] update pagination offset
- [manganelo] update domain to 'chapmanganato.com'
- [pixiv] use 'exact_match_for_tags' as default search mode
- [redgifs] fix 'token' extraction
- [skeb] fix extraction
- improve compatibility of DownloadArchive

## 1.23.3 - 2022-10-15
### Additions
- [2chen] Add `2chen.moe` extractor
- [8chan] add `thread` and `board` extractors
- [deviantart] add `group` option
- [fanbox] add `content` metadata field
- [instagram] restore `cursor` functionality
- [instagram] restore warnings for private profiles
- [nana] add `nana` extractors
- [nijie] add `feed` and `followed` extractors
- [tumblr] support `https://www.tumblr.com/BLOGNAME` URLs
- [tumblr] add `offset` option
- [vk] add `tagged` extractor
- add `path-extended` option
- emit debug logging messages before calling time.sleep()

### Changes
- [postprocessor:metadata] assume `"mode": "custom"` when \ 
`format` is given

### Fixes
- [artstation] skip missing projects
- [danbooru] fix ugoira metadata extraction
- [deviantart] fix `deviation` extraction
- [hitomi] fall back to `webp` when selected format is not available
- [imagefap] fix and improve folder extraction and gallery pagination
- [instagram] fix login
- [nozomi] fix extraction
- [redgifs] fix extraction
- [tumblr] sleep between fallback retries
- [vk] unescape error messages
- fix duplicated metadata bug with `-j`
- fix bug when processing input file comments

## 1.23.2 - 2022-10-01
### Additions
- [artstation] support search filters
- [blogger] add `label` and `query` metadata fields
- [exhentai] add a slash to the end of gallery URLs
- [instagram] add `count` metadata field
- [instagram] add `api` option
- [kemonoparty] add `count` metadata field
- [mastodon] warn about moved accounts
- [newgrounds] add `games` extractor
- [newgrounds] extract `type` metadata
- [pixiv] add `series` extractor
- [sankaku] implement `refresh` option
- [skeb] add `search` extractor and `filters` option
### Fixes
- [deviantart] fix extraction
- [fappic] fix extraction
- [instagram] extract higher-resolution photos
- [instagram] fix `username` and `fullname` metadata for saved posts
- [instagram] update API headers
- [kemonoparty] send `Referer` headers
- [kemonoparty] restore `favorites` API endpoints
- [myportfolio] use fallback when no images are found
- [plurk] fix extraction
- [sankaku] detect expired links
- [tumblr] retry extraction of failed higher-resolution images

Files:
RevisionActionfile
1.89modifypkgsrc/net/gallery-dl/Makefile
1.63modifypkgsrc/net/gallery-dl/PLIST
1.85modifypkgsrc/net/gallery-dl/distinfo