./news/sfeed, RSS and Atom parser

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 2.0, Package name: sfeed-2.0, Maintainer: leot

RSS and Atom parser (and some format programs).


Required to build:
[pkgtools/cwrappers]

Master sites:

Filesize: 66 KB

Version history: (Expand)


CVS history: (Expand)


   2024-01-01 16:13:05 by Leonardo Taccari | Files touched by this commit (2) | Package updated
Log message:
sfeed: Update to 2.0

Changes:
2.0
---
Features:
* sfeed_update: use xargs -P for improved job handling when updating feeds in
  parallel.
* sfeed_update: update default maxjobs from 8 to 16.

Fixes:
* sfeed_json: fix name of "url" field for attachments.
* sfeed_opml_export and sfeed_update: allow only regular files.
* sfeed_update: improve compatibility with zsh as a non-interactive shell,
  because some variables likes like $path and $status are reserved.
* sfeed_update:
  when the feeds() function is not defined then exit with status code 1 (this
  was incorrectly status code 0).
* sfeed_update: improvements in the cleanup of temporary files for edge-cases.

Code improvements:
* sfeed_curses/sfeed_gopher: mark functions as static.
* sfeed_gopher: fix a clang-analyzer warning (no difference in behaviour).
* sfeed_update: in the order() function suppress output to stderr, like merge()
  already does.
* Some small documentation improvements.

There was some work in adding tests for sfeed_update. This makes testing for
regressions on many platforms faster to spot.
   2023-08-21 15:07:15 by Leonardo Taccari | Files touched by this commit (2) | Package updated
Log message:
sfeed: Update to 1.9

Changes:
1.9
---
Features:
* sfeed_{curses,frames,gopher,html,plain}: add $SFEED_NEW_MAX_SECS

  By introducing the new environment variable $SFEED_NEW_MAX_SECS in some
  sfeed_* utilities marking feeds as new based on comparing their age, it is now
  possible to override this age limit. The default limit was the last day (86400
  seconds).

  This allows, for example, to be notified about new feeds within the last
  hour with by prefixing new items with " N ":

        SFEED_NEW_MAX_SECS=3600 sfeed_plain ~/.sfeed/feeds/*

  While creating a web report for last week's news by:

        SFEED_NEW_MAX_SECS=604800 sfeed_html ~/.sfeed/feeds/*

  This marks the items of the last week as bold in HTML.

  Based on the initial patch by Alvar Penning, thanks!

* sfeed_update/sfeedrc: add url a as parameter to the filter() and order()
  function This makes it easier to set filters or ordering by pattern matching on
  a group of feeds by the feed URL. For example for Youtube or Reddit feeds.

* sfeed_curses: move one line down when marking an item as read or unread.
  I don't mind either behaviour, but it has been suggested by a few people. For
  example the mutt mail client also has this behaviour.

Fixes:
* Improve to use proper includes.
  Reduce using some of the unneeded sys/* headers too. Using the C99 includes.
* sfeed_atom: for gmtime_r() make the error message consistent with sfeed_mbox.
* Makefile: change Gentoo commented example from -lcurses to -lncurses.
* sfeed_markread: fail early if creating a temporary file failed.

Code-cleaning / pedantic fixes:
* sfeed: datetounix: code-style, change , to separate lines (-Wcomma).
* sfeed_curses: make struct urls static like the other variables.
* sfeed_gopher: reduce scope and shadowing of a variable (no effective change \ 
though).
* xml.h: _XML_H_: macro name with an underscore is a reserved identifier.

Documentation:
* Improve note about CDNs and HTTP User-Agent blocking and change the example
  in sfeedrc.5 by setting a User-Agent.
* sfeedrc.example: add comment to reference to the man pages and README file.
* README: RSS 0.90+ is supported (not 0.91+).
* Typo fixes, consistency and structure fixes and some rewording.
   2023-04-28 15:42:01 by Leonardo Taccari | Files touched by this commit (3) | Package updated
Log message:
sfeed: Update to 1.8

Changes:
1.8
---
Fixes:
* sfeed_update: fail early if creating a temporary directory or status file fails.
* sfeed_atom, sfeed_json, sfeed_mbox:
  Fix reading past the buffer with an escaped NUL byte (\ NUL).
  Note that this could not happen with output from sfeed itself. Only if it was
  manipulated.
* sfeed_curses: fix (very hard to trigger) memleak when getline() returns EOF
  for lazyloaded items.
* sfeed parser:
  * Improve parsing RFC2822 obsolete short year.
  * Use errno ENOMEM instead of EOVERFLOW.
    This matches the behaviour of setting errno for malloc/calloc on the following
    systems too: glibc, musl libc, OpenBSD libc.
  * date to unix timestamp: fix incorrect int type to long.
    Found while testing sfeed on 16-bit MS-DOS with Open Watcom (for \ 
"fun" :)).
* Makefile: remove duplicate CPPFLAGS for sfeed_curses

Features:
* sfeed_json: add JSON output format tool. This formats the TSV data to JSON.
  It uses a subset of JSON Feed 1.1: https://www.jsonfeed.org/version/1.1/

Optimizations:
* sfeed_atom: save a few bytes in the output by removing the type="text"
  attribute, because for Atom the default for the type is text.
* Slightly reduce stack size for translating XML entities.
  A numeric entity could use 5 bytes, so use a round number of 8 bytes.

Misc:
* sfeed: simplify time calculation and make it slightly easier to read.
  This also fixes a calculation (possibly a compiler bug) with Open Watcom 1.9.
* Remove the sfeed name in some outputs ("branding").

Documentation:
* Fix some typos and improve code comments, in particular about the time parsing.
* README: sfeed_download example: change youtube-dl to yt-dlp:
  this is an active maintained fork.
* README: add error checking for temporary files in the examples.

Aside from the above changes there have been lots of testing on different and
strange systems and the test-cases have been expanded to cover some cases.
These tests are in a separate repo.

Some tested strange systems:

* SerenityOS: gophers://codemadness.org/9/paste/sfeed-serenityos.webm partially
  works.  The core base utilities like sort are limited and non-POSIX though, so
  sfeed_update doesn't work directly there.

* MS-DOS (16-bit and 32-bit extender) using Open Watcom 1.9. Works fine using a
  32-bit extender.  No 2038-support with 64-bit time_t though, just _unsigned_
  32-bit :)

Tested these less-used systems or compilers can be fun and also discover
(pedantic) bugs.

While adding JSON Feed output support (sfeed_json) I also wrote a parser for it
to read the data.  It can convert from JSON Feed to Atom. It can also convert
directly from JSON Feed to sfeed(5). It is available at:

        https://git.codemadness.org/jfconvert/files.html

1.7
---
Changes:
sfeed_curses:
* Add SCO keys for next, prior (CSI I and CSI G).
  Tested on DragonFlyBSD (cons25 console).
* Add SUN keys support.
  Tested on OpenIndiana.

sfeed_gopher:
* Remove unnecesary PATH_MAX restricting the path length.
  This also makes it compile cleanly on GNU/Hurd.

* Man page and documentation improvements.
   2022-08-17 17:21:12 by Leonardo Taccari | Files touched by this commit (2)
Log message:
sfeed: Update to 1.6

Changes:
1.6
---
* Code cleanups: unused code and refactored the code for reading read items
  from a plain-text URL file list.
* Fix typos in the code and documentation.
* Improve clarity of some code comments.
* Add setsid example to the README.
   2022-05-28 20:42:30 by Leonardo Taccari | Files touched by this commit (2) | Package updated
Log message:
sfeed: Update to 1.5

Changes:
1.5
---
* sfeed_curses: interrupt waitpid while interactive child program is running
  This now handles SIGTERM on sfeed_curses while an interactive child program is
  running.
* sfeed_curses: close stdin before spawning a plumb program in non-interactive
  mode, which is more intuitive: the program doesn't seem to hang when it expects
  input in this case since there is no way to send input anyway.
* Properly escape backslashes in the man pages (thanks adc!).
* Documentation improvements to the man pages and a progress indicator example
  script for sfeed_update.

1.4
---
* Improve read and write error handling in streams in many programs.
* sfeed_update:
  * Improve error handling and return a non-zero status if any feed fails.
    NOTE: this changes the way of using:
        sfeed_update && pkill -SIGHUP sfeed_curses
  * Separate errors to stderr and "OK" messages to stdout.
* sfeed_curses:
  * Avoid processes becoming a zombie (plumb, pipe, yank).
  * Line editor: temporarily disable the mouse when searching with /.
  * Improved signal handling.
  * Improved waiting on processes and status handling.
* sfeed_html/sfeed_frames: add dark mode support to the example stylesheet.
* sfeed_opml_export: use a control-character separator for converting the list.
* General code cleanup improvements.
* Compatibility: reduce the assumption the builtin libc locale is ASCII-compatible
  noticed on OpenBSD 3.8 where iscntrl detected C1 as control-characters,
  breaking UTF-8.
* General documentation improvements.
* Makefile: rebuild sfeed_curses if the same theme changes.
* README: improve error handling in downloader example when a feed fails.
   2022-03-07 23:47:34 by Leonardo Taccari | Files touched by this commit (2)
Log message:
sfeed: Update to 1.3

Changes:
Fixes
-----
* Fix a compiler warning with some curses implementations, like NetBSD curses.
* sfeed_curses: add keybinds for the home key and the default home and end key
  for urxvt.
* sfeed_curses: fix a redraw when reloading a file with a feed file read from
  stdin and using an URL file and changing this URL file externally.
* sfeed_curses: cast character for SFEED_AUTOCMD to unsigned char to allow
  character sequences outside the ASCII range.

Documentation
-------------
* README: add an example script to count new and unread items. This can be
  useful for some statusbar indicator (asked about by e-mail).
* Small code-style, comments and documentation improvements and fixes.
   2022-02-10 12:35:16 by Leonardo Taccari | Files touched by this commit (2)
Log message:
sfeed: Update to 1.2

Changes:
1.2
---
Fixes:
* sfeed parser: extend the time range to atleast 64-bit (long long).
* sfeed parser: allow leap second like 23:59:60 (RFC2822 Section 3.3).
* sfeed_curses: pedantic fix for UB with an empty URL file (qsort and bsearch).
* sfeed_curses: fix a compile warning with tparm() on some systems.

Features:
* sfeed_atom: add category field(s).
* sfeed_curses: add feature to go to the next bold row and previous bold row
with J and K.  This is useful to quickly jump to the next new item or new feed.
Thanks to IanJ for the suggestion and feedback!
* sfeed_content: allow to set HTML converter with env variable $SFEED_HTMLCONV.
For example: SFEED_HTMLCONV="w3m -I UTF-8 -O UTF-8 -T text/html -dump"
* Add compile-time option to improve output on dumb non-UTF8 terminals
(-DSFEED_DUMBTERM).

Misc:
* Documentation improvements to the man page and a sfeed_download example in
the README: this is a parallel downloader/extractor example script.
* Code-style improvements.
   2021-12-18 15:14:24 by Leonardo Taccari | Files touched by this commit (3)
Log message:
sfeed: Update to 1.1

Changes:
1.1
---
 * The formatting program sfeed_curses is now merged in the sfeed repo.
   This is a program for viewing items in a curses UI.
   sfeed can still be built without it if its considered bloat with:
         make SFEED_CURSES="".
 * sfeed_mbox: escape the link and enclosure text when using HTML content
   with the option SFEED_MBOX_CONTENT="1"
 * sfeed_mbox: add link as base URL for HTML content.
 * sfeed_mbox: use 64-bit for the checksum number on 32-bit systems.
 * sfeed_content: optimizations to the awk script for the UTF-8 locale and
   string replacements.
 * sfeed_content: add link as base URL for HTML content.
 * Documentation improvements and more examples in the man page.
 * Small code-style improvements.