./net/ndpi, Library for deep-packet inspection

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 3.2, Package name: ndpi-3.2, Maintainer: adam

nDPI is an open source LGPLv3 library for deep-packet inspection. Based on
OpenDPI it includes ntop extensions. We have tried to push them into the OpenDPI
source tree but nobody answered emails so we have decided to create our own
source tree.


Required to run:
[textproc/json-c]

Required to build:
[pkgtools/cwrappers]

Master sites:

SHA1: 47ea3ac4a17afba1c7dd54bc8ec7a116031f16ef
RMD160: aaabdaf8b626925b85b880617b55143b9f6a3e8e
Filesize: 28892.626 KB

Version history: (Expand)


CVS history: (Expand)


   2020-05-25 22:25:22 by Adam Ciarcinski | Files touched by this commit (6) | Package updated
Log message:
ndpi: updated to 3.2

nDPI 3.2:

New Features
* New API calls
  * Protocol detection: ndpi_is_protocol_detected
  * Categories: ndpi_load_categories_file / ndpi_load_category
  * JSON/TLV serialization: ndpi_serialize_string_boolean / \ 
ndpi_serialize_uint32_boolean
  * Patricia tree: ndpi_load_ipv4_ptree
  * Module initialization: ndpi_init_detection_module / ndpi_finalize_initalization
  * Base64 encoding: ndpi_base64_encode
  * JSON exprot: ndpi_flow2json
  * Print protocol: ndpi_get_l4_proto_name / ndpi_get_l4_proto_info
* Libfuzz integration
* Implemented Community ID hash (API call ndpi_flowv6_flow_hash and \ 
ndpi_flowv4_flow_hash)
* Detection of RCE in HTTP GET requests via PCRE
* Integration of the libinjection library to detect SQL injections and XSS type \ 
attacks in HTTP requests

New Supported Protocols and Services
* TLS
  * Added ALPN support
  * Added export of supported version in TLS header
* Added Telnet dissector with metadata extraction
* Added Zabbix dissector
* Added POP3/IMAP metadata extraction
* Added FTP user/password extraction
* Added NetBIOS metadata extraction
* Added Kerberos metadata extraction
* Implemented SQL Injection and XSS attack detection
* Host-based detection improvements and changes
  * Added Microsoft range
  * Added twitch.tv website
  * Added brasilbandalarga.com.br and .eaqbr.com.br as EAQ
  * Added 20.180.0.0/14, 20.184.0.0/13 range as Skype
  * Added 52.84.0.0/14 range as Amazon
  * Added ^pastebin.com
  * Changed 13.64.0.0/11 range from Skype to Microsoft
  * Refreshed Whatsapp server list, added *whatsapp-*.fbcdn.net IPs
* Added public DNSoverHTTPS servers

Improvements
* Reworked and improved the TLS dissector
* Reworked Kerberos dissector
* Improved DNS response decoding
* Support for DNS continuous flow dissection
* Improved Python bindings
* Improved Ethereum support
* Improved categories detection with streaming and HTTP
* Support for IP-based detection to compute the application protocol
* Renamed protocol 104 to IEC60870 (more meaningful)
* Added failed authentication support with FTP
* Renamed DNSoverHTTPS to handle bot DoH and DoT
* Implemented stacked DPI decoding
* Improvements for CapWAP and Bloomberg
* Improved SMB dissection
* Improved SSH dissection
* Added capwap support
* Modified API signatures for ndpi_ssl_version2str / ndpi_detection_giveup
* Removed ndpi_pref_http_dont_dissect_response / \ 
ndpi_pref_dns_dont_dissect_response (replaced by ndpi_extra_dissection_possible)

Fixes
* Fixed memory invalid access in SMTP and leaks in TLS
* Fixed a few memory leaks
* Fixrd invalid memory access in a few protocol dissectors (HTTP, memcached, \ 
Citrix, STUN, DNS, Amazon Video, TLS, Viber)
* Fixed IPv6 address format across the various platforms/distributions
* Fixed infinite loop in ndpi_workflow_process_packet
* Fixed SHA1 certificate detection
* Fixed custom protocol detection
* Fixed SMTP dissection (including email)
* Fixed Telnet dissection and invalid password report
* Fixed invalid category matching in HTTP
* Fixed Skype and STUN false positives
* Fixed SQL Injection detection
* Fixed invalid SMBv1 detection
* Fixed SSH dissection
* Fixed ndpi_ssl_version2str
* Fixed ndpi_extra_dissection_possible
* Fixed out of bounds read in ndpi_match_custom_category

Misc
* ndpiReader
  * CSV output enhancements
  * Added tunnelling decapsulation
  * Improved HTTP reporting

nDPI 3.0:

New Features
* nDPI now reports the protocol ASAP even when specific fields have not yet been \ 
dissected because such packets have not yet been observed. This is important for \ 
inline applications that can immediately act on traffic. Applications that need \ 
full dissection need to call the new API function \ 
ndpi_extra_dissection_possible() to check if metadata dissection has been \ 
completely performed or if there is more to read before declaring it completed.
* TLS (formerly identified as SSL in nDPI v2.x) is now dissected more deeply, \ 
certificate validity is extracted as well certificate SHA-1.
* nDPIreader can now export data in CSV format with option `-C`
* Implemented Sequence of Packet Length and Time (SPLT) and Byte Distribution \ 
(BD) as specified by Cisco Joy (https://github.com/cisco/joy). This allows \ 
malware activities on encrypted TLS streams. Read more at \ 
https://blogs.cisco.com/security/detect … decryption
  * Available as library and in `ndpiReader` with option `-J`
* Promoted usage of protocol categories rather than protocol identifiers in \ 
order to classify protocols. This allows application protocols to be clustered \ 
in families and thus better managed by users/developers rather than using \ 
hundred of protocols unknown to most of the people.
* Added Inter-Arrival Time (IAT) calculation used to detect protocol \ 
misbehaviour (e.g. slow-DoS detection)
* Added data analysis features for computign metrics such as entropy, average, \ 
stddev, variance on a single and consistent place that will prevent when \ 
possible. This should ease traffic analysis on monitoring/security applications. \ 
New API calls have been implemented such as ndpi_data_XXX() to handle these \ 
calculations.
* Initial release of Python bindings available under nDPI/python.
* Implemented search of human readable strings for promoting data exfiltration \ 
detection
  * Available as library and in `ndpiReader` with option `-e`
* Fingerprints
  * JA3 (https://github.com/salesforce/ja3)
  * HASSH (https://github.com/salesforce/hassh)
  * DHCP
* Implemented a library to serialize/deserialize data in both Type-Length-Value \ 
(TLV) and JSON format
  * Used by nProbe/ntopng to exchange data via ZMQ

New Supported Protocols and Services

* DTLS (i.e. TLS over UDP)
* Hulu
* TikTok/Musical.ly
* WhatsApp Video
* DNSoverHTTPS
* Datasaver
* Line protocol
* Google Duo and Hangout merged
* WireGuard VPN
* IMO
* Zoom.us

Improvements

* TLS
  * Organizations
  * Ciphers
  * Certificate analysis
* Added PUBLISH/SUBSCRIBE methods to SIP
* Implemented STUN cache to enhance matching of STUN-based protocols
* Dissection improvements
  * Viber
  * WhatsApp
  * AmazonVideo
  * SnapChat
  * FTP
  * QUIC
  * OpenVPN support for UDP-based VPNs
  * Facebook Messenger mobile
  * Various improvements for STUN, Hangout and Duo
* Added new categories: CUSTOM_CATEGORY_ANTIMALWARE, \ 
NDPI_PROTOCOL_CATEGORY_MUSIC, NDPI_PROTOCOL_CATEGORY_VIDEO, \ 
NDPI_PROTOCOL_CATEGORY_SHOPPING, NDPI_PROTOCOL_CATEGORY_PRODUCTIVITY and \ 
NDPI_PROTOCOL_CATEGORY_FILE_SHARING
* Added NDPI_PROTOCOL_DANGEROUS classification

Fixes

* Fixed the dissection of certain invalid DNS responses
* Fixed Spotify dissection
* Fixed false positives with FTP and FTP_DATA
* Fix to discard STUN over TCP flows
* Fixed MySQL dissector
* Fix category detection due to missing initialization
* Fix DNS rsp_addr missing in some tiny responses
* Various hardening fixes
   2020-05-19 14:09:09 by Nia Alarie | Files touched by this commit (23)
Log message:
Recursive revbump for json-c-0.14
   2019-07-12 11:30:34 by Adam Ciarcinski | Files touched by this commit (10) | Package updated
Log message:
ndpi: updated to 2.8

2.8 Stable

New Supported Protocols and Services
* Added Modbus over TCP dissector

Improvements
* Wireshark Lua plugin compatibility with Wireshark 3
* Improved MDNS dissection
* Improved HTTP response code handling
* Full dissection of HTTP responses

Fixes
* Fixed false positive mining detection
* Fixed invalid TCP DNS dissection
* Releasing buffers upon realloc failures
* ndpiReader: Prevents references after free
* Endianness fixes
* Fixed IPv6 HTTP traffic dissection
* Fixed H.323 detection

Other
* Disabled ookla statistics which need to be improved
* Support for custom protocol files of arbitrary length
* Update radius.c to RFC2865
   2018-10-18 18:24:29 by Adam Ciarcinski | Files touched by this commit (9) | Package updated
Log message:
ndpi: updated to 2.4

2.4 Stable:
New Supported Protocols and Services
------------------------------------
Showmax.com
Musical.ly
RapidVideo
VidTO streaming service
Apache JServ Protocol
Facebook Messenger
FacebookZero protocol

Improvements
------------
Improved YouTube support
Improved Netflix support
Updated Google Hangout detection
Updated Twitter address range
Updated Viber ports, subnet and domain
Updated AmazonVideo detection
Updated list of FaceBook sites
Initial Skype in/out support
Improved Tor detection
Improved hyperscan support and category definition
Custom categories loading, extended ndpiReader (-c <file>) for loading \ 
name-based categories

Fixes
-----
Fixes for Instagram flows classified as Facebook
Fixed Spotify detection
Fixed minimum packet payload length for SSDP
Fixed length check in MSN, x-steam-sid, Tor certificate name
Increase client's maximum payload length for SSH
Fixed end-of-line bounds handling
Fixed substring matching
Fix for handling IP address based custom categories
Repaired wrong timestamp calculation
Fixed memory leak
Optimized memory usage

Other/Changes
-------------
New API calls:
ndpi_set_detection_preferences()
ndpi_load_hostname_category()
ndpi_enable_loaded_categories()
ndpi_fill_protocol_category()
ndpi_process_extra_packet()
Skype CallIn/CallOut are now set as Skype.SkypeCallOut Skype.SkypeCallIn
Added support for SMTPS on port 587
Changed RTP from VoIP to Media category
Added site unavailable category
Added custom categories CUSTOM_CATEGORY_MINING, CUSTOM_CATEGORY_MALWARE, \ 
CUSTOM_CATEGORY_ADVERTISEMENT, CUSTOM_CATEGORY_BANNED_SITE
Implemented hash-based categories
Converted some not popular protocols to NDPI_PROTOCOL_GENERIC with category detection
   2017-11-06 14:29:38 by Thomas Klausner | Files touched by this commit (1)
Log message:
ndpi: follow redirect
   2017-06-15 12:06:39 by Filip Hajny | Files touched by this commit (3)
Log message:
Fix ndpi build on SunOS.
   2016-07-01 06:51:57 by Adam Ciarcinski | Files touched by this commit (1)
Log message:
Added patches/patch-configure.ac
   2016-07-01 06:51:15 by Adam Ciarcinski | Files touched by this commit (10)
Log message:
Changes 2.4:
* Memory-management, stability and speed have been fundamentally improved
* We have kept an eye on security and hardened the code to prevent privileges \ 
escalation and XSS
* Alerts have been extended to include support for
  . Re-arming to avoid raising trains of identical alerts in short periods of time
  . Alert propagation to the infrastructure monitoring software Nagios
  . CIDR-based triggers to monitor the behavior of whole networks
  . The detection of suspicious probing attempts
* Netfilter support has been added together with optional packet dropping features
* Routing visibility is now possible through RIPE RIS
* Availability of fine-grained historical data drill-down features, including \ 
top talkers, top applications, and interactions between hosts (more details \ 
here)
* Integrations with other software
  . LDAP authentication support
  . alerts forwarding/withdrawal to Nagios
  . nBox integration to request full packet pcaps of monitored flows
  . Data export to Apache Kafka
* We have extended and improved traffic monitoring
  . Visibility of TCP sessions throughput estimations and state breakdown (e.g., \ 
connections established, connections reset, etc.)
  . Goodput monitoring
  . Trends detection
  . Highlight of low-goodput flows and hosts
  . Visibility of hosts top-visited sites
* Built-in support is now included for
  . GRE detunnelling
  . per-VLAN historical statistics
  . ICMP and ICMPv6 dissection
* We have extended the set of supported OSes to include: Ubuntu 16, Debian 7, EdgeOS
* There is also an optional support for hosts categorization via service \ 
flashstart.it