$Id: NEWS,v 1.279 2008/03/13 12:04:06 vanbaal Exp $ <-- be friendly for people pasting this text into email: 72 cols max --> LogReport Lire NEWS - user visible changes (and some other changes too). Refer to ChangeLog for detailed per-file info. Copyright (C) 2000, 2001, 2002, 2003, 2004 Stichting LogReport Foundation Copyright (C) 2006, 2007, 2008 Joost van Baal version 2.0.3 - March 13 2008 - New stable release. No user visible changes since 2.0.2.99.1 prerelease. version 2.0.2.99.2 - March 3 2008 - This is a prerelease for upcoming 2.0.3. - No user visible changes. version 2.0.2.99.1 - March 3 2008 - This version is not publicly released. - Bug fixes: * Various fixes in merging and generating PDF reports. Thanks to Stefan Schuermans for the patches. * Fix in lire's behaviour when Report Jobs are incorrectly named. Thanks to Konstantinos Koukopoulos for the patches. - New DLF converter contribued by Lire's users: * FortigateDlfConverter: converts FortiGate 3.x packet log to firewall DLF format. Thanks to Jean Benoit Marzio for this contribution. version 2.0.2 - July 26 2006 - New stable release. No big changes since 2.0.1.99.3. version 2.0.1.99.3 - July 22 2006 - Another prerelease for upcoming 2.0.2. Various documentation improvements. - Bug fix * Lire now processes new-style CUPS page_log files fine. Thanks Marco Gaiarin for the bug report (http://bugs.debian.org/313545). version 2.0.1.99.2 - July 16 2006 - Another prerelease for upcoming 2.0.2. Various documentation improvements. - Bug fix * Lire no longer crashes on Apache combined logs with referrer URLs featuring non-utf8 encoded %-escaped characters. Thanks Laurent Fousse for the bug report (http://bugs.debian.org/291063). version 2.0.1.99.1 - May 29 2006 - This is a prerelease for upcoming 2.0.2. This release includes more than a year of work by Francis J. Lacoste, Wolfgang Sourdeau, Wytze van der Raay and Joost van Baal. Most of the work is bugfixes and documentation improvements. See ChangeLog for the details. version 2.0.1 - October 9 2004 - Lire now requires DBD::SQLite2. (The DBD::SQLite module switched to the SQLite 3.0 starting from version 1.00. DBD::SQLite2 must be used for the old database format.) - Bug fixes * lr_cron will generate a report even if not all the DLF streams required by the report are available. (This prevented the default www report from working with CLF log file.) * lr_cron now set the period filter on the each report specs instead of the section. (The former behavior broke the default www report.) version 2.0 - September 3 2004 - Bug fixes * Fixed double encoding problems when local codeset is UTF-8. * Updated User and Developer Manuals. Added categoriser example to doc/examples. * Do not try to include empty charts in HTML and PDF reports. * PDF reports which contained doubly nested groups were failing. version 2.0rc2 - September 2 2004 - Bug fixes * Some www DlfAnalysers were left out of the package. * Non-ASCII characters are now displayed correctly in localized strings and in XML reports. * No more rotated pages in the PDF output. * postfix DLF converter wasn't working under lr_cron command. * ms_isa DLF converter wasn't working. version 2.0rc1 - August 30 2004 - Changes to output formats * DocBook and all the extra tools are no longer required for any output format. * New HTML format which is a lot nicer than the previous one. * PDF output format is now generated through LaTeX. The only requirements are a recent TeX installation and GhostScript. Latex, dvi and ps were added as available output formats. * The dvi, latex, html, pdf and ps output formats contain the appropriate hyperlinked schemas reference. * The 'docbookx' and 'rtf' output formats were removed. * The 'html_onepage', 'xhtml' and 'xhtml_output' were removed. (The 'html' format can generate all of these through options.) - Changes to the lire command * It is now possible to edit report configuration files through the lire interface. Report configurations are now stored in XML. The older report configuration files are obsolete and won't be supported in the next Lire releases. There is a lr_report_cfg2xml script to convert these files to XML templates. Old report configuration files used by configured jobs will be automatically converted when you use the lire tool. * It is now possible to have more than one superservice in a report. Each report section can use a different superservice. * NB: It is suggested to use a terminal with a height greater than 24 lines for maximum usability. * If you are using a terminal with resizing capacity (like xterm), the lire command will take advantage of the new size. (It is possible that you encounter a "screen too small" message while resizing, but trying different size can make the message disappear.) - Changes to lr_log2report, lr_log2mail, lr_xml2mail and lr_xml2report * These commands were rewritten in Perl and now use the API directly. As a consequence most of the internal commands they used were removed: lr_check_prereq, lr_dlf2xml, lr_inflate, lr_log2xml, lr_store, lr_xml2ascii, lr_xml2chart, lr_xml2dbx, lr_xml2html, lr_xml2pdf, lr_xml2xls, lr_xml_merge and lr_xslt are no longer shipped as separate utilities. * The command line usage was improved. (Long options are now available and all commands can operate on other things then STDIN and STDOUT). Consult the man pages for details on the new features. * Old command line usage still works for now, with two exceptions: 1. The second argument to lr_xml2report is the destination file. To merge additional reports use the --merge option. 2. The first argument to lr_xml2mail is now the report to format. (It had to be specified using -x before.) * The commands now output legible messages on STDERR (they do not use the old lire logging format). As a consequence, they shouldn't be run under lr_run. - Changes to charts * Three new types of charts are supported: scatterplot, stacked plots and distribution plot. * Most chart types now allow the plotting of more than one variable. * Charts can be configured through the new report configuration interface. * The ploticus tool is now required to generate charts. Support for GD::Chart was dropped. * ChartTypes are now Lire::Plugin, making it possible to add new chart types easily. - Changes to XML report * Support for 1.0 DTD XML reports (as generated by Lire version < 1.2) was dropped. * The chartype attribute, image and file elements were removed from the LRML. Charts are now specified in a chart-configs element which contains the chart configuraiton in LRCML. The charttype attribute is still supported for older reports. - Configuration API changes * It is now possible to set default values directly in the specification file. As a consequence, Lire doesn't install any files in the share/lire/config-defaults directory. * There is a report-config element that can be used to define a report configuration. The default report configuration for each superservice will be looked up under _default_report in the configuration specification. * The new_from_config() method used to instantiate a custom object now takes the Lire::Config::Value as parameter and not its as_value() result. * All specification elements now have an 'obsolete' and 'required' attribute. * Compound objects now also support the is_valid() method which makes sure that all required contained parameters are valid. * String specifications now have an optional 'valid-re' attribute which should contain a regular expression that the is_valid() method will use for validation. - New Lire::DlfAnalyser API * This API replaces the old undocumented DerivedFieldCreator and ExtendedFieldCreator. This API is similar to the Lire::DlfConverter one. Consult the Developer's manual for more information. * Analysers are now run after the log importation. The analysers to run can be configured by DlfStore through the lire command. - New Lire::OutputFormat API * All output formats are now implemented as Lire::Plugin. - API and other framework changes * Lire::DlfConverterManager was removed and replaced by Lire::PluginManager. As a result, the way a DlfConverter is initialised and registered changed: it is the responsability of the registration script to call register_plugin() to register its plugin. Furthermore, the plugin registration script should be located in the directories contained in the 'plugins_init_path' configuration variable. Registration scripts located in 'lr_converters_init_path' will still be loaded using the old mechanism as a backward compatibility measure. - Bug fixes * Merging of first and last operator summary data was only working when the field used was the first in the schema. * Lots of other bugs were fixed by this major overhaul. Comparing the BUGS file with the previous release will give you insight. - Other minor changes * Lire::WWW::UserAgent was updated, your www reports will give more information on user agents. Contributed by Egon Willighagen. version 1.5 - April 12 2004 Release 1.5 is the last interim release before the Lire 2.0 release. (See http://www.logreport.org/lire/roadmap-2.0.php). It completes the Internationalisation and Store Configuration items on the 2.0 roadmap. - Store Configuration The lr_config command which was used to configure Lire and schedule report generation was replaced by a new Curses-based command : lire. Like the old lr_config command, it can be used to change Lire global settings as well as scheduling jobs that will be executed by crond(8). But unlike the old lr_config command, the way jobs are configured changed radically. Consult the README file for more information. Support for the old lr_config's jobs was dropped and that configuration cannot be migrated to the new paradigm. - I18N * Lire now supports non-ascii log files. With perl 5.8.0 and later, 8bit characters will no longer be replaced with '?'. * The encoding of log files can be specified when they are in a different encoding than the system's default. * Any XML::Parser supported encoding can be used in XML files. * There is a lr_spec2pot command that can be used to extract translatable strings from XML specifications. * Many Lire commands were internationalized. - Requirements changes: * Perl >= 5.6.1 is now required. To support non-ascii encoding perl >= 5.8.1 is required. * The Curses::UI perl module is required (this module also required the Curses and TermReadKey modules.) * The libintl-perl module. - API and other framework changes * Three new configuration types are available: record, object and plugin. A record is a configuration type that contains many fields. An object is like a record but instead of being instantiated as a hash reference, it specifies a factory method which will be called in as_value(). A plugin is a type similar to a select, but which has additional properties based on the selected value. * Most XML specs have a new 'i18n_domain' attribute which can be used to specify the Gettext domain in which translations should be looked for. * DlfConverter's should specify their configuration variables using a record or objet specification. Users will be able to set the converter properties from the user interface. * Each stores has two DLF streams called "lire_import_stats" and "lire_import_log". The former contains stats about the execution of ImportJobs and the latter is used to store continuation lines, error messages and other such information. - Incompatible changes * lr_config was removed. * Many global configuration variables were moved to by-job parameters. * Support for DocBook's "userlevel" attribute was removed. The whole description will always be shown from now on. * The RTF output format was removed. * Support for migration of old environment based configuration (pre-1.3) was removed. * Files and categories were removed from lr_store. * Many other bug fixes. version 1.4.1 - January 7 2004 - Cisco syslog format understands the optinal TZ. - Requires DBD::SQLite 0.29 or later. version 1.4 - 15 November 2003 Release 1.4 introduces a new report generation algorithm which improves performance a lot. DLF data is now stored in an SQLite database and the report is generated directly from the database. Tests show performance increases of ~18%-55% depending on the superservice used (superservices like www, which uses a lot of analyzers, will see less benefits than superservices like firewall which doesn't because the bulk of the time is spent in the analyzers). The new report generation algorithm has some impact on the requirements and introduces other incompatible changes: - Requirement changes: * The DBI perl module is required. * The DBD-SQLite module is required. At this time, a patch (extras/DBD-SQLite-0.28.patch) needs to be applied. Versions 0.29 and above of DBD::SQLite will not need this patch. A version with the patch already applied is available from http://download.logreport.org/ * The Digest::MD5 perl module is required (it is included with perl 5.8.0). * There is no lire-full version available anymore. Expat and XML::Parser need to be installed separately before installing Lire. - Changes to the reports: * Reports do not include 'empty groups' anymore. For example, in Lire 1.3 and before, if you generated a 'Requests by hour' report and you had requests during hours 14:00 and 16:00 but not during 15:00, 15:00 would appear with 0 in the report. In Lire 1.4, the 15:00 hour will not appear in the report at all. Merging such reports will reintroduce the empty groups though. * The ranges generated by rangegroup operator which uses a 'size-scale' attribute != 1 are now different. For example, Lire 1.3 would generate the following groups for a scale of 5: [0-1k> [1k-5k> [5k-25> whereas Lire 1.4 will generate [0-1k> [1k-6k> [6k-31k> which was really the advertised behaviour. * The column's name automatically generated for aggregator changed. For example, in Lire 1.3, the column of the timegroup operator took the name of the DLF schema's timestamp field. It's now called 'timegroup:field'. Similar changes affect the column generated by the rangegroup, timeslot and records aggregators. - Other incompatible changes: * Because of the changes related to reports' content, new reports can't be merged with old ones. For example, you can't merge XML reports generated by Lire 1.3 with reports generated by Lire 1.4. * The service_aliases.cf file is not used anymore. As a consequence, you will need to use the canonical names instead of the old ones. The aliases which were removed are 'bind8', 'bind9', 'cisco', 'cups', 'lprng' and 'squid'. Use 'bind8_query', 'bind9_query', 'cisco_ios', 'cups_pagelog', 'lprng_account' and 'squid_access' instead. * The experimental MySQL archiving support (lr_dlf2sql, lr_sql2dlf and lr_sql2report) was removed. * The lr_viconf configuration frontend was removed. * Some configuration variables are obsolete: lr_sql_archive, lr_sql_max_size, lr_old_service_aliases_file, lr_dbi_uri, lr_dbi_user, lr_paswd_file. - This release has undergone extensive testing on lots of platforms. We've been able to add FreeBSD, OpenBSD, NetBSD and Mac OS/X to our group of test platforms. - Many bug fixes. version 1.3 - 17 May 2003 - New improved lr_config(1) command which makes it easier to configure periodical reports. - Documentation was improved. The internal APIs which should be used by extension developers are documented in man pages. Tutorial on how to to write new DLF converters and DLF schemas were added to the Developer's manual. There is also sample code in doc/examples. - Several bug fixes: * Fixed merging problem with report using the timegroup operator. * Fixed merging problem with report containg missing subreport. * Fixed merging problem with report that were generated with 0 records. * Encode all outgoing mail using Base64 or quote-printable (no more problems related to long lines). * ISO week numbers are supported everywhere Lire is. - Incompatible change: * Configuration is now held in XML files. Some of the old-shell based configuration can be migrated using the lr_env2conf(1) command. The periodical report configuration isn't migrated though. You will need to enter it anew using the improved lr_config(1). * Some internal commands were removed (lr_archive_log, lr_check_errlines, lr_db_fetch, lr_db_store, lr_db_flush, lr_run_tests). If you used those, you will need to modify the relevant scripts. * The old archiving behavior is removed. Only the responder now saves the log files and reports. If you used LR_ARCHIVE while not running a responder, you will need to take care of archiving your reports and log files yourself for now. - Many enhancements to Lire's core: * New configuration API making it easy to make extensions (DLF converter, analyzers, etc.) configurable. * New API for DLF converter with support for multiple DLF schemas and log continuation. * Extensive suite of unit tests and regression tests were added to Lire, making it easier to modify. - New DLF converters contribued by Lire's users: * tinydns - DNS DLF converter for DJB TinyDNS. Contributed by Christopher Boumenot * s1ms - Email DLF converter for SunONE Messaging Server. Contributed by Jean-Yves Monnier. * snort - Firewall DLF converter for Snort IDS. Contributed by Torsten Fellhauer. * fw1_lea - Firewall DLF converter for CheckPoint Firewall-1 5.0. This DLF converter isn't activated by default. See the notes in the fw1_lea2dlf(1) man page for activation instructions. Contributed by Torsten Fellhauer. - Other minor stuff: * added lr_vendor_cron script (mainly interesting for people making Lire packages for specific Unix and GNU/Linux distributions). version 1.2.1 - 3 December 2002 * Fixed broken variable substitution in defaults which prevents loading of "local" files created by lr_config in @sysconfdir@/lire. version 1.2 - 29 November 2002 - Many enhancements to the report engine: * New ratio attribute to the sum and count operators to express their value as a percentage of the table's or group's total. All reports where upgraded to support this new XML Report format; they now include labels and ratio columns (whenever appropriate) * New weight attribute to the sum and avg operator to compute weighted average and sums. * New first and last operators that can be used to the first or last field's value in a group. (See the top-user-leftover-bytes msgstore report for an example of lire:last.) * Report specifications can set columns label using the label attribute. * Reports now include summary information (group's and table's total for example). - Major upgrade of the output layer, output formats * All typesetters can now format reports containing complex table containing multiple columns and group by entry. * All typesetters support column labels and group and table summary information. * Many improvements to the DTD and to the ReportParser API to make it easier to work with Lire XML Reports. * One can now typeset PDF reports using PassiveTeX instead of jade/DSSSL. * Support for output in LogML ( http://www.cs.rpi.edu/projects/pb/WWWPal/logml.html ) format is removed. (LogML is an XML based language to compress log files in smaller files, enabling data mining techniques.) The Lire code for LogML has been badly maintained for some time; interest seems low. However, if you'd like to use this functionality, we'll gladly accept code contributions. Contact the LogReport team at the development@lists.logreport.org list for hints on bringing LogML output back into Lire. * Support for generating graphs using ploticus added. See the Lire User Manual for details. - New services and superservices * New syslog format supported: WTsyslog. * Support for DBMAIL ( http://www.dbmail.org/ ) POP3 log files, thanks to a code contribution by Edwin Groothuis. * New superservice: spamfilter, for email filters like those used for blocking UCE. Support for the spamassassin spam filter, thanks to a code contribution by Edwin Groothuis. * The nmsstore service will now parse log entries coming from the webmail interface, thanks to a fix provided by Jean-Yves Monnier. - New reports * WWW report Client hosts by Size added * msgstore top-user-leftover reports added: diskspace used after session (thanks Edwin Groothuis) - Documentation Lots of documentation updates and improvements: We're shipping more than 1MB of documentation in man page format now. Apart from this, we still offer documentation in our Lire User Manual, FAQ and Developers Manual, of course. * Some contributions to manpages by Edwin Groothuis. * A lot of Lire modules have been properly documented in manpages now, which is especially useful for Lire hackers. * Since we have been asked for this over and over again, a lire(7) manpage is added. * We now ship a ChangeLog with Lire. - Small bugs fixed * Bug fixes in the Kiwi syslog parser. * Bug fix in the mysql parser (thanks Fatih Gey). * Bug fixes in the dbmail service and msgstore superservice (thanks Edwin Groothuis) * Some other small bugs fixed (see ChangeLog for details) * Fixed building the lire-full distribution with Solaris' make. - Lire developers * Hook for typesetting documentation using pdfxmltex and xsltproc instead of pdfjadetex and (open)jade added. * New superservice `test' added, to debug changes to the Lire core. * All files in $sysconfdir/lire/ ending in defaults and defaults.local will be loaded. This makes it easier to add configuration hooks for external modules. * New lr_env(1) command which can be used to find the location where various Lire extensions should be installed. - Other business * Small helper script for setting up an online responder added: lr_setup_responder. * If you have written your own report definition files, or if you're using Lire's report merging capabilities, you might get hit by some backwards incompatible changes. Refer to the `upgrading from Lire 1.2 and earlier' section in the INSTALL file for details. version 1.1 - 19 August 2002 - New superservices * dialup, including a isdnlog2dlf convertor for log files of isdn4linux.de's isdn4k-utils. The superservice has a few report types on total costs, connection time, and connections by period. * dnszone, for DNS log files about zone transfers. Lire now supports Bind 8's named log files. Thanks to a code contribution from Edwin Groothuis. * msgstore, including support for Netscape Message Store (nmsstore) and Netscape Messaging Multiplexor (nmsmms). Thanks to a code contribution from Arnaud Taddei and Arnaud Gaillard. * syslog. It is now possible to have some general overview of your syslog log file. - New services * Lire now supports logs from the WatchGuard Firebox firewall solution, thanks to a code contribution from Joost Bekkers. * Overhauled Syslog parser. Lire now supports log files coming from various syslog implementations. In addition to the BSD, Solaris 8 and Netscape Messaging Server logging daemon already supported, we now support 5 other formats: Kiwi Syslog Daemon (ISO, US and DD/MM date formats), Sendmail Switch Log daemon and an unknown daemon that includes the encoded priority and the year in its format. * The bind9 service supports the Bind 9.3 log format too. * The squid service now supports the Netcache network appliance 'squid logging format'. Thanks to Arnaud Gaillard for sending us a sample log file. * The cisco service now supports all the syslog log formats the syslog parser supports. * Both Cisco and PIX services now support a variety of timestamp settings. - Other improvements * Much improved Netscape Messaging support. Thanks to Arnaud Taddei and Arnaud Gaillard! * Bugfix to xferlog parser which now support the "completion_status" field. * Generate catalog.xml at runtime so that it correctly reflects the configured location of the DocBook DTD. * Include an XML catalog for our DTDs. You shouldn't see the "Attempt to load network entity..." message from xsltproc anymore. * The image format of the charts is now configurable through the LR_IMAGE_FORMAT variable. We don't rely on ImageMagick's convert(1) anymore. All formats supported by the GD library are available (which means jpeg, png and/or gif depending on your version). * Superservice aren't statically defined anymore. The list of available superservices is now determined at runtime. * New email reports: Largest Email Exchange Per Relay Pair, Status Summary, Top Status Per Domain and Domains By Period. Enable by adapting your email.cfg file. * Work on using an SQL database for storing internal data has started. If you're interested in Lire's development, you might like to take a look at the new experimental lr_sql2dlf(1), lr_dlf2sql(1) and lr_sql2report(1) scripts. Thanks to Arnaud Gaillard for this spiffy code contribution! * New script lr_check_service (internally used). - Documentation * Completely rewritten chapter on Lire's architecture in the Developer's Manual. * Documentation on all current DLF schemas is included in the Developer's Manual. * Updated development roadmap. - Other business * Superservices and services are now found at runtime. This makes Lire much more flexible: writing your own service plugin is now as easy as sticking it in libexec/lire/convertors. Furthermore, the user interfaces to lots of scripts are now simpler: no more need to supply a redundant superservice argument. Among these are lr_log2mail and lr_log2report. The subject of emails, sent by lr_log2mail (and by Online Responders) have changed: these no longer list the superservice name. A jobfile, as used by lr_cron(1) no longer needs to set the superservice environment variable. We have tried to remain backwards compatible. However, you're strongly advised to move to the new UI asap. If you have made lots of local changes to your Lire installation, you're very likely gonna get hit by some changes. Refer to the 'Upgrading from Lire 1.0' section in the INSTALL file for more details. version 1.0 - 23 June 2002 * It is now possible to merge reports over time and over different servers: generating weekly and monthly reports, as well as site-wide reports can now be done in a much more easy and efficient way. Refer to the "Merging Reports" section in Chapter 3 of the Lire User's Manual for details. * lr_xml2mail and lr_xml2report supports multiple XML reports as input, by calling lr_xml_merge: this offers a user interface to the XML report merging functionality. To support this, the interfaces of lr_archive_log, lr_xml2report and lr_xml2mail are slightly changed. Since you probably don't call these scripts directly, this will not cause you any trouble. * New firewall service: `pix', for Cisco PIX firewalls. New database service `pgsql', for PostgreSQL query log files. The list of supported services now is: superservice services ============ ======== database mysql, pgsql dns bind8, bind9 email argomail, exim, nms, postfix, qmail, sendmail firewall cisco, ipchains, ipfilter, iptables, pix, welf ftp iis_ftp, xferlog print cups, lprng proxy ms_isa, squid, welf_proxy www combined, common, modgzip, referer, w3c_extended * New output format: excel95. This outputs report in a Excel95 spreadsheet. This can be useful if you want to generate sexy 3d charts from your Lire reports. You need the Spreadsheet::WriteExcel CPAN module for this. * Renamed lr_anonimize and lr_deanonimize to lr_anonymize and lr_deanonymize: fixed spelling bug. * Fixed bugs in sendmail2dlf: 1) which caused from= to get filed under the 'bar.com___' domain. 2) which caused messages from <> to a pipe (e.g. procmail) to get filed as being sent _from_ the controlling user (while it should be _to_ this user, in this case) Tnx Edward Eldred for reporting these bugs. * For Top-reports (e.g. `Top 10 Referring Pages' in the www report), specifying `0' in the superservice.cfg file as the parameter for this report (e.g. `referer_to_show=0' in www.cfg) will cause all available data to get displayed. * Added keyword_to_show and engine_to_show variables to the www "Requests By Keywords" and "Requests by Search Engine with Keywords" reports: now these reports no longer show _all_ keywords. Set the variables to 0 in www.cfg to get back the old behaviour. version 20020415 * New email service: ArGoSoft Mail Server Pro log files are now supported through the argomail service. * Memory footprint of the report generation process is now limited. Lire will not memory DoS your server because it takes too much memory. It can now process very big log files without taking too much memory or disk space. * lr_log2mail and lr_rawmail2mail now supports -i and -o options. The report will be sent using proper MIME encoding for the output format. (X)HTML reports should be displayed inline by all mailers supporting HTML mail. Beware! When specifying multiple destination addresses on the lr_log2mail command line, separate them by _spaces_. Separating these by commas no longer does the trick. Furthermore, lr_log2mail's last optional `flags' argument is no longer supported. * To use the responder, the lr_log2mail and the lr_rawmail2mail command, you will need to install the MIME::Tools CPAN package: We now use the more complete and reliable MIME::Tools package instead of the mpack(1) and munpack(1) command line tools. * There is a new lr_xml2report command which acts as a frontend to all the supported output formats. * lr_log2report doesn't need any error file argument anymore: Call it using lr_log2report superservice service < logfile > report. * The report configuration file to use can be specified to lr_log2report and lr_log2mail by using the -c option. * New output formats: html_page and xhtml_page which generates a one page (X)HTML report. * There is now a records element that can be used to include fields from DLF records matching certain criteria in a report specification. * timegroup, rangegroup and timeslot element can now be nested like the group element could. * The uniq filter element was removed. You will want to use the fields attribute of the count element instead. * Improvements to the ASCII formatter. Better formatting of DocBook itemized lists. Long names are wrapped instead of being cropped. * lr_log2report now handles compressed log file. * Removed obsoleted *-by-day report specifications. * The postfix2dlf convertor has been redesigned: it now no longer calls sort(1), this makes it more flexible: it will be easier to support non-syslog timestamped logs. It now creates a time-sorted DLF. It now fully supports postfix logs timestamped by the Sun Solaris syslog. * Improvements and bug fixes to the LPR-NG DLF converter. * Many improvements to the www report. The report was reorganised in several sections. New reports were added: top-traversals, user_session-visit_number, top-referring_sites, keywords-by-search_engine, requests-by-search_engine. * Important bug fixes in the www session analyzer. For one thing, all unifinished session were ignored. Images and stylesheets are now excluded from the page_count and other fields in sessions. * Some report specifications were obsoleted by the use of global filters. If you used them in custom report configuration, you will have to update your configuration. The User's Manual explain which report to use instead. They will be removed in a future release. (www): top-pages-by-client_host top-tracked_pages-by-client_host tracked_pages-by-period user_session-finished (dns): requests-by-period-by-method requesttype-distribution-by-method top-requested-names-by-method top-requesting-hosts-by-method * Using `apache' as a service name is deprecated. Instead of `www apache', you will have to use `www common'. Furthermore, using `boa' as a service name is deprecated: `www boa' is now called `www combined'. Finally, `www apachemodgzip' is now called `www modgzip'. * A lot of environment variables are renamed: an `LR_' prefix was added to the names, to prevent nameclashes. If you have made some local configuration changes, you will have to adapt these: this change is NOT backwards compatible. See INSTALL for upgrade details. version 20020214 * Output when requesting HTML or XHTML was changed. The output is now always an uncompressed tar file (like when generating images previously). This makes HTML and XHTML output identical whether or not images are generated. Also, when report sections are used, the (X)HTML reports will be broken into several files. * The HTML manuals are now splitted in several files. * lr_config can now be run by any user (not only the Lire's administrative user). * Reports can now be subdivided into sections. This can be achieved by using =section directives in the report's configuration file. Consult the User's Manual for the complete story. * Sections can contain filters that will be applied to the input of all the section's subreports. Consult the User's Manual for the complete story. * New proxy superservice, with support for MS ISA, WELF and squid logs and a lot of reports. * New database superservice, with service MySQL and reports top-users, top-databases, actions-by-period, top-querytypes. * New FTP service: IIS FTP Logs (iis_ftp). * New firewall services: iptables, welf. WELF is the WebTrends Enhanced Log Format supported by several firewall products (see http://www.webtrends.com/partners/firewall.htm for a list). Common firewalls using this format: SunScreen, SonicWall, Raptor, etc. This makes Lire support a lot of firewalls. We didn't test it with all of those products though. So we greatly appreciate all feedback regarding how Lire behaves with those firewalls' logs. * The firewall superservice can also be used for network intrusion detection type of logs. * The firewall superservice now includes a lot of new reports. * The incompabitilities between the Cisco service and the other firewall services were resolved. * New ftp reports: bytes-by-user-by-period, bytes-by-dir-by-user-by-period, top-users-bytes. * New www reports: requests-by-keywords, top-referers. * New email service: Netscape Messaging Server (nms). * New dns reports: requests-summary, requests-summary-by-method, requesttype-by-method, requests-by-timeslot, req-by-period-by-method, req-by-timeslot-by-method * Support for Netscape Messaging Server log daemon. * Improvements in postfix email service: more robust qmgr line handling (no longer creates bogus size fields in dlf), support for postfix/virtual lines. (Thanks Cedric Gross for helping us!) * Online responder more robust against evil characters in subject of received email message. Directories in var/lib/lire/data/{email,log,report}, named after ID tags, which are contructed from these subjects, could contain \ in their names. This is ugly, and breaks non-robust scripts. It even might have had security implications. People running an online responder should upgrade. The change is in lr_processmail. * Changes to the LRSML DTD: You may have to modify custom report specifications you wrote. - The 'sort' attribute was removed from the filter-spec element. * The WWW::* perl modules, like WWW::Useragent and WWW::Domain, are moved to Lire::WWW::*, in order to avoid future nameclashes. version 20011205.1 * Fixed distclean target. * Fixed typo in lr_xml2logml. * Added missing doc/lire.ent in tarball. * Fixed sendmail queue identifier parsing for very old and very new versions. * Fixed a timestamp bug in exim converter. version 20011205 * Only RTF and PDF output now requires Jade. HTML and XHTML output can be generated with xsltproc and Norman Walsh's DocBook XSL Stylesheets. IMPORTANT: this means that you need to install Norman Walsh's XSL Stylesheets available from http://docbook.sourceforge.net/projects/xsl/index.html and libxslt version 1.0.4 or later to generate HTML output. * New firewall service: Linux 2.2 ipchains log, as well as Darren Reed's IP Filter logs, as shipped with *BSD's. NOTE: there are some "known issues" with the IOS Cisco service in this release. The informations it gives isn't really equivalent to the other firewall services. You may consult http://www.nlnet.nl/projects/logreport/hypermail/logreport/development/0518.html for more informations. * New superservice "print" with CUPS (page_log) and LPRng (lp-acct) plugins. * Syslog parser now handles Solaris 8 syslog tag: [ID ...] * lire:timegroup aggregator can now aggregate records by months or years. * New aggregators available to build report specification: lire:summary, lire:rangegroup and lire:timeslot * New www service: W3C Extended Log Format (Used by Microsoft's Internet Information Server IIS 4.0 and IIS 5.0). * New www reports: requests-summary, requests-by-size, requests-by-timeslot * More www user session reports. * Speed improvements to the www robot, country and OS analyzers. * New robots and Nimda attack detected in www reports. * New ftp reports: tracked-users, tracked-files. * New email reports: deliveries-by-delay, deliveries-by-size, tracked_senders, tracked_recipients. * Sendmail converter is now more robust and interpret more lines. (Anti-spam messages will now get into the reports) * The Lire client no longer is configured by running ./configure --disable-server. If you want to run Lire as a client for an online responder, you just install Lire in the regular way. * Various improvements to most DLF converters. (DNS converters were optimized, all converters were updated to the new DLF API). * All "by-day" reports have "by-period" equivalent: (www): bytes-by-period, bytes-by-result-by-period, clienthost-by-period, requests-by-period, requests-by-result-by-period; (ftp): bytes-by-period. The following report specification are now obsolete and will be removed in a future Lire release: (www): bytes-by-day, bytes-by-result-by-day, clienthost-by-day, requests-by-day, requests-by-result-by-day; (ftp): bytes-by-day You should update your report configuration accordingly. * Bytes, seconds and numbers can be scaled to more human readable format by setting the variables LR_SCALE_BYTES, LR_SCALE_SEC and LR_SCALE_NUMBER to 'yes'. By default, bytes and seconds are scaled into '1k' and '2m' in the reports. * Fixed various small bugs (e.g. the one which caused .org to get skipped in the www 'Requests By Top Level Domain' report, as well as the one which caused requests from .net clients to get filed under .et, when using perl versions << 5.6.1 (5.005_03 is reported to behave in such a way)) version 20011017 * The www report is more exhaustive now: New www reports: requests-by-attack, tracked_pages-by-period, top-last_page, user_sessions-by-period reports. Added detection of Code Red attack in WWW reports. Added detection of country where client is located (WWW superservice). Added detection of GNU/Hurd OS and Links browser in WWW reports. * Added volume per hour email report. Reordered email reports, as suggested by Thierry Montigneaux (thanks Thierry!) * Added firewall superservice, added cisco service, for cisco acl logfiles (Thanks Joost Bekkers) * Added FTP superservice. Xferlog file format is now supported. * Overall performance improvement. * This version introduces a complete overhaul of the reporting engine which will make customization and extension of Lire much easier. If you are upgrading from a previous version, you probably want to read the "Upgrading From Lire's Version 20010903 or Earlier" section of the INSTALL file which has important informations about how to migrate your configurations. Some changes in this release (especially the way configuration files are handled) are _not_ backwards compatible. * The xml report format has changed. Therefore, if you install this version on a responder, submitters should install this version if they want to be able to typeset your reports, after deanonymizing them. * XML::Parser and expat (http://expat.sourceforge.net/) are now required to build and use Lire. You can install those components separately or you can get the lire-full tarball which includes and build them as part of the normal Lire installation process. The only supported XSLT processor with this version is xsltproc which comes with the XSLT C library for Gnome (http://xmlsoft.org/XSLT/) * Reports are now written using the XML Lire's Report Specification Markup Language (LRSML). DLF formats are specified using the XML Lire's DLF Schema Markup Language. * ASCII reports generated by the builtin formatter or the XSL tools are now identical. * Informations included with a report can now be tweaked according to the intended audience. You can set LR_TARGET_USER in your $HOME/.lire/etc/defaults file to 'sysadmin' or 'manager'. There is also the LR_USERLEVEL variable which can be set to either 'normal' or 'advanced'. Those variables will modify the description accordingly. Note that not all reports have different informations based on those variables. * There is a LR_MAX_MEMORY variable which you can set to process big log files without trashing. The default is 40Megs. For optimal performance, you should set this to half your available RAM (unless you are usually short on memory). * All Lire's programs now log performance information: time taken, memory used (on Linux only). * Reports which need missing information in the DLF file are now automatically skipped. * Report dates are now in the local time zone. * 8bits characters are escaped in the reports to ?. * New supported output format: XHTML and RTF. * Improvement to charts: Height increased to 300 pixels. X labels are drawn vertically making it possible to read e.g. URLs. * Charts for each report that support it are now generated when LR_INCLUDEIMAGES is set to 1. (Not only the first one). * New chart type: histogram (bars now have space between the bars) * When generating an HTML, XHTML or DocBook report with images, the result is a tar gzipped file that contains the report file and the images. When generating an RTF report with images, the result is a zip file that contains the report file and the images. * The user's and developer's manuals are reorganized. There are still a lot of the new intended sections uncompleted, though. * The apachemodgzip superservice is now a regular www service called 'modgzip'. The apache service was split into three different ones: common, combined, referer. * Fixed bugs (as always ;) version 20010903 * Lot of bugs removed * BEWARE! The file ~/.lirerc is obsolete. If you use it, please move it to ~/.lire/etc/defaults. * Fixed "bogus message: mkdir: cannot create directory `/usr/local/var': Permission denied"-bug. (This one occured when one did a systeminstall, and ran the lire scripts as a non-priviliged user afterwards.) * Now setting print-category and print-severity in named.conf is optional: a wider range of dns / bind9 logs gets accepted. * Added Apache referer parser: now one can see which links brought visitors to your site. * For www user agents newly detected is: operating system, browser type, user browser UI language and web robots * Added 6 new "www" reports: http version, user agents, bytes per directory, top referer-page connection and http method * More reports are now customizable * Added new output formats: LogML (an experimental XML application) and DocBook * Responder use munpack for all MIME messages now, not just multipart one. * Security check can be disabled by setting LR_SKIP_ID_TEST. * Default report format (txt, xml, pdf or html) can be configured via LR_DEFAULT_OUTPUT_FORMAT in $sysconfdir/defaults. * XML/SGML environment can be reconfigured at runtime from $sysconfir/defaults. * Support more XSLT processor : Sablotron and XsltProc (from libxslt). Xalan-C can be found by more names (testXSLT or xalan-c) version 20010629.1 * developer.html and developer.txt now distributed again with the tarball. version 20010629 * Cleaned up configure.in, to offer more hooks via the environment. This is nice for package builders. version 20010626 * Moved /usr/local/lib/xml to /usr/local/share/lire, to adhere to FHS. * Added reports for www superservice: report_bytesperresultperday (this was report_bytesperday, but one does no longer split things up) and report_totalperday and report_iptotalperday. * Reduce warnings generated by postfix2dlf. * Responder should reply with a proper error message when it fails to generate a report. * XML and SGML support should now be more portable between various environment. * PDF reports now can include images. This can be turned on in the .../etc/lire/defaults file, by setting LR_INCLUDEIMAGES to 1. * Fixed possible security problem: we now run with umask 037, so that copied logfiles no longer possibly get world readable. * Fixed bug in apache common convertor: now processes lines with empty http request ok. * Fixed bug which got sendmail2dlf spit evil perl error messages when fed an empty logfile. Thanks to Claire Holleman for finding it. * Fixed bug which caused postfix2dlf to produce bogus dlf from logs containing qmgr lines caused by failed deliveries, e.g. in case of 'invalid recipient syntax'. Thanks to Mark 'Xaa' Huizer for reporting this. * Keeps old reports and dlf's in a Lire archive, if variable LR_ARCHIVE is set. * Stores metainfo on logs and dlf's in a Lire database. * Added report_sizeperfromdomain and report_sizepertodomain email subreports * Added lr_xml2html script. * Fixed problem with Jade (now uses xml declaration) * Description blurbs in email report resurrected. Added description blurbs to email report. * Lire's own logfile now easier machineparsable: added LR_ID job tags * Default spooldir for responder setups is moved from /usr/local/var/spool/logreport to /usr/local/var/spool/lire. THIS CAN BREAK UPGRADES! When you're upgrading a responder, i.e. a Lire system which automatically processes logfiles which get received in email messages, move your spool manually to the new location before restarting lr_spoold. One should also fix the .forward's (or any other mechanism your mta uses) to deliver mail to Maildirs in the new spool location. version 20010509 * Fixed long line problems in ascii.xsl and docbook.xsl. * Added lr_xml2pdf script. * Fixed bug in "bytes-per-http-result" report. * Info from README, doc/developer.txt converted to xml. Now shipped as {developer,manual}.{html,pdf,txt}. Per default gets installed in /usr/local/share/doc/lire/. version 20010418 * Pointrelease for debian package. version 20010407 * Fixed bugs and improved time handling. We now take year and timezone in account, if available in the logfile. Some code cleanup. * Internals: raw intermediate format replaced by xml format. * Fixed bug which blew up Apache.pm while dealing with clients who's ip resolves to a hostname only. Tnx Wytze! * Fixed some bugs and added new elements to the logreport xml stylesheet, to gain flexibility. * Added Exim2DLF: convertor for logfiles from the exim Mail Transport Agent. ( http://www.exim.org/ .) * Added per-user ~/.lirerc file. version 20010318 * Changed package name from lr to lire. * Fixed anoying bug in lr_anonymize: now cleans up tmpfiles. * Added option to generate XML output. * Added lr_config script, for easier configuration. * Responder now is capable of sending reports to more than one Reply-To address. * Added lr_getbody manpage. * More sane defaults for things like e.g. From-address for emails sent by lr_log2mail. * Various bugfixes, especially in dealing with bogus logs. * Responder now sends informative message in case the submitted log was hardly parsable. * Added responder for access logs from the boa ( http://www.boa.org/ ) webserver. * More configuration hooks for www reports: one can specify not to get top 30, but top n. * Added ability to track specific webpages. version 20010219 * Added local site config file defaults.local, kept between upgrades. * New www reports, added apache mod_gzip engine, added responder for apache "combined" logfiles. * Added lr_run script: running the scripts will no longer spam your terminal with debug output. * Client software now includes tools (lr_rawmail2mail and friends) to automatically de-anonymize received report. Added notes about this to README.lr-client. * Added notes about running lr-client software on Solaris to README.lr-client. * Some cleanup of filesystem layout: got rid of some directories, merged config files. * Some finetuning of email reports. version 20010116 * No longer depends on GNU make. * Finds perl and sendmail path during configure: now builds, installs and runs out of the box on Solaris. * Scripts respect --prefix and other configure flags. We're really using autoconf and automake now, which makes the package act sanely in other directory hierarchies. No longer uses LR_HOME and other environment variables: no need to set them any longer. * Now package behaves sanely when configure is run without any arguments. * Integrated a client package. During configure, one can choose to install just the client, or the complete suite. The client enables one to send and receive logs and reports in an anonymized format. * Bugfixes in postfix and apache convertors. * Documentation updated. version 20001216 * Now runs on SunOS 5.7: fixed /bin/sh scripts and awk invocations. version 20001213 * Various bugfixes, especially in sendmail2dlf * Documentation updated * New email dlf format, so that relay's ip number and fqdn can be used separately * postfix engine supports nqmgr * Fixed awk scripts, to be able to run with the original awk, as shipped with OpenBSD * lr_log2mail no longer relies on mutt(1), but uses the more generally available /usr/sbin/sendmail version 20001211 * Various bugfixes * Documentation updated version 20001205 * Bugfixes in qmail convertor * An IPfilter convertor has been added * A postfix convertor and report engine are added version 20001130 * The responder uses a new mail receiving engine * The install now uses automake and autoconf * Extra reports for www added * Added manpages * New setup of report scripts * Scheduler no longer needed; more lean design * Fixed various bugs version 0.0.1 - september 2000 * First public release