LIBARCHIVE(3)            BSD Library Functions Manual            LIBARCHIVE(3)

NAME
     libarchive_internals - description of libarchive internal
     interfaces

OVERVIEW
     The libarchive library provides a flexible interface for
     reading and writing streaming archive files such as tar
     and cpio.  Internally, it follows a modular layered design
     that should make it easy to add new archive and compres-
     sion formats.

GENERAL ARCHITECTURE
     Externally, libarchive exposes most operations through an
     opaque, object-style interface.  The archive_entry(1)
     objects store information about a single filesystem
     object.  The rest of the library provides facilities to
     write archive_entry(1) objects to archive files, read them
     from archive files, and write them to disk.  (There are
     plans to add a facility to read archive_entry(1) objects
     from disk as well.)

     The read and write APIs each have four layers: a public
     API layer, a format layer that understands the archive
     file format, a compression layer, and an I/O layer.  The
     I/O layer is completely exposed to clients who can replace
     it entirely with their own functions.

     In order to provide as much consistency as possible for
     clients, some public functions are virtualized.  Eventu-
     ally, it should be possible for clients to open an archive
     or disk writer, and then use a single set of code to
     select and write entries, regardless of the target.

READ ARCHITECTURE
     From the outside, clients use the archive_read(3) API to
     manipulate an archive object to read entries and bodies
     from an archive stream.  Internally, the archive object is
     cast to an archive_read object, which holds all read-spe-
     cific data.  The API has four layers: The lowest layer is
     the I/O layer.  This layer can be overridden by clients,
     but most clients use the packaged I/O callbacks provided,
     for example, by archive_read_open_memory(3), and
     archive_read_open_fd(3).  The compression layer calls the
     I/O layer to read bytes and decompresses them for the for-
     mat layer.  The format layer unpacks a stream of uncom-
     pressed bytes and creates archive_entry objects from the
     incoming data.  The API layer tracks overall state (for
     example, it prevents clients from reading data before
     reading a header) and invokes the format and compression
     layer operations through registered function pointers.  In
     particular, the API layer drives the format-detection pro-
     cess: When opening the archive, it reads an initial block
     of data and offers it to each registered compression han-
     dler.  The one with the highest bid is initialized with
     the first block.  Similarly, the format handlers are
     polled to see which handler is the best for each archive.
     (Prior to 2.4.0, the format bidders were invoked for each
     entry, but this design hindered error recovery.)

   I/O Layer and Client Callbacks
     The read API goes to some lengths to be nice to clients.
     As a result, there are few restrictions on the behavior of
     the client callbacks.

     The client read callback is expected to provide a block of
     data on each call.  A zero-length return does indicate end
     of file, but otherwise blocks may be as small as one byte
     or as large as the entire file.  In particular, blocks may
     be of different sizes.

     The client skip callback returns the number of bytes actu-
     ally skipped, which may be much smaller than the skip
     requested.  The only requirement is that the skip not be
     larger.  In particular, clients are allowed to return zero
     for any skip that they don't want to handle.  The skip
     callback must never be invoked with a negative value.

     Keep in mind that not all clients are reading from disk:
     clients reading from networks may provide different-sized
     blocks on every request and cannot skip at all; advanced
     clients may use mmap(2) to read the entire file into mem-
     ory at once and return the entire file to libarchive as a
     single block; other clients may begin asynchronous I/O
     operations for the next block on each request.

   Decompresssion Layer
     The decompression layer not only handles decompression, it
     also buffers data so that the format handlers see a much
     nicer I/O model.  The decompression API is a two stage
     peek/consume model.  A read_ahead request specifies a min-
     imum read amount; the decompression layer must provide a
     pointer to at least that much data.  If more data is imme-
     diately available, it should return more: the format layer
     handles bulk data reads by asking for a minimum of one
     byte and then copying as much data as is available.

     A subsequent call to the consume() function advances the
     read pointer.  Note that data returned from a read_ahead()
     call is guaranteed to remain in place until the next call
     to read_ahead().  Intervening calls to consume() should
     not cause the data to move.

     Skip requests must always be handled exactly.  Decompres-
     sion handlers that cannot seek forward should not register
     a skip handler; the API layer fills in a generic skip han-
     dler that reads and discards data.

     A decompression handler has a specific lifecycle:
     Registration/Configuration
             When the client invokes the public support func-
             tion, the decompression handler invokes the inter-
             nal __archive_read_register_compression() function
             to provide bid and initialization functions.  This
             function returns NULL on error or else a pointer
             to a struct decompressor_t.  This structure con-
             tains a void * config slot that can be used for
             storing any customization information.
     Bid     The bid function is invoked with a pointer and
             size of a block of data.  The decompressor can
             access its config data through the decompressor
             element of the archive_read object.  The bid func-
             tion is otherwise stateless.  In particular, it
             must not perform any I/O operations.

             The value returned by the bid function indicates
             its suitability for handling this data stream.  A
             bid of zero will ensure that this decompressor is
             never invoked.  Return zero if magic number checks
             fail.  Otherwise, your initial implementation
             should return the number of bits actually checked.
             For example, if you verify two full bytes and
             three bits of another byte, bid 19.  Note that the
             initial block may be very short; be careful to
             only inspect the data you are given.  (The current
             decompressors require two bytes for correct bid-
             ding.)
     Initialize
             The winning bidder will have its init function
             called.  This function should initialize the
             remaining slots of the struct decompressor_t
             object pointed to by the decompressor element of
             the archive_read object.  In particular, it should
             allocate any working data it needs in the data
             slot of that structure.  The init function is
             called with the block of data that was used for
             tasting.  At this point, the decompressor is
             responsible for all I/O requests to the client
             callbacks.  The decompressor is free to read more
             data as and when necessary.
     Satisfy I/O requests
             The format handler will invoke the read_ahead,
             consume, and skip functions as needed.
     Finish  The finish method is called only once when the
             archive is closed.  It should release anything
             stored in the data and config slots of the
             decompressor object.  It should not invoke the
             client close callback.

   Format Layer
     The read formats have a similar lifecycle to the decom-
     pression handlers:
     Registration
             Allocate your private data and initialize your
             pointers.
     Bid     Formats bid by invoking the read_ahead() decom-
             pression method but not calling the consume()
             method.  This allows each bidder to look ahead in
             the input stream.  Bidders should not look further
             ahead than necessary, as long look aheads put
             pressure on the decompression layer to buffer lots
             of data.  Most formats only require a few hundred
             bytes of look ahead; look aheads of a few kilo-
             bytes are reasonable.  (The ISO9660 reader some-
             times looks ahead by 48k, which should be consid-
             ered an upper limit.)
     Read header
             The header read is usually the most complex part
             of any format.  There are a few strategies worth
             mentioning: For formats such as tar or cpio, read-
             ing and parsing the header is straightforward
             since headers alternate with data.  For formats
             that store all header data at the beginning of the
             file, the first header read request may have to
             read all headers into memory and store that data,
             sorted by the location of the file data.  Subse-
             quent header read requests will skip forward to
             the beginning of the file data and return the cor-
             responding header.
     Read Data
             The read data interface supports sparse files;
             this requires that each call return a block of
             data specifying the file offset and size.  This
             may require you to carefully track the location so
             that you can return accurate file offsets for each
             read.  Remember that the decompressor will return
             as much data as it has.  Generally, you will want
             to request one byte, examine the return value to
             see how much data is available, and possibly trim
             that to the amount you can use.  You should invoke
             consume for each block just before you return it.
     Skip All Data
             The skip data call should skip over all file data
             and trailing padding.  This is called automati-
             cally by the API layer just before each header
             read.  It is also called in response to the client
             calling the public data_skip() function.
     Cleanup
             On cleanup, the format should release all of its
             allocated memory.

   API Layer
     XXX to do XXX

WRITE ARCHITECTURE
     The write API has a similar set of four layers: an API
     layer, a format layer, a compression layer, and an I/O
     layer.  The registration here is much simpler because only
     one format and one compression can be registered at a
     time.

   I/O Layer and Client Callbacks
     XXX To be written XXX

   Compression Layer
     XXX To be written XXX

   Format Layer
     XXX To be written XXX

   API Layer
     XXX To be written XXX

WRITE_DISK ARCHITECTURE
     The write_disk API is intended to look just like the write
     API to clients.  Since it does not handle multiple formats
     or compression, it is not layered internally.

GENERAL SERVICES
     The archive_read, archive_write, and archive_write_disk
     objects all contain an initial archive object which pro-
     vides common support for a set of standard services.
     (Recall that ANSI/ISO C90 guarantees that you can cast
     freely between a pointer to a structure and a pointer to
     the first element of that structure.)  The archive object
     has a magic value that indicates which API this object is
     associated with, slots for storing error information, and
     function pointers for virtualized API functions.

MISCELLANEOUS NOTES
     Connecting existing archiving libraries into libarchive is
     generally quite difficult.  In particular, many existing
     libraries strongly assume that you are reading from a
     file; they seek forwards and backwards as necessary to
     locate various pieces of information.  In contrast,
     libarchive never seeks backwards in its input, which some-
     times requires very different approaches.

     For example, libarchive's ISO9660 support operates very
     differently from most ISO9660 readers.  The libarchive
     support utilizes a work-queue design that keeps a list of
     known entries sorted by their location in the input.
     Whenever libarchive's ISO9660 implementation is asked for
     the next header, checks this list to find the next item on
     the disk.  Directories are parsed when they are encoun-
     tered and new items are added to the list.  This design
     relies heavily on the ISO9660 image being optimized so
     that directories always occur earlier on the disk than the
     files they describe.

     Depending on the specific format, such approaches may not
     be possible.  The ZIP format specification, for example,
     allows archivers to store key information only at the end
     of the file.  In theory, it is possible to create ZIP
     archives that cannot be read without seeking.  Fortu-
     nately, such archives are very rare, and libarchive can
     read most ZIP archives, though it cannot always extract as
     much information as a dedicated ZIP program.

SEE ALSO
     archive(3), archive_entry(3), archive_read(3),
     archive_write(3), archive_write_disk(3)

HISTORY
     The libarchive library first appeared in FreeBSD 5.3.

AUTHORS
     The libarchive library was written by Tim Kientzle
     <kientzle@acm.org>.

BUGS
BSD                             April 16, 2007                             BSD
