3 ms·
The WARC (Web ARChive) file format offers a convention for concatenating multiple resource records (data objects), each consisting of a set of simple text hea
by ferfumarma 3y ago
The WARC (Web ARChive) file format offers a convention for concatenating multiple
resource records (data objects), each consisting of a set of simple text headers
and an arbitrary data block into one long file. The WARC format is an extension
of the ARC file format (ARC) that has
traditionally been used to store “web crawls” as sequences of content blocks
harvested from the World Wide Web. Each capture in an ARC file is preceded by a
one-line header that very briefly describes the harvested content and its length.
This is directly followed by the retrieval protocol response messages and content.
The original ARC format file has been used by the Internet Archive (IA) since 1996
for managing billions of objects, and by several national libraries.