Re: [RFC] LKML Archive in Maildir Format
OK, so I understand how to clone archives from lore.kernel.org and how to convert a git archive to a maildir (thanks, Konstantin!) What I *don't* understand is how to effectively read this locally. Ideally I'd like to run mutt, possibly with notmuch for indexing. But a maildir with 3M files seems impractical. I did actually try it (without notmuch), but it takes mutt about 5 minutes to start up. And the maildir is about 23G, compared with 7.5G for the git archive. Any pointers? I guess there's no mutt backend that can read a public-inbox archive directly?
Bjorn Helgaas <bhelgaas@google.com> wrote:
OK, so I understand how to clone archives from lore.kernel.org and how to convert a git archive to a maildir (thanks, Konstantin!)
What I *don't* understand is how to effectively read this locally. Ideally I'd like to run mutt, possibly with notmuch for indexing. But a maildir with 3M files seems impractical. I did actually try it (without notmuch), but it takes mutt about 5 minutes to start up. And the maildir is about 23G, compared with 7.5G for the git archive.
Right, relying on Maildir for long-term storage of giant archives is not a usable solution with any general purpose FSes I know about. git itself had the same problem with loose object scalability in the old days and packs were invented as a result.
Any pointers? I guess there's no mutt backend that can read a public-inbox archive directly?
There's mutt patches to support reading over NNTP, so that works: mutt -f news://$INBOX_HOST/$INBOX_NEWSGROUP I don't think mutt handles mboxrd 100% correctly, but it's close enough that you can can download the gzipped mboxrd of a search query and open it via "mutt -f /path/to/downloaded/mbox.gz" curl -XPOST -OJ "$INBOX_URL/?q=$SEARCH_QUERY&x=m" POST is required(*), and -OJ lets it use the Content-Disposition: header for a meaningful server-generated name, but you can also redirect the result to whatever you want. For all messages since March 1, you could use: SEARCH_QUERY=d:20190301.. All the supported search queries are documented in $INBOX_URL/_/text/help/ and the search prefixes (e.g. "d:", "s:", "b:") are modeled after what's in mairix. You'll need to escape the queries for URIs (e.g. " " => "+", and so on). Xapian requires date ranges to be denoted with ".." whereas mairix uses "-" for ranges. The main thing public-inbox search misses from mairix is support for "-t" which grabs non-matching messages from the same thread. I would like to support that someday, but don't have enough time (or funding) to make it happen at the moment. (*) to reliably avoid wasting resources from spiders/prefetchers
On Tue, Mar 5, 2019 at 5:26 PM Eric Wong <e@80x24.org> wrote:
Bjorn Helgaas <bhelgaas@google.com> wrote:
Any pointers? I guess there's no mutt backend that can read a public-inbox archive directly?
There's mutt patches to support reading over NNTP, so that works:
mutt -f news://$INBOX_HOST/$INBOX_NEWSGROUP
Neomutt includes NNTP support, so I tried this: neomutt -f news://nntp.lore.kernel.org/org.kernel.vger.linux-kernel which worked OK but (1) I only see the most recent 1000 messages and (2) obviously isn't reading a *local* archive. Neomutt took about 45 seconds to start up over my wimpy ISP. I assume I could probably have a local archive and run a local NNTP server and point neomutt at that local server. But I don't know how full-archive searching would work there.
I don't think mutt handles mboxrd 100% correctly, but it's close enough that you can can download the gzipped mboxrd of a search query and open it via "mutt -f /path/to/downloaded/mbox.gz"
curl -XPOST -OJ "$INBOX_URL/?q=$SEARCH_QUERY&x=m"
I got nothing at all with -XPOST, but this: curl -OJ "https://lore.kernel.org/linux-pci/?q=d:20190301..&x=m" got me the HTML source. Nothing that looks like mboxrd. I assume this is stupid user error on my part, but even with that resolved, it wouldn't have the nice git fetch properties of the git archive, i.e., incremental updates of only new stuff, would it? I think my ideal solution would be a mutt that could read the git archive directly, plus a notmuch index. But AFAIK, mutt can't do that, and notmuch only works with one message per file, not with the git archive. Something that might work would be to use Konstantin's "git archive to maildir" hint but shard into a bunch of smaller maildirs instead of one big one, then have notmuch index those, and use mutt or vim with notmuch queries instead of having it read in a maildir. But I feel like I must be missing the solution that's obvious to everybody but me. Bjorn
Bjorn Helgaas <bhelgaas@google.com> wrote:
On Tue, Mar 5, 2019 at 5:26 PM Eric Wong <e@80x24.org> wrote:
Bjorn Helgaas <bhelgaas@google.com> wrote:
Any pointers? I guess there's no mutt backend that can read a public-inbox archive directly?
There's mutt patches to support reading over NNTP, so that works:
mutt -f news://$INBOX_HOST/$INBOX_NEWSGROUP
Neomutt includes NNTP support, so I tried this:
neomutt -f news://nntp.lore.kernel.org/org.kernel.vger.linux-kernel
which worked OK but (1) I only see the most recent 1000 messages and (2) obviously isn't reading a *local* archive. Neomutt took about 45 seconds to start up over my wimpy ISP.
I assume I could probably have a local archive and run a local NNTP server and point neomutt at that local server. But I don't know how full-archive searching would work there.
Right. AFAIK there isn't a good solution for search via NNTP.
I don't think mutt handles mboxrd 100% correctly, but it's close enough that you can can download the gzipped mboxrd of a search query and open it via "mutt -f /path/to/downloaded/mbox.gz"
curl -XPOST -OJ "$INBOX_URL/?q=$SEARCH_QUERY&x=m"
I got nothing at all with -XPOST, but this:
Ah, I guess nginx (or something in AWS) rejects POST without Content-Length headers. Adding "-HContent-Length:0" to the command-line with -XPOST works for lore.
curl -OJ "https://lore.kernel.org/linux-pci/?q=d:20190301..&x=m"
got me the HTML source. Nothing that looks like mboxrd. I assume
Right. The "x=m" requests an mbox; but it's only available via POST requests (to prevent search engine spiders from wasting time on non-HTML content). With the HTML output in a browser, the "mbox.gz" button makes the POST request and allows you to download the mbox.
this is stupid user error on my part, but even with that resolved, it wouldn't have the nice git fetch properties of the git archive, i.e., incremental updates of only new stuff, would it?
You could bump d:YYYYMMDD (there's also "dt:" for date-time if you need more precision).
I think my ideal solution would be a mutt that could read the git archive directly, plus a notmuch index. But AFAIK, mutt can't do that, and notmuch only works with one message per file, not with the git archive.
Something that might work would be to use Konstantin's "git archive to maildir" hint but shard into a bunch of smaller maildirs instead of one big one, then have notmuch index those, and use mutt or vim with notmuch queries instead of having it read in a maildir.
Small Maildirs work great, but large ones fall over. I don't think having a bunch of smaller Maildirs would help notmuch since notmuch still needs to know each file path. The only way I could see notmuch/Maildir working well is to keep the overall number of messages relatively small. One of my longer-term goals is to write a mairix-like tool in Perl which works with public-inbox archives; but I barely have enough time for public-inbox these days :< mairix works with gzipped mboxes, which is great for large archives; but the indexing falls over since it rewrites the entire search index every time. SSDs have died as a result :<
But I feel like I must be missing the solution that's obvious to everybody but me.
Nope, you're not alone :) There's not a lot of mail software which can handle LKML-sized histories efficiently.
participants (2)
-
Bjorn Helgaas -
Eric Wong