License: MIT
Acknowledgements: Maxim Khitrov
imap2json converts an imap URL of a folder to a HTML conversation view (Organised by thread)
Dependencies:
- (Dovecot) IMAP server for http://tools.ietf.org/html/rfc5256
- Golang for sanitising and serialising IMAP response to JSON
- Web interface to render the JSON, between familiar index and expanded conversation views
Example use case:
Ruth and Kai have an email exchange arranging a movie. A friend Jamie wants to catch up on the conversation and perhaps join. Kai emails Jamie the link to the start of the conversation.
The URL points to the Web email archive produced imap2json of the entire conversation. Now Jamie knows all the details, in a familiar Web conversation view, that is mails of the same topic are grouped together, of the email exchange. There is no need for Kai or Ruth to use the clumsy "Forward all" mechanism on their email client.
Example usage:
- go get github.com/kaihendry/imap2json
- $GOPATH/bin/imap2json imap://imap.dabase.com
This will generate mail.json from our test repo imap://imap.dabase.com
imap.dabase.com is a documented working read only IMAP server, serving an example Maildir.
For a simple Web interface (json2html) for mail.json see serve our html directory from your Webserver.
Dovecot implements RFC5256 threading which can be used to group emails into conversations like Gmail does.
This saves us from having to code this tricky part ourselves by tracking Message-ID: and References: / In-Reply-To:, which can be rather complex.
IMAP also gives us a uniform interface at an IMAP URL, e.g.
imap://$USER[@]imap.dabase.com[/folder] to generate an archive from.
On a local dovecot instance using Maildir and minimal Dovecot configuration:
printf "1 select inbox\n2 UID THREAD references us-ascii all\n3 fetch 1:* envelope\n4 logout\n" |
/usr/lib/dovecot/imap
Or using netcat with our read only test IMAP service, so you don't need to setup Dovecot:
nc imap.dabase.com imap << EOF
1 authenticate anonymous foobar
2 select inbox
3 UID THREAD references UTF-8 all
4 fetch 1:* envelope
5 logout
EOF
Dovecot's THREAD rfc5256
THREAD (1 2 3 4)(5)
We can see messages 1,2,3,4 are a single conversation about a "Movie".
5 is a separate conversation about a Dentist appointment.
-- 1
\-- 2
|-- 3
\-- 4
-- 5
Mutt is an excellent MUA and reference point.
mutt -f Maildir
Mutt conjures a great threaded conversation view as a reference independent of Dovecot's THREAD rfc5256 support. So it would be interesting to eventually compare the threading structure.
mutt needs to be compiled with --enable-debug.
$ mutt -v | grep -i debug
Configure options: '--prefix=/usr' '--sysconfdir=/etc' '--enable-imap' '--enable-debug' '--enable-smtp' '--enable-hcache' '--with-curses=/usr' '--with-regex' '--with-gss=/usr' '--with-ssl=/usr' '--with-sasl' '--with-idn' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,--hash-style=gnu'
+DEBUG
We need a mutt configuration for testing:
$ cat mutt-imap2json
set spoolfile=imap://anonymous@imap.dabase.com
set folder=imap://anonymous@imap.dabase.com
set sort=threads
set sort_aux=reverse-last-date-received
Now run mutt in debug mode with the above configuration:
mutt -d2 -F mutt-imap2json
Look at .muttdebug0 for a complete log
grep '4>' .muttdebug0 # show the IMAP commands issued by mutt
Going to use Golang to interface with Dovecot IMAP using http://godoc.org/code.google.com/p/go-imap
http://golang.org/pkg/net/mail/ can help parse/sanitise addresses, text bodies and convert Dates to Epoch.
Then json.MarshalIndent the sanitised email metadata & body into JSON.
Hoping to statically generate a large JSON file per mailbox. This will be compressed on delivery to the Web interface.
Javascript needs to be written to AJAX the JSON and produce the output. Initially as close to Gmail's or Fastmail's conversation view as possible.
Perhaps a NodeJS can be written to statically generate the HTML too, so no Javascript is required, making it Web crawler friendly.
The index & expanded views are probably most important.
Gmail uses 16 character alphanumeric string as a conversation id, e.g. https://mail.google.com/mail/u/0/#inbox/1424243f629686c1
Fastmail uses two 16 character alphanumeric strings with a dash in between, e.g. https://www.fastmail.fm/mail/Inbox/5cf2a643c23b82dc-f65945369u632?u=148c411e
The sha1sum of the first message (\r\n line endings) of a conversation could be used as the id of the conversation.
$ unix2dos 'Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S'
unix2dos: converting file Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S to DOS format ...
$ sha1sum 'Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S'
5048d370149a7a5d25dc17869cb1404cf747b6bb Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S
So the "movie" conversation would be identified by 5048d370149a7a5d25dc17869cb1404cf747b6bb
http://imap2json.dabase.com/#5048d370149a7a5d25dc17869cb1404cf747b6bb or rather some short, easily sharable version like: http://imap2json.dabase.com/#5048d
Mailman's hellokitty demo