As explained on Dovecot Wiki, dovecot imap server support FTS, one of them use Solr (from Apache Lucene project).
apt-get install openjdk-7-jdk mkdir /usr/java ln -s /usr/lib/jvm/java-7-openjdk-amd64 /usr/java/default
Now that Java is installed, let’s install Solr (with Jetty web server embedded):
cd /opt/ wget http://apache.mirrors.tds.net/lucene/solr/4.10.2/solr-4.10.2.tgz tar xzf solr-4.10.2.tgz mkdir -p solr cp -R solr-4.10.2/example/* /opt/solr/
Then you can test your Solr installation, first launch it:
cd /opt/solr java -jar start.jar
Check if it works by visiting http://YOUR_IP:8983/solr.
When it works, go back into your SSH session and close the window with Ctrl+C. The integrated Jetty server that comes with Solr is configured to bind to port 8983 on all IP adresses by default. This configuration is unsafe: anyone could clear your Solr index! Let’s make Jetty listen only on localhost by nano /opt/etc/jetty.xml and add default=”127.0.0.1″ to:
<Call name="addConnector"> <Arg> <New class="org.eclipse.jetty.server.bio.SocketConnector"> <Set name="host"><SystemProperty name="jetty.host" default="127.0.0.1" /></Set> ... ...
Save. Next time you want to access the solr admin, first open a SSH tunnel with your server: ssh -L localhost:8080:127.0.0.1:8983 user@server -N -C and then set firefox with SOCKS 5 proxy to go through port 8080, and finally in the firefox address bar type http://localhost:8080/solr .
Jetty (java webserver)
Let’s continue with some Jetty config. Edit nano /etc/default/jetty and add this:
NO_START=0 # Start on boot JAVA_OPTIONS="-Dsolr.solr.home=/opt/solr/solr $JAVA_OPTIONS" JAVA_HOME=/usr/java/default JETTY_HOME=/opt/solr JETTY_USER=solr JETTY_LOGS=/opt/solr/logs
Now nano /opt/solr/etc/jetty-logging.xml and add:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">
<!-- =============================================================== -->
<!-- Configure stderr and stdout to a Jetty rollover log file -->
<!-- this configuration file should be used in combination with -->
<!-- other configuration files. e.g. -->
<!-- java -jar start.jar etc/jetty-logging.xml etc/jetty.xml -->
<!-- =============================================================== -->
<Configure id="Server" class="org.mortbay.jetty.Server">
<New id="ServerLog" class="java.io.PrintStream">
<Arg>
<New class="org.mortbay.util.RolloverFileOutputStream">
<Arg><SystemProperty name="jetty.logs" default="."/>/yyyy_mm_dd.stderrout.log</Arg>
<Arg type="boolean">false</Arg>
<Arg type="int">90</Arg>
<Arg><Call class="java.util.TimeZone" name="getTimeZone"><Arg>GMT</Arg></Call></Arg>
<Get id="ServerLogName" name="datedFilename"/>
</New>
</Arg>
</New>
<Call class="org.mortbay.log.Log" name="info"><Arg>Redirecting stderr/stdout to <Ref id="ServerLogName"/></Arg></Call>
<Call class="java.lang.System" name="setErr"><Arg><Ref id="ServerLog"/></Arg></Call>
<Call class="java.lang.System" name="setOut"><Arg><Ref id="ServerLog"/></Arg></Call></Configure>
Then, create the Solr user and grant it permissions:
sudo useradd -d /opt/solr -s /sbin/false solr sudo chown solr:solr -R /opt/solr
After that, download the start file and set it to automatically start up:
sudo wget -O /etc/init.d/jetty http://dev.eclipse.org/svnroot/rt/org.eclipse.jetty/jetty/trunk/jetty-distribution/src/main/resources/bin/jetty.sh sudo chmod a+x /etc/init.d/jetty sudo update-rc.d jetty defaults
Finally, try to start Jetty/Solr again with the new way: sudo service jetty start and try to access the admin panel. Nice!
Solr 4.10
Now let’s configure Solr. We’ll rename the default “Schema” and download a new one that is customized for Dovecot.
mv /opt/solr/solr/collection1/conf/schema.xml opt/solr/solr/collection1/conf/schema.xml-dist cd opt/solr/solr/collection1/conf/ wget https://raw.githubusercontent.com/extremeshok/solr-dovecot2/master/schema.xml
That’s good thing done. Now as I store my email in an encrypted (ecryptfs) Private folder, I also want my solr index to be encrypted, right? So let’s symlink the data folder into my encrypted Private folder: ln -s /home/user/Private/mail/solr/data /opt/solr/solr/collection1/data and be sure to chown solr:solr the new folders properly.
Dovecot 2.2
Now let’s configure Dovecot. First they mention some pre-requisite on the wiki page, so let’s install them (they may already be installed) :
apt-get install libcurl4-openssl-dev libexpat1-dev
then let’s enable solr in Dovecot config by nano /etc/dovecot/conf.d/10-mail.conf and find/add this line mail_plugins = fts fts_solr . And set the plugin settings by nano /etc/dovecot/conf.d/90-plugins and add/edit/replace with
plugin {
fts = solr
fts_solr = url=http://127.0.0.1:8983/solr/ break-imap-search
fts_autoindex = yes
}
break-imap-search : Use Solr also for indexing TEXT and BODY searches. This makes your server non-IMAP-compliant. (This is always enabled in v2.1+) It’s a good thing.
Now we need to compile Dovecot as for me, under a fresh Ubuntu 14.04 my Dovecot didn’t have Solr support ! but it’s very simple, so don’t be scared (it will take around 10~15mn with a low config server):
apt-get source dovecot cd dovecot-2.2.9 sudo apt-get build-dep package debuild -us -uc -b cd .. dpkg -i dovecot-core_2.2.9-1ubuntu2.1_amd64.deb dovecot-solr_2.2.9-1ubuntu2.1_amd64.deb
After having reinstalled Dovecot and installed Solr support, you can verify your dovecot conf directory and version is still the same as before:
doveconf -n | head -n1 # 2.2.9: /etc/dovecot/dovecot.conf
Rock it!
Restart everything,
service jetty restart service dovecot restart
and try the new Solr search by logging into your imap server a1 login user@server password and searching with a3 SEARCH text “test” see below for the full process:
openssl s_client -connect localhost:993 * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot (Ubuntu) ready. a1 login user@server password a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY] Logged in a2 select Inbox a3 SEARCH text "test" * OK Indexed 29% of the mailbox, ETA 0:25 * OK Indexed 37% of the mailbox, ETA 0:34 * OK Indexed 89% of the mailbox, ETA 0:03 * OK Indexed 97% of the mailbox, ETA 0:01 * OK Indexed 97% of the mailbox, ETA 0:01 * OK Indexed 97% of the mailbox, ETA 0:01 * OK Indexed 97% of the mailbox, ETA 0:02 * OK Indexed 97% of the mailbox, ETA 0:02 * OK Indexed 97% of the mailbox, ETA 0:02 * OK Indexed 97% of the mailbox, ETA 0:03 * OK Indexed 97% of the mailbox, ETA 0:03 * OK Mailbox indexing finished * SEARCH 1 2 3 4 7 8 9 10 11 12 13 14 17 18 22 23 26 27 33 34 56 57 58 59 60 77 83 85 88 118 122 123 126 128 156 178 179 183 186 191 199 200 212 213 225 245 247 254 281 282 286 287 289 305 309 311 312 315 317 320 329 333 357 358 361 364 374 375 379 387 390 401 405 408 414 415 416 428 433 439 442 444 445 465 494 495 496 500 504 533 537 538 542 543 549 557 561 566 567 568 569 575 580 603 620 621 636 641 642 643 668 677 679 684 685 686 687 691 701 705 710 715 718 721 722 734 736 743 747 748 753 754 773 774 775 776 777 778 779 780 784 785 786 787 792 795 813 820 821 829 830 831 843 844 855 861 862 863 864 865 870 871 886 889 890 891 892 893 894 895 899 906 914 915 924 926 930 933 937 938 939 940 941 946 947 979 987 988 990 991 1004 1005 1006 1007 1012 1013 1014 1018 1027 1028 1030 1037 1047 1054 1055 1059 1060 1061 1063 1064 1068 1069 1070 1071 1072 1074 1080 1081 1082 1083 1085 1087 1092 1093 1104 1111 1112 1113 1117 1118 1120 1140 1142 1143 1146 1152 1153 1154 1165 1176 1184 1185 1189 1201 1202 1210 1211 1212 1217 1228 1237 1238 1240 1243 1244 1245 1246 1247 1248 1249 1250 1252 1258 1264 1265 1268 1269 1279 1280 1284 1291 1298 1299 1300 1302 1306 1314 1315 1317 1322 1331 1338 a3 OK Search completed (117.978 secs). a4 SEARCH text "test" * SEARCH 1 2 3 4 7 8 9 10 11 12 13 14 17 18 22 23 26 27 33 34 56 57 58 59 60 77 83 85 88 118 122 123 126 128 156 178 179 183 186 191 199 200 212 213 225 245 247 254 281 282 286 287 289 305 309 311 312 315 317 320 329 333 357 358 361 364 374 375 379 387 390 401 405 408 414 415 416 428 433 439 442 444 445 465 494 495 496 500 504 533 537 538 542 543 549 557 561 566 567 568 569 575 580 603 620 621 636 641 642 643 668 677 679 684 685 686 687 691 701 705 710 715 718 721 722 734 736 743 747 748 753 754 773 774 775 776 777 778 779 780 784 785 786 787 792 795 813 820 821 829 830 831 843 844 855 861 862 863 864 865 870 871 886 889 890 891 892 893 894 895 899 906 914 915 924 926 930 933 937 938 939 940 941 946 947 979 987 988 990 991 1004 1005 1006 1007 1012 1013 1014 1018 1027 1028 1030 1037 1047 1054 1055 1059 1060 1061 1063 1064 1068 1069 1070 1071 1072 1074 1080 1081 1082 1083 1085 1087 1092 1093 1104 1111 1112 1113 1117 1118 1120 1140 1142 1143 1146 1152 1153 1154 1165 1176 1184 1185 1189 1201 1202 1210 1211 1212 1217 1228 1237 1238 1240 1243 1244 1245 1246 1247 1248 1249 1250 1252 1258 1264 1265 1268 1269 1279 1280 1284 1291 1298 1299 1300 1302 1306 1314 1315 1317 1322 1331 1338 a4 OK Search completed (0.002 secs). a5 LOGOUT
Yeah! 0,002 secs instead of 117,98 secs. Now you can go log in Roundcube or open Thunderbird, and start leverage the FTS to datamine your mails.
Because Solr index needs to be optimized, and Dovecot doesn’t tell Solr to do it, you should add a cron job crontab -e with these commands:
# Optimize should be run somewhat rarely, e.g. once a day at 2:30 am 30 2 * * * curl http://localhost:8983/solr/update?optimize=true # Commit should be run pretty often, e.g. every minute */1 * * * * curl http://localhost:8983/solr/update?commit=true
If you want to index again your mailbox (for whatever reason), use this command doveadm fts rescan -u <username> and then log into dovecot server and do a search (as above). It will start indexing the mailbox.
Note : Thunderbird builds its own local index, if you want to send a IMAP search to Dovecot (that will search using Solr) you can launch the search box using ctrl+shift+f
Thanks
This article was inspired from Extremshok and digitalocean. All the mail configuration was done by mailinabox. Thanks to them.
iRedMail can also be used to automagically setup the mailserver.