Ubuntu server 14.04 + OpenVPN + Android client

We are going to install Openvpn on Ubuntu server 14.04 and then use Android to connect to it (so you can bypass Chinese firewall for example ūüėČ

Install OpenVPN

apt-get install openvpn libssl-dev openssl

We need to allow IPv4 forwarding so the server can send out packets on the VPN‚Äôs behalf. let’s¬†nano /etc/sysctl.conf¬†and uncomment this line:

net.ipv4.ip_forward=1

Then sudo sysctl -p to reload the modified conf. Then nano /etc/default/ufw and edit this line:

DEFAULT_FORWARD_POLICY="ACCEPT"

finally nano /etc/ufw/before.rules and edit like this:

# START OPENVPN RULES

# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0]
# Allow traffic from OpenVPN client to eth0
-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
COMMIT
#END OPENVPN RULES


# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines

#OpenVPN rules
-A ufw-before-input -i tun+ -j ACCEPT
-A ufw-before-output -i tun+ -j ACCEPT
-A ufw-before-forward -s 10.8.0.0/24 -j ACCEPT
-A ufw-before-forward -d 10.8.0.0/24 -j ACCEPT

Open the openvpn port with ufw allow 1194/udp and restart ufw service ufw restart .

Create Server Keys

We will use Easy-RSA to generate the server side keys.

apt-get install easy-rsa
make-cadir /etc/openvpn/easy-rsa
cd /etc/openvpn/easy-rsa/
source vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server
cd keys
openvpn --genkey --secret ta.key

Now copy the server keys to the root of the openvpn directory:

cp server.crt server.key ca.crt dh2048.pem ta.key /etc/openvpn/

Create Client Certificates

cd /etc/openvpn/easy-rsa/
source vars
./pkitool client-name

Those commands will create new files int the easy-rsa/keys directory called client-name.crt and client-name.key.

/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/client-name.crt
/etc/openvpn/easy-rsa/keys/client-name.key

These 3 files need to be copied to the client, so the client software can use them to make the connection with the server.  You can use filezilla to download them on the client for example.

Tweak OpenVPN config

Server side

Copy/extract the default Openvpn conf file:

cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
gzip -d /etc/openvpn/server.conf.gz
nano /etc/openvpn/server.conf

nano /etc/openvpn/server.conf  and make the following changes:

;dh dh1024.pem
;change to
dh dh2048.pem

Change this so all your client traffic passes through the VPN.

;push "redirect-gateway def1 bypass-dhcp"
;change to
push "redirect-gateway def1 bypass-dhcp"

Push specific DNS address to your clients.

;push "dhcp-option DNS 208.67.222.222"
;push "dhcp-option DNS 208.67.220.220"
;uncomment and e3dit these as needed to:
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

Change the following to increase security so the VPN service has restricted access

;user nobody
;group nogroup
;change to
user nobody
group nogroup

Specify where to output the log of openvpn

log-append /var/log/openvpn.log

finally:

service openvpn restart

Android side

Now on your android device, download the official OpenVPN client from F-Droid market (https://f-droid.org/wiki/page/de.blinkt.openvpn), add a profile, and edit the server address in the config (your server IP or FQDN). Give the android the path to the 3 files you previously downloaded. And start the connection!

home openvpn-main-page settings

That’s it, you can enjoy Youtube & co in China.

Add Full Text Search (FTS) to your Dovecot using Solr 4.10 on Ubuntu 14.04

As explained on Dovecot Wiki, dovecot imap server support FTS, one of them use Solr (from Apache Lucene project).

apt-get install openjdk-7-jdk
mkdir /usr/java
ln -s /usr/lib/jvm/java-7-openjdk-amd64 /usr/java/default

Now that Java is installed, let’s install Solr (with Jetty web server embedded):

cd /opt/
wget http://apache.mirrors.tds.net/lucene/solr/4.10.2/solr-4.10.2.tgz
tar xzf solr-4.10.2.tgz
mkdir -p solr
cp -R solr-4.10.2/example/* /opt/solr/

Then you can test your Solr installation, first launch it:

cd /opt/solr
java -jar start.jar

Check if it works by visiting http://YOUR_IP:8983/solr.

Screenshot from 2014-11-05 09:54:48

When it works, go back into your SSH session and close the window with Ctrl+C. The integrated Jetty server that comes with Solr is configured to bind to port 8983 on all IP adresses by default. This configuration is unsafe: anyone could clear your Solr index! Let’s make Jetty listen only on localhost by nano /opt/etc/jetty.xml and add default=”127.0.0.1″ to:

<Call name="addConnector"> <Arg> <New class="org.eclipse.jetty.server.bio.SocketConnector"> <Set name="host"><SystemProperty name="jetty.host" default="127.0.0.1" /></Set> ... ...

Save. Next time you want to access the solr admin, first open a SSH tunnel with your server: ssh -L localhost:8080:127.0.0.1:8983 user@server -N -C   and then set firefox with SOCKS 5 proxy to go through port 8080, and finally in the firefox address bar type http://localhost:8080/solr .

Jetty (java webserver)

Let’s continue with some Jetty config. Edit nano /etc/default/jetty and add this:

NO_START=0 # Start on boot
JAVA_OPTIONS="-Dsolr.solr.home=/opt/solr/solr $JAVA_OPTIONS"
JAVA_HOME=/usr/java/default
JETTY_HOME=/opt/solr
JETTY_USER=solr
JETTY_LOGS=/opt/solr/logs

Now nano /opt/solr/etc/jetty-logging.xml  and add:

<?xml version="1.0"?>
  <!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">
  <!-- =============================================================== -->
  <!-- Configure stderr and stdout to a Jetty rollover log file -->
  <!-- this configuration file should be used in combination with -->
  <!-- other configuration files.  e.g. -->
  <!--    java -jar start.jar etc/jetty-logging.xml etc/jetty.xml -->
  <!-- =============================================================== -->
  <Configure id="Server" class="org.mortbay.jetty.Server">

      <New id="ServerLog" class="java.io.PrintStream">
        <Arg>
          <New class="org.mortbay.util.RolloverFileOutputStream">
            <Arg><SystemProperty name="jetty.logs" default="."/>/yyyy_mm_dd.stderrout.log</Arg>
            <Arg type="boolean">false</Arg>
            <Arg type="int">90</Arg>
            <Arg><Call class="java.util.TimeZone" name="getTimeZone"><Arg>GMT</Arg></Call></Arg>
            <Get id="ServerLogName" name="datedFilename"/>
          </New>
        </Arg>
      </New>

      <Call class="org.mortbay.log.Log" name="info"><Arg>Redirecting stderr/stdout to <Ref id="ServerLogName"/></Arg></Call>
      <Call class="java.lang.System" name="setErr"><Arg><Ref id="ServerLog"/></Arg></Call>
      <Call class="java.lang.System" name="setOut"><Arg><Ref id="ServerLog"/></Arg></Call></Configure>

Then, create the Solr user and grant it permissions:

sudo useradd -d /opt/solr -s /sbin/false solr
sudo chown solr:solr -R /opt/solr

After that, download the start file and set it to automatically start up:

sudo wget -O /etc/init.d/jetty http://dev.eclipse.org/svnroot/rt/org.eclipse.jetty/jetty/trunk/jetty-distribution/src/main/resources/bin/jetty.sh
sudo chmod a+x /etc/init.d/jetty
sudo update-rc.d jetty defaults

Finally, try to start Jetty/Solr again with the new way: sudo service jetty start and try to access the admin panel. Nice!

Solr 4.10

Now let’s configure Solr. We’ll rename the default “Schema” and download a new one that is customized for Dovecot.

mv /opt/solr/solr/collection1/conf/schema.xml opt/solr/solr/collection1/conf/schema.xml-dist
cd opt/solr/solr/collection1/conf/
wget https://raw.githubusercontent.com/extremeshok/solr-dovecot2/master/schema.xml

That’s good thing done. Now as I store my email in an encrypted (ecryptfs) Private folder, I also want my solr index to be encrypted, right? So let’s symlink the data folder into my encrypted Private folder: ln -s /home/user/Private/mail/solr/data /opt/solr/solr/collection1/data and be sure to chown solr:solr the new folders properly.

Dovecot 2.2

Now let’s configure Dovecot. First they mention some pre-requisite on the wiki page, so let’s install them (they may already be installed) :

apt-get install libcurl4-openssl-dev libexpat1-dev

then let’s enable solr in Dovecot config by nano /etc/dovecot/conf.d/10-mail.conf and find/add this line mail_plugins = fts fts_solr . And set the plugin settings by nano /etc/dovecot/conf.d/90-plugins and add/edit/replace with

plugin {
  fts = solr
  fts_solr = url=http://127.0.0.1:8983/solr/ break-imap-search
  fts_autoindex = yes
}

break-imap-search : Use Solr also for indexing TEXT and BODY searches. This makes your server non-IMAP-compliant. (This is always enabled in v2.1+)¬† It’s a good thing.

Now we need to compile Dovecot as for me, under ¬†a fresh Ubuntu 14.04 my Dovecot didn’t have Solr support ! but it’s very simple, so don’t be scared (it will take around 10~15mn with a low config server):

apt-get source dovecot
cd dovecot-2.2.9
sudo apt-get build-dep package
debuild -us -uc -b
cd ..
dpkg -i dovecot-core_2.2.9-1ubuntu2.1_amd64.deb dovecot-solr_2.2.9-1ubuntu2.1_amd64.deb

After having reinstalled Dovecot and installed Solr support, you can verify your dovecot conf directory and version is still the same as before:

doveconf -n | head -n1
# 2.2.9: /etc/dovecot/dovecot.conf

Rock it!

Restart everything,

service jetty restart
service dovecot restart

and try the new Solr search by logging into your imap server a1 login user@server password¬†and searching with a3 SEARCH text “test”¬†see below for the full process:

openssl s_client -connect localhost:993
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot (Ubuntu) ready.
a1 login user@server password
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY] Logged in
a2 select Inbox
a3 SEARCH text "test"
* OK Indexed 29% of the mailbox, ETA 0:25
* OK Indexed 37% of the mailbox, ETA 0:34
* OK Indexed 89% of the mailbox, ETA 0:03
* OK Indexed 97% of the mailbox, ETA 0:01
* OK Indexed 97% of the mailbox, ETA 0:01
* OK Indexed 97% of the mailbox, ETA 0:01
* OK Indexed 97% of the mailbox, ETA 0:02
* OK Indexed 97% of the mailbox, ETA 0:02
* OK Indexed 97% of the mailbox, ETA 0:02
* OK Indexed 97% of the mailbox, ETA 0:03
* OK Indexed 97% of the mailbox, ETA 0:03
* OK Mailbox indexing finished
* SEARCH 1 2 3 4 7 8 9 10 11 12 13 14 17 18 22 23 26 27 33 34 56 57 58 59 60 77 83 85 88 118 122 123 126 128 156 178 179 183 186 191 199 200 212 213 225 245 247 254 281 282 286 287 289 305 309 311 312 315 317 320 329 333 357 358 361 364 374 375 379 387 390 401 405 408 414 415 416 428 433 439 442 444 445 465 494 495 496 500 504 533 537 538 542 543 549 557 561 566 567 568 569 575 580 603 620 621 636 641 642 643 668 677 679 684 685 686 687 691 701 705 710 715 718 721 722 734 736 743 747 748 753 754 773 774 775 776 777 778 779 780 784 785 786 787 792 795 813 820 821 829 830 831 843 844 855 861 862 863 864 865 870 871 886 889 890 891 892 893 894 895 899 906 914 915 924 926 930 933 937 938 939 940 941 946 947 979 987 988 990 991 1004 1005 1006 1007 1012 1013 1014 1018 1027 1028 1030 1037 1047 1054 1055 1059 1060 1061 1063 1064 1068 1069 1070 1071 1072 1074 1080 1081 1082 1083 1085 1087 1092 1093 1104 1111 1112 1113 1117 1118 1120 1140 1142 1143 1146 1152 1153 1154 1165 1176 1184 1185 1189 1201 1202 1210 1211 1212 1217 1228 1237 1238 1240 1243 1244 1245 1246 1247 1248 1249 1250 1252 1258 1264 1265 1268 1269 1279 1280 1284 1291 1298 1299 1300 1302 1306 1314 1315 1317 1322 1331 1338
a3 OK Search completed (117.978 secs).
a4 SEARCH text "test"
* SEARCH 1 2 3 4 7 8 9 10 11 12 13 14 17 18 22 23 26 27 33 34 56 57 58 59 60 77 83 85 88 118 122 123 126 128 156 178 179 183 186 191 199 200 212 213 225 245 247 254 281 282 286 287 289 305 309 311 312 315 317 320 329 333 357 358 361 364 374 375 379 387 390 401 405 408 414 415 416 428 433 439 442 444 445 465 494 495 496 500 504 533 537 538 542 543 549 557 561 566 567 568 569 575 580 603 620 621 636 641 642 643 668 677 679 684 685 686 687 691 701 705 710 715 718 721 722 734 736 743 747 748 753 754 773 774 775 776 777 778 779 780 784 785 786 787 792 795 813 820 821 829 830 831 843 844 855 861 862 863 864 865 870 871 886 889 890 891 892 893 894 895 899 906 914 915 924 926 930 933 937 938 939 940 941 946 947 979 987 988 990 991 1004 1005 1006 1007 1012 1013 1014 1018 1027 1028 1030 1037 1047 1054 1055 1059 1060 1061 1063 1064 1068 1069 1070 1071 1072 1074 1080 1081 1082 1083 1085 1087 1092 1093 1104 1111 1112 1113 1117 1118 1120 1140 1142 1143 1146 1152 1153 1154 1165 1176 1184 1185 1189 1201 1202 1210 1211 1212 1217 1228 1237 1238 1240 1243 1244 1245 1246 1247 1248 1249 1250 1252 1258 1264 1265 1268 1269 1279 1280 1284 1291 1298 1299 1300 1302 1306 1314 1315 1317 1322 1331 1338
a4 OK Search completed (0.002 secs).
a5 LOGOUT

Yeah! 0,002 secs instead of 117,98 secs. Now you can go log in Roundcube or open Thunderbird, and start leverage the FTS to datamine your mails.

Because Solr index needs to be optimized, and Dovecot doesn’t tell Solr to do it, you should add a cron job crontab -e¬†with these commands:

# Optimize should be run somewhat rarely, e.g. once a day at 2:30 am
30 2 * * * curl http://localhost:8983/solr/update?optimize=true
# Commit should be run pretty often, e.g. every minute
*/1 * * * * curl http://localhost:8983/solr/update?commit=true

If you want to index again your mailbox (for whatever reason), use this command doveadm fts rescan -u <username> and then log into dovecot server and do a search (as above). It will start indexing the mailbox.

Note : Thunderbird builds its own local index, if you want to send a IMAP search to Dovecot (that will search using Solr) you can launch the search box using ctrl+shift+f

 Thanks

This article was inspired from Extremshok and digitalocean. All the mail configuration was done by mailinabox. Thanks to them.

iRedMail can also be used to automagically setup the mailserver.