Share your VPN tun0 with a network interface eth0

My laptop and a home server are both in the same room, behind a very restrictive NAT. My VPN provider just provide a GUI interface, so my laptop can connect to google & Co, but my server NOT! So no apt-get update for my home server…so, should we give up? No, a few commands and the problem will be solved, letting your home server (odroid c1) surfing the waves of the free web. Let’s start by allowing forwading in the system:

echo 1 > /proc/sys/net/ipv4/ip_forward
sysctl net.netfilter.nf_conntrack_acct=1

Now we will allow tun0 to forward data, and then create the rule to forward all the packet coming from eth0 to the VPN tun0 (yes, iptable is magic).

sudo iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
sudo iptables -A FORWARD -i eth0 -o tun0 -m conntrack --ctstate NEW -j ACCEPT

Now we will install a simple DHCP server on the laptop, so it will give IP address to the server when connecting the server to the laptop.

sudo apt-get install isc-dhcp-server

Edit the /etc/dhcp/dhcpd.conf  to tell the DHCP server how to attribute the IP to the server (that will connect through eth0 with ethernet cable):

subnet 10.10.0.0 netmask 255.255.255.0 {
range 10.10.0.25 10.10.0.50;
option domain-name-servers 8.8.4.4;
option routers 10.10.0.1;
}

Edit /etc/default/isc-dhcp-server  to tell the DHCP server which Network interface to use:

INTERFACES="eth0"

then check your config is correct with dhcpd -t /etc/dhcp/dhcpd.conf and start/restart the DHCP server:

sudo start isc-dhcp-server

Now set your eth0  ip to the same IP as the gateway IP defined in the DHCP.conf using this command:

sudo ifconfig eth0 10.10.0.1 netmask 255.255.255.0

Only then connect, your server ethernet port to your laptop ethernet port, the server is dhcp client for me, so it will be dispatched an IP address from the DHCP server running on the laptop. Most probably it will be 10.10.0.25. You can connect to your server from your laptop using:

ssh user@10.10.0.25

and from your server, running apt-get update && apt-get dist-upgrade  which is going through your laptop VPN !! BRAVO !!

credit to this and this.

Note : In case your ISC-DHCP-SERVER won’t start

It happened to me the second time I followed this procedure, impossible to get isc-dhcp-server to work, so I ditched it apt-get remove isc-dhcp-server and installed dnsmasq instead apt-get install dnsmasq then edit

nano /etc/dnsmasq.conf

find these lines and edit them like this:

interface=eth0
dhcp-range=10.10.0.50,10.10.0.150,255.255.255.0,12h

then manually set you laptop a fixed ip (10.10.0.1). finally open the needed ports in the firewall with sudo ufw allow bootps and ufw enable before connecting the ethernet of the server to your laptop. You can check the logs by tail -f /var/log/syslog . You can run service dnsmasq restart .

credit for dnsmasq config.

Bonus

To find the IP you server was attributed (so you can ssh into it), you can use

sudo nmap -sP 10.10.0.0/24

To find the gateway of you server (properly redirected to your laptop IP through eth0) do route -n .

because i didn’t research it, dnsmasq do not attribute dns name to the dhcp clients, so you will have to add it with echo “nameserver 8.8.8.8” > /etc/resolv.conf

Extend a (not LVM) partition to use full space of the SDcard

We will use the fdisk utility to delete the too small partition and then recreate a bigger one, making full use of the SDcard space. After what we will resize the file system. All the details are explained here, and below is the set of commands (Credits to mdrjr):

#!/bin/bash

fdisk_first() {
		p2_start=`fdisk -l /dev/mmcblk0 | grep mmcblk0p2 | awk '{print $2}'`
		echo "Found the start point of mmcblk0p2: $p2_start"
		fdisk /dev/mmcblk0 << __EOF__ >> /dev/null
d
2
n
p
2
$p2_start

p
w
__EOF__

		sync
		touch /root/.resize
		echo "Ok, Partition resized, please reboot now"
		echo "Once the reboot is completed please run this script again"
}

resize_fs() {
	echo "Activating the new size"
	resize2fs /dev/mmcblk0p2 >> /dev/null
	echo "Done!"
	echo "Enjoy your new space!"
	rm -rf /root/.resize
}


if [ -f /root/.resize ]; then
	resize_fs
else
	fdisk_first
fi

 

Extend a LVM partition after increasing its virtual disk on Virtualbox

No Linux machine at work? the easy way could be to simply install Virtualbox in one of the PC, create a VDI and install Ubuntu 14.04 in it. But the day will come when you need more space! Here is how to resize it:

Resize Virtualbox VDI

Open Virtualbox, make sure to shutdown your virtual machine. Then open a terminal (here windows):

cd “C:\Program Files\Oracle\VirtualBox”
VBoxManage list vms
VBoxManage clonehd uuid 0000000000000000 backup.vdi
VBoxManage modifyhd uuid 0000000000000000 --resize 204800

This is the Virtualbox official manual of available commands. And this is a website to convert Gb into Mb (–resize takes Mb as input, 200Gb = 204800Mb).

Boot Gparted to resize the partition

After a default install of Ubuntu Server on one physical disk, you will have a SWAP partition, and then an extended partition in which you’ll have your LVM partition (So 3 partitions). Download Gparted .iso, then in your Virtualbox VM settings add a optical drive that point to Gparted.iso, start the VM, press F12 and choose to boot on CD-ROM. Then in Gparted resize first the extended partition to take all the available space, and then same for the LVM partition. Confirm changes, and reboot the VM on its hard drive.

Resize the LVM stack

Let’s resize the PV (Physical Volume) so it takes all the

user@ubuntuServer: sudo su
root@ubuntuServer# pvs
  PV         VG              Fmt  Attr PSize   PFree
  /dev/sda5  ubuntuServer-vg lvm2 a--  199.76g 98.16g
root@ubuntuServer# pvresize /dev/sda5
  Physical volume "/dev/sda5" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Now let’s extend the LV (Logical Volume) to the full size of the PV. First display its name:

root@ubuntuServer:# lvdisplay
  --- Logical volume ---
  LV Path                /dev/ubuntuServer-vg/root
  LV Name                root
  VG Name                ubuntuServer-vg
  LV UUID                HLv3Z1-50dU-h5Cv-w3mD-3025-4CS1-0WBxd4
  LV Write Access        read/write
  LV Creation host, time ubuntuServer, 2014-08-13 13:08:17 +0800
  LV Status              available
  # open                 1
  LV Size                100.59 GiB
  Current LE             25752
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:0

  --- Logical volume ---
  LV Path                /dev/ubuntuServer-vg/swap_1
  LV Name                swap_1
  VG Name                ubuntuServer-vg
  LV UUID                sAT582-dGyq-UeNy-9hgL-p6bY-MkTy-L1EMmN
  LV Write Access        read/write
  LV Creation host, time ubuntuServer, 2014-08-13 13:08:17 +0800
  LV Status              available
  # open                 2
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:1

And then extend it to full size available (100%):

root@ubuntuServer:/home/cartolux# lvextend -l+100%FREE /dev/ubuntuServer-vg/root     
  Extending logical volume root to 198.76 GiB
  Logical volume root successfully resized

Now let’s check filesystem of partition:

root@ubuntuServer:# df -Th
Filesystem                        Type      Size  Used Avail Use% Mounted on
/dev/mapper/ubuntuServer--vg-root ext4      99G   12G  83G   12% /
none                              tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
udev                              devtmpfs  487M  4.0K  487M   1% /dev
tmpfs                             tmpfs     100M  432K   99M   1% /run
none                              tmpfs     5.0M     0  5.0M   0% /run/lock
none                              tmpfs     497M     0  497M   0% /run/shm
none                              tmpfs     100M     0  100M   0% /run/user
/dev/sda1                         ext2      236M  125M   99M  56% /boot

It’s ext4, and as you can see, the filesystem size is still 99G (and not 200G as wanted). So the last step is to extend the filesystem on the whole LV:

root@ubuntuServer:# resize2fs /dev/ubuntuServer-vg/root
resize2fs 1.42.9 (4-Feb-2014)
Filesystem at /dev/ubuntuServer-vg/root is mounted on /; on-line resizing required
old_desc_blocks = 7, new_desc_blocks = 13
The filesystem on /dev/ubuntuServer-vg/root is now 52103168 blocks long.

That’s it! Now you can run df -Th again and see that the available space has increased. Congrats, job done!

Ubuntu server 14.04 + OpenVPN + Android client

We are going to install Openvpn on Ubuntu server 14.04 and then use Android to connect to it (so you can bypass Chinese firewall for example 😉

Install OpenVPN

apt-get install openvpn libssl-dev openssl

We need to allow IPv4 forwarding so the server can send out packets on the VPN’s behalf. let’s nano /etc/sysctl.conf and uncomment this line:

net.ipv4.ip_forward=1

Then sudo sysctl -p to reload the modified conf. Then nano /etc/default/ufw and edit this line:

DEFAULT_FORWARD_POLICY="ACCEPT"

finally nano /etc/ufw/before.rules and edit like this:

# START OPENVPN RULES

# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0]
# Allow traffic from OpenVPN client to eth0
-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
COMMIT
#END OPENVPN RULES


# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines

#OpenVPN rules
-A ufw-before-input -i tun+ -j ACCEPT
-A ufw-before-output -i tun+ -j ACCEPT
-A ufw-before-forward -s 10.8.0.0/24 -j ACCEPT
-A ufw-before-forward -d 10.8.0.0/24 -j ACCEPT

Open the openvpn port with ufw allow 1194/udp and restart ufw service ufw restart .

Create Server Keys

We will use Easy-RSA to generate the server side keys.

apt-get install easy-rsa
make-cadir /etc/openvpn/easy-rsa
cd /etc/openvpn/easy-rsa/
source vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server
cd keys
openvpn --genkey --secret ta.key

Now copy the server keys to the root of the openvpn directory:

cp server.crt server.key ca.crt dh2048.pem ta.key /etc/openvpn/

Create Client Certificates

cd /etc/openvpn/easy-rsa/
source vars
./pkitool client-name

Those commands will create new files int the easy-rsa/keys directory called client-name.crt and client-name.key.

/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/client-name.crt
/etc/openvpn/easy-rsa/keys/client-name.key

These 3 files need to be copied to the client, so the client software can use them to make the connection with the server.  You can use filezilla to download them on the client for example.

Tweak OpenVPN config

Server side

Copy/extract the default Openvpn conf file:

cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
gzip -d /etc/openvpn/server.conf.gz
nano /etc/openvpn/server.conf

nano /etc/openvpn/server.conf  and make the following changes:

;dh dh1024.pem
;change to
dh dh2048.pem

Change this so all your client traffic passes through the VPN.

;push "redirect-gateway def1 bypass-dhcp"
;change to
push "redirect-gateway def1 bypass-dhcp"

Push specific DNS address to your clients.

;push "dhcp-option DNS 208.67.222.222"
;push "dhcp-option DNS 208.67.220.220"
;uncomment and e3dit these as needed to:
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"

Change the following to increase security so the VPN service has restricted access

;user nobody
;group nogroup
;change to
user nobody
group nogroup

Specify where to output the log of openvpn

log-append /var/log/openvpn.log

finally:

service openvpn restart

Android side

Now on your android device, download the official OpenVPN client from F-Droid market (https://f-droid.org/wiki/page/de.blinkt.openvpn), add a profile, and edit the server address in the config (your server IP or FQDN). Give the android the path to the 3 files you previously downloaded. And start the connection!

home openvpn-main-page settings

That’s it, you can enjoy Youtube & co in China.

Add Full Text Search (FTS) to your Dovecot using Solr 4.10 on Ubuntu 14.04

As explained on Dovecot Wiki, dovecot imap server support FTS, one of them use Solr (from Apache Lucene project).

apt-get install openjdk-7-jdk
mkdir /usr/java
ln -s /usr/lib/jvm/java-7-openjdk-amd64 /usr/java/default

Now that Java is installed, let’s install Solr (with Jetty web server embedded):

cd /opt/
wget http://apache.mirrors.tds.net/lucene/solr/4.10.2/solr-4.10.2.tgz
tar xzf solr-4.10.2.tgz
mkdir -p solr
cp -R solr-4.10.2/example/* /opt/solr/

Then you can test your Solr installation, first launch it:

cd /opt/solr
java -jar start.jar

Check if it works by visiting http://YOUR_IP:8983/solr.

Screenshot from 2014-11-05 09:54:48

When it works, go back into your SSH session and close the window with Ctrl+C. The integrated Jetty server that comes with Solr is configured to bind to port 8983 on all IP adresses by default. This configuration is unsafe: anyone could clear your Solr index! Let’s make Jetty listen only on localhost by nano /opt/etc/jetty.xml and add default=”127.0.0.1″ to:

<Call name="addConnector"> <Arg> <New class="org.eclipse.jetty.server.bio.SocketConnector"> <Set name="host"><SystemProperty name="jetty.host" default="127.0.0.1" /></Set> ... ...

Save. Next time you want to access the solr admin, first open a SSH tunnel with your server: ssh -L localhost:8080:127.0.0.1:8983 user@server -N -C   and then set firefox with SOCKS 5 proxy to go through port 8080, and finally in the firefox address bar type http://localhost:8080/solr .

Jetty (java webserver)

Let’s continue with some Jetty config. Edit nano /etc/default/jetty and add this:

NO_START=0 # Start on boot
JAVA_OPTIONS="-Dsolr.solr.home=/opt/solr/solr $JAVA_OPTIONS"
JAVA_HOME=/usr/java/default
JETTY_HOME=/opt/solr
JETTY_USER=solr
JETTY_LOGS=/opt/solr/logs

Now nano /opt/solr/etc/jetty-logging.xml  and add:

<?xml version="1.0"?>
  <!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">
  <!-- =============================================================== -->
  <!-- Configure stderr and stdout to a Jetty rollover log file -->
  <!-- this configuration file should be used in combination with -->
  <!-- other configuration files.  e.g. -->
  <!--    java -jar start.jar etc/jetty-logging.xml etc/jetty.xml -->
  <!-- =============================================================== -->
  <Configure id="Server" class="org.mortbay.jetty.Server">

      <New id="ServerLog" class="java.io.PrintStream">
        <Arg>
          <New class="org.mortbay.util.RolloverFileOutputStream">
            <Arg><SystemProperty name="jetty.logs" default="."/>/yyyy_mm_dd.stderrout.log</Arg>
            <Arg type="boolean">false</Arg>
            <Arg type="int">90</Arg>
            <Arg><Call class="java.util.TimeZone" name="getTimeZone"><Arg>GMT</Arg></Call></Arg>
            <Get id="ServerLogName" name="datedFilename"/>
          </New>
        </Arg>
      </New>

      <Call class="org.mortbay.log.Log" name="info"><Arg>Redirecting stderr/stdout to <Ref id="ServerLogName"/></Arg></Call>
      <Call class="java.lang.System" name="setErr"><Arg><Ref id="ServerLog"/></Arg></Call>
      <Call class="java.lang.System" name="setOut"><Arg><Ref id="ServerLog"/></Arg></Call></Configure>

Then, create the Solr user and grant it permissions:

sudo useradd -d /opt/solr -s /sbin/false solr
sudo chown solr:solr -R /opt/solr

After that, download the start file and set it to automatically start up:

sudo wget -O /etc/init.d/jetty http://dev.eclipse.org/svnroot/rt/org.eclipse.jetty/jetty/trunk/jetty-distribution/src/main/resources/bin/jetty.sh
sudo chmod a+x /etc/init.d/jetty
sudo update-rc.d jetty defaults

Finally, try to start Jetty/Solr again with the new way: sudo service jetty start and try to access the admin panel. Nice!

Solr 4.10

Now let’s configure Solr. We’ll rename the default “Schema” and download a new one that is customized for Dovecot.

mv /opt/solr/solr/collection1/conf/schema.xml opt/solr/solr/collection1/conf/schema.xml-dist
cd opt/solr/solr/collection1/conf/
wget https://raw.githubusercontent.com/extremeshok/solr-dovecot2/master/schema.xml

That’s good thing done. Now as I store my email in an encrypted (ecryptfs) Private folder, I also want my solr index to be encrypted, right? So let’s symlink the data folder into my encrypted Private folder: ln -s /home/user/Private/mail/solr/data /opt/solr/solr/collection1/data and be sure to chown solr:solr the new folders properly.

Dovecot 2.2

Now let’s configure Dovecot. First they mention some pre-requisite on the wiki page, so let’s install them (they may already be installed) :

apt-get install libcurl4-openssl-dev libexpat1-dev

then let’s enable solr in Dovecot config by nano /etc/dovecot/conf.d/10-mail.conf and find/add this line mail_plugins = fts fts_solr . And set the plugin settings by nano /etc/dovecot/conf.d/90-plugins and add/edit/replace with

plugin {
  fts = solr
  fts_solr = url=http://127.0.0.1:8983/solr/ break-imap-search
  fts_autoindex = yes
}

break-imap-search : Use Solr also for indexing TEXT and BODY searches. This makes your server non-IMAP-compliant. (This is always enabled in v2.1+)  It’s a good thing.

Now we need to compile Dovecot as for me, under  a fresh Ubuntu 14.04 my Dovecot didn’t have Solr support ! but it’s very simple, so don’t be scared (it will take around 10~15mn with a low config server):

apt-get source dovecot
cd dovecot-2.2.9
sudo apt-get build-dep package
debuild -us -uc -b
cd ..
dpkg -i dovecot-core_2.2.9-1ubuntu2.1_amd64.deb dovecot-solr_2.2.9-1ubuntu2.1_amd64.deb

After having reinstalled Dovecot and installed Solr support, you can verify your dovecot conf directory and version is still the same as before:

doveconf -n | head -n1
# 2.2.9: /etc/dovecot/dovecot.conf

Rock it!

Restart everything,

service jetty restart
service dovecot restart

and try the new Solr search by logging into your imap server a1 login user@server password and searching with a3 SEARCH text “test” see below for the full process:

openssl s_client -connect localhost:993
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot (Ubuntu) ready.
a1 login user@server password
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY] Logged in
a2 select Inbox
a3 SEARCH text "test"
* OK Indexed 29% of the mailbox, ETA 0:25
* OK Indexed 37% of the mailbox, ETA 0:34
* OK Indexed 89% of the mailbox, ETA 0:03
* OK Indexed 97% of the mailbox, ETA 0:01
* OK Indexed 97% of the mailbox, ETA 0:01
* OK Indexed 97% of the mailbox, ETA 0:01
* OK Indexed 97% of the mailbox, ETA 0:02
* OK Indexed 97% of the mailbox, ETA 0:02
* OK Indexed 97% of the mailbox, ETA 0:02
* OK Indexed 97% of the mailbox, ETA 0:03
* OK Indexed 97% of the mailbox, ETA 0:03
* OK Mailbox indexing finished
* SEARCH 1 2 3 4 7 8 9 10 11 12 13 14 17 18 22 23 26 27 33 34 56 57 58 59 60 77 83 85 88 118 122 123 126 128 156 178 179 183 186 191 199 200 212 213 225 245 247 254 281 282 286 287 289 305 309 311 312 315 317 320 329 333 357 358 361 364 374 375 379 387 390 401 405 408 414 415 416 428 433 439 442 444 445 465 494 495 496 500 504 533 537 538 542 543 549 557 561 566 567 568 569 575 580 603 620 621 636 641 642 643 668 677 679 684 685 686 687 691 701 705 710 715 718 721 722 734 736 743 747 748 753 754 773 774 775 776 777 778 779 780 784 785 786 787 792 795 813 820 821 829 830 831 843 844 855 861 862 863 864 865 870 871 886 889 890 891 892 893 894 895 899 906 914 915 924 926 930 933 937 938 939 940 941 946 947 979 987 988 990 991 1004 1005 1006 1007 1012 1013 1014 1018 1027 1028 1030 1037 1047 1054 1055 1059 1060 1061 1063 1064 1068 1069 1070 1071 1072 1074 1080 1081 1082 1083 1085 1087 1092 1093 1104 1111 1112 1113 1117 1118 1120 1140 1142 1143 1146 1152 1153 1154 1165 1176 1184 1185 1189 1201 1202 1210 1211 1212 1217 1228 1237 1238 1240 1243 1244 1245 1246 1247 1248 1249 1250 1252 1258 1264 1265 1268 1269 1279 1280 1284 1291 1298 1299 1300 1302 1306 1314 1315 1317 1322 1331 1338
a3 OK Search completed (117.978 secs).
a4 SEARCH text "test"
* SEARCH 1 2 3 4 7 8 9 10 11 12 13 14 17 18 22 23 26 27 33 34 56 57 58 59 60 77 83 85 88 118 122 123 126 128 156 178 179 183 186 191 199 200 212 213 225 245 247 254 281 282 286 287 289 305 309 311 312 315 317 320 329 333 357 358 361 364 374 375 379 387 390 401 405 408 414 415 416 428 433 439 442 444 445 465 494 495 496 500 504 533 537 538 542 543 549 557 561 566 567 568 569 575 580 603 620 621 636 641 642 643 668 677 679 684 685 686 687 691 701 705 710 715 718 721 722 734 736 743 747 748 753 754 773 774 775 776 777 778 779 780 784 785 786 787 792 795 813 820 821 829 830 831 843 844 855 861 862 863 864 865 870 871 886 889 890 891 892 893 894 895 899 906 914 915 924 926 930 933 937 938 939 940 941 946 947 979 987 988 990 991 1004 1005 1006 1007 1012 1013 1014 1018 1027 1028 1030 1037 1047 1054 1055 1059 1060 1061 1063 1064 1068 1069 1070 1071 1072 1074 1080 1081 1082 1083 1085 1087 1092 1093 1104 1111 1112 1113 1117 1118 1120 1140 1142 1143 1146 1152 1153 1154 1165 1176 1184 1185 1189 1201 1202 1210 1211 1212 1217 1228 1237 1238 1240 1243 1244 1245 1246 1247 1248 1249 1250 1252 1258 1264 1265 1268 1269 1279 1280 1284 1291 1298 1299 1300 1302 1306 1314 1315 1317 1322 1331 1338
a4 OK Search completed (0.002 secs).
a5 LOGOUT

Yeah! 0,002 secs instead of 117,98 secs. Now you can go log in Roundcube or open Thunderbird, and start leverage the FTS to datamine your mails.

Because Solr index needs to be optimized, and Dovecot doesn’t tell Solr to do it, you should add a cron job crontab -e with these commands:

# Optimize should be run somewhat rarely, e.g. once a day at 2:30 am
30 2 * * * curl http://localhost:8983/solr/update?optimize=true
# Commit should be run pretty often, e.g. every minute
*/1 * * * * curl http://localhost:8983/solr/update?commit=true

If you want to index again your mailbox (for whatever reason), use this command doveadm fts rescan -u <username> and then log into dovecot server and do a search (as above). It will start indexing the mailbox.

Note : Thunderbird builds its own local index, if you want to send a IMAP search to Dovecot (that will search using Solr) you can launch the search box using ctrl+shift+f

 Thanks

This article was inspired from Extremshok and digitalocean. All the mail configuration was done by mailinabox. Thanks to them.

iRedMail can also be used to automagically setup the mailserver.

 

Debugging the iRedmail stack

iRedMail is a script that will take care of all the configuration of a mail server. That’s a lot less trouble for the admin as it’s really automagical! But recently after upgrading Ubunt server 12.04 to 14.04, I couldn’t send/receive my mails anymore.

And as everytime, I completly forgot where are the proper log to consult, which program were installed by iRedMail etc…Here is a post to simplify this next time I’ll have a problem:

iRedMail Debugging

If not able to send/receive mails, first try to connect the IMAP server (dovecot) using K9-mail or Roundcube. (Dovecot log : /etc/log/dovecot.log)

If login is OK, try sending a mail. And look at /etc/log/mail.log and /etc/log/mail.err

If you see :  connect to 127.0.0.1[127.0.0.1]:10024: Connection refused the it means that Amavis is not running (port 10024). Try restarting  /etc/init.d/amavis restart   and check that it’s well running netstat -tap | grep amavisd   if not, it could be that you server do not have enough RAM (iRedMail needs 1Go minimum). In such case, you still can run the mail server, but you’ll have to disable Amavis/ClamAV/SpamAssassin.

If your server has 1Go, then you should run Amavis in debug mode using /etc/init.d/amavis debug   and see the output.

In my case, a folder in /var/lib/amavis didn’t had the appropriate rights and Amavis daemon couldn’t write inside (see below debug output)

/usr/sbin/amavisd-new[29699]: (!!)TROUBLE in pre_loop_hook: QUARANTINEDIR directory /var/lib/amavis/virusmails not writable at /usr/sbin/amavisd-new line 10926.

So I just chown -R amavis: /var/lib/amavis/virusmails   and restarted my server (thanks to ZhangHuangBin). It works great now! Below, a summary of the whole iRedMail stack:

The iRedMail stack

Details of iRedMail installation with Conf files, logs, RC-start scripts…etc

SSL cert keys (size: 2048):
– /etc/ssl/certs/iRedMail_CA.pem
– /etc/ssl/private/iRedMail.key

Mail Storage:
– Root directory: /var/vmail
– Mailboxes: /var/vmail/vmail1
– Backup scripts and copies: /var/vmail/backup

PHP:
* Configuration file: /etc/php5/apache2/php.ini
* Disabled functions: show_source,system,shell_exec,passthru,exec,phpinfo,proc_open

OpenLDAP:
* Configuration files:
– /etc/ldap
– /etc/ldap/slapd.conf
– /etc/ldap/ldap.conf
– /etc/ldap/schema/iredmail.schema
* Log file related:
– /etc/rsyslog.d/iredmail.conf
– /var/log/openldap.log
– /etc/logrotate.d/openldap
* Data dir and files:
– /var/lib/ldap
– /var/lib/ldap/vbonhomme.fr
– /var/lib/ldap/vbonhomme.fr/DB_CONFIG
* RC script:
– /etc/init.d/slapd
* See also:
– /root/iRedMail-0.8.6/conf/ldap_init.ldif

Postfix (basic):
* Configuration files:
– /etc/postfix
– /etc/postfix/aliases
– /etc/postfix/main.cf
– /etc/postfix/master.cf

Postfix (LDAP):
* Configuration files:
– /etc/postfix/ldap/virtual_mailbox_domains.cf
– /etc/postfix/ldap/relay_domains.cf
– /etc/postfix/ldap/transport_maps_domain.cf
– /etc/postfix/ldap/transport_maps_user.cf
– /etc/postfix/ldap/virtual_mailbox_maps.cf
– /etc/postfix/ldap/sender_login_maps.cf
– /etc/postfix/ldap/virtual_alias_maps.cf
– /etc/postfix/ldap/virtual_group_maps.cf
– /etc/postfix/ldap/virtual_group_members_maps.cf
– /etc/postfix/ldap/catchall_maps.cf
– /etc/postfix/ldap/recipient_bcc_maps_domain.cf
– /etc/postfix/ldap/recipient_bcc_maps_user.cf
– /etc/postfix/ldap/sender_bcc_maps_domain.cf
– /etc/postfix/ldap/sender_bcc_maps_user.cf

Policyd (cluebringer):
* Web UI:
– URL: httpS://xxxxxxxxxxx/cluebringer/
– Username: xxxxxXXXXX
– Password: xxxxxXXXXX
* Configuration files:
– /etc/cluebringer/cluebringer.conf
– /etc/cluebringer/cluebringer-webui.conf
* RC script:
– /etc/init.d/postfix-cluebringer
* Database:
– Database name: cluebringer
– Database user: cluebringer
– Database password: xxxxxxXXXXXXXX

* Log file:
– /etc/rsyslog.d/iredmail.conf

Dovecot:
* Configuration files:
– /etc/dovecot/dovecot.conf
– /etc/dovecot/dovecot-ldap.conf (For OpenLDAP backend)
– /etc/dovecot/dovecot-mysql.conf (For MySQL backend)
– /etc/dovecot/dovecot-pgsql.conf (For PostgreSQL backend)
– /etc/dovecot/dovecot-used-quota.conf (For real-time quota usage)
– /etc/dovecot/dovecot-share-folder.conf (For IMAP sharing folder)
* RC script: /etc/init.d/dovecot
* Log files:
– /var/log/dovecot.log
– /var/log/sieve.log
* See also:
– /var/vmail/sieve/dovecot.sieve
– Logrotate config file: /etc/logrotate.d/dovecot

ClamAV:
* Configuration files:
– /etc/clamav/clamd.conf
– /etc/clamav/freshclam.conf
– /etc/logrotate.d/clamav
* RC scripts:
+ /etc/init.d/clamav-daemon
+ /etc/init.d/clamav-freshclam
* Log files:
– /var/log/clamav/clamd.log
– /var/log/clamav/freshclam.log

Amavisd-new:
* Configuration files:
– /etc/amavis/conf.d/50-user
– /etc/postfix/master.cf
– /etc/postfix/main.cf
* RC script:
– /etc/init.d/amavis
* MySQL Database:
– Database name: amavisd
– Database user: amavisd
– Database password: xxxxxxXXXXXX

SpamAssassin:
* Configuration files and rules:
– /etc/mail/spamassassin
– /etc/mail/spamassassin/local.cf

iRedAPD – Postfix Policy Daemon:
* Version: 1.4.2
* Listen address: 127.0.0.1, port: 7777
* Related files:
– /opt/iRedAPD-1.4.2/
– /opt/iredapd/
– /opt/iredapd/etc/settings.py

iRedAdmin – official web-based admin panel:
* Version: 0.3
* Configuration files:
– /usr/share/apache2/iRedAdmin-0.3/
– /usr/share/apache2/iRedAdmin-0.3/settings.py*
* URL:
– https://xxxxxxxxxxx/iredadmin/
* Login account:
– Username: xxxxxxxxxxx, password: xxxxxxxxxxxx
* SQL database account:
– Database name: iredadmin
– Username: iredadmin
– Password: xxxxxxxxxxxxxxxxxxx
* Settings:
– /usr/share/apache2/iRedAdmin-0.3/settings.py
* See also:
– /etc/apache2/conf.d/iredadmin.conf

Roundcube webmail:
* Configuration files:
– /usr/share/apache2/roundcubemail-0.9.5/
– /usr/share/apache2/roundcubemail-0.9.5/config/
* URL:
– http://xxxxxxxxxxxxx/mail/
– https://xxxxxxxxxxxx/mail/ (Over SSL/TLS)
– http://xxxxxxxxxxxxx/webmail/
– https://xxxxxxxxxxxxx/webmail/ (Over SSL/TLS)
* See also:
– /etc/apache2/conf.d/roundcubemail.conf

phpLDAPadmin:
* Configuration files:
– /usr/share/phpldapadmin/config/config.php
* See also:
– /etc/apache2/conf.d/phpldapadmin.conf

 

Hope it will be helpful to you.

Move your mails from one IMAP server to another using IMAPSYNC

I lived for almost 5 years with a @gmail address, and I was happy with that. But recently, personal concerns about privacy convinced me to switch to a self-run mail server (easy install thanks to iRedMail, mail.dir being stored in a ecryptfs container). It works perfectly, but I was still frequently logging back in my Gmail account to search my mail history. So I decided to move all my @gmail mails to my email server…And fortunately IMAPSYNC comes in handy for this task. Here are the few steps  I had to follow to assure a successful migration:

Install dependencies

Download the source code from git

In your /root directory for example:

git clone git://github.com/imapsync/imapsync.git

Then cd into the imapsync directory, and try to compile.

cd imapsync
mkdir dist
make install

The compiling should fail…it’s normal, perl still misses some dependencies:

perl -MCPAN -e "install Authen::NTLM"
perl -MCPAN -e "install Data::Uniqid"
perl -MCPAN -e "install File::Copy::Recursive"
perl -MCPAN -e "install IO::Tee"

Then run the following command to be sure that all the needed dependencies are well installed, if yes, the return will be empty:

perl -mMail::IMAPClient -mDigest::MD5 -mTerm::ReadKey -mIO::Socket::SSL        -mDigest::HMAC_MD5 -mAuthen::NTLM -e -mTime::HiRes        -mData::Uniqid -mURI::Escape -mFile::Copy::Recursive -mIO::Tee ""

Then sudo make install  again, and confirm the installation went smoothly by finding out the version of imapsync  imapsync -version  .

Migrate your emails

Now it’s time to move your email from Gmail to your self-run IMAP server.

New mail account

I personally decided to create a new email on my mail server to specifically store all my gmail mails. So I had to create this new mail, which is a very simple process thanks to the iredadmin web interface bundled in iRedMail.

Tweak some Gmail settings

Gmail runs on top of an IMAP server, but it has its own way to classify mails, beginning with the folders. You can go in the settings of your Gmail account and disable some “custom gmail folder” if you don’t want them to be copied by imapsync during the migration.

Note: You can actually tell imapsync to map specific Gmail IMAP server folder to another folder on your new IMAP server, as well as tell him to not copy some folders, which let you avoid the step below. Refer to this post to see the script.

Selection_040-1024x406

I also deleted all my gmail labels because the “tag” concept doesn’t exists in the IMAP, and so I suspected imapsync would maybe try to convert them into folders (but maybe not!). Anyway, I never really used these labels.

My IMAP server installed by iRedMail is Dovecot (which is supported by IMAPSYNC). All right, here is the final one command that worked for me (omit  –dry –justfolders   if you want to start the migration for real):

imapsync --host1 imap.gmail.com --authmech1 LOGIN --port1 993 --user1 vxxxxxxx.xxxxxxe@gmail.com --password1 "your_gmail_password" --host2 mail.vxxxxxxxe.fr --user2 vxxxxxx-gmail@vxxxxx.fr --password2 "password_of_new_email" --syncinternaldates --ssl1 -ssl2 --noauthmd5 --split1 100 --split2 100 --exclude "All Mail|Spam|Trash" --allowsizemismatch --useheader Message-ID --dry --justfolders

If you want to know more about each statement purpose, refer to this article.

I only had one small problem as I didn’t use the command to map folders together, it created a [GMAIL]/[SENT] folder in my new IMAP server. I used Thunderbird with a DRAGn’DROP to move them into the usual SENT folder of my IMAP server. But next time, I will add this to the command to map the Google directories with the dovecot architecture:

--folder "[Gmail]/Sent Mail" --regextrans2 's/Sent Mail/Sent/' --prefix1 '[Gmail]/' --folder "INBOX"

So I successfully moved 28 000+ emails to my new IMAP server ! 😉

Cisco router basic command and config

Show the hardware + software specifications of the router: show version

Show the current global configuration: show running-config

NAT

Enter the config mode  configure terminal   and only then add a static NAT rule: ip nat inside source static tcp <inside_local_ip> <port> <outside_global_ip> <port>

If you want to remove this NAT rule, run the same command but adding no at the beggining (in config mode as well!).

Refer to this article for more details.

Note: There are two main copies of Cisco Router configuration file. The configuration file where router stores the configuration changes when the router is up and running is called the “running-config” file. The “running-config” file is NOT persistent (stored in RAM), which means that the changes made in the “running-config” while the router is running are not retained after a reboot. To make it persistent after a reboot, we need to save it into the “startup-config” file using this command:

copy running-config startup-config 

Show interface

The show interface  command displays the status of the router’s interfaces. For a summary we can use show ip interface brief

SECURITY

Disable HTTP access to the Cisco router :

configure terminal
no ip http server

Setup Nginx + Passenger + RVM (ruby and rails) + Redmine 2.5

Recently I had to move a Redmine instance on a new server (Ubuntu 14.04).  When I was lost in process overwhelmed by some ruby, bundler and rake commands, I swore to myself that I would write down al the steps once succeeded. Here we are:

RVM

RVM is a program that let you install a precise version of Ruby (the one required by Redmine for example) and then let you partition all ruby gem in a specific folder (as once agin, Redmine required a precise version of each gem).

Run this as a regular user in your home folder:

\curl -sSL https://get.rvm.io | bash -s stable --rails

This will install RVM in one command. Then, you have to source it to use it:

source ~/.rvm/scripts/rvm

Then install Ruby 2.0.0 as required by Redmine:

rvm install 2.0.0

If you install multiple version of Ruby, use rvm use ruby_version  to switch the version of Ruby you want to work with. Then we will create a specific folder called gemset to store Redmine gems and switch RVM focus on this folder:

rvm gemset create gemset_name    # create a gemset
rvm ruby_version@gemset_name  # specify Ruby version and our new gemset

You will have something similar to this .rvm/gems/ruby-2.0.0-p481@redmine2.5/gems/

Note: Running rvm ruby_version@gemset_name  as your regular user before installing any gem (using gem install  or bundle install  is very important. Moreover, you shouldn’t run the command from Byobu/Screen, or you will have this error like “rvm is not a function”.

REDMINE 2.5

Prior to the Redmine install, you can install these program as you will need them (some headers are required for the gem compilation below):

apt-get install git subversion imagemagick libmagickwand-dev mysql-server libmysqlclient-dev

I recommend following the official guide for the installation. The point is to give right on the Redmine2.5 folder to the user that is used by passenger to run your app. You can check which user run your app using ps aux | grep -i passenger  which should be the user who installed RVM, then apply the proper rights:

mkdir -p tmp tmp/pdf public/plugin_assets
sudo chown -R <regular_user>: files log tmp public/plugin_assets
sudo chmod -R 755 files log tmp public/plugin_assets

And then be sure to have select the right version of ruby and the right gemset, and to be in the Redmine2.5 folder before running these commands:

gem install bundler
bundle install --without development test
bundle exec rake generate_secret_token
RAILS_ENV=production bundle exec rake db:migrate
RAILS_ENV=production bundle exec rake redmine:load_default_data

 

first let’s install a version of Nginx compiled with passenger using the official guide of Passenger project.

Passenger help Nginx to handle ruby code. We need to tell Nginx which version of Ruby to use when running the app.

Run this comand /usr/bin/passenger-config –root  and copy its result in the file /etc/nginx/nginx.conf  It should look something like passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;  . Then from your gemset dir run: passenger-config –ruby-command  that will tell you where is the wrapper for the correct version of ruby that you want Nginx+Pasenger to use! According to this, edit you nginx vhost, and add the line:

server {
   listen 80;
   root /var/wwww/redmine2.5/web/public;
   passenger_enabled on;
   passenger_ruby /home/user/.rvm/wrappers/ruby-2.0.0-p247@your_gemset/ruby;
}

So you can run a different version of ruby and a different set gem for every app using a different passenger_ruby line in your nginx vhost. That’s the trick!

BONUS PROXY

If when running gem or bundler the gem server is unreachable because you are behing a proxy (or in China 🙂 then you should tell your bash to go through a proxy using this command:

export http_proxy=http://user:password@host:port

BONUS LDAP Active Directory

Maybe you want to authenticate your redmine users against your company AD LDAP (windows server…bouh). Then I advise you to first discover the LDAP server architecture using a LDAP browser from Linux or Windows. And maybe setup a user than has read access on the LDAP schema, as Redmine ned one. Here is the official documentation to setup LDAP authentication.

And here is my config :

Name : Ldap
Host : ldap server IP
Port : 389
Account : redmine@AD.LOCAL
Password : redmine user password (redmine user should have read access on the ldap)
Base DN : OU=Users_Custom,DC=CTXSUZ,DC=LOCAL
LDAP filter : Empty
Timeout (in seconds) : 5
On-the-fly user creation : checked
 
Attributes
 
Login attribute : sAMAccountName
Firstname attribute : givenName
Lastname attribute : sN
Email attribute : mail

Enjoy 😉

Ubuntu 12.04 + nginx + passenger + SPDY

SPDY is a improvment on HTTP and will soon become HTTP 2.0, most of the major browser support SPDY so if you enable it on your webserver, you will save a hundred milliseconds to your visitor (and to yourself). It cannot hurt right?

But here is the problem, Nginx do not support loadable module, which means that you have to add the SPDY option when compiling it. I personnally don’t like compiling stuff on my small VPS, and I always miss some dependencies. And last, the Nginx you get when doing apt-get install nginx on Ubuntu 12.04 has Passenger support, but not SPDY.

so the cool guys at Phusion provide us an alternative : Nginx downloadable binary with Passenger and SPDY already included. Everything is explained on this page.

The idea is to copy their binary over the apt-get provided nginx binary. And everytime apt-get wants to upgrade you nginx, copy over again.

That simple, and then you can say that your blog is “SPDY READY“!