Summer Nights

Let's make the world a better place using digital technologies!

Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04

Solr Configuration

  • Download solr binary from http://lucene.apache.org/solr/downloads.html, extract into a temporary location and run (you still need to reference to the original binary!)
    ./install_solr_service.sh solr-6.5.0.zip -i /opt -d /var/solr -u solr -s solr -p YOUR_SOLR_PORT

    to install the solr and run as a service. You can choose the solr port as you like.

  • Important: the binaries are located in the /opt/solr (which is a symbolic link to /opt/solr-6.5.0) and the data as well as the logs are located in /var/solr
  • Make sure that you are running the latest version of dovecot. I tested it with 2.2.22 and it worked but did not work with 2.2.12.
  • Create a “solr” user and group, and make a home folder for the user:
    • groupadd solr
    • useradd -g solr solr
    • mkdir /home/solr
    • chown solr:solr /home/solr
  • Configure some basic solr settings in  /etc/default/solr.in.sh file.
  • Mine are shown below:
  • SOLR_PID_DIR="/var/solr"                 
    SOLR_HOME="/var/solr/data"               
    LOG4J_PROPS="/var/solr/log4j.properties"
    SOLR_LOGS_DIR="/var/solr/logs"           
    SOLR_PORT="YOUR_SOLR_PORT"                         
    HOME="/home/solr"
  • Restart solr by /etc/init.d/solr resart and verify that it is running by opening
    http://SERVERIP:YOUR_SOLR_PORT/solr
  • Now we need to create a solr core in order to process the data.
  •  solr create_core -c dovecot
  • This will create a solr core in /var/solr/data/data. Verify that a new folder dovecot has been created there.
  • Go to /var/solr/data/dovecot/conf, deleted the managed-schema.xml and create a new schema.xml file and paste the following schema into it.
  • <?xml version="1.0" encoding="UTF-8" ?>                                                                  
                                                                                                             
    <!--                                                                                                     
    For fts-solr:                                                                                            
                                                                                                             
    This is the Solr schema file, place it into solr/conf/schema.xml. You may                                
    want to modify the tokenizers and filters.                                                               
    -->                                                                                                      
    <schema name="dovecot" version="1.5">                                                                    
    <types>                                                                                                  
    <!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs -->                             
    <fieldType name="string" class="solr.StrField" />                                                        
    <fieldType name="long" class="solr.TrieLongField" />                                                     
    <fieldType name="boolean" class="solr.BoolField" />                                                      
                                                                                                             
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">                                
    <analyzer type="index">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    <analyzer type="query">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>      
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    </fieldType>                                                                                             
    </types>                                                                                                 
    <fields>                                                                                                 
    <field name="id" type="string" indexed="true" stored="true" required="true" />                           
    <field name="uid" type="long" indexed="true" stored="true" required="true" />                            
    <field name="box" type="string" indexed="true" stored="true" required="true" />                          
    <field name="user" type="string" indexed="true" stored="true" required="true" />                         
                                                                                                             
    <field name="hdr" type="text" indexed="true" stored="false" />                                           
    <field name="body" type="text" indexed="true" stored="false" />                                          
                                                                                                             
    <field name="from" type="text" indexed="true" stored="false" />                                          
    <field name="to" type="text" indexed="true" stored="false" />                                            
    <field name="cc" type="text" indexed="true" stored="false" />                                            
    <field name="bcc" type="text" indexed="true" stored="false" />                                           
    <field name="subject" type="text" indexed="true" stored="false" />                                       
                                                                                                             
    <!-- Used by Solr internally: -->                                                                        
    <field name="_version_" type="long" indexed="true" stored="true"/>                                       
    </fields>                                                                                                
                                                                                                             
    <uniqueKey>id</uniqueKey>                                                                                
    </schema>
  • Now (following the instructions from here), edit the solrconfig.xml file and edit as below:
  • Completely remove this section
    <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
    <str name="defaultFieldType">strings</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Boolean</str>
    <str name="fieldType">booleans</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.util.Date</str>
    <str name="fieldType">tdates</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Long</str>
    <str name="valueClass">java.lang.Integer</str>
    <str name="fieldType">tlongs</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Number</str>
    <str name="fieldType">tdoubles</str>
    </lst>
    </processor>
  • Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260
    ........
    See http://wiki.apache.org/solr/GuessingFieldTypes-->
    <schemaFactory class="ClassicIndexSchemaFactory">
    </schemaFactory>
  • Restart solr.

Dovecot Configuration

  • First install the solr plugin for dovecot using:
  • apt install dovecot-solr
  • The relevant configuration files for dovecot are located in /etc/dovecot/conf.d/ folder.
  • In the 10-mail.conf file, insert the following:
    mail_plugins = fts fts_solr
  • In the 90-plugin.conf file, insert the following:
  • plugin {                                                                          
      fts = solr                                                                      
      fts_autoindex = yes                                                             
      fts_solr = url=http://SERVERIP:YOUR_SOLR_PORT/solr/dovecot/ break-imap-search debug
    }
  • Restart dovecot (/etc/init.d/dovecot restart)
  • Index the mail by using, for example:
  • doveadm -v  index -u YOUR_USERNAME Inbox
  • Solr should now be functional. Execute:
  • tail -n100 /var/log/mail.log

    to see the access to solr working by issuing a search command in your favorite mail program (roundcube, horde, etc.)

  • Also the /var/solr/logs contains useful solr log information in case you need to investigate.
Continue Reading

Setting Up SVN on Local Network

Assuming svn is installed on both computers (use apt-get install subversion)

On the Server:

  • Create the root repository container folder, e.g., /usr/src/my_data/svnrepos
  • Create the user that you’ll use to access the repository remotely.
  • adduser --no-create-home --ingroup subversion rpi (select passowrd. Create group subversion beforehand if needed)
  • smbpasswd -a rpi(use the same password if you like)
  • chown -R root:subversion svnrepos (chown to root and subversion group where the user should also belong)
  • Share the above folder using samba with the “force group” and “valid users” commands. Also change the masks to 0775.
  • Create svn respositry: svnadmin create raspi
  • Share top svn folder (/usr/src/my_data/svnrepos) using samba

On the Client:

  • Mount shared folder under, e.g., /mnt/svnrepos using, e.g.:
    //192.168.0.10/svnrepos /mnt/svnrepos cifs user=rpi,pass=password 0 0
  • Mount it using:
    mount -a
  • Go to where you want to create the repositorycheckout the respository, e.g. /usr/src
  • Check out the repository using:
    svn checkout file:///mnt/svnrepos/raspi
Continue Reading

Installing a new ssl Certificate on your own Linux Server

I use startssl as they provide an excellent free ssl certificate for private use. You can choose any that you like.

  • Register on startssl and start domain validation process
  • Once, the domain is validated, e.g., via email, follow these steps to create an ssl CSR (certificate request)
    openssl req -newkey rsa:2048 -keyout yourdomainname.key -out yourdomainname.csr
  • Note: the key and csr files can be named as you like.
  • Unencrypt the yourdomainname.key file using the following command:
    openssl rsa -in yourdomainname.key -out yourdomanprivate.key
  • Copy the yourdomainprivate.key file to your Apache installation (e.g., /etc/apache2/ssl)
  • Start the process of ssl certificate creation on startssl and paste your csr in the box provided.
  • Download the generated zip file and unzip to find the Apache (or whatever server your are using) version.
  • copy the 1_root_bundle.crt and yourdomainname.crt files to your apache ssl directory (e.g., /etc/apache2/ssl)
  • Update the key names in the sites-enabled config file like:
    SSLCertificateFile /etc/apache2/ssl/yourdomainname.crt                           
    SSLCertificateKeyFile /etc/apache2/ssl/yourdomainprivate.key                        
    SSLCertificateChainFile /etc/apache2/ssl/1_root_bundle.crt
  • Restart Apache by:
    service apache2 restart
  • Note: Even when you are renewing with startssl, the process is the same as first time certificate creation.
Continue Reading

Using sshfs to mount Linux shares on Windows over ssh/Putty

For really long time, I’d been looking for a simpler way to access Linux shares on Windows over ssh. It’s tedious process to map them via Samba/CIFS driver on Windows that requires setting up a virtual network adapter to route the requests via Putty to the remote server. In enterprise environment, you’ve issues with this approach when the sharing is combined with local network shares.

Fortunately, some geeky souls have been working on this problem and we now have a good solution. I’m particulary thankful to the maintainers of the following projects:

Dokany is a windows file driver that provides file system functions to a file system application.
WinSshFS is a windows application that, together with Dokany, maps a remote directory over ssh as a local windows drive.

Steps:

  • Configure ssh access on Linux
  • Confirm that it works using Putty on Windows
  • Configure Pageant with the ssh private key of the Linux user who will be accessing the share
  • Make sure that user has access to the share on Linux
  • Download and install Dokany driver on Windows
  • Download and run WinSshFS
  • Map the drive in the WinSshFS, see image below for hint.

Important: Make sure you use mutually compatible versions of both Dokany and WinSshFS.

Continue Reading

Share Data using FTP Server

First create an appropriate user, with no shell access, belonging to ftp-users group and chrooted jail at /srv/ftp/xxxx:

useradd ftp-userxx -G ftp-users -s /bin/false -d /srv/ftp/xxxx

Now assign a password to the user, and make it the owner of the shared folder:
passwd ftp-userxx
chown -R ftp-userxx /srv/ftp/xxxx

Now copy the files to be shared into the respective folder. After that change the permissions to read-only. Be careful! we need +x in order to be able to do chroot:
chmod -R 544 /srv/ftp/xxxx

Assuming we are doing it using proftp, the following should be given:

  • proftpd running
  • Port 21 open
  • In /etc/proftpd/proftpd.conf file: RequireValidShell off
  • And to keep the users in the chrooted jail: DefaultRoot ~
Continue Reading

Reset USB 3.0 Device from the Linux Shell

If you are getting errors like:
cdc_acm 3-11:1.2: acm_submit_read_urb - usb_submit_urb failed: -12
or
xhci_hcd 0000:00:14.0: ERROR no room on ep ring
or
tty_port_close_start: tty->count = 1 port count = 0.

then most probably xhci (USB3) module is causing this problem. Generally you can fix this problem (at least temporarily) by removing and re-inserting the USB connector. However, it would be much better to do it from the command line. It is possible by doing:
echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/unbind
and then
echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/bind

Note that “0000:00:14.0” is the id of the malfunctioning USB device that can be different in your case. You can easily find this by inspecting your log (use dmesg).

Continue Reading

php does not show any output

If the php from the linux cli is not showing any output, then most probably it is crashing with a fatal error. It will not show any output if the errors are turned off in /etc/php5/cli/php.ini.

Turn on the errors by setting display_errors = On and you will find out what is causing the problem

Note: If php cli is not working because of a bad configuration, then pear will also not show any output.

Continue Reading

Using svn+ssh from Eclipse through Pageant & Plink under Windows 7 64-bit

So I just want to use my svn repositories that are already configured and ready to use with svn+ssh protocol. I can check them out through TortoiseSVN as I have explained in one of my previous posts. Now the next step is to integrate it directly in the Eclipse IDE. Here are the steps needed to achieve that:

  • Add new site http://subclipse.tigris.org/update_1.8.x through Help->Install New Software in Eclipse.
  • Install both Subclipse and SVNKit.
  • Create a new system variable SVN_SSH with the following value:
    C:\\Program Files\\TortoiseSVN\\bin\\TortoisePlink.exe -ssh -2 -A -l username.
    Take care to match the path and the username to your needs.
  • Restart Eclipse
  • Go to Window->Show View->Other and choose SVN Repositories.

  • Now go in the SVN Repositories tab on the lower side of the IDE and add the path to your repository (svn+ssh://yoursite/repo). If the authentication is working properly, you will now see you remote repository:
  • Now you can right-click on the desired project and Checkout to Eclipse to work on it. All the usual SVN commands will be available through Team menu.
Continue Reading

backur: A Configurable Python Utility for Optimized Incremental Backups

I have been on the lookout for an efficient small utility to incrementally backup my Linux server to an external drive. I have looked at various options like Back in Time and Backup PC. They are however too advanced and too automated for my purpose. My requirements are the following:

  1. Incremental backups
  2. Console based
  3. Preservation of file times, permissions, user and group
  4. Easy configuration
  5. Logging support
  6. Mirror mode

I have been using rsync for some time in bash scripts. As Python is becoming more and more my language of scripting, so I wanted to write it in Python. With that background information, here comes “backur”. backur is very easy to configure and use. backur.conf, as given below, is self-explanatory. It also provides customizable logging including log-file rotation. Th backup and source paths are very easy to enter, and allow spaces, thanks to shlex split method. You can run backur either manually, or through as a cron job, as it suits you.

#! /usr/bin/python

# Title:    backur.py
# Author:   Asif Iqbal (aiqbalrana@gmail.com)
# Date:     11OCT2012
# Info:     "backur" can be used to automatically backup given folder locations
#           to another backup location. backur is easily configurable through a
#           configuration file. backur uses rsync to achieve highly optimized
#           data transfers by doing only the incremental updates.    

import ConfigParser
import re, os
import glob
import logging
import logging.handlers
import subprocess
import datetime
import shlex

def splitNStripArgs(inputStr, charToStrip):
    data = inputStr.split(',')
    retData = []
    for item in data:
        retData.append((item.strip()).strip(charToStrip))
    return retData

# Read configuration data
confpath = os.path.dirname(__file__)
config = ConfigParser.ConfigParser()
config.read(confpath + "/backur.conf")

rsyncflags = splitNStripArgs((config.get("Config", "RSYNCFlags")),'"')
rsyncexe   = config.get("Config", "RSYNC")
folderlist = splitNStripArgs((config.get("Folders", "FolderList")),'"')
backuploc  = config.get("Config","BackupLocation")
backupsize = int(config.get("Config","MaxLogFileSize"))
numbackups = int(config.get("Config","NumBackups"))
logfile    = config.get("Config","LogFile")

# Setup logging
my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler(logfile, maxBytes=backupsize, backupCount=numbackups)
my_logger.addHandler(handler)

# Do the job
my_logger.info("#########################################################################################")
my_logger.info("-----------------------------------------------------------------------------------------")
my_logger.info("Starting backup")
my_logger.info("Time: " + datetime.datetime.now().strftime("%Y-%m-%d %H:%M"))
my_logger.info("-----------------------------------------------------------------------------------------")

# Run individual backups
for jobfolder in folderlist:
    command = rsyncexe + " " + ' '.join(rsyncflags) + " -s \"" + jobfolder + "\" \"" + backuploc + "\""
    my_logger.info("Executing: " + command)
    out = subprocess.check_output(shlex.split(command))
    my_logger.info(out)

my_logger.info("-----------------------------------------------------------------------------------------")
my_logger.info("Backup Complete")
my_logger.info("Time: " + datetime.datetime.now().strftime("%Y-%m-%d %H:%M"))
my_logger.info("-----------------------------------------------------------------------------------------")
my_logger.info("#########################################################################################")

backur can be configure using backur.conf which look like the following:

[Config]
BackupLocation: /media/usb/backup/
LogFile: /var/log/backur.log
MaxLogFileSize: 2097152
NumBackups: 5
RSYNCFlags: "-a",
	        "-v",
	        "--delete"
RSYNC: /usr/bin/rsync

[Folders]
FolderList: "/mnt/Drv1/pics",
	        "/mnt/Drv2/my data",
	        "/mnt/Drv3/docs"

backur can also be downloaded in a zip file. I have written and tested backur only on Linux. However, it should also run on Windows provided that Python is installed on the system. Do let me know about your experiences or issues if you use backur.

Continue Reading

Setting up svn+ssh on Ubuntu Server and Tortoise SVN as client on Windows PC

  1. First you need to setup the basic ssh connectivity using e.g. putty and sshd. This is not the focus of this short report.
  2. On the Windows, use Pageant to cache the ssh key. The target in the “Shortcut to Pageant” should be similar to:
    "C:\Program Files (x86)\PuTTY\pageant.exe" C:\path\private_sshkey
  3. Check again that you can now connect to the server without having to enter your password in the putty.
  4. Install subversion on the server using:
    apt-get install subversion
  5. Create the repositories in a location, let’s say /usr/local/svn.
  6. Make your first repository under svn, lets say proj1. Simply create a folder using mkdir.
  7. Now mark this folder as a repository using:
    svnadmin create /usr/local/svn/proj1
  8. Create a new group called subversion and add yourself and www-data to this group:
    groupadd subversion
    adduser youruid subversion
    adduser root subversion
    adduser www-data subversion
  9. Assign the owner rights of the repositories to the www-data user and the subversion group, as well as make all the subsequently added files belong to the subversion group.
    cd /usr/local/svn
    chown -R www-data:subversion proj1
    chmod -R g+rws proj1
  10. That completes the basic subversion setup. You can check the basic setup by checking-out the newly setup repo locally on the linux server by:
    svn co file:///usr/local/svn/proj1
  11. For the ssh setup, first open the /etc/subversion/config file and uncomment this line:
    ssh = $SVN_SSH ssh -q -o ControlMaster=no
  12. Now open the /usr/local/svn/proj1/conf/svnserve.conf file and make changes according to the following, where realm is whatever authentication realm you want to name it. The realm is not very important as long as you keep it same across different repos where you want to use the same authentication:
    anon-access = none
    auth-access = write
    realm = xxxx
  13. Now go to /etc/init.d and copy the skeleton file to svnserve file. Make appropriate changes in the beginning of  the svnserve file as follows:
    PATH=/sbin:/usr/sbin:/bin:/usr/bin
    DESC="Subversion Server"
    NAME=svnserve
    DAEMON=/usr/bin/$NAME
    DAEMON_ARGS="-d -r /usr/local/svn"
    PIDFILE=/var/run/$NAME.pid
    SCRIPTNAME=/etc/init.d/$NAME
  14. Set the execution bit by: chmod +x svnserve
  15. Now issue the following command to automatically start the svnserve at the bootup:
    update-rc.d svnserve defaults
  16. That is all for the settings on the server side. Start the svnserve using: /etc/init.d/svnserve start
  17. On the windows side, do the following:
    1. Start Pageant and enter the password.
    2. Open Tortoise SVN settings and ask it to use Plink as shown in the image below:
  18. Note that the -l parameter provides the username that should be used for the repository. The ssh key must belong to this user name.
  19. Now checkout the repository by providing the link in the following manner:
    svn+ssh://yourhost/usr/local/svn/proj1
    You must take care to provide the full path to the svn repository. This is because ssh is only tunneling the svn access which means it is similar to accessing it locally on the server itself. Other authentication methods like sasl, etc. allow accessing the repos in a simpler way like svn://yourhost/proj1 but this is not the case with ssh. There might be a way to achieve this also with ssh but I am not aware of it.
  20. That is it. Now you have a functional svn server which can be securely accessed through ssh from anywhere.
Continue Reading