Let's make the world a better place using digital technologies!

Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04

Solr Configuration

  • Download solr binary from http://lucene.apache.org/solr/downloads.html, extract into a temporary location and run (you still need to reference to the original binary!)
    ./install_solr_service.sh solr-6.5.0.zip -i /opt -d /var/solr -u solr -s solr -p YOUR_SOLR_PORT

    to install the solr and run as a service. You can choose the solr port as you like.

  • Important: the binaries are located in the /opt/solr (which is a symbolic link to /opt/solr-6.5.0) and the data as well as the logs are located in /var/solr
  • Make sure that you are running the latest version of dovecot. I tested it with 2.2.22 and it worked but did not work with 2.2.12.
  • Create a “solr” user and group, and make a home folder for the user:
    • groupadd solr
    • useradd -g solr solr
    • mkdir /home/solr
    • chown solr:solr /home/solr
  • Configure some basic solr settings in  /etc/default/solr.in.sh file.
  • Mine are shown below:
  • SOLR_PID_DIR="/var/solr"                 
    SOLR_HOME="/var/solr/data"               
    LOG4J_PROPS="/var/solr/log4j.properties"
    SOLR_LOGS_DIR="/var/solr/logs"           
    SOLR_PORT="YOUR_SOLR_PORT"                         
    HOME="/home/solr"
  • Restart solr by /etc/init.d/solr resart and verify that it is running by opening
    http://SERVERIP:YOUR_SOLR_PORT/solr
  • Now we need to create a solr core in order to process the data.
  •  solr create_core -c dovecot
  • This will create a solr core in /var/solr/data/data. Verify that a new folder dovecot has been created there.
  • Go to /var/solr/data/dovecot/conf, deleted the managed-schema.xml and create a new schema.xml file and paste the following schema into it.
  • <?xml version="1.0" encoding="UTF-8" ?>                                                                  
                                                                                                             
    <!--                                                                                                     
    For fts-solr:                                                                                            
                                                                                                             
    This is the Solr schema file, place it into solr/conf/schema.xml. You may                                
    want to modify the tokenizers and filters.                                                               
    -->                                                                                                      
    <schema name="dovecot" version="1.5">                                                                    
    <types>                                                                                                  
    <!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs -->                             
    <fieldType name="string" class="solr.StrField" />                                                        
    <fieldType name="long" class="solr.TrieLongField" />                                                     
    <fieldType name="boolean" class="solr.BoolField" />                                                      
                                                                                                             
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">                                
    <analyzer type="index">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    <analyzer type="query">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>      
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    </fieldType>                                                                                             
    </types>                                                                                                 
    <fields>                                                                                                 
    <field name="id" type="string" indexed="true" stored="true" required="true" />                           
    <field name="uid" type="long" indexed="true" stored="true" required="true" />                            
    <field name="box" type="string" indexed="true" stored="true" required="true" />                          
    <field name="user" type="string" indexed="true" stored="true" required="true" />                         
                                                                                                             
    <field name="hdr" type="text" indexed="true" stored="false" />                                           
    <field name="body" type="text" indexed="true" stored="false" />                                          
                                                                                                             
    <field name="from" type="text" indexed="true" stored="false" />                                          
    <field name="to" type="text" indexed="true" stored="false" />                                            
    <field name="cc" type="text" indexed="true" stored="false" />                                            
    <field name="bcc" type="text" indexed="true" stored="false" />                                           
    <field name="subject" type="text" indexed="true" stored="false" />                                       
                                                                                                             
    <!-- Used by Solr internally: -->                                                                        
    <field name="_version_" type="long" indexed="true" stored="true"/>                                       
    </fields>                                                                                                
                                                                                                             
    <uniqueKey>id</uniqueKey>                                                                                
    </schema>
  • Now (following the instructions from here), edit the solrconfig.xml file and edit as below:
  • Completely remove this section
    <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
    <str name="defaultFieldType">strings</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Boolean</str>
    <str name="fieldType">booleans</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.util.Date</str>
    <str name="fieldType">tdates</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Long</str>
    <str name="valueClass">java.lang.Integer</str>
    <str name="fieldType">tlongs</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Number</str>
    <str name="fieldType">tdoubles</str>
    </lst>
    </processor>
  • Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260
    ........
    See http://wiki.apache.org/solr/GuessingFieldTypes-->
    <schemaFactory class="ClassicIndexSchemaFactory">
    </schemaFactory>
  • Restart solr.

Dovecot Configuration

  • First install the solr plugin for dovecot using:
  • apt install dovecot-solr
  • The relevant configuration files for dovecot are located in /etc/dovecot/conf.d/ folder.
  • In the 10-mail.conf file, insert the following:
    mail_plugins = fts fts_solr
  • In the 90-plugin.conf file, insert the following:
  • plugin {                                                                          
      fts = solr                                                                      
      fts_autoindex = yes                                                             
      fts_solr = url=http://SERVERIP:YOUR_SOLR_PORT/solr/dovecot/ break-imap-search debug
    }
  • Restart dovecot (/etc/init.d/dovecot restart)
  • Index the mail by using, for example:
  • doveadm -v  index -u YOUR_USERNAME Inbox
  • Solr should now be functional. Execute:
  • tail -n100 /var/log/mail.log

    to see the access to solr working by issuing a search command in your favorite mail program (roundcube, horde, etc.)

  • Also the /var/solr/logs contains useful solr log information in case you need to investigate.

Asif

http://things.m31.ch

View more posts from this author
2 thoughts on “Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04
  1. Znuff

    This also works on Ubuntu 14.04.

    Make sure you run the ‘solr’ binary to create the core as the solr user (if you installed it via the supplied install_solr_service.sh), it refused to work as root (with -force) for me.

     
    Reply
  2. Paweł

    Good work. I’ve installed solr accroding your instruction and it’s working perfectly.
    I have only one issue.
    I got hundreds of new emails every day in /var/solr/Maildir/new/

    A content of the mails is:
    Return-Path:
    X-Original-To: solr
    Delivered-To: solr@mydomain.com
    Received: by mydomain.com (Postfix, from userid 119)
    id CA3E59F72F; Tue, 14 Nov 2017 16:25:01 +0100 (CET)
    DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mydomain.com;
    s=mail; t=1510673101;
    bh=ILrc8aExL+GbrEJvBrMmhn+dNprqWFuXK9xMYm27zjQ=;
    h=From:To:Subject:Date:From;
    b=osxUNX7Fdysu+tLmLpZuN9lmzBTu9hEr/8jiFvVjAVd09Im5WRjv8f04xIto7buT7
    IfhDSm72co+il2PP9rdJPcJOFAjHafdhhJpwuq7OXQYrPG/bZVrfAyHzdEKll+YAZA
    RXDyzx8Z7oD157zCE6MdCnROcst8QF/T4GylJp60=
    From: root@mydomain.com (Cron Daemon)
    To: solr@mydomain.com
    Subject: Cron curl -s http://localhost:8983/solr/dovecot2/update?commit=true
    MIME-Version: 1.0
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: 8bit
    X-Cron-Env:
    X-Cron-Env:
    X-Cron-Env:
    X-Cron-Env:
    Message-Id:
    Date: Tue, 14 Nov 2017 16:25:01 +0100 (CET)

    08

    How can I get rid of them?

     
    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *