Summer Nights

A blog on embedded and real-time systems as well as on general computing issues

Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04

Solr Configuration

  • Download solr binary from http://lucene.apache.org/solr/downloads.html, extract into a temporary location and run (you still need to reference to the original binary!)
    ./install_solr_service.sh solr-6.5.0.zip -i /opt -d /var/solr -u solr -s solr -p YOUR_SOLR_PORT

    to install the solr and run as a service. You can choose the solr port as you like.

  • Important: the binaries are located in the /opt/solr (which is a symbolic link to /opt/solr-6.5.0) and the data as well as the logs are located in /var/solr
  • Make sure that you are running the latest version of dovecot. I tested it with 2.2.22 and it worked but did not work with 2.2.12.
  • Create a “solr” user and group, and make a home folder for the user:
    • groupadd solr
    • useradd -g solr solr
    • mkdir /home/solr
    • chown solr:solr /home/solr
  • Configure some basic solr settings in  /etc/default/solr.in.sh file.
  • Mine are shown below:
  • SOLR_PID_DIR="/var/solr"                 
    SOLR_HOME="/var/solr/data"               
    LOG4J_PROPS="/var/solr/log4j.properties"
    SOLR_LOGS_DIR="/var/solr/logs"           
    SOLR_PORT="YOUR_SOLR_PORT"                         
    HOME="/home/solr"
  • Restart solr by /etc/init.d/solr resart and verify that it is running by opening
    http://SERVERIP:YOUR_SOLR_PORT/solr
  • Now we need to create a solr core in order to process the data.
  •  solr create_core -c dovecot
  • This will create a solr core in /var/solr/data/data. Verify that a new folder dovecot has been created there.
  • Go to /var/solr/data/dovecot/conf, deleted the managed-schema.xml and create a new schema.xml file and paste the following schema into it.
  • <?xml version="1.0" encoding="UTF-8" ?>                                                                  
                                                                                                             
    <!--                                                                                                     
    For fts-solr:                                                                                            
                                                                                                             
    This is the Solr schema file, place it into solr/conf/schema.xml. You may                                
    want to modify the tokenizers and filters.                                                               
    -->                                                                                                      
    <schema name="dovecot" version="1.5">                                                                    
    <types>                                                                                                  
    <!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs -->                             
    <fieldType name="string" class="solr.StrField" />                                                        
    <fieldType name="long" class="solr.TrieLongField" />                                                     
    <fieldType name="boolean" class="solr.BoolField" />                                                      
                                                                                                             
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">                                
    <analyzer type="index">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    <analyzer type="query">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>      
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    </fieldType>                                                                                             
    </types>                                                                                                 
    <fields>                                                                                                 
    <field name="id" type="string" indexed="true" stored="true" required="true" />                           
    <field name="uid" type="long" indexed="true" stored="true" required="true" />                            
    <field name="box" type="string" indexed="true" stored="true" required="true" />                          
    <field name="user" type="string" indexed="true" stored="true" required="true" />                         
                                                                                                             
    <field name="hdr" type="text" indexed="true" stored="false" />                                           
    <field name="body" type="text" indexed="true" stored="false" />                                          
                                                                                                             
    <field name="from" type="text" indexed="true" stored="false" />                                          
    <field name="to" type="text" indexed="true" stored="false" />                                            
    <field name="cc" type="text" indexed="true" stored="false" />                                            
    <field name="bcc" type="text" indexed="true" stored="false" />                                           
    <field name="subject" type="text" indexed="true" stored="false" />                                       
                                                                                                             
    <!-- Used by Solr internally: -->                                                                        
    <field name="_version_" type="long" indexed="true" stored="true"/>                                       
    </fields>                                                                                                
                                                                                                             
    <uniqueKey>id</uniqueKey>                                                                                
    </schema>
  • Now (following the instructions from here), edit the solrconfig.xml file and edit as below:
  • Completely remove this section
    <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
    <str name="defaultFieldType">strings</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Boolean</str>
    <str name="fieldType">booleans</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.util.Date</str>
    <str name="fieldType">tdates</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Long</str>
    <str name="valueClass">java.lang.Integer</str>
    <str name="fieldType">tlongs</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Number</str>
    <str name="fieldType">tdoubles</str>
    </lst>
    </processor>
  • Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260
    ........
    See http://wiki.apache.org/solr/GuessingFieldTypes-->
    <schemaFactory class="ClassicIndexSchemaFactory">
    </schemaFactory>
  • Restart solr.

Dovecot Configuration

  • First install the solr plugin for dovecot using:
  • apt install dovecot-solr
  • The relevant configuration files for dovecot are located in /etc/dovecot/conf.d/ folder.
  • In the 10-mail.conf file, insert the following:
    mail_plugins = fts fts_solr
  • In the 90-plugin.conf file, insert the following:
  • plugin {                                                                          
      fts = solr                                                                      
      fts_autoindex = yes                                                             
      fts_solr = url=http://SERVERIP:YOUR_SOLR_PORT/solr/dovecot/ break-imap-search debug
    }
  • Restart dovecot (/etc/init.d/dovecot restart)
  • Index the mail by using, for example:
  • doveadm -v  index -u YOUR_USERNAME Inbox
  • Solr should now be functional. Execute:
  • tail -n100 /var/log/mail.log

    to see the access to solr working by issuing a search command in your favorite mail program (roundcube, horde, etc.)

  • Also the /var/solr/logs contains useful solr log information in case you need to investigate.

Leave a Reply

Your email address will not be published. Required fields are marked *