Summer Nights

Let's make the world a better place using digital technologies!

Manually Upgrading the owncloud Instance on Your Linux Server

As I have to do it every now and then, here are the important steps of the upgrade workflow:

  • Backup the whole folder (/var/www/owncloud) using, e.g.
    cp -a owncloud backup_folder
  • Activate the maintenance mode (in owncloud folder):
    sudo -u www-data php occ maintenance:mode --on
  • Download the new owncloud archive and “replace” the existing folder (including .htaccess file)
  • Replace back the config/config.php file from the backup_folder and the data folder
  • Run the upgrade:
    sudo -u www-data php occ upgrade
  • Switch back to normal mode:
    sudo -u www-data php occ maintenance:mode --off
Continue Reading

Accessing Remote Desktop Served by VBoxHeadless via SSH

– Make sure Xming is installed and connectable via SSH (putty) on Windows
– Execute the following, given that the name of the virtual machine is “Windows 7”:
VBoxHeadless --startvm "Windows 7"

You can connect to this machine via rdesktop locally, using swiss german keyboard, and forward it over ssh through xming by:
rdesktop -k de-ch localhost:3389

In case, there is an “unable to connect” error when you try to connect to the machine on port 3389, this could be due to outdated extensions pack. Solve it by following these steps:

Download the extension pack for your version from http://download.virtualbox.org/virtualbox.

Uninstall the old one:

VBoxManage extpack uninstall "Oracle VM VirtualBox Extension Pack"

Install the new one:

VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.1.xx.vbox-extpack

This should solve the issue.

In case, you need to update the guest additions on the guest OS (e.g., Windows), download the matching guest additions and then install directly from within the guest OS. You would most probably need to expand the ISO image.

Continue Reading

Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04

Solr Configuration

  • Download solr binary from http://lucene.apache.org/solr/downloads.html, extract into a temporary location and run (you still need to reference to the original binary!)
    ./install_solr_service.sh solr-6.5.0.zip -i /opt -d /var/solr -u solr -s solr -p YOUR_SOLR_PORT

    to install the solr and run as a service. You can choose the solr port as you like.

  • Important: the binaries are located in the /opt/solr (which is a symbolic link to /opt/solr-6.5.0) and the data as well as the logs are located in /var/solr
  • Make sure that you are running the latest version of dovecot. I tested it with 2.2.22 and it worked but did not work with 2.2.12.
  • Create a “solr” user and group, and make a home folder for the user:
    • groupadd solr
    • useradd -g solr solr
    • mkdir /home/solr
    • chown solr:solr /home/solr
  • Configure some basic solr settings in  /etc/default/solr.in.sh file.
  • Mine are shown below:
  • SOLR_PID_DIR="/var/solr"                 
    SOLR_HOME="/var/solr/data"               
    LOG4J_PROPS="/var/solr/log4j.properties"
    SOLR_LOGS_DIR="/var/solr/logs"           
    SOLR_PORT="YOUR_SOLR_PORT"                         
    HOME="/home/solr"
  • Restart solr by /etc/init.d/solr resart and verify that it is running by opening
    http://SERVERIP:YOUR_SOLR_PORT/solr
  • Now we need to create a solr core in order to process the data.
  •  solr create_core -c dovecot
  • This will create a solr core in /var/solr/data/data. Verify that a new folder dovecot has been created there.
  • Go to /var/solr/data/dovecot/conf, deleted the managed-schema.xml and create a new schema.xml file and paste the following schema into it.
  • <?xml version="1.0" encoding="UTF-8" ?>                                                                  
                                                                                                             
    <!--                                                                                                     
    For fts-solr:                                                                                            
                                                                                                             
    This is the Solr schema file, place it into solr/conf/schema.xml. You may                                
    want to modify the tokenizers and filters.                                                               
    -->                                                                                                      
    <schema name="dovecot" version="1.5">                                                                    
    <types>                                                                                                  
    <!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs -->                             
    <fieldType name="string" class="solr.StrField" />                                                        
    <fieldType name="long" class="solr.TrieLongField" />                                                     
    <fieldType name="boolean" class="solr.BoolField" />                                                      
                                                                                                             
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">                                
    <analyzer type="index">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    <analyzer type="query">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>      
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    </analyzer>                                                                                              
    </fieldType>                                                                                             
    </types>                                                                                                 
    <fields>                                                                                                 
    <field name="id" type="string" indexed="true" stored="true" required="true" />                           
    <field name="uid" type="long" indexed="true" stored="true" required="true" />                            
    <field name="box" type="string" indexed="true" stored="true" required="true" />                          
    <field name="user" type="string" indexed="true" stored="true" required="true" />                         
                                                                                                             
    <field name="hdr" type="text" indexed="true" stored="false" />                                           
    <field name="body" type="text" indexed="true" stored="false" />                                          
                                                                                                             
    <field name="from" type="text" indexed="true" stored="false" />                                          
    <field name="to" type="text" indexed="true" stored="false" />                                            
    <field name="cc" type="text" indexed="true" stored="false" />                                            
    <field name="bcc" type="text" indexed="true" stored="false" />                                           
    <field name="subject" type="text" indexed="true" stored="false" />                                       
                                                                                                             
    <!-- Used by Solr internally: -->                                                                        
    <field name="_version_" type="long" indexed="true" stored="true"/>                                       
    </fields>                                                                                                
                                                                                                             
    <uniqueKey>id</uniqueKey>                                                                                
    </schema>
  • Now (following the instructions from here), edit the solrconfig.xml file and edit as below:
  • Completely remove this section
    <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
    <str name="defaultFieldType">strings</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Boolean</str>
    <str name="fieldType">booleans</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.util.Date</str>
    <str name="fieldType">tdates</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Long</str>
    <str name="valueClass">java.lang.Integer</str>
    <str name="fieldType">tlongs</str>
    </lst>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Number</str>
    <str name="fieldType">tdoubles</str>
    </lst>
    </processor>
  • Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260
    ........
    See http://wiki.apache.org/solr/GuessingFieldTypes-->
    <schemaFactory class="ClassicIndexSchemaFactory">
    </schemaFactory>
  • Restart solr.

Dovecot Configuration

  • First install the solr plugin for dovecot using:
  • apt install dovecot-solr
  • The relevant configuration files for dovecot are located in /etc/dovecot/conf.d/ folder.
  • In the 10-mail.conf file, insert the following:
    mail_plugins = fts fts_solr
  • In the 90-plugin.conf file, insert the following:
  • plugin {                                                                          
      fts = solr                                                                      
      fts_autoindex = yes                                                             
      fts_solr = url=http://SERVERIP:YOUR_SOLR_PORT/solr/dovecot/ break-imap-search debug
    }
  • Restart dovecot (/etc/init.d/dovecot restart)
  • Index the mail by using, for example:
  • doveadm -v  index -u YOUR_USERNAME Inbox
  • Solr should now be functional. Execute:
  • tail -n100 /var/log/mail.log

    to see the access to solr working by issuing a search command in your favorite mail program (roundcube, horde, etc.)

  • Also the /var/solr/logs contains useful solr log information in case you need to investigate.
Continue Reading

Setting Up SVN on Local Network

Assuming svn is installed on both computers (use apt-get install subversion)

On the Server:

  • Create the root repository container folder, e.g., /usr/src/my_data/svnrepos
  • Create the user that you’ll use to access the repository remotely.
  • adduser --no-create-home --ingroup subversion rpi (select passowrd. Create group subversion beforehand if needed)
  • smbpasswd -a rpi(use the same password if you like)
  • chown -R root:subversion svnrepos (chown to root and subversion group where the user should also belong)
  • Share the above folder using samba with the “force group” and “valid users” commands. Also change the masks to 0775.
  • Create svn respositry: svnadmin create raspi
  • Share top svn folder (/usr/src/my_data/svnrepos) using samba

On the Client:

  • Mount shared folder under, e.g., /mnt/svnrepos using, e.g.:
    //192.168.0.10/svnrepos /mnt/svnrepos cifs user=rpi,pass=password 0 0
  • Mount it using:
    mount -a
  • Go to where you want to create the repositorycheckout the respository, e.g. /usr/src
  • Check out the repository using:
    svn checkout file:///mnt/svnrepos/raspi
Continue Reading

Installing a new ssl Certificate on your own Linux Server

I use startssl as they provide an excellent free ssl certificate for private use. You can choose any that you like.

  • Register on startssl and start domain validation process
  • Once, the domain is validated, e.g., via email, follow these steps to create an ssl CSR (certificate request)
    openssl req -newkey rsa:2048 -keyout yourdomainname.key -out yourdomainname.csr
  • Note: the key and csr files can be named as you like.
  • Unencrypt the yourdomainname.key file using the following command:
    openssl rsa -in yourdomainname.key -out yourdomanprivate.key
  • Copy the yourdomainprivate.key file to your Apache installation (e.g., /etc/apache2/ssl)
  • Start the process of ssl certificate creation on startssl and paste your csr in the box provided.
  • Download the generated zip file and unzip to find the Apache (or whatever server your are using) version.
  • copy the 1_root_bundle.crt and yourdomainname.crt files to your apache ssl directory (e.g., /etc/apache2/ssl)
  • Update the key names in the sites-enabled config file like:
    SSLCertificateFile /etc/apache2/ssl/yourdomainname.crt                           
    SSLCertificateKeyFile /etc/apache2/ssl/yourdomainprivate.key                        
    SSLCertificateChainFile /etc/apache2/ssl/1_root_bundle.crt
  • Restart Apache by:
    service apache2 restart
  • Note: Even when you are renewing with startssl, the process is the same as first time certificate creation.
Continue Reading

Using sshfs to mount Linux shares on Windows over ssh/Putty

For really long time, I’d been looking for a simpler way to access Linux shares on Windows over ssh. It’s tedious process to map them via Samba/CIFS driver on Windows that requires setting up a virtual network adapter to route the requests via Putty to the remote server. In enterprise environment, you’ve issues with this approach when the sharing is combined with local network shares.

Fortunately, some geeky souls have been working on this problem and we now have a good solution. I’m particulary thankful to the maintainers of the following projects:

Dokany is a windows file driver that provides file system functions to a file system application.
WinSshFS is a windows application that, together with Dokany, maps a remote directory over ssh as a local windows drive.

Steps:

  • Configure ssh access on Linux
  • Confirm that it works using Putty on Windows
  • Configure Pageant with the ssh private key of the Linux user who will be accessing the share
  • Make sure that user has access to the share on Linux
  • Download and install Dokany driver on Windows
  • Download and run WinSshFS
  • Map the drive in the WinSshFS, see image below for hint.

Important: Make sure you use mutually compatible versions of both Dokany and WinSshFS.

Continue Reading

Share Data using FTP Server

First create an appropriate user, with no shell access, belonging to ftp-users group and chrooted jail at /srv/ftp/xxxx:

useradd ftp-userxx -G ftp-users -s /bin/false -d /srv/ftp/xxxx

Now assign a password to the user, and make it the owner of the shared folder:
passwd ftp-userxx
chown -R ftp-userxx /srv/ftp/xxxx

Now copy the files to be shared into the respective folder. After that change the permissions to read-only. Be careful! we need +x in order to be able to do chroot:
chmod -R 544 /srv/ftp/xxxx

Assuming we are doing it using proftp, the following should be given:

  • proftpd running
  • Port 21 open
  • In /etc/proftpd/proftpd.conf file: RequireValidShell off
  • And to keep the users in the chrooted jail: DefaultRoot ~
Continue Reading

Reset USB 3.0 Device from the Linux Shell

If you are getting errors like:
cdc_acm 3-11:1.2: acm_submit_read_urb - usb_submit_urb failed: -12
or
xhci_hcd 0000:00:14.0: ERROR no room on ep ring
or
tty_port_close_start: tty->count = 1 port count = 0.

then most probably xhci (USB3) module is causing this problem. Generally you can fix this problem (at least temporarily) by removing and re-inserting the USB connector. However, it would be much better to do it from the command line. It is possible by doing:
echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/unbind
and then
echo -n "0000:00:14.0" | tee /sys/bus/pci/drivers/xhci_hcd/bind

Note that “0000:00:14.0” is the id of the malfunctioning USB device that can be different in your case. You can easily find this by inspecting your log (use dmesg).

Continue Reading

php does not show any output

If the php from the linux cli is not showing any output, then most probably it is crashing with a fatal error. It will not show any output if the errors are turned off in /etc/php5/cli/php.ini.

Turn on the errors by setting display_errors = On and you will find out what is causing the problem

Note: If php cli is not working because of a bad configuration, then pear will also not show any output.

Continue Reading

Using svn+ssh from Eclipse through Pageant & Plink under Windows 7 64-bit

So I just want to use my svn repositories that are already configured and ready to use with svn+ssh protocol. I can check them out through TortoiseSVN as I have explained in one of my previous posts. Now the next step is to integrate it directly in the Eclipse IDE. Here are the steps needed to achieve that:

  • Add new site http://subclipse.tigris.org/update_1.8.x through Help->Install New Software in Eclipse.
  • Install both Subclipse and SVNKit.
  • Create a new system variable SVN_SSH with the following value:
    C:\\Program Files\\TortoiseSVN\\bin\\TortoisePlink.exe -ssh -2 -A -l username.
    Take care to match the path and the username to your needs.
  • Restart Eclipse
  • Go to Window->Show View->Other and choose SVN Repositories.

  • Now go in the SVN Repositories tab on the lower side of the IDE and add the path to your repository (svn+ssh://yoursite/repo). If the authentication is working properly, you will now see you remote repository:
  • Now you can right-click on the desired project and Checkout to Eclipse to work on it. All the usual SVN commands will be available through Team menu.
Continue Reading