Let's make the world a better place using digital technologies!

Category: Linux

Simple URL Monitor and Mobile Push Notifications using Pushover

Often we would like to monitor the online status of one or more websites or urls in order to be able to fix any issues as soon as a problem arises. I created the following simple mechanism to probe a list of url’s and send push notifications in case of an error.

#!/usr/bin/env python3

# Script to check url health and send push notifications using pushover servrice
# Author: Dr. Asif Rana (aiqbalrana@gmail.com)
# License: MIT License 
# Date: 20180213, 12:06 CET

import urllib.request
import http.client, urllib
import sys
sitelist = {
  'http://site1url/' : 'site1 label',
  'http://site2url/' : 'site2 label', }
for x in sitelist:
  statuslive = 0
    url = urllib.request.urlopen(x)
    code = url.getcode()
    if (code == 200):
      statuslive = 1
    statuslive = 0
  if (statuslive == 0):
    conn = http.client.HTTPSConnection("api.pushover.net:443")
    conn.request("POST", "/1/messages.json", urllib.parse.urlencode({
    "token": "your pushover.net application token",
    "user": "your pushover.net user key", "
    message": "Site down: " + sitelist[x], }), { "Content-type": "application/x-www-form-urlencoded" })

Ideally, you would run this script on a different server than the site you would like to monitor. A pushover.net account will be needed to get your token and key. In addition, you would need to install the pushover app on your iOS or Android device to get the notifications.

Tested and works great for a simple URL monitoring and getting notifications about website crashes, etc.!

You need to add this script as a cronjob using crontab tool and run it on regular intervals (e.g., every 5 minutes) to check the urls.

Continue Reading

Managing Repeat Offender in OSSEC

In ossec.conf:


where 60, 120, 1440 indicate minutes banned after first, second, and third offense. Adapt these values to your taste.

Continue Reading

Create REST Service on Raspberry Pi with Flask

Creating the REST script using Flask

  • First install Flask on raspi:
pip3 install flask
  • Create a REST service using the following code. This is a very simple service that provides one command. In addition, I’ve created another interface also for providing psutil functions, as a demo.
# Script to provide a rest interface to raspberry 
# Author: Dr. Asif Rana (aiqbalrana@gmail.com) 
# License: MIT License
# Date: 20180217 
from flask import Flask, url_for 
import os 
import sys 
import psutil 
raspicmds = { 
 'reboot' : '/sbin/reboot' 
app = Flask(__name__) 
@app.route('/osinfo/<cmd>', methods=['GET', 'POST']) 
def api_osinfo(cmd): 
 cmdstr = 'psutil.' + cmd 
 cmdresp = eval(cmdstr) 
 return str(cmdresp) 
@app.route('/raspi/<kcmd>', methods=['GET', 'POST']) 
def api_raspicmd(kcmd): 
 if kcmd in raspicmds: 
 os.system("sudo " + raspicmds[kcmd]) 
 return str(kcmd + 'executed') 
if __name__ == '__main__': 
 app.run(debug=True, host='') 
  • Caution: do not expose your raspi to the internet without taking good care of security. This example is just for illustration purposes.
  • Making the script a service (daemon)

  • Create this file: /lib/systemd/system/raspirest.service with following contents:
Description=Raspi REST Interface
  • Add and enable the service to start at boot:
sudo systemctl daemon-reload
sudo systemctl enable raspirest
  • Check if the service is running by calling it at: http://IP:Port/raspi/<yourcmd>
Continue Reading

Using Let’s Encrypt Free SSL Certificates

apt install software-properties-common
add-apt-repository ppa:certbot/certbot
apt update
apt-get install python-certbot-apache
certbot-auto -d www.domain.com -d subdomain.domain.com -d subdomain2.domain.com  -d domain.com

This should install the certificates for all the domains in their respective sections (443) in the apache config file.

Now the certificates should be renewed every 90 days: Test renewal by doing a dry run:
certbot renew --dry-run

If everything works, then configure a fully automatic renewal via a cron job, e.g.,
0 0 15 * * /usr/bin/certbot renew --quiet

Continue Reading
Continue Reading

Creating systemd services for Confluence and Jira

Place the following two files (confluence.service and jira.service) under /etc/systemd/system folder.




After=network.target ossec.service apache2.service mysql.service confluence.service     

Enable and start services using the following commands. The services will be enabled on next bootup.

systemctl enable jira.service
systemctl enable confluence.service
systemctl start jira.service
systemctl start confluence.service

Some other useful commands:

systemctl disable jira.service
systemctl daemon-reload
Continue Reading

Installing Redis for Increasing Owncloud and Gallery Performance

  • Download, make, test, and install latest stable Redis:
cd /tmp
curl -O http://download.redis.io/redis-stable.tar.gz
tar -xvf redis-stable.tar.gz
cd redis-stable
make test
make install
  • Check that it is running by:apt install php-redis
redis-server -v
  • Install Redis support for php
apt install php-redis
  • Check that its working and enabled:
    php --ri redis
  • Add the following to /var/www/owncloud/config/config.php:
'memcache.local' => 'OC\\Memcache\\Redis', 
'filelocking.enabled' => 'true', 
'memcache.locking' => 'OC\\Memcache\\Redis', 
'redis' => 
 array ( 
 'host' => 'localhost', 
 'port' => REDIS_PORT, 

In addition, enable asset pipelining in owncloud by putting in:

$CONFIG = array (

the following line:

asset-pipeline.enabled' => true,
  • Open Admin panel in owncloud and see the logs for any anomalies or to confirm good status.
Continue Reading

Manually Upgrading the owncloud Instance on Your Linux Server

As I have to do it every now and then, here are the important steps of the upgrade workflow:

  • Backup the whole folder (/var/www/owncloud) using, e.g.
    cp -a owncloud backup_folder
  • Activate the maintenance mode (in owncloud folder):
    sudo -u www-data php occ maintenance:mode --on
  • Download the new owncloud archive and “replace” the existing folder (including .htaccess file)
  • Replace back the config/config.php file from the backup_folder and the data folder
  • Run the upgrade:
    sudo -u www-data php occ upgrade
  • Switch back to normal mode:
    sudo -u www-data php occ maintenance:mode --off
Continue Reading

Accessing Remote Desktop Served by VBoxHeadless via SSH

– Make sure Xming is installed and connectable via SSH (putty) on Windows
– Execute the following, given that the name of the virtual machine is “Windows 7”:
VBoxHeadless --startvm "Windows 7"

You can connect to this machine via rdesktop locally, using swiss german keyboard, and forward it over ssh through xming by:
rdesktop -k de-ch localhost:3389

In case, there is an “unable to connect” error when you try to connect to the machine on port 3389, this could be due to outdated extensions pack. Solve it by following these steps:

Download the extension pack for your version from http://download.virtualbox.org/virtualbox.

Uninstall the old one:

VBoxManage extpack uninstall "Oracle VM VirtualBox Extension Pack"

Install the new one:

VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.1.xx.vbox-extpack

This should solve the issue.

In case, you need to update the guest additions on the guest OS (e.g., Windows), download the matching guest additions and then install directly from within the guest OS. You would most probably need to expand the ISO image.

Continue Reading

Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04

Solr Configuration

  • Download solr binary from http://lucene.apache.org/solr/downloads.html, extract into a temporary location and run (you still need to reference to the original binary!)
    ./install_solr_service.sh solr-6.5.0.zip -i /opt -d /var/solr -u solr -s solr -p YOUR_SOLR_PORT

    to install the solr and run as a service. You can choose the solr port as you like.

  • Important: the binaries are located in the /opt/solr (which is a symbolic link to /opt/solr-6.5.0) and the data as well as the logs are located in /var/solr
  • Make sure that you are running the latest version of dovecot. I tested it with 2.2.22 and it worked but did not work with 2.2.12.
  • Create a “solr” user and group, and make a home folder for the user:
    • groupadd solr
    • useradd -g solr solr
    • mkdir /home/solr
    • chown solr:solr /home/solr
  • Configure some basic solr settings in  /etc/default/solr.in.sh file.
  • Mine are shown below:
  • SOLR_PID_DIR="/var/solr"                 
  • Restart solr by /etc/init.d/solr resart and verify that it is running by opening
  • Now we need to create a solr core in order to process the data.
  •  solr create_core -c dovecot
  • This will create a solr core in /var/solr/data/data. Verify that a new folder dovecot has been created there.
  • Go to /var/solr/data/dovecot/conf, deleted the managed-schema.xml and create a new schema.xml file and paste the following schema into it.
  • <?xml version="1.0" encoding="UTF-8" ?>                                                                  
    For fts-solr:                                                                                            
    This is the Solr schema file, place it into solr/conf/schema.xml. You may                                
    want to modify the tokenizers and filters.                                                               
    <schema name="dovecot" version="1.5">                                                                    
    <!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs -->                             
    <fieldType name="string" class="solr.StrField" />                                                        
    <fieldType name="long" class="solr.TrieLongField" />                                                     
    <fieldType name="boolean" class="solr.BoolField" />                                                      
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">                                
    <analyzer type="index">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    <analyzer type="query">                                                                                  
    <tokenizer class="solr.StandardTokenizerFactory"/>                                                       
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>      
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>                 
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>                                      
    <filter class="solr.LowerCaseFilterFactory"/>                                                            
    <filter class="solr.EnglishPossessiveFilterFactory"/>                                                    
    <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>                              
    <filter class="solr.EnglishMinimalStemFilterFactory"/>                                                   
    <field name="id" type="string" indexed="true" stored="true" required="true" />                           
    <field name="uid" type="long" indexed="true" stored="true" required="true" />                            
    <field name="box" type="string" indexed="true" stored="true" required="true" />                          
    <field name="user" type="string" indexed="true" stored="true" required="true" />                         
    <field name="hdr" type="text" indexed="true" stored="false" />                                           
    <field name="body" type="text" indexed="true" stored="false" />                                          
    <field name="from" type="text" indexed="true" stored="false" />                                          
    <field name="to" type="text" indexed="true" stored="false" />                                            
    <field name="cc" type="text" indexed="true" stored="false" />                                            
    <field name="bcc" type="text" indexed="true" stored="false" />                                           
    <field name="subject" type="text" indexed="true" stored="false" />                                       
    <!-- Used by Solr internally: -->                                                                        
    <field name="_version_" type="long" indexed="true" stored="true"/>                                       
  • Now (following the instructions from here), edit the solrconfig.xml file and edit as below:
  • Completely remove this section
    <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
    <str name="defaultFieldType">strings</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Boolean</str>
    <str name="fieldType">booleans</str>
    <lst name="typeMapping">
    <str name="valueClass">java.util.Date</str>
    <str name="fieldType">tdates</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Long</str>
    <str name="valueClass">java.lang.Integer</str>
    <str name="fieldType">tlongs</str>
    <lst name="typeMapping">
    <str name="valueClass">java.lang.Number</str>
    <str name="fieldType">tdoubles</str>
  • Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260
    See http://wiki.apache.org/solr/GuessingFieldTypes-->
    <schemaFactory class="ClassicIndexSchemaFactory">
  • Restart solr.

Dovecot Configuration

  • First install the solr plugin for dovecot using:
  • apt install dovecot-solr
  • The relevant configuration files for dovecot are located in /etc/dovecot/conf.d/ folder.
  • In the 10-mail.conf file, insert the following:
    mail_plugins = fts fts_solr
  • In the 90-plugin.conf file, insert the following:
  • plugin {                                                                          
      fts = solr                                                                      
      fts_autoindex = yes                                                             
      fts_solr = url=http://SERVERIP:YOUR_SOLR_PORT/solr/dovecot/ break-imap-search debug
  • Restart dovecot (/etc/init.d/dovecot restart)
  • Index the mail by using, for example:
  • doveadm -v  index -u YOUR_USERNAME Inbox
  • Solr should now be functional. Execute:
  • tail -n100 /var/log/mail.log

    to see the access to solr working by issuing a search command in your favorite mail program (roundcube, horde, etc.)

  • Also the /var/solr/logs contains useful solr log information in case you need to investigate.
Continue Reading