February 18, 2018
Simple URL Monitor and Mobile Push Notifications using Pushover
Often we would like to monitor the online status of one or more websites or urls in order to be able to fix any issues as soon as a problem arises. I created the following simple mechanism to probe a list of url’s and send push notifications in case of an error.
#!/usr/bin/env python3
# Script to check url health and send push notifications using pushover servrice
# Author: Dr. Asif Rana (aiqbalrana@gmail.com)
# License: MIT License
# Date: 20180213, 12:06 CET
import urllib.request
import http.client, urllib
import sys
sitelist = {
'http://site1url/' : 'site1 label',
'http://site2url/' : 'site2 label', }
for x in sitelist:
statuslive = 0
try:
url = urllib.request.urlopen(x)
code = url.getcode()
if (code == 200):
statuslive = 1
except:
statuslive = 0
if (statuslive == 0):
conn = http.client.HTTPSConnection("api.pushover.net:443")
conn.request("POST", "/1/messages.json", urllib.parse.urlencode({
"token": "your pushover.net application token",
"user": "your pushover.net user key", "
message": "Site down: " + sitelist[x], }), { "Content-type": "application/x-www-form-urlencoded" })
conn.getresponse()
Ideally, you would run this script on a different server than the site you would like to monitor. A pushover.net account will be needed to get your token and key. In addition, you would need to install the pushover app on your iOS or Android device to get the notifications.
Tested and works great for a simple URL monitoring and getting notifications about website crashes, etc.!
You need to add this script as a cronjob using crontab tool and run it on regular intervals (e.g., every 5 minutes) to check the urls.
February 18, 2018
Managing Repeat Offender in OSSEC
In ossec.conf:
<active-response> <repeated_offenders>60,120,1440</repeated_offenders> </active-response>
where 60, 120, 1440 indicate minutes banned after first, second, and third offense. Adapt these values to your taste.
February 18, 2018
Create REST Service on Raspberry Pi with Flask
Creating the REST script using Flask
- First install Flask on raspi:
pip3 install flask
- Create a REST service using the following code. This is a very simple service that provides one command. In addition, I’ve created another interface also for providing psutil functions, as a demo.
# Script to provide a rest interface to raspberry
# Author: Dr. Asif Rana (aiqbalrana@gmail.com)
# License: MIT License
# Date: 20180217
from flask import Flask, url_for
import os
import sys
import psutil
raspicmds = {
'reboot' : '/sbin/reboot'
}
app = Flask(__name__)
@app.route('/osinfo/<cmd>', methods=['GET', 'POST'])
def api_osinfo(cmd):
cmdstr = 'psutil.' + cmd
cmdresp = eval(cmdstr)
return str(cmdresp)
@app.route('/raspi/<kcmd>', methods=['GET', 'POST'])
def api_raspicmd(kcmd):
if kcmd in raspicmds:
os.system("sudo " + raspicmds[kcmd])
return str(kcmd + 'executed')
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
- Caution: do not expose your raspi to the internet without taking good care of security. This example is just for illustration purposes.
-
Making the script a service (daemon)
-
Create this file: /lib/systemd/system/raspirest.service with following contents:
[Unit] Description=Raspi REST Interface [Service] Type=simple ExecStart=/usr/src/scripts/raspirest.py [Install] WantedBy=multi-user.target
- Add and enable the service to start at boot:
sudo systemctl daemon-reload sudo systemctl enable raspirest
- Check if the service is running by calling it at: http://IP:Port/raspi/<yourcmd>
September 10, 2017
Using Let’s Encrypt Free SSL Certificates
apt install software-properties-common
add-apt-repository ppa:certbot/certbot
apt update
apt-get install python-certbot-apache
certbot-auto -d www.domain.com -d subdomain.domain.com -d subdomain2.domain.com -d domain.com
This should install the certificates for all the domains in their respective sections (443) in the apache config file.
Now the certificates should be renewed every 90 days: Test renewal by doing a dry run:
certbot renew --dry-run
If everything works, then configure a fully automatic renewal via a cron job, e.g.,
0 0 15 * * /usr/bin/certbot renew --quiet
August 6, 2017
Updating Nameserver (bind9) Entries for a Sub-Domain in Ubuntu
- Edit
/etc/bind/db.domainname.com - Increase the serial number and add the following line:
subdomain IN A xx.xx.xx.xx named-checkconfservice bind9 restart
May 23, 2017
Creating systemd services for Confluence and Jira
Place the following two files (confluence.service and jira.service) under /etc/systemd/system folder.
confluence.service
[Unit]
Description=Confluence
After=mysql.service
[Service]
Type=forking
User=confluence
PIDFile=/opt/atlassian/confluence/work/catalina.pid
ExecStart=/opt/atlassian/confluence/bin/start-confluence.sh
ExecStop=/opt/atlassian/confluence/bin/stop-confluence.sh
[Install]
WantedBy=default.target
jira.service
[Unit]
Description=Jira
After=network.target ossec.service apache2.service mysql.service confluence.service
[Service]
Type=forking
User=jira
PIDFile=/opt/atlassian/jira/work/catalina.pid
ExecStart=/opt/atlassian/jira/bin/start-jira.sh
ExecStop=/opt/atlassian/jira/bin/stop-jira.sh
[Install]
WantedBy=default.target
Enable and start services using the following commands. The services will be enabled on next bootup.
systemctl enable jira.service systemctl enable confluence.service systemctl start jira.service systemctl start confluence.service
Some other useful commands:
systemctl disable jira.service systemctl daemon-reload
April 26, 2017
Installing Redis for Increasing Owncloud and Gallery Performance
- Download, make, test, and install latest stable Redis:
cd /tmp curl -O http://download.redis.io/redis-stable.tar.gz tar -xvf redis-stable.tar.gz cd redis-stable make make test make install
- Check that it is running by:apt install php-redis
redis-server -v
- Install Redis support for php
apt install php-redis
- Check that its working and enabled:
php --ri redis
- Add the following to /var/www/owncloud/config/config.php:
'memcache.local' => 'OC\\Memcache\\Redis', 'filelocking.enabled' => 'true', 'memcache.locking' => 'OC\\Memcache\\Redis', 'redis' => array ( 'host' => 'localhost', 'port' => REDIS_PORT, ),
In addition, enable asset pipelining in owncloud by putting in:
$CONFIG = array (
the following line:
asset-pipeline.enabled' => true,
- Open Admin panel in owncloud and see the logs for any anomalies or to confirm good status.
April 23, 2017
Manually Upgrading the owncloud Instance on Your Linux Server
As I have to do it every now and then, here are the important steps of the upgrade workflow:
- Backup the whole folder (/var/www/owncloud) using, e.g.
cp -a owncloud backup_folder
- Activate the maintenance mode (in owncloud folder):
sudo -u www-data php occ maintenance:mode --on
- Download the new owncloud archive and “replace” the existing folder (including .htaccess file)
- Replace back the config/config.php file from the backup_folder and the data folder
- Run the upgrade:
sudo -u www-data php occ upgrade
- Switch back to normal mode:
sudo -u www-data php occ maintenance:mode --off
April 13, 2017
Accessing Remote Desktop Served by VBoxHeadless via SSH
– Make sure Xming is installed and connectable via SSH (putty) on Windows
– Execute the following, given that the name of the virtual machine is “Windows 7”:
VBoxHeadless --startvm "Windows 7"
You can connect to this machine via rdesktop locally, using swiss german keyboard, and forward it over ssh through xming by:
rdesktop -k de-ch localhost:3389
In case, there is an “unable to connect” error when you try to connect to the machine on port 3389, this could be due to outdated extensions pack. Solve it by following these steps:
Download the extension pack for your version from http://download.virtualbox.org/virtualbox.
Uninstall the old one:
VBoxManage extpack uninstall "Oracle VM VirtualBox Extension Pack"
Install the new one:
VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.1.xx.vbox-extpack
This should solve the issue.
In case, you need to update the guest additions on the guest OS (e.g., Windows), download the matching guest additions and then install directly from within the guest OS. You would most probably need to expand the ISO image.
April 9, 2017
Solr (6.5.0) and Dovecot (2.2.22) on Ubuntu 16.04
Solr Configuration
- Download solr binary from http://lucene.apache.org/solr/downloads.html, extract into a temporary location and run (you still need to reference to the original binary!)
./install_solr_service.sh solr-6.5.0.zip -i /opt -d /var/solr -u solr -s solr -p YOUR_SOLR_PORT
to install the solr and run as a service. You can choose the solr port as you like.
- Important: the binaries are located in the /opt/solr (which is a symbolic link to /opt/solr-6.5.0) and the data as well as the logs are located in /var/solr
- Make sure that you are running the latest version of dovecot. I tested it with 2.2.22 and it worked but did not work with 2.2.12.
- Create a “solr” user and group, and make a home folder for the user:
- groupadd solr
- useradd -g solr solr
- mkdir /home/solr
- chown solr:solr /home/solr
- Configure some basic solr settings in /etc/default/solr.in.sh file.
- Mine are shown below:
-
SOLR_PID_DIR="/var/solr" SOLR_HOME="/var/solr/data" LOG4J_PROPS="/var/solr/log4j.properties" SOLR_LOGS_DIR="/var/solr/logs" SOLR_PORT="YOUR_SOLR_PORT" HOME="/home/solr"
- Restart solr by /etc/init.d/solr resart and verify that it is running by opening
http://SERVERIP:YOUR_SOLR_PORT/solr
- Now we need to create a solr core in order to process the data.
-
solr create_core -c dovecot
- This will create a solr core in /var/solr/data/data. Verify that a new folder dovecot has been created there.
- Go to /var/solr/data/dovecot/conf, deleted the managed-schema.xml and create a new schema.xml file and paste the following schema into it.
-
<?xml version="1.0" encoding="UTF-8" ?> <!-- For fts-solr: This is the Solr schema file, place it into solr/conf/schema.xml. You may want to modify the tokenizers and filters. --> <schema name="dovecot" version="1.5"> <types> <!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs --> <fieldType name="string" class="solr.StrField" /> <fieldType name="long" class="solr.TrieLongField" /> <fieldType name="boolean" class="solr.BoolField" /> <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPossessiveFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.EnglishMinimalStemFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPossessiveFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.EnglishMinimalStemFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="uid" type="long" indexed="true" stored="true" required="true" /> <field name="box" type="string" indexed="true" stored="true" required="true" /> <field name="user" type="string" indexed="true" stored="true" required="true" /> <field name="hdr" type="text" indexed="true" stored="false" /> <field name="body" type="text" indexed="true" stored="false" /> <field name="from" type="text" indexed="true" stored="false" /> <field name="to" type="text" indexed="true" stored="false" /> <field name="cc" type="text" indexed="true" stored="false" /> <field name="bcc" type="text" indexed="true" stored="false" /> <field name="subject" type="text" indexed="true" stored="false" /> <!-- Used by Solr internally: --> <field name="_version_" type="long" indexed="true" stored="true"/> </fields> <uniqueKey>id</uniqueKey> </schema> - Now (following the instructions from here), edit the solrconfig.xml file and edit as below:
- Completely remove this section
<processor class="solr.AddSchemaFieldsUpdateProcessorFactory"> <str name="defaultFieldType">strings</str> <lst name="typeMapping"> <str name="valueClass">java.lang.Boolean</str> <str name="fieldType">booleans</str> </lst> <lst name="typeMapping"> <str name="valueClass">java.util.Date</str> <str name="fieldType">tdates</str> </lst> <lst name="typeMapping"> <str name="valueClass">java.lang.Long</str> <str name="valueClass">java.lang.Integer</str> <str name="fieldType">tlongs</str> </lst> <lst name="typeMapping"> <str name="valueClass">java.lang.Number</str> <str name="fieldType">tdoubles</str> </lst> </processor>
- Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260
........ See http://wiki.apache.org/solr/GuessingFieldTypes--> <schemaFactory class="ClassicIndexSchemaFactory"> </schemaFactory>
- Restart solr.
Dovecot Configuration
- First install the solr plugin for dovecot using:
-
apt install dovecot-solr
- The relevant configuration files for dovecot are located in /etc/dovecot/conf.d/ folder.
- In the 10-mail.conf file, insert the following:
mail_plugins = fts fts_solr
- In the 90-plugin.conf file, insert the following:
-
plugin { fts = solr fts_autoindex = yes fts_solr = url=http://SERVERIP:YOUR_SOLR_PORT/solr/dovecot/ break-imap-search debug } - Restart dovecot (/etc/init.d/dovecot restart)
- Index the mail by using, for example:
-
doveadm -v index -u YOUR_USERNAME Inbox
- Solr should now be functional. Execute:
-
tail -n100 /var/log/mail.log
to see the access to solr working by issuing a search command in your favorite mail program (roundcube, horde, etc.)
- Also the /var/solr/logs contains useful solr log information in case you need to investigate.