Summer Nights

Let's make the world a better place using digital technologies!

backur: A Configurable Python Utility for Optimized Incremental Backups

I have been on the lookout for an efficient small utility to incrementally backup my Linux server to an external drive. I have looked at various options like Back in Time and Backup PC. They are however too advanced and too automated for my purpose. My requirements are the following:

  1. Incremental backups
  2. Console based
  3. Preservation of file times, permissions, user and group
  4. Easy configuration
  5. Logging support
  6. Mirror mode

I have been using rsync for some time in bash scripts. As Python is becoming more and more my language of scripting, so I wanted to write it in Python. With that background information, here comes “backur”. backur is very easy to configure and use. backur.conf, as given below, is self-explanatory. It also provides customizable logging including log-file rotation. Th backup and source paths are very easy to enter, and allow spaces, thanks to shlex split method. You can run backur either manually, or through as a cron job, as it suits you.

#! /usr/bin/python

# Title:
# Author:   Asif Iqbal (
# Date:     11OCT2012
# Info:     "backur" can be used to automatically backup given folder locations
#           to another backup location. backur is easily configurable through a
#           configuration file. backur uses rsync to achieve highly optimized
#           data transfers by doing only the incremental updates.    

import ConfigParser
import re, os
import glob
import logging
import logging.handlers
import subprocess
import datetime
import shlex

def splitNStripArgs(inputStr, charToStrip):
    data = inputStr.split(',')
    retData = []
    for item in data:
    return retData

# Read configuration data
confpath = os.path.dirname(__file__)
config = ConfigParser.ConfigParser() + "/backur.conf")

rsyncflags = splitNStripArgs((config.get("Config", "RSYNCFlags")),'"')
rsyncexe   = config.get("Config", "RSYNC")
folderlist = splitNStripArgs((config.get("Folders", "FolderList")),'"')
backuploc  = config.get("Config","BackupLocation")
backupsize = int(config.get("Config","MaxLogFileSize"))
numbackups = int(config.get("Config","NumBackups"))
logfile    = config.get("Config","LogFile")

# Setup logging
my_logger = logging.getLogger('MyLogger')
handler = logging.handlers.RotatingFileHandler(logfile, maxBytes=backupsize, backupCount=numbackups)

# Do the job"#########################################################################################")"-----------------------------------------------------------------------------------------")"Starting backup")"Time: " +"%Y-%m-%d %H:%M"))"-----------------------------------------------------------------------------------------")

# Run individual backups
for jobfolder in folderlist:
    command = rsyncexe + " " + ' '.join(rsyncflags) + " -s \"" + jobfolder + "\" \"" + backuploc + "\"""Executing: " + command)
    out = subprocess.check_output(shlex.split(command))"-----------------------------------------------------------------------------------------")"Backup Complete")"Time: " +"%Y-%m-%d %H:%M"))"-----------------------------------------------------------------------------------------")"#########################################################################################")

backur can be configure using backur.conf which look like the following:

BackupLocation: /media/usb/backup/
LogFile: /var/log/backur.log
MaxLogFileSize: 2097152
NumBackups: 5
RSYNCFlags: "-a",
RSYNC: /usr/bin/rsync

FolderList: "/mnt/Drv1/pics",
	        "/mnt/Drv2/my data",

backur can also be downloaded in a zip file. I have written and tested backur only on Linux. However, it should also run on Windows provided that Python is installed on the system. Do let me know about your experiences or issues if you use backur.

Continue Reading

Setting up svn+ssh on Ubuntu Server and Tortoise SVN as client on Windows PC

  1. First you need to setup the basic ssh connectivity using e.g. putty and sshd. This is not the focus of this short report.
  2. On the Windows, use Pageant to cache the ssh key. The target in the “Shortcut to Pageant” should be similar to:
    "C:\Program Files (x86)\PuTTY\pageant.exe" C:\path\private_sshkey
  3. Check again that you can now connect to the server without having to enter your password in the putty.
  4. Install subversion on the server using:
    apt-get install subversion
  5. Create the repositories in a location, let’s say /usr/local/svn.
  6. Make your first repository under svn, lets say proj1. Simply create a folder using mkdir.
  7. Now mark this folder as a repository using:
    svnadmin create /usr/local/svn/proj1
  8. Create a new group called subversion and add yourself and www-data to this group:
    groupadd subversion
    adduser youruid subversion
    adduser root subversion
    adduser www-data subversion
  9. Assign the owner rights of the repositories to the www-data user and the subversion group, as well as make all the subsequently added files belong to the subversion group.
    cd /usr/local/svn
    chown -R www-data:subversion proj1
    chmod -R g+rws proj1
  10. That completes the basic subversion setup. You can check the basic setup by checking-out the newly setup repo locally on the linux server by:
    svn co file:///usr/local/svn/proj1
  11. For the ssh setup, first open the /etc/subversion/config file and uncomment this line:
    ssh = $SVN_SSH ssh -q -o ControlMaster=no
  12. Now open the /usr/local/svn/proj1/conf/svnserve.conf file and make changes according to the following, where realm is whatever authentication realm you want to name it. The realm is not very important as long as you keep it same across different repos where you want to use the same authentication:
    anon-access = none
    auth-access = write
    realm = xxxx
  13. Now go to /etc/init.d and copy the skeleton file to svnserve file. Make appropriate changes in the beginning of  the svnserve file as follows:
    DESC="Subversion Server"
    DAEMON_ARGS="-d -r /usr/local/svn"
  14. Set the execution bit by: chmod +x svnserve
  15. Now issue the following command to automatically start the svnserve at the bootup:
    update-rc.d svnserve defaults
  16. That is all for the settings on the server side. Start the svnserve using: /etc/init.d/svnserve start
  17. On the windows side, do the following:
    1. Start Pageant and enter the password.
    2. Open Tortoise SVN settings and ask it to use Plink as shown in the image below:
  18. Note that the -l parameter provides the username that should be used for the repository. The ssh key must belong to this user name.
  19. Now checkout the repository by providing the link in the following manner:
    You must take care to provide the full path to the svn repository. This is because ssh is only tunneling the svn access which means it is similar to accessing it locally on the server itself. Other authentication methods like sasl, etc. allow accessing the repos in a simpler way like svn://yourhost/proj1 but this is not the case with ssh. There might be a way to achieve this also with ssh but I am not aware of it.
  20. That is it. Now you have a functional svn server which can be securely accessed through ssh from anywhere.
Continue Reading

Write your own Python Plugin for Dreambox Newnigma2 and Map to a QuickButton

I have a Samsung TV that is connected to the Dreambox 800SE. The audio output of the dreambox goes to a Sony Blu Ray player. I can turn on the Sony Blu Ray player through Samsung TV remote control but not through Dreambox remote. This always requires also having the Samsung remote handy. What I wanted was to simply map a long press of a quickbutton (e.g., color buttons on the dreambox remote) to a plugin that will send Samsung remote control commands to the Samsung TV. I have written a python script that shows how you can achieve this in another post. The following diagram will explain the setup that I just described.

So actually what I want is to send remote control commands to the Samsung TV through a plugin from the Dreambox which will in turn use my python script to send them to the TV. The TV will then execute them to connect over CEC to the Sony Blu Ray player.

To write the plugin for dreambox, I have used Python. The script development has been explained nicely in this guide which I also used as guideline. The main  component of the plugin is the file which for my case is very simple as shown below.

# My own small Plugin
# Author: Asif Iqbal
# Date: 06 APR 2012

from Plugins.Plugin import PluginDescriptor
import os

def main(session, **kwargs):
        os.system('python /etc/')

def Plugins(**kwargs):
        return PluginDescriptor(
                description="Samsung TV Remote",
                where = PluginDescriptor.WHERE_PLUGINMENU,

As you can see the plugin itself is very simple. It just calls your python script. The Python script as I have explained above can be found in one of my other posts. To use the plugin, follow these steps:

  • Create a folder according to the name of your plugin (in my case ExecuteMyScript)  /usr/lib/enigma2/python/Plugins/Extensions/
  • Put the above explained plugin script under /usr/lib/enigma2/python/Plugins/Extensions/ExecuteMyScript or the folder name that you chose.
  • You also need to place an empty file named in this folder.
  • Make sure that the link to your own script is correct in the file.
  • Now you need to reboot the Dreambox (or restart GUI).
  • After reboot, use the QuickButton Plugin to choose your Plugin (ExecuteMyScript, in my case) to map to the long press of a color button of your choice.
  • That’s it. You have created your own Dreambox plugin and have successfully mapped to a quick button.

A zip file containing both the plugin files ( and the can be found on my download page.

Continue Reading

Samsung TV Remote Control over LAN/WLAN using Python

For quite some time, I have been looking for a method to control my Samsung UE55C8000 TV using Linux scripts. Finally, I found this forum post on SamyGo that provides an example using Perl. As I like to do scripting in Python, so I have translated the script to Python with some minor changes. As you can see in the following Python code, the script contains Key names for all the remote control keys and even some extra. You should be able to easily modify the script to your needs. The script can also be downloaded on my download page.

#!  /usr/bin/python
#   Title:
#   Author: Asif Iqbal
#   Date: 05APR2012
#   Info: To send remote control commands to the Samsung tv over LAN
#   TODO:

import socket
import base64
import time, datetime

#IP Address of TV
tvip = ""
#IP Address of TV
myip = ""
#Used for the access control/validation, but not after that AFAIK
mymac = "00-0c-29-3e-b1-4f"
#What the iPhone app reports
appstring = ""
#Might need changing to match your TV type
tvappstring = ""
#What gets reported when it asks for permission
remotename = "Python Samsung Remote" 

# Function to send keys
def sendKey(skey, dataSock, appstring):
 messagepart3 = chr(0x00) + chr(0x00) + chr(0x00) + chr(len(
base64.b64encode(skey))) + chr(0x00) + base64.b64encode(skey);
 part3 = chr(0x00) + chr(len(appstring)) + chr(0x00) \
+ appstring + chr(len(messagepart3)) + chr(0x00) + messagepart3

# Open Socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((tvip, 55000))

# Key Reference
# Normal remote keys
 #KEY_W_LINK #Media P
 #KEY_RSS #Internet
 #KEY_MTS #Dual
# Bonus buttons not on the normal remote:
#Don't work/wrong codes:

# First configure the connection
ipencoded = base64.b64encode(myip)
macencoded = base64.b64encode(mymac)
messagepart1 = chr(0x64) + chr(0x00) + chr(len(ipencoded)) \
+ chr(0x00) + ipencoded + chr(len(macencoded)) + chr(0x00) \
+ macencoded + chr(len(base64.b64encode(remotename))) + chr(0x00) \
+ base64.b64encode(remotename)

part1 = chr(0x00) + chr(len(appstring)) + chr(0x00) + appstring \
+ chr(len(messagepart1)) + chr(0x00) + messagepart1

messagepart2 = chr(0xc8) + chr(0x00)
part2 = chr(0x00) + chr(len(appstring)) + chr(0x00) + appstring \
+ chr(len(messagepart2)) + chr(0x00) + messagepart2

# Now send the keys as you like, e.g.,

# Close the socket when done


Continue Reading

Remote Desktop to a Virtual Machine on VirtualBox running on a Headless Server

I have followed the excellent directions given by Falko Timme at howtoforge. I am copying his directions here with some changes to install a Windows 7 system.

First you need to add the VirtualBox repo to the Ubuntu resources.lst:
emacs /etc/apt/sources.list
Add the following line to it:
deb oneiric contrib

Then download and install the VirtualBox public key:
wget -q -O- | sudo apt-key add -
Update our package database:
apt-get update
Afterwards, install the VirtualBox 4.1 as follows:
apt-get install linux-headers-$(uname -r) build-essential virtualbox-4.1 dkms

Now install the VirtualBox extensions.

cd /tmp
VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.1.8-75467.vbox-extpack

Now add the user who will run VirtualBox to the vboxusers group:
adduser root vboxusers

Now get a Windows 7 iso and do the following steps to install it to a virtual machine:

VBoxManage createvm --name "Windows 7" --register
VBoxManage modifyvm "Windows 7" --memory 2048 --acpi on --boot1 dvd --nic1 bridged --bridgeadapter1 eth0
VBoxManage createhd --filename windows7.vdi --size 50000
VBoxManage storagectl "Windows 7" --name "IDE Controller" --add ide
VBoxManage storageattach "Windows 7" --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium /mnt/Windows/windows7.vdi
VBoxManage storageattach "Windows 7" --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium /mnt/Windows/Windows7Ultimate.iso
VBoxHeadless --startvm "Windows 7"

You can now connect using Remote Desktop to this virtual machine.

Continue Reading

Useful tmux Commands

tmux is a powerful console window manager. The most valuable feature of tmux is the ability to detach from a running tmux session, exit the ssh session, relogin to ssh again and resume the tmux session. This is simply beautiful. Here I list some handy tmux commands.

  • tmux
    starts the tmux.
  • ^b d
    detaches from a running tmux session.
  • tmux attach
    attaches to a running tmux session.
  • ^b ?
    lists all the key-bindings. Great help.
  • ^b %
    splits the window into two vertical panes.
  • ^b c
    creates a new window.
    switches to right/left panes.
    scrolls the buffer up/down.
  • ^b d
    detaches the current session.
  • tmux attach -d
    attaches to a tmux session, detaching the other clients.

Thank you tmux-team for developing such a useful program.

Continue Reading

Useful rsync Commands

rsync -avn /pathtofolder/'Source Folder' /pathtofolder/'Destination Folder'
dry-runs a mirror of Source Folder to the Destination Folder.

rsync -av /pathtofolder/'Source Folder' /pathtofolder/'Destination Folder'
mirrors Source Folder to the Destination Folder.

rsync -avt --progress /pathtofolder/'Source Folder' /pathtofolder/'Destination Folder'
does the same as above, in addition maintaining time-stamps and printing progress.

Continue Reading

MTS/M2TS to MKV (Especially for Sony HDR-SR12)

After spending many frustrating hours, I finally found a solution to easily re-mux m2ts/mts files produced by Sony HDR-SR12 or other similar camcoders in order to stream them smoothly over DLNA to Samsung C-Series (55C8000) TV in full HD quality. But the solution is generic and can be used for any rendered.


First the problem: Above mentioned Sony camcoder produces 1920×1080, 60i video. However, the time-stamps on the frames in each interlaced field are wrong, both frames of a filed having same time-stamp. This is very problematic for almost all rendering devices.
My first approach was to re-encode these mts files using Handbrake to convert to m4v. This worked though at a heavy computing cost. The conversion is very intensive and for 50GB of videos could easily take more than 12 hours on a core-2 processor. One day, I was browsing Ubuntu forums, I came across this thread. This actually produced the re-muxed file in a fraction of a second as opposed to re-encoding with Handbrake that took very long time.
But there are two problems in the above solution:

  • It is using mencoder + lavf, a known broken combination in presence of B-frames.
  • It does not allow to copy the subtitle track (PGS) that Sony camcoder produces. This subtitle track is very important as it contains the recording date/time
    information which will be valuable when using these videos in future.


The following solution overcomes all the above-mentioned problems, is rock-stable and preserves the sub-title stream as well. I use ffmpeg to simply copy all the streams and output them in an mkv container. No information is lost/re-encoded. Here is the simple answer (as often observed in scientific research, the useful answer to a complex problem can often be stated very simply ):
ffmpeg -i input.mt2s -scodec copy -acodec copy -vcodec copy -f matroska input.mkv

I have written a script that converts all the m2ts files in a subfolder to mkv. The script is as below:
# m2tstomkv_subs (Fix the faulty fps (dual frames in the interlaced fields with same time-stamps) and repackage in matroska mkv container)
time {
# remux m2ts movies into mkv with ffmpeg
for i in *.m2ts; do
ffmpeg -i $i -scodec copy -acodec copy -vcodec copy -f matroska `basename $i .m2ts`.mkv;
echo "Conversion done";

The conversion is lightening fast. Finally I am able to watch the videos in their true quality. I will be glad if this information helps someone facing similar issues.

Continue Reading

Horde Keeps Rejecting the Login

If you unable to login to your horde (groupware webmail edition, or other versions as well) installation and receive messages like:

Login failed because your username or password was entered incorrectly.

or in /tmp/horde.log file you have something similar to:

Oct 29 12:18:42 HORDE [error] [imp] FAILED LOGIN for iqa [100.xx.xx.xx] to {localhost:143 [imap/notls]} [pid xxxxx on line 139 of “/var/www/horde/imp/lib/Auth/imp.php”]

then before digging deeper into horde internals, it is worthwhile to check whether your imap daemon (dovecot-imapd in my case) is running because if you have configured in many installations imap is the default authentication mechanism that horde uses. It can be checked quite easily by issuing:
telnet localhost 143
And if the response is like the following:
telnet: Unable to connect to remote host: Connection refused

then you do not have an imap daemon running. And that is preventing the horde to authenticate. In this case, you could install dovecot-imapd which will serve as the imap daemon and would provide the desired authentication mechanism for horde.

Continue Reading

Tutorial: Setting-up ARM Cortex-M3 Development Enviroment using Codesourcery Toolchain, Eclipse, and Segger J-Link Debugger

As can also be inferred from the title, we need three basic components:

  1. Eclipse CDT
  2. Codesourcery Lite for ARM Toolchain
  3. Segger J-Link Debugger/Emulator

Eclipse CDT is a C/C++ development IDE based on Eclipse (which defaults to Java development).

Codesourcery Lite is gcc-based ARM toolchain which provides all the basic ingredients (compiler, assembler, archiver, linker, libraries, newlib, binutils, etc.) in console based executables. The best thing about Codesourcery is that they also offer commercial solutions which result in quarterly updates to the Lite tool-chain also. This way you always stay inline with the latest gcc developments.

Segger J-Link is a very good JTAG/SWD debugger for ARM targets. If you are not for the commercial development, you can purchase an EDU version which costs around 70 Euro (incl. postage) and is worth every penny.


  • Download and install Eclipse CDT on your computer. In most cases, installation is merely creating a shortcut to eclipse.exe in the downloaded and uncompressed eclipse folder.
  • Install all necessary components in Eclipse CDT required for embedded development and debugging as shown in the following image. These can be installed from “Help->Install New Software”. Especially important are “GDB Hardware Debugging” and “Eclipse Debugger for C/C++”.

  • Download and install Codesourcery. Let it change your PATH so that Eclipse can find the proper tools it needs during compilation.
  • Test the Codesourcery install by opening a command prompt and typing the following:

    C:\>arm-none-eabi-gcc --version
    arm-none-eabi-gcc (Sourcery G++ Lite 2010q1-188) 4.4.1
    Copyright (C) 2009 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. There is NO

    You should see a response similar to given above. If windows complains about unknown or unrecognized command, then Codesourcery has not been properly installed or the PATH information is not updated. Please follow Codesourcery Getting Started Guide for further information to solve this problem.

  • Now download and import (hint: import existing projects into workspace) the following two projects in an Eclipse workspace from here:
    1. libSTM32F103
    2. Blinky_STM32F103
  • After import, the workspace should look similar to the following:

    As you can see, it contains the library for STM32 device and a blinky project to flash an LED.

  • If you right click on the libSTM32F103 (or any project for that sake), you can go into “Build Configuration -> Set Active” to set Debug or Release configuration as active. Here we choose Debug as shown in the figure below:

  • Now if you right click on libSTM32F103 project and select “Build Project”, you should see the output similar to the following in Console:

    **** Build of configuration Debug for project libSTM32F103 ****

    cs-make all
    Building file: stm32f10x_adc.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_adc.o ../src/stm32f10x_adc.c
    Finished building: stm32f10x_adc.c

    Building file: stm32f10x_bkp.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_bkp.o ../src/stm32f10x_bkp.c
    Finished building: stm32f10x_bkp.c

    Building file: stm32f10x_can.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_can.o ../src/stm32f10x_can.c
    Finished building: stm32f10x_can.c

    Building file: stm32f10x_dma.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_dma.o ../src/stm32f10x_dma.c
    Finished building: stm32f10x_dma.c

    Building file: stm32f10x_exti.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_exti.o ../src/stm32f10x_exti.c
    Finished building: stm32f10x_exti.c

    Building file: stm32f10x_flash.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_flash.o ../src/stm32f10x_flash.c
    Finished building: stm32f10x_flash.c

    Building file: stm32f10x_gpio.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_gpio.o ../src/stm32f10x_gpio.c
    Finished building: stm32f10x_gpio.c

    Building file: stm32f10x_i2c.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_i2c.o ../src/stm32f10x_i2c.c
    Finished building: stm32f10x_i2c.c

    Building file: stm32f10x_iwdg.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_iwdg.o ../src/stm32f10x_iwdg.c
    Finished building: stm32f10x_iwdg.c

    Building file: stm32f10x_lib.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_lib.o ../src/stm32f10x_lib.c
    Finished building: stm32f10x_lib.c

    Building file: stm32f10x_nvic.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_nvic.o ../src/stm32f10x_nvic.c
    Finished building: stm32f10x_nvic.c

    Building file: stm32f10x_pwr.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_pwr.o ../src/stm32f10x_pwr.c
    Finished building: stm32f10x_pwr.c

    Building file: stm32f10x_rcc.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_rcc.o ../src/stm32f10x_rcc.c
    Finished building: stm32f10x_rcc.c

    Building file: stm32f10x_rtc.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_rtc.o ../src/stm32f10x_rtc.c
    Finished building: stm32f10x_rtc.c

    Building file: stm32f10x_spi.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_spi.o ../src/stm32f10x_spi.c
    Finished building: stm32f10x_spi.c

    Building file: stm32f10x_systick.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_systick.o ../src/stm32f10x_systick.c
    Finished building: stm32f10x_systick.c

    Building file: stm32f10x_tim.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_tim.o ../src/stm32f10x_tim.c
    Finished building: stm32f10x_tim.c

    Building file: stm32f10x_tim1.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_tim1.o ../src/stm32f10x_tim1.c
    Finished building: stm32f10x_tim1.c

    Building file: stm32f10x_usart.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_usart.o ../src/stm32f10x_usart.c
    Finished building: stm32f10x_usart.c

    Building file: stm32f10x_wwdg.c
    Invoking: MCU C Compiler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -I .. -I ../inc -c -o objs/stm32f10x_wwdg.o ../src/stm32f10x_wwdg.c
    Finished building: stm32f10x_wwdg.c

    Building file: cortexm3_macro.s
    Invoking: MCU Assembler
    arm-none-eabi-gcc -O0 -ggdb -Wall -fno-strict-aliasing -fmessage-length=0 -fno-builtin -mthumb -mcpu=cortex-m3 -c -o objs/cortexm3_macro.o ../src/cortexm3_macro.s
    Finished building: cortexm3_macro.s

    arm-none-eabi-ar cr libstm32f103.a objs/stm32f10x_adc.o objs/stm32f10x_bkp.o objs/stm32f10x_can.o objs/stm32f10x_dma.o objs/stm32f10x_exti.o objs/stm32f10x_flash.o objs/stm32f10x_gpio.o objs/stm32f10x_i2c.o objs/stm32f10x_iwdg.o objs/stm32f10x_lib.o objs/stm32f10x_nvic.o objs/stm32f10x_pwr.o objs/stm32f10x_rcc.o objs/stm32f10x_rtc.o objs/stm32f10x_spi.o objs/stm32f10x_systick.o objs/stm32f10x_tim.o objs/stm32f10x_tim1.o objs/stm32f10x_usart.o objs/stm32f10x_wwdg.o objs/cortexm3_macro.o
    Finished building target: libstm32f103.a

    cs-make –no-print-directory post-build
    Performing post-build steps
    arm-none-eabi-size –totals libstm32f103.a
    text data bss dec hex filename
    2856 0 0 2856 b28 stm32f10x_adc.o (ex libstm32f103.a)
    460 0 0 460 1cc stm32f10x_bkp.o (ex libstm32f103.a)
    4088 0 0 4088 ff8 stm32f10x_can.o (ex libstm32f103.a)
    940 0 0 940 3ac stm32f10x_dma.o (ex libstm32f103.a)
    676 0 0 676 2a4 stm32f10x_exti.o (ex libstm32f103.a)
    204 0 0 204 cc stm32f10x_flash.o (ex libstm32f103.a)
    1652 0 0 1652 674 stm32f10x_gpio.o (ex libstm32f103.a)
    2776 0 0 2776 ad8 stm32f10x_i2c.o (ex libstm32f103.a)
    208 0 0 208 d0 stm32f10x_iwdg.o (ex libstm32f103.a)
    0 0 0 0 0 stm32f10x_lib.o (ex libstm32f103.a)
    2420 0 0 2420 974 stm32f10x_nvic.o (ex libstm32f103.a)
    500 0 0 500 1f4 stm32f10x_pwr.o (ex libstm32f103.a)
    2620 0 8 2628 a44 stm32f10x_rcc.o (ex libstm32f103.a)
    880 0 0 880 370 stm32f10x_rtc.o (ex libstm32f103.a)
    1520 0 0 1520 5f0 stm32f10x_spi.o (ex libstm32f103.a)
    428 0 0 428 1ac stm32f10x_systick.o (ex libstm32f103.a)
    6308 0 0 6308 18a4 stm32f10x_tim.o (ex libstm32f103.a)
    6488 0 0 6488 1958 stm32f10x_tim1.o (ex libstm32f103.a)
    2296 0 0 2296 8f8 stm32f10x_usart.o (ex libstm32f103.a)
    316 0 0 316 13c stm32f10x_wwdg.o (ex libstm32f103.a)
    110 0 0 110 6e cortexm3_macro.o (ex libstm32f103.a)
    37746 0 8 37754 937a (TOTALS)

    I have intentionally put the complete output because it can help to compare your output while looking for possible problems. This shows that the library has been properly compiled and put in an archive.

  • Now that everything is working properly, we need to know how we came to realize it. The magic lies in the Makefile which contains all the necessary commands to set the compiler, their flags, paths, etc. Beside the Makefile that we will discuss shortly, there are only two settings required in Eclipse for the compilation to occur correctly. The first is to make Eclipse aware of the Codesourcery make (called cs-make) and the second is to let Eclipse know where the “make” should be called. Both of these settings are shown in the figure below:

    As can be seen, we use Eclipse macros to define the location of the Makefile as:
    ${workspace_loc:/${ProjName}/Debug}. Similarly, for Release configuration, the location would change to:

  • In addition to that, we must also choose proper Binary Parsers and addr2line tools. To this end, please make appropriate changes according to the following image:

  • The Makefile is the most important file in the build process. In our case, for compilation of the library, we use the following Makefile:

    # Makefile to generate “archived” library for embedded targets.
    # Author: Asif Iqbal (
    # Date: 6th October, 2010

    CPU := cortex-m3
    TARGET := libstm32f103
    TARGET_EXT := a
    COMPILER_OPTIONS := -O0 -ggdb -Wall -fno-strict-aliasing \
    -fmessage-length=0 -fno-builtin -m$(INSTRUCTION_MODE) -mcpu=$(CPU)
    INCLUDE := -I ..
    INCLUDE += -I ../inc
    SRC_FOLDER := ../src
    OBJ_FOLDER := objs

    CC = arm-none-eabi-gcc

    CXX = arm-none-eabi-g++

    AS = arm-none-eabi-gcc

    OBJCP = arm-none-eabi-objcopy
    OBJCPFLAGS = -O ihex

    AR = arm-none-eabi-ar
    ARFLAGS = cr

    RM := rm -rf

    O_SRCS :=
    C_SRCS :=
    S_SRCS :=
    C_OBJS :=
    S_OBJS :=

    # All of the sources participating in the build are defined here

    C_SRCS += $(wildcard $(SRC_FOLDER)/*.c)
    S_SRCS += $(wildcard $(SRC_FOLDER)/*.s)
    C_OBJS += $(notdir $(patsubst %.c,%.o,$(C_SRCS)))
    S_OBJS += $(notdir $(OBJ_FOLDER)/$(patsubst %.s,%.o,$(S_SRCS)))

    # First compile C sources
    $(C_OBJS) : $(C_SRCS)
    @echo ‘Building file: $(@:%.o=%.c)’
    @echo ‘Invoking: MCU C Compiler’
    $(CC) $(CFLAGS) -o $(OBJ_FOLDER)/$@ $(SRC_FOLDER)/$(@:%.o=%.c)
    @echo ‘Finished building: $(@:%.o=%.c)’
    @echo ‘ ‘
    # Now compile ASS sources
    $(S_OBJS) : $(S_SRCS)
    @echo ‘Building file: $(@:%.o=%.s)’
    @echo ‘Invoking: MCU Assembler’
    $(AS) $(ASFLAGS) -o $(OBJ_FOLDER)/$@ $(SRC_FOLDER)/$(@:%.o=%.s)
    @echo ‘Finished building: $(@:%.o=%.s)’
    @echo ‘ ‘

    # All Target
    all: $(TARGET).$(TARGET_EXT)

    # Tool invocations
    $(AR) $(ARFLAGS) $@ $(addprefix $(OBJ_FOLDER)/,$(C_OBJS)) $(addprefix $(OBJ_FOLDER)/,$(S_OBJS))
    @echo ‘Finished building target: $@’
    @echo ‘ ‘
    $(MAKE) –no-print-directory post-build

    # Other Targets
    -$(RM) $(TARGET).$(TARGET_EXT) $(addprefix $(OBJ_FOLDER)/,$(C_OBJS)) $(addprefix $(OBJ_FOLDER)/,$(S_OBJS))
    -@echo ‘ ‘

    -@echo ‘Performing post-build steps’
    arm-none-eabi-size –totals $(TARGET).$(TARGET_EXT)
    -@echo ‘ ‘

    .PHONY: all clean dependents
    .SECONDARY: post-build

    The Makefile first defines the CPU and instruction mode. Then it defines the compiler options. As the compiler options are dependent on the output, so we have a different Makefile for each Debug and Release target. Please refer to GNU Make to understand the Makefile syntax.

  • The Blinky project flashes an LED and uses libstm32f103.a for accessing the microcontroller. In this case, the Makefile is a bit different and is shown below:

    # Makefile to generate “elf” compiled binary for embedded targets.
    # Author: Asif Iqbal (
    # Date: 7th October, 2010

    CPU := cortex-m3
    TARGET := blinky_stm32f103
    TARGET_EXT := elf
    LD_SCRIPT := ../stm32.ld

    LIBS := ../../libSTM32F103/Debug/libstm32f103.a

    INCLUDE += -I ../src
    INCLUDE += -I ../../libSTM32F103
    INCLUDE += -I ../../libSTM32F103/inc
    INCLUDE += -I “C:\Program Files\CodeSourcery\Sourcery G++ Lite\lib\gcc\arm-none-eabi\4.4.1\include”

    OBJ_FOLDER := objs

    COMPILER_OPTIONS = -O0 -g -ggdb -Wall -fno-strict-aliasing \
    -fmessage-length=0 -fno-builtin -m$(INSTRUCTION_MODE) \
    -mcpu=$(CPU) -MMD -MP

    DEPEND_OPTS = -MF $(OBJ_FOLDER)/$(patsubst %.o,%.d,$(notdir $@)) \
    -MT $(OBJ_FOLDER)/$(patsubst %.o,%.d,$(notdir $@))

    CC = arm-none-eabi-gcc

    CXX = arm-none-eabi-g++
    AS = arm-none-eabi-gcc

    LD = arm-none-eabi-gcc

    OBJCP = arm-none-eabi-objcopy
    OBJCPFLAGS = -O ihex

    AR = arm-none-eabi-ar
    ARFLAGS = cr

    RM := rm -rf

    USER_OBJS :=
    C_SRCS :=
    S_SRCS :=
    C_OBJS :=
    S_OBJS :=

    # Every subdirectory with source files must be described here
    SUBDIRS := ../src

    C_SRCS := $(foreach dir,$(SUBDIRS),$(wildcard $(dir)/*.c))
    C_OBJS := $(patsubst %.c,$(OBJ_FOLDER)/%.o,$(notdir $(C_SRCS)))
    S_SRCS := $(foreach dir,$(SUBDIRS),$(wildcard $(dir)/*.s))
    S_OBJS := $(patsubst %.c,$(OBJ_FOLDER)/%.o,$(notdir $(S_SRCS)))


    $(OBJ_FOLDER)/%.o : %.c
    @echo ‘Building file: $(@F)’
    @echo ‘Invoking: MCU C Compiler’
    $(CC) $(CFLAGS) $< -o $@ @echo 'Finished building: $(@F)' @echo ' ' $(OBJ_FOLDER)/%.o : %.s @echo 'Building file: $(@F)' @echo 'Invoking: MCU Assembler' $(AS) $(ASFLAGS) $< -o $@ @echo 'Finished building: $(@F)' @echo ' ' # All Target all: $(TARGET).$(TARGET_EXT) # Tool invocations $(TARGET).$(TARGET_EXT): $(C_OBJS) $(S_OBJS) @echo 'Building target: $@' @echo 'Invoking: MCU Linker' $(LD) $(LD_OPTIONS) $(C_OBJS) $(S_OBJS) $(USER_OBJS) $(LIBS) -o$(TARGET).$(TARGET_EXT) @echo 'Finished building target: $@' @echo ' ' $(MAKE) --no-print-directory post-build # Other Targets clean: -$(RM) $(TARGET).$(TARGET_EXT) $(TARGET).bin $(TARGET).map $(OBJ_FOLDER)/*.* -@echo ' ' post-build: -@echo 'Performing post-build steps' arm-none-eabi-size --totals $(TARGET).$(TARGET_EXT); arm-none-eabi-objcopy -O binary $(TARGET).$(TARGET_EXT) $(TARGET).bin ; -@echo ' ' .PHONY: all clean dependents .SECONDARY: post-build

    As we can see, it references the library using LIBS variable. The compiler options must be noted as they are optimized for debugging. During my development, I have found that using -ffunction-sections and -fdata-sections for compilation as well as –gc-sections for linking results in very hard to debug code and sometimes even wrong output. So stay away from these gcc options during debugging. They can be safely added to the Release version and result in considerably smaller binaries. The Release Makefile reflects this point.

  • Now you should also build the Blinky project by right-clicking and Build Project command. After the build has been completed successfully, you have a blinky_stm32f103.elf file in the Debug folder of the project. We need to download this to the target and execute it. Here we need the Segger debugger.
  • To use the debugger, download and install the J-Link drivers from the SEGGER website.
  • Now connect your target using SWD connections and start the SEGGER GDB server as following:

    C:\>cd “C:\Program Files\SEGGER\JLinkARM_V420e”
    C:\Program Files\SEGGER\JLinkARM_V420e>
    C:\Program Files\SEGGER\JLinkARM_V420e>JLinkGDBServer.exe -if swd
    C:\Program Files\SEGGER\JLinkARM_V420e>

    Do not forget to include the -if swd switch as it instructs the J-Link to use the SWD interface. Once the debugger is correctly connected to the device and the device is powered, the J-Link interface look similar to the following image:

    This shows that SEGGER GDB server is ready and connected to the device. The server is waiting on localhost:2331 to accept remote GDB connections.

  • Now we must configure Eclipse to use GDB server to download and debug our Blinky application to the target. For this open “Debug Configurations” as shown below:

  • Create a new node under “GDB Hardware Debugging” named “Blinky_STM32F103 Debug” with settings as shown in the following figures:

    The script in the “Startup” tab is not visible completely and is therefore given in the following:

    target remote localhost:2331
    monitor interface SWD
    monitor speed 2000
    monitor flash device = STM32F103CB
    monitor flash download = 1
    monitor flash breakpoints = 1
    monitor endian little
    monitor reset

    Save this configuration.

  • Once all the settings are made, click next to the “green bug” icon and select “Blinky_STM32F103 Debug”. Eclipse will open the Debug perspective similar to the following:

  • Now that is really something. You have access to the disassembly, registery, breakpoints and if you hover over a variable, you will see its value, live! You have a fully-functional Cortex-M IDE without any limitations on compiling, debugging, downloading, etc. And the best part is, it is open-source.

Continue Reading