GpxPlot : first glimpse

GpxPlot software is a project to:

  • Display GPS tracks in 3D (from .gpx files)
  • edit them (cutting a link, moving or removing a point,  merge tracks, edit waypoints…)
  • Display a digital elevation model (DEM) for the map
  • Display satellite/aerial images mapped onto the DEM
  • Display OpenStreetMap vector data on the DEM (maybe also Mapnik-style OSM tiles)

GpxPlot is a follow-up of my OSM and SRTM data test using Python.

GpxPlot-2014-09-21GpxPlot-2014-09-21b

This is a 3D image of  Mount Rachais (near Grenoble, France) including:

  • 3D DEM from NASA SRTM data
  • Satellite/aerial (Bing Maps) image mapped onto the surface, but larger. Altitude is 0 outside SRTM data; this explains the “walls” all around the 3D surface. There is still a bug that creates an offset between the image and the surface (look at the shape of the Isère river crossing the roads on the left)
  • In red, a GPX track recorded while going up the moutain
  • Other tracks are roads and paths from OpenStreetMap.

The tools used are not ideal yet (3D display is a little slow and not always easy to use…), but here they are:

Python, PyQt4 for the GUI, Mayavi Mlab for 3D display, gpxpy to read GPX files, OpenStreetMap Python API, Bing Maps API, SRTM data

Here’s hoping I will have some time to continue this, and add open data from the city and region, thanks to the work of the GreLibre groups for around Grenoble (hoping for better DEM and images…)

Posted in Misc. | Leave a comment

Sciences en Marche

Le mouvement Sciences en marche est parti !

Voici la carte finale pour le trajet en vélo Grenoble-Lyon samedi 4 octobre :

Voir en plein écran

Posted in Politics & society | Leave a comment

Drobo after power cut : mstatus 98304

After an unexpected power cut during the weekend, both Drobo FS at work restarted (21 hours later when the power was restored) but the samba/windows shares could not be accessed.

When I tried my droboStatus script, everything was displayed normally, except for one line:

Overall status of the device 98304 = 11000000000000000b -> UNKNOWN STATUS – please see http://code.google.com/p/drobowebdashboard/wiki/ESATMUpdate#ESATMUpdate/mStatus

As the linked page does not give an answer, it seems that it means that everything is ok after a file system check (fsck) but the shares do not work.

A simple reboot with the button at the back of the Drobos was enough to get everything on line again.

Posted in TechTips | Leave a comment

Coconut MRI

When trying to construct a head model for MRI studies, I did some experiments with coconuts.

Here are some images using BrainVisa/Anatomist for display of the acquisitions from a coconut still half full of juice.

coco1-c coco2-c coco3-c coco4-c coco5-c

Posted in Misc. | 2 Comments

WordPress behind proxy does not detect updates

WordPress is used for a small website in my workplace. The server must use a proxy to connect to the web, and this is necessary for many plugins and for updates.

As explained in the documentation, the proxy is declared in wp-config.php

define('WP_PROXY_HOST', 'myCache.fr');
define('WP_PROXY_PORT', '3128');

Since 3.8.1, the Dashboard does not show any update available (even if now 3.9.1 is out). No error message is displayed either.
I had to run wireshark to see what was happening :

  1. WordPress sends a POST request to https://api.wordpress.org/plugins/info/1.0
  2. Squid proxy answers Error 501 : “Unsupported Request Method and Protocol”

It seems that the proxy does not support HTTPS POST requests. So I had to find where the https URL was defined in WordPress. It turns out it is in wp-includes/update.php. But the URLs are defined as http, and the following code is used (in three different places) to convert it to https if ssl is available :

if ( $ssl = wp_http_supports( array( 'ssl' ) ) )
                $url = set_url_scheme( $url, 'https' );

I thus commented those :

/**if ( $ssl = wp_http_supports( array( 'ssl' ) ) )
 $url = set_url_scheme( $url, 'https' );*/

And I could update the server. Of course, I will have to do that again after each update…

Posted in TechTips | 3 Comments

Dying hard disk : getting my data back (GNU/Linux)

Five years ago I made backups of my data to three hard drives. When my desktop died, I used a small laptop where I could not copy the 1+Tb of data stored on the backups.

Last week, I tried to copy back all the data from my laptop and the old data from my Backups onto a new computer.
Of the three backup disks, one was completely dead (it did not even power up), the second one worked but contained only a partial backup (it was a smaller disk), and the last one was apparently unreadable.

When this disk was plugged in, ‘dmesg’ was printing ‘Read errors’ all over the place, and Linux did not detect any partition on the disk. I thought I had lost 1Tb of data…

Then I found this, this and this.

And here is how I got almost everything back :

  1. Plug in the dying disk (/dev/sdi)
  2. In dmesg, I had hundreds of lines such as :
    [200135.764454] Buffer I/O error on device sdi, logical block 9464
    [200160.490454] sd 9:0:0:0: [sdi] Unhandled sense code
    [200160.490457] sd 9:0:0:0: [sdi]  
    [200160.490458] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    [200160.490460] sd 9:0:0:0: [sdi]  
    [200160.490461] Sense Key : Medium Error [current] 
    [200160.490463] sd 9:0:0:0: [sdi]  
    [200160.490465] Add. Sense: Data phase error
    [200160.490466] sd 9:0:0:0: [sdi] CDB: 
    [200160.490467] Read(10): 28 00 00 01 27 c0 00 00 08 00
    [200160.490472] end_request: I/O error, dev sdi, sector 75712
  3. Make an image of the disk, ignoring read errors (replaced by 0’s in the file). This took 5 days for 1Tb.
    sudo dd if=/dev/sdi of=/home/myself/backupdisk.bin conv=noerror,sync
  4. Repair the partition table using testdisk (sudo apt-get install testdisk)
    It found one Linux (Ext3fs) partition filling up the disk image, asked to rewrite the partition table and saved the result.

    testdisk /home/myself/backupdisk.bin
  5. Display the partition table in the disk image
    sudo fdisk -lu /home/myself/backupdisk.bin
    Disk /home/myself/backupdisk.bin: 1000.2 GB, 1000204886016 bytes
    255 têtes, 63 secteurs/piste, 121601 cylindres, total 1953525168 secteurs
    Unités = secteurs de 1 * 512 = 512 octets
    Taille de secteur (logique / physique) : 512 octets / 512 octets
    taille d'E/S (minimale / optimale) : 512 octets / 512 octets
    Identifiant de disque : 0x00000000
    Périphérique Amorçage  Début         Fin      Blocs    Id. Système
    /home/myself/backupdisk.bin1   *          63  1953520064   976760001   83  Linux
  6. Set up the partition using the loopback device
    From the partition table, the partition begins at block 63 and units are 512 bytes. So the offset of the partition in the image is 32256 = 63*512. We can now set the /dev/loop0 partition as the first partition of the disk image

    sudo losetup -o 32256 /dev/loop0  /home/myself/backupdisk.bin
  7. Check and repair the partition
    Option -y is used to force fsck to correct errors (without it, it asked for confirmation thousands of times, such as “Free inodes count is wrong for group n°7230 (16384, should be=16378). Correct ?)

    sudo fsck  -y /dev/loop0
    
    #  This ends with :
    /dev/loop0: ***** FILE SYSTEM WAS MODIFIED *****
    /dev/loop0 : 968660/122109952 files (3.9% non contiguous), 234454083/244190000 blocks
  8. Mount the partition and access files
    sudo mkdir /mnt/backupdisk
    sudo mount /dev/loop0 /mnt/backupdisk
  9. Explore files and copy them from /mnt/backupdisk
  10. Unmount the partition et free the loopack device
    sudo umount /mnt/backupdisk
    sudo losetup -d /dev/loop0
Posted in TechTips | Leave a comment

Pineapple MRI

Around 2010, while developping MRI sequences for brain imaging, we used an pineapple to test the effect of some parameters on the images.

Here are some images rendered with Anatomist (from a Philips 3 Tesla scanner). Most are T1-weighted 3D sequences, the most beautiful ones are T2 2D multi-slices sequences

T2yellowred3-c T2red-c T2blackwhite2-c T2actif-ret-c T1blackwhite3-c T1blackwhite2-c T1blackwhite1-c

Posted in Misc. | Leave a comment

Ending tactical voting

[To be translated…]

Tactical voting

Before each election in France, the largest political parties call for “tactical voting” (‘vote utile’, litterally ‘useful vote’) in the first round of our two-round elections to avoid a risk, e.g. the left being removed for the second round (like in Presidential elections of 2002 which saw a right/far-right second round) or the election of the left with a right majority as the vote is split between multiple right-wind candidates. The good citizen should thus vote for the main party of its side of the political spectrum.
Lastly in the 2014 local elections, the same idea took the form of “republican withdrawal” mostly defended by the parti socialiste : if the far right might win in the second round against a right-wing and a left-wing candidate (with just 34% of votes), the least successful in the first round should withdraw and ask voters to vote for the other. This idea was not really successful as it was not accepted by UMP (main right-wing party), so PS withdrew in favor of UMP when they had the lowest score, not the other way around.
Thus, largest political parties are blackmailing citizens : if you don’t vote for us, you may get the worst (far-right). How strange for the very idea of democracy ! Voters wouldn’t be free to vote for the party they like the most, but should feel responsible enough to support the major parties as a lesser evil. So “voting against” instead of “voting for”. Where is this coming from ?

Voting system : the problem and a solution

In the current voting system (in France) if there are two left-wing candidates L1 and L2, a right-wing one R and a far-right one FR in the second round, if the results are 25.9%(L1), 26.1%(L2), 21%(R), 27%(FR), far-right will govern, despite the fact that 52% of voters wanted a left-wing candidate !
The solution is simple : changing the voting system !
There are numerous voting systems which don’t have this problem. I’ll present only one here : Instant run-off voting.

  1. Each voter ranks candidates from the best to the worst (it is not necessary to rank all candidates) :
    • a left-wing voter’s ballot could be L1, L2, R
    •  a right-winger who rejects the far-right could choose R, L2, L1
    • a right-winger who prefers the far-right to the left :  R, FR
  2. To count the results is rather simple : start by counting the first choice in all ballots. You get the same result as with the current system : 25.9%(L1), 26.1%(L2), 21%(R), 27%(FR)
  3. Remove the least popular candidate, in this example R with 21%. The ballots of the removed candidate are attributed to the other candidates according to the second choice of the voters (if there isn’t a second choice, just drop the ballot).
    We get a new result : 35.9% (L1), 30.1% (L2), 32% (FR). The total is not 100% because some voters for R did not choose a second candidate.
  4. L2 having the lowest result is removed, and the ballots are attributed to the next choice (if it is D, already eliminated, just get the next choice).
    The final score is thus :
    61% (L1) et 33% (FR).
    The winner is L1.

So, allowing voters to rank candidates allows a free expression of voters’opinion, without the fear of having cast a “wasted vote”, a vote without any influence on the choice of the winner because their preferred option lost the election.
It is of course possible to add a blank vote in the list, to express disatisfaction with all candidates, but to follow it by you ranking of the candidates, thus giving your voice for the “least bad”.

When will this come ?

If there are solutions, why are they not implemented ? In fact they have been in many countries (Australia, New Zealand, Ireland, India…).

The main beneficiaries of the current system are the largest parties (PS and UMP in France), so it would be suprising that their members risk changing what works for them : they want to keep their seats in the Parliament (where they are dominant thanks to the voting system)

And that’s how you convert the diversity of opinions in a two-party system !

As a conclusion, here is a relevant excerpt of  The Hitchiker’s Guide to the Galaxy by Douglas Adams :

On its world, the people are people. The leaders are lizards. The people hate the lizards and the lizards rule the people.”
“Odd,” said Arthur, “I thought you said it was a democracy.”
“I did,” said Ford. “It is.”
“So,” said Arthur,[…] “why don’t people get rid of the lizards?”
“It honestly doesn’t occur to them,” said Ford. “They’ve all got the vote, so they all pretty much assume that the government they’ve voted in more or less approximates to the government they want.”
“You mean they actually vote for the lizards?”
“Oh yes,” said Ford with a shrug, “of course.”
“But,” said Arthur,[…] “why?”
“Because if they didn’t vote for a lizard,” said Ford, “the wrong lizard might get in.

Posted in news, Politics & society | Leave a comment

Alpha web version of PyVotons!

At last, I put online a web version of PyVotons!.

It is alpha software, not easy to run, lacking a lot of features.

You should read the Documentation page before trying to install it.
You need Python and python-openssl, on a system running on GNU/Linux.

Here it is :

pyvotons-2014-04-15.tar.bz2

Posted in Development, news | Leave a comment

Backup all MySQL databases

Goal : Saving MySQL databases to a samba (windows share) server (Ubuntu Linux).

Relatively easy thing to do, but I will keep this here if I need to do it again someday… Here is the code of a small bash script used to do that (not optimal, I know). But first, to get things started  :

# Create a backup user to get read access to the database
mysql -u root -p
mysql> GRANT SELECT,RELOAD,LOCK TABLES ON *.* TO ‘backup_user’@’localhost’;

# Add the root user to the mysql group (as cron jobs run as root)
sudo usermod -a -G mysql root

# Allows read access to users of the mysql group for the directories of mysql data files
sudo find /var/lib/mysql -maxdepth 1 -type d -exec chmod g+r \{\} \;
# List all databases
mysql -u backup_user -B -e ‘show databases;’
# Backup one database
mysqlhotcopy -u backup_user databaseName /TheBackupDir/
# Backup all databases -> this does not work as some databases listed do not really exist !
for i in `mysql -u backup_user -B -e ‘show databases;’`; do   echo $i; mysqlhotcopy -u backup_user “$i” /TheBackupDir/ ; done

On the samba share, create a directory WebServerBackup and in this directory, an empty file autoBackup to allow the script to confirm that the right directory is mounted before backing the database to it.

The samba share used here does not need a username or password, but adding user=”myuser”,password=”mypassword” to the list of options when mounting the share should work nicely in other cases.

The /var/www dir will be saved as is (no old versions) in the same directory and kept syncrhonized by rsync. A log will be written in /var/log/backup.log

Here is the script itself :

#!/bin/bash
echo >> /var/log/backup.log
echo >> /var/log/backup.log
day=$(date +%d-%m-%Y_%Hh%M)
# List of databases to backup. Unfortunately, 
#   mysql -u backup_user -B -e 'show databases;'
#   lists system databases that cannot be saved. Using manual list instead in the for loop
echo "-------------------------------------------------" >> /var/log/backup.log
echo "Beginning WebServer Backup -> Launching backupWebsite.sh" >> /var/log/backup.log
date -R >> /var/log/backup.log
mount -t cifs -o iocharset=utf8,file_mode=0777,dir_mode=0777 //MySambaServer/MySambaShare  /backupDisk
if [ -f "/backupDisk/WebServerBackup/autoBackup" ]
then
  rsync -rltp --del --stats  /var/www /backupDisk/WebServerBackup/www >> /var/log/backup.log # --del
  mkdir "/backupDisk/WebServerBackup/mysql_$day"
  # Backup all databases
  for i in 'mybase' 'wordpress' 'mysql'; 
    do 
      echo $i; mysqlhotcopy -u backup_user "$i" "/backupDisk/WebServerBackup/mysql_$day" >> /var/log/backup.log ;
    done
  # Remove the old database backups (more than 30 days)
  find /backupDisk/WebServerBackup -maxdepth 1 -type d -name "mysql_*" -mtime +30 -exec rm -rf \{\} \;
else
  echo "Echec du montage ! ***Pas de sauvegarde du serveur web !***" >> /var/log/backup.log
fi

umount /backupDisk

echo "WebServer BACKUP FINISHED (in //MySambaServer/MySambaShare/WebServerBackup )" >> /var/log/backup.log
date -R >> /var/log/backup.log
echo "**************************************************" >> /var/log/backup.log

Finally, the script (executable, thanks to a sudo chmod a+x /usr/bin/backupWebsite.sh) is linked to from the directory /etc/cron.daily/  so it will launch automatically once a day. The .sh is removed in the link name as some versions of cron don’t execute .sh files.

sudo ln -s /usr/bin/backupWebsite.sh /etc/cron.daily/backupWebsite

 

Posted in TechTips | Leave a comment