Monthly Archives: February 2010

System & Shell commands

Posted on by
3
  • Run a process detached to the current terminal:
    nohup my_command &
  • Get the exit code of the latest runned command:
    echo $?
  • Run the last command as root (source):
    sudo !!
  • Show the user under which I’m currently logged in:
    whoami
  • If you have the following error:
    -bash: ./myscript.sh: /bin/bash^M: bad interpreter: No such file or directory

    Then the fix consist of removing the bad characters:

    sed -i 's/\r//' ./myscript.sh
  • Free up some memory by clearing RAM caches (source):
    sync ; echo 3 > /proc/sys/vm/drop_caches
  • Display which distro is running the system (source):
    lsb_release -a

    or

    cat /etc/lsb-release
  • List of most used commands:
    history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
  • Disable a service on Debian/Ubuntu, then re-enable it:
    update-rc.d my-service-name remove
update-rc.d my-service-name defaults
  • Same thing as above but on a RedHat-like system:
    chkconfig sshd --del
chkconfig sshd --add

MySQL commands

Posted on by
1
  • List all users in the current MySQL server:
    mysql> SELECT User, Host FROM mysql.user;
  • Remove a user from the server:
    mysql> DROP USER '<User>'@'<Host>';
  • Remove all binary logs:
    mysql> RESET MASTER;
  • Delete all binary logs but keep a week worth of logs:
    mysql --verbose --execute="PURGE BINARY LOGS BEFORE '`date +"%Y-%m-%d" -d last-week`';"

    And if you put this in a cron-tab, don’t forget to escape percents:

    mysql --verbose --execute="PURGE BINARY LOGS BEFORE '`date +\%Y-\%m-\%d -d last-week`';"
  • Check, auto-repair and optimize all databases:
    mysqlcheck --auto-repair --optimize --all-databases
  • Export a database:
    mysqldump -u my_user "my-database" > data.sql
  • Here is a cron-able command to restart a MySQL service if no process found active:
    [ `ps axu | grep -v "grep" | grep --count "mysql"` -le 0 ] && /etc/init.d/mysql restart
  • Monitor the queries being run (source):
    watch -n 1 mysqladmin --user=XXXXX --password=XXXXX processlist
  • Get the list of default configuration parameters the server will use regardless of the values set in config files (source):
    mysqld --no-defaults --verbose --help
  • Migrate all tables of all databases from MyISAM to InnoDB:
    mysql --skip-column-names --silent --raw --execute="SELECT CONCAT(table_schema , '.', table_name) FROM INFORMATION_SCHEMA.tables WHERE table_type='BASE TABLE' AND engine='MyISAM';" | xargs -I '{}' mysql --verbose --execute="ALTER TABLE {} ENGINE=InnoDB;"

How-to fork a CVS project with Git

Posted on by
Reply

This week I’ve decided to put my work on Cool Cavemen’s concert videos aside, and work instead on refreshing our online store. After all, fans are requesting this, so I can’t escape my duty…

The theme the store is based on is Drupify, an adaptation of the RokWebify Joomla theme. All these themes are licensed under the GPL, so I have to share all my modifications with the community. This is a great opportunity to seriously experiment with Git (at last !).

Here is my plan:

  1. Make an exact copy of Drupify’s code base in my GitHub repository.
  2. Hack it in this playground.
  3. ???
  4. Profit ! :D

Problem: Drupify lives in a CVS repository.

Solution: Git features a cvsimport command.

Before going further, we need to install cvsps. For MacPorts users, this is as simple as:

~$ sudo port install cvsps
Password:
--->  Computing dependencies for cvsps
--->  Fetching cvsps
--->  Attempting to fetch cvsps-2.1.tar.gz from http://arn.se.distfiles.macports.org/cvsps
--->  Attempting to fetch cvsps-2.1.tar.gz from http://distfiles.macports.org/cvsps
--->  Verifying checksum(s) for cvsps
--->  Extracting cvsps
--->  Applying patches to cvsps
--->  Configuring cvsps
--->  Building cvsps
--->  Staging cvsps into destroot
--->  Installing cvsps @2.1_1
--->  Activating cvsps @2.1_1
--->  Cleaning cvsps

Then we create a temporary copy of Drupify’s CVS repository:

~$ git cvsimport -a -k -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal-contrib -C drupify-copy contributions/themes/drupify
Initialized empty Git repository in /Users/kevin/drupify-copy/.git/
parse error on user@server in pserver
cvs rlog: Logging contributions/themes/drupify
cvs rlog: Logging contributions/themes/drupify/css
cvs rlog: Logging contributions/themes/drupify/images

The new Git repository automatically created is named drupify-copy. Here is how it looks like in GitX (notice tags and branches):

To keep things clean and tidy, I want to relocate all the content of this repository to a drupify-fork folder. Inspired by Pedro Melo, we’ll use the git filter-branch to do this job:

~$ cd drupify-copy
drupify-copy$ git filter-branch -f --prune-empty --tree-filter '
  mkdir -p /tmp/drupify-fork;
  mv $(ls -A) /tmp/drupify-fork;
  mv /tmp/drupify-fork drupify-fork
' -- --all
Rewrite a9319cebb234c46cc8e0ada95ffb2cd81b87993c (17/17)
Ref 'refs/heads/DRUPAL-5' was rewritten
Ref 'refs/heads/DRUPAL-6--1' was rewritten
Ref 'refs/heads/master' was rewritten
Ref 'refs/heads/origin' was rewritten
Ref 'refs/tags/DRUPAL-6--1-0' was rewritten

The command we just used alter all the commits, in a way that let Drupify act as if it was located, since the beginning of its history, in the drupify-fork sub-directory.

By default, filter-branch creates a backup of the tree using references prefixed by refs/original/:

drupify-copy$ git show-ref
4c33470f0f59bcfe7d0d88ee64945bb5625d6d02 refs/heads/DRUPAL-5
8930672eaf97eefa8f9d4ed9f5144f466a97728f refs/heads/DRUPAL-6--1
e5907fac0160febbd91f0cda73633b3e6eafa2a9 refs/heads/master
e5907fac0160febbd91f0cda73633b3e6eafa2a9 refs/heads/origin
af9786625a280930b532541722806739e221ebda refs/original/refs/heads/DRUPAL-5
a9319cebb234c46cc8e0ada95ffb2cd81b87993c refs/original/refs/heads/DRUPAL-6--1
328f3440e202ed72253974dbbbd45f39db23ea4a refs/original/refs/heads/master
328f3440e202ed72253974dbbbd45f39db23ea4a refs/original/refs/heads/origin
957bb22704bc8188c0421b68cbb2f52a3fdcdef6 refs/original/refs/tags/DRUPAL-6--1-0
ee44c42250a2552c1dbef2f7165d65179e1d19c6 refs/tags/DRUPAL-6--1-0

We’re not the only ones to not see through this mess. GitX seems to be confused too:

But according Jakub Narębski on the Git mailing-list, we can safely removes Git’s backups:

drupify-copy$ rm -Rf .git/refs/original
drupify-copy$ git gc --prune=now
Counting objects: 106, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (88/88), done.
Writing objects: 100% (106/106), done.
Total 106 (delta 40), reused 0 (delta 0)

After the cleaning, references are back to normal:

drupify-copy$ git show-ref
4c33470f0f59bcfe7d0d88ee64945bb5625d6d02 refs/heads/DRUPAL-5
8930672eaf97eefa8f9d4ed9f5144f466a97728f refs/heads/DRUPAL-6--1
e5907fac0160febbd91f0cda73633b3e6eafa2a9 refs/heads/master
e5907fac0160febbd91f0cda73633b3e6eafa2a9 refs/heads/origin
ee44c42250a2552c1dbef2f7165d65179e1d19c6 refs/tags/DRUPAL-6--1-0

We can then fire up GitX to get the ultimate proof that the relocation operation didn’t change anything, but the base folder (and SHA hashes):

It’s time to import all this code in our main repository. First, get a local copy of our public GitHub code base:

drupify-copy$ cd
~$ git clone git@github.com:kdeldycke/kev-code.git
Initialized empty Git repository in /Users/kevin/kev-code/.git/
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (5/5), done.

Now let’s include our temporary drupify-copy as a tracked remote repository:

~$ cd kev-code/
kev-code$ git remote add drupify ../drupify-copy
kev-code$ git fetch drupify
warning: no common commits
remote: Counting objects: 106, done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 106 (delta 40), reused 106 (delta 40)
Receiving objects: 100% (106/106), 61.62 KiB, done.
Resolving deltas: 100% (40/40), done.
From ../drupify-copy
 * [new branch]      DRUPAL-5   -> drupify/DRUPAL-5
 * [new branch]      DRUPAL-6--1 -> drupify/DRUPAL-6--1
 * [new branch]      master     -> drupify/master
 * [new branch]      origin     -> drupify/origin
From ../drupify-copy
 * [new tag]         DRUPAL-6--1-0 -> DRUPAL-6--1-0

As you can see, all the little particularities of the remote repository are well tracked (HEAD, branches and tags are there):

kev-code$ git remote show drupify
* remote drupify
  Fetch URL: ../drupify-copy
  Push  URL: ../drupify-copy
  HEAD branch (remote HEAD is ambiguous, may be one of the following):
    master
    origin
  Remote branches:
    DRUPAL-5    tracked
    DRUPAL-6--1 tracked
    master      tracked
    origin      tracked
  Local ref configured for 'git push':
    master pushes to master (local out of date)

Another way to check this is to list all tracked remote branches:

kev-code$ git branch -r
  drupify/DRUPAL-5
  drupify/DRUPAL-6--1
  drupify/master
  drupify/origin
  origin/HEAD -> origin/master

It’s time to merge all our tracked remote code (from drupify-copy) in our local repository (kev-code). The branch I’m interested in is DRUPAL-6--1, as it holds the latest Drupify code for Drupal 6.x:

kev-code$ git merge drupify/DRUPAL-6--1
Merge made by recursive.
 drupify-fork/README.txt               |   16 +
 drupify-fork/css/editor_content.css   |    7 +
 drupify-fork/css/index.html           |    1 +
 drupify-fork/css/template_ie.css      |   55 +++
 drupify-fork/drupify.info             |   11 +
 drupify-fork/images/arrow.png         |  Bin 0 -> 278 bytes
 drupify-fork/images/bg.png            |  Bin 0 -> 315 bytes
 drupify-fork/images/bottom-bg.png     |  Bin 0 -> 583 bytes
 drupify-fork/images/col-divider.png   |  Bin 0 -> 200 bytes
 drupify-fork/images/emailButton.png   |  Bin 0 -> 454 bytes
 drupify-fork/images/favicon.ico       |  Bin 0 -> 1150 bytes
 drupify-fork/images/footer-bg.png     |  Bin 0 -> 544 bytes
 drupify-fork/images/footer-l.png      |  Bin 0 -> 593 bytes
 drupify-fork/images/footer-r.png      |  Bin 0 -> 592 bytes
 drupify-fork/images/footer-rocket.png |  Bin 0 -> 3391 bytes
 drupify-fork/images/header-bg.png     |  Bin 0 -> 638 bytes
 drupify-fork/images/header-l.png      |  Bin 0 -> 430 bytes
 drupify-fork/images/header-r.png      |  Bin 0 -> 444 bytes
 drupify-fork/images/indent1.png       |  Bin 0 -> 214 bytes
 drupify-fork/images/indent2.png       |  Bin 0 -> 214 bytes
 drupify-fork/images/indent3.png       |  Bin 0 -> 214 bytes
 drupify-fork/images/indent4.png       |  Bin 0 -> 214 bytes
 drupify-fork/images/index.html        |    1 +
 drupify-fork/images/menu-bg.png       |  Bin 0 -> 343 bytes
 drupify-fork/images/menu-bullet.png   |  Bin 0 -> 933 bytes
 drupify-fork/images/menu-divider.png  |  Bin 0 -> 200 bytes
 drupify-fork/images/pdf_button.png    |  Bin 0 -> 482 bytes
 drupify-fork/images/printButton.png   |  Bin 0 -> 467 bytes
 drupify-fork/logo.png                 |  Bin 0 -> 27829 bytes
 drupify-fork/node.tpl.php             |   31 ++
 drupify-fork/page.tpl.php             |  173 ++++++++++
 drupify-fork/screenshot.png           |  Bin 0 -> 11289 bytes
 drupify-fork/style.css                |  593 +++++++++++++++++++++++++++++++++
 drupify-fork/template.php             |   10 +
 34 files changed, 898 insertions(+), 0 deletions(-)
 create mode 100644 drupify-fork/README.txt
 create mode 100644 drupify-fork/css/editor_content.css
 create mode 100644 drupify-fork/css/index.html
 create mode 100644 drupify-fork/css/template_ie.css
 create mode 100644 drupify-fork/drupify.info
 create mode 100644 drupify-fork/images/arrow.png
 create mode 100644 drupify-fork/images/bg.png
 create mode 100644 drupify-fork/images/bottom-bg.png
 create mode 100644 drupify-fork/images/col-divider.png
 create mode 100644 drupify-fork/images/emailButton.png
 create mode 100644 drupify-fork/images/favicon.ico
 create mode 100644 drupify-fork/images/footer-bg.png
 create mode 100644 drupify-fork/images/footer-l.png
 create mode 100644 drupify-fork/images/footer-r.png
 create mode 100644 drupify-fork/images/footer-rocket.png
 create mode 100644 drupify-fork/images/header-bg.png
 create mode 100644 drupify-fork/images/header-l.png
 create mode 100644 drupify-fork/images/header-r.png
 create mode 100644 drupify-fork/images/indent1.png
 create mode 100644 drupify-fork/images/indent2.png
 create mode 100644 drupify-fork/images/indent3.png
 create mode 100644 drupify-fork/images/indent4.png
 create mode 100644 drupify-fork/images/index.html
 create mode 100644 drupify-fork/images/menu-bg.png
 create mode 100644 drupify-fork/images/menu-bullet.png
 create mode 100644 drupify-fork/images/menu-divider.png
 create mode 100644 drupify-fork/images/pdf_button.png
 create mode 100644 drupify-fork/images/printButton.png
 create mode 100644 drupify-fork/logo.png
 create mode 100644 drupify-fork/node.tpl.php
 create mode 100644 drupify-fork/page.tpl.php
 create mode 100644 drupify-fork/screenshot.png
 create mode 100644 drupify-fork/style.css
 create mode 100644 drupify-fork/template.php

We can remove the attached drupify repository and its local drupify-copy source:

kev-code$ git remote rm drupify
kev-code$ cd ..
~$ rm -rf ./drupify-copy

At this stage, here is what our repository looks like:

To keep all the details that were created by git cvsimport, we can add by hand all the missing refs. The only difference with the original ones is that I unified the namespace with a drupify/ prefix:

kev-code$ git update-ref refs/heads/drupify/DRUPAL-6--1 8930672
kev-code$ git update-ref refs/heads/drupify/master e5907fa
kev-code$ git update-ref refs/heads/drupify/origin e5907fa
kev-code$ git tag drupify/DRUPAL-6--1-0 DRUPAL-6--1-0
kev-code$ git tag -d DRUPAL-6--1-0
Deleted tag 'DRUPAL-6--1-0'

And finally, we can contemplate our work:

This let me work cleanly on the CVS branch I wanted to in the first place. But there is one missing thing: all other tracked remote branches were not merged properly. I really wanted to import all of them (especially the DRUPAL-5 branch), to keep a perfect copy of the original CSV tree. But I failed to find a way. Does anyone have a clue ?

Git commands

Posted on by
2
  • Get a clean local copy of my GitHub repository with read & write access:
    git clone git@github.com:kdeldycke/scripts.git
  • Switch to another branch:
    git checkout another_branch
  • Set the current repository in the state it was at commit 1234567:
    git checkout 1234567
  • Get the current commit number:
    git rev-parse HEAD
  • Print a nice graph of your commits sorted by date:
    git log --graph --all --pretty=oneline --abbrev-commit --date-order
  • Destroy all your local changes and get back a sane repository:
    git reset --hard
  • Send local repository modifications to remote one:
    git push origin
  • Attach a tag to a given commit:
    git tag "1.2.3" 8fe2934d1552c97246836987f0ea08e10ba749ae
  • Publish all tags to the remote repository:
    git push --tags
  • Add a remote repository located on GitHub as a submodule in the ./folder/project-copy folder:
    git submodule add https://github.com/my-id/project.git ./folder/project-copy
  • While playing with backups of a local repository, you may encounter this error:
    Cannot rewrite branch(es) with a dirty working directory.

    In this case, you can get back a clean repository by removing all the unstaged changes:

    git stash

Cool Cavemen live at Gayant Expo, part II.

Posted on by
5

Here is “Funky Cops”, the second Cool Cavemen’s live song at Gayant Expo:

I released this video two weeks ago for Cool Cavemen. As I try to release one video every week, I give a high priority to the editing work. This leaves me with little time to write on this blog.

But starting from now, I plan to publish a blog post for each video. I’ll use these articles to write about one aspect of the work involved behind the scene.

In the first post of the series, I gave you the context in which the concert was performed. Today the post is dedicated to video formats. First, let’s talk about the video sources…

The concert was shot with 4 cameras. Among them, only two were of the same kinds. Those were part of the live broadcasting system of the event. This explain the “mise en abyme” effect in the background screen:

At the end of the gig, I wasn’t be able to retrieve two continuous feeds. Instead I got an already-edited video corresponding to what was projected live (*sigh*).

As a result, I ended with 3 video sources:

  1. A DVD-like video stream (576i) produced by my consumer-grade camera (now for sale at 0.01€ on ebay). It produces 720×576 pixels interlaced frames at 25 fps, with a pixel ratio of 16:15 (giving 768×576 pixels frames at 1:1) and a final display ratio of 4:3. All encoded as a 9 Mbps MPEG-2 stream in a MPEG-PS container.
  2. A 720p video stream: 1280×720 pixels progressive frames at 30 fps, with 1:1 pixel ratio and 16:9 display ratio, encoded as variable bitrate MJPEG stream in a QuickTime container.
  3. The already-edited video stream (Half-D1) from unidentified Sony cameras: 352×576 pixels interlaced frames at 25 fps, with a pixel ratio of 24:11 (giving 768×576 pixels frames at 1:1) and a final display ratio of 4:3. The file was a 6 Mbps MPEG-2 stream in a MPEG-PS container.

All those informations were extracted thanks to ffmeg, mplayer and tcprobe (see all the command lines involved).

As you can see, this is an absolute mess ! There is no consistency ! And now, before starting the video editing itself, I have this important decision to make: choose the final video format, in which my project will be rendered.

Let me explain how I did it. But before, I have to tell you something. To me, an interlaced video at 25 fps is just a 50 fps stream with half the vertical resolution. This is important for you to know if you want to understand how I perceive quality. I’ll probably explain it in details in a future article. But for now, this should give you enough insights on how I came up with my two strategies.

The first one is the “maximizing” strategy. It consists of keeping the best parts from all video sources. Based on formats described above, this means 1280×720 pixels progressive frames at 50 fps, with 1:1 pixel ratio and 16:9 display ratio. In this process we create non-existent informations by scaling and interpolating spatial and temporal data.

The second strategy is the “minimizing” strategy which, you can guess from its name, is the exact opposite of the first one. Here we discard spatial and temporal informations until we reach a sub-format shared by all sources. In our example, this gives 352×288 pixels frames at 30 fps, with a pixel ratio of 24:22 and a display ratio of 4:3. There, 288 is half 576, which is the result of using a deinterlacing “bob” filter on video streams #1 and #3 to get 50 fps. And for the pixel ratio, as we “bobbed” the interlaced videos, we keep the worst horizontal scaling and multiply the vertical scaling by two, which give us 24:22.

For this project, I finally went by the first stategy. I choosed to render the project to a 720p video at 25 fps, with a 1:1 pixel ratio and 16:9 display ratio. Also known as… HD-Ready !

Why this format ? It’s the most popular one that closely match the characteristics we established three paragraphs above. It’s also quite standard, and “gives a chance” to the second video source to display in full resolution. I also felt that it will cause less pain when confronted to the wide range of software video players out there.

Now that I have decided which format to use, I can create a project in my video editor with the right parameters and start the editing process. But this is another topic for another post !