Creating a RAMFS

With the previous method i found it slow for being a ram disk, i read that ramfs does no swapping, i want to test it:

mount -t ramfs -o size=50m ramfs /ramcache

This is a good resource:

http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/ 

Linux RAM filesystem

I do datamining, and the first thing learned is that 80% of this wonderful science is datapreparation.  I am looking for cheap wais to manage large about of data, and today i want to try how a RAM disk in Linux performs to create a dataset for weka.

As root, i want a 10MB file (yes, even though it is far away from being big data, the data preparation was taking several minutes)

root@talento:~# mkfs -q /dev/ram1 10240
root@talento:~# mkdir -p /ramcache
root@talento:~# mount /dev/ram1 /ramcache
root@talento:~# df -H | grep ramcache
/dev/ram1               11M    95k   9.6M   1% /ramcache

How to normalize a variable

This is simple procedure, but i usually find difficult to remember simple things.


To normalize a variable and bring it from a scale to another, this is the procedure:

The minimum values of A and maximum values are obvious (i hope),
D and C are the new ranges.  Being D the maximum from the new range and C the minimum from the new range.

This is an example:

Perl Date manipulation as is

Today i had to program a little script, in perl, and had to manipulate some dates.  To convert the AM/PM part i used this previously written function (here).

But then i need to do some delta operation on the dates, for whom i new packages like Date::Calc.

The surprise was that i couldn't install new packages in the server, it is a remote server and the less problems one causes, the better... because the server is shared, there was no gcc, bla, bla..

I found this package:
use Date::Manip;

Which seems to be included with Perl.  I couldn't confirm it but it saved the day, here is an example:

my $ffin = &DateCalc("$startdate", "+8 days", \$err);
my ($y, $m, $d) = $ffin =~ m/([0-9]{4})([0-9]{2})([0-9]{2}).*/g;


More about it:
http://snowhare.com/utilities/perldoc2tree/example/Date/Manip.html

glib lotus notes java memory corruption

Today i updated some packets related to IBM lotus notes and a problem with libraries caused a great and long investigation process to have lotus notes back.


I tried lot of things: reinstalling, purging the packets, installing the same version again... nothing worked. Today i found the solution here.


It says:

[...] After some research I found the problem: libnuma. If this lib exists in the system the Notes Client loads data forever. [...]


the solution is:
apt-get remove libnuma1





Standard disclosure: this is my opinion and not necessarily that of my employer or anyone else.

Decision Trees with r project on Linux

Chaid Trees with R Project on Linux

Start R and install Partykit:
$ R
> install.packages("partykit")

On a command line download the chaid sources:
svn checkout svn://svn.r-forge.r-project.org/svnroot/chaid/

$ cd chaid
$ R CMD INSTALL pkg/

On R:
> library("CHAID")
Loading required package: partykit
Loading required package: grid



Chaid project:
http://r-forge.r-project.org/scm/?group_id=343

How to obtain the flash videos in Firefox 4

Flash videos were stored in /tmp/FlashXXXX and i was able to copy them to another place like cp /tmp/FlashXXX ~/Desktop/new.video.mpeg.

Since Firefox 4 i thought for a moment that everything changed and my happy days were over but i was wrong, i remembered that with Linux there are no sad days.

How to obtain the flash videos with Firefox 4:

When the video already finished loading in the browser (being Youtube or any other), do this:

ps -ef | grep libflashplayer.so | grep -v grep

walter   20040 19812  9 11:52 ?        00:53:34 /usr/lib/firefox-3.6.17/plugin-container /home/walter/.mozilla/plugins/libflashplayer.so 19812 plugin true

Now, with the PID, inspect what it is opening:


root@talento:~# lsof -p 20040 | grep Flash
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system /home/walter/.gvfs
      Output information may be incomplete.
plugin-co 20040 walter   16u   REG        8,2  1249930 21708868 /tmp/FlashXXQANNm2 (deleted)
root@talento:~#

You are interested in the file descriptor id, these are the file descriptors from that process:


root@talento:~# ls -la /proc/20040/fd/
total 0
dr-x------ 2 walter walter  0 2011-04-30 21:12 .
dr-xr-xr-x 7 walter walter  0 2011-04-30 21:06 ..
lr-x------ 1 walter walter 64 2011-04-30 21:12 0 -> /dev/null
lrwx------ 1 walter walter 64 2011-04-30 21:12 1 -> /home/walter/.xsession-errors
l-wx------ 1 walter walter 64 2011-04-30 21:12 10 -> pipe:[18679239]
lr-x------ 1 walter walter 64 2011-04-30 21:12 11 -> pipe:[18679240]
l-wx------ 1 walter walter 64 2011-04-30 21:12 12 -> pipe:[18679240]
lrwx------ 1 walter walter 64 2011-04-30 21:12 13 -> socket:[18679243]
lr-x------ 1 walter walter 64 2011-04-30 21:12 14 -> /home/walter/.mozilla/firefox/2te1hj2x.default/cert8.db
lr-x------ 1 walter walter 64 2011-04-30 21:12 15 -> /home/walter/.mozilla/firefox/2te1hj2x.default/key3.db
lrwx------ 1 walter walter 64 2011-04-30 21:15 16 -> /tmp/FlashXXQANNm2 (deleted)
lr-x------ 1 walter walter 64 2011-04-30 21:16 17 -> pipe:[19890054]
l-wx------ 1 walter walter 64 2011-04-30 21:16 18 -> pipe:[19890054]
lr-x------ 1 walter walter 64 2011-04-30 21:16 19 -> pipe:[19890055]
lrwx------ 1 walter walter 64 2011-04-30 21:12 2 -> /home/walter/.xsession-errors
l-wx------ 1 walter walter 64 2011-04-30 21:16 20 -> pipe:[19890055]
lrwx------ 1 walter walter 64 2011-04-30 21:16 21 -> socket:[19890059]
lrwx------ 1 walter walter 64 2011-04-30 21:12 22 -> socket:[19338003]
lr-x------ 1 walter walter 64 2011-04-30 21:12 23 -> anon_inode:inotify
lrwx------ 1 walter walter 64 2011-04-30 21:12 3 -> socket:[18679123]
lrwx------ 1 walter walter 64 2011-04-30 21:12 4 -> anon_inode:[eventpoll]
lrwx------ 1 walter walter 64 2011-04-30 21:12 42 -> socket:[18679124]
lrwx------ 1 walter walter 64 2011-04-30 21:12 5 -> socket:[18679236]
lrwx------ 1 walter walter 64 2011-04-30 21:12 6 -> socket:[18679237]
lr-x------ 1 walter walter 64 2011-04-30 21:12 7 -> pipe:[18679238]
l-wx------ 1 walter walter 64 2011-04-30 21:12 8 -> pipe:[18679238]
lr-x------ 1 walter walter 64 2011-04-30 21:12 9 -> pipe:[18679239]

What is of our interest if the Flash process, displayed in the previous lines.

Now, you can copy the file to a safe place, because, if you close Firefox, those files are released and lost.

cp /proc/20040/fd/16 ~/Desktop/A.Movie.mpeg

Now if you want to put everything in a script:

root@talento:~# cat saveflash.sh
#!/bin/bash

PID=`ps -ef | grep libflashplayer.so | grep -v grep | awk '{print $2}'`
FD=`lsof -p $PID | grep Flash | awk '{print $4}' | sed 's/u$//'`
cp /proc/$PID/fd/$FD "$1"

And remember, you can "invite me a coffee" :)

Yahoo KDD Cup

In a few days starts the Yahoo KDD Cup, i'll participate in this one.  It is an interesting task where the rate of users has to be predicted.  The prediction task is based on previous users behaviour, finding patterns and common music/artists/genre and profiling new customers based on the music they do like.  I hope i have fun during this competition and keep learning from these experiences.

If possible, i would keep doing all my mining/machine learning scripts in perl, so i will share the solutions for the problems i had to face during this experience.

I wanted to search an id inside an array, to do this i did:

open TODO, "todo" or die $!;
my @todo = ;
close TODO;
@array = ();
foreach (@todo) {
   my ($id, $type) = split(/\t/);
   chop ($type);
   $array[$id] = $type;
}

my ($id, $rank, $day, $time) = split(/\t/, $linea);