the boredom of ramfree17

The SOA server I am currently working with had a nasty quirk on its services that I havent figured out yet on how to fix: it fails on requests with an XML comment. We use SOAPUI to trigger requests and the quirk requires most of us to strip the comments that is automatically generated by the tool. This quirk however gives me a good segue on this IT tipjar: how to leverage pattern matching to batch remove comments. This should serve as an introduction in other pattern matching applications when dealing with text/ascii content.

A normal SOAP request template instance will normally contain XML comments for optional items similar to this:

Normally most programmer’s text editor will have an option to use regular expressions when doing file search and replace. As can be seen in the above, XML comments have a pattern wherein they start with “<!–” and then end with “–>” (note there should be 2 dashes and not a single long line). In these examples each comment goes into a single line which makes it easier to replace them.

Now to the useful stuff, here are a few ways to quickly remove those comments:

On your programmer’s text editor (e.g. PsPad, Geany,etc.)

• open the search&replace editor
• tick on the option for the regular expression
• Enter the express <!–.*–> in the search field. (The dot and asterisk in the middle is important!)



• Press on the OK or whatever button to make it go, and…


TADA!

Another beauty of regular expressions is that a lot of utilities have support for it. If you have access to a Unix/Linux shell then chances are you have access to the sed (stream editor) and grep (global/regular expression/print) utilities.

A quick cat command to display the contents of the original request:

The sed option is nice as it will work even if the comments are inline with other texts. Be wary though that if there are multiple - -> in the same line then the example shown above will greedily replace up to the last ending match. If you are pretty sure the string to be searched for is unique then you can even specify a partial match (see second invocation example):

Those limited to a very restricted Windows operating system need not worry. Microsoft provides the findstr utility that provides limited support for pattern matching.

Microsoft has introduced the PowerShell which has a more robust support for regular expressions and other unix utilities. Below is an example of a grep-like functionality. Take note of the ! before the $_.Contains function as it serves as a negation so only non-matching lines will be printed out.

This is just a basic introduction to text replacement via regular expressions. Regexps can provide more functionality for those who have the interest and time to learn some of its quirks.

Yesterday pixie, my RPi serving as the torrent/dlna box, stopped responding. Rebooting it doesnt help as it eventually reverts to only having the red LED on which I have started to interpret as the system is not yet booted.

I pulled pixie out of its nook and hooked it to my monitor before rebooting it. It showed that the boot process is encountering errors when reading from the SD card. The process stopped while asking for the root password to start the file check maintenance, or Ctrl-D to continue the boot process. I plugged a keyboard and here lies the conundrum: this is a debian system and I have been administering it as the pi user. I have been relying on the sudo mechanism and never replaced the root password so I cant provide it. That realization blows.

No other recourse now but to pull out the SD card and have the partition checked on my desktop. A “sudo fsck /dev/sdb2 -y” command (because sdb2 is the partition assigned to it by udev) and ten minutes of automated fixing later pixie is back online serving its DLNA goodness.

Now I made sure I have changed the root password so this can be fixed without booting the desktop. Come to think of it, I am doing it also on my Ubuntu desktop.

ciao!

The pragmatic in me kicked in and I realized again that problems are what defines our existence. Only the  dead are free of them.

With that said I wish your problems will be sur mount able,  your challenges manageable,  trespasses against you forgivable,  and your heartaches forgettable.

Year 2013,  bring your best and worst.  We already have gone through the millennium bug and the mayan calendar end. We are as ready as we are going to be for the next round. 

ciao!

Hectic schedule but I finally finished having my own TorrentBox/DLNA/UPnP server out of the Raspberry Pis. One concern that got answered is if the system is usable as a media streaming server as there has been reports of slow throughput since the Pi’s ethernet (network) and USB system share the same memory space. Early on the file I/O throughput on the SD card I am using as the BIOS and operating system is a little bit slow, and as a techie friend (named Gideon[1]) pointed out its the class of the card itself that is the bottleneck.

The instructions are all over the net so Ill just reference the articles I used and add the other stuff I did to make it work on Pixie (the name of this Pi).

Mounting the external file systems
Before I can move all of my torrenting stuff into the Pi I first need to ensure that the partitions of my external USB hard drive is mounted properly, and with the proper file system labels to avoid confusing myself. I used a slightly modified udev automated mounting configuration posted in http://kissmyarch.blogspot.be/2011/10/usb-automount.html

The debian linux base already uses udev and the only change I made is changing the first uncommented line from:

KERNEL!="sd[b-z]*", GOTO="my_media_automount_end"

to

KERNEL!="sd[a-z]*", GOTO="my_media_automount_end"

The single character change is because the USB storage in the Pis will start at sda while the original post assumes that they will start at sdb because sda is already used by the host system. Tested the rules with a short reboot (and because I dont expect Pixie to be operational 24×7) to ensure that the partitions on the external 1TB disk gets mounted properly under /media.

Headless torrent box

I am already using Transmission as my bit torrent client in my desktop. To minimize the learning curve (or so I initially thought) I used the elinux.org torrent setup guide which is pretty straight forward. The guide contains a section on mounting an NTFS drive but I skipped it as I have already configured the Pi to auto-mount its filesystem in the previous section. I also skipped the virtual host configuration because of two reasons:

1. I dont expect to control transmission outside of my home network, and
2. I cant immediately map out that functionality in my router.

This is where most of my time got spent because everytime I access the Transmission web front end I always get a forbidden error message because I am accessing from a machine that is not in the whitelist. I have tried setting the whitelist to a specific IP address or via asterisk masks, or disabling the whitelist altogether but I still cant progress past this error message. I modified the rpc port to another non-standard port but the front end still listens on the default 9091. Then it dawned on me that IT was using another config file somewhere rather the one in /etc/transmission-daemon/settings.json. Running the command “find / -type f -name “*.json” -exec grep 9091 {} \;” quickly showed that there is another setting file existed in /root/.config/transmission-daemon/settings.json. I stopped the daemon and replaced the rogue settings file with a symlink to the one I maintain in /etc/transmission-daemon/settings.json. I also found another one in /var/lib/transmission-daemon/info/settings.json which I also replaced with a symlink. Everything looked good so I placed the snippet below at the end of the /etc/rc.local to ensure that the transmission daemon only gets started if the FileStorage partition is mounted properly


df -kh | grep FileStorage 1>/dev/null
if [ $? -eq 0 ]
then
transmission-daemon &
fi


Until I rebooted the Pi and the issue came back again. The culprit is that the command I used cannot read the default configuration file in /var/lib/transmission-daemon/info/settings.json and created its own in /.config/transmission-daemon/info/settings.json. There are three ways to fix this: (1) fix the permission going to the of the /var/lib settings file, (2) replace the /.config/ settings file with a symlink, or (3) replace the transmission invoke command in the rc.local file with one that specifies a config directory.

I chickend out and went for #2. I am convincing myself that this is a temporary fix because I really wanted to validate that I can do the next section.

Stream multimedia!!!!

Streaming the multimedia files directly means less time for me to transfer files into the android devices in the house. This would also minimize the inopportune times that my wife asks me to replenish her playlist.

The install and configuration is the easiest so far. It involves installing a DLNA server and the MiniDLNA project already has a ready package for the ARM platform so it all boils down to invoking:

$ sudo apt-get install minidlna

and editing /etc/minidlna.conf to reflect all the directories in the attached hard drive that I want to serve in the house. The salient points in my config follow:


media_dir=/var/lib/minidlna
media_dir=V,/media/Multimedia/TV
media_dir=V,/media/Multimedia/Movies
media_dir=A,/media/Multimedia/Audio
media_dir=/media/Multimedia/Streams
media_dir=/media/Multimedia/Comics

media_dir=V,/media/Blackboard/QueueVids
media_dir=V,/media/Blackboard/TV
media_dir=V,/media/Blackboard/tor/complete
media_dir=V,/media/Blackboard/Movies
friendly_name=PixieDust


The local MiniDLNA database needs to be updated after each configuration change and the commands are:


$ sudo service minidlna restart
$ sudo service minidlna force-reload


The force-reload will cause the minidlna.conf file to be re-read and in my case it spewed out errors that some of my defined media directories are not readable. It turned out that all some of the directories are owned by another user. My first impulse was to change the permission bits to 755 but it would have been of no use since the minidlna daemon couldn’t even get past the middle directory.

After this its a matter of choosing the DLNA client on my Android phone. My Galaxy S2 has a built-in client named AllShare but the first few tries was a bust since it can only play the video files encoded in MP4 format which is barely 20% of all of my video files. I tested various clients before settling with BubbleUPnP combined with MX Player. The BubbleUPnP fetches the files from the DLNA server and hands it off to third party renderers like the MX player. As long as I configure the MX Player to use its software decoder I can even play the MKV files. BubbleUPnP also allows downloading of the content for later playback which is a bonus for my needs.

Not bad for half-a-day’s work and it shows that the RPi can be used as a streaming server. Granted that I dont have anything greater than 720p files but I dont see why it cant do 1080p as well if all are being served within the same home network.

[1] The G+ posting is going to be unviewable for most not on my Plus circles. I dont think Plus allows the changing of a post into a Public setting.

ciao!

Thats the difference between 21 and 32 inches at 2 feet away.

Sounds great in theory but sucks a bit in actual use.

ciao!