Everyone loves graphs...

This really is depressing. Being Canadian and all I really want to root for the home team. Moreover I hate polluting my blog with complaints about BlackBerry development. Alas, here we go again…

Todays topic: BlackBerry Push Service. BlackBerry push can be routed through BlackBerry Internet Service or it can route through a BlackBerry Enterprise Server. The difference? BIS is the one you want to use if you are a developer targeting the consumer market at large. To route push notifications through BIS, you must request to evaluate the push service. It also appears you must submit another request when you want to go live, but I haven’t made it that far.

This is where the problem lies. I have submitted three requests in the past two weeks with no response. Each time I submit one, I’m greeted with a friendly message:

“Thank you, someone will contact you in two business days“

Bottom line I can’t finish OR test my app without access to this service.

What about the other guys?

This is where it gets sad. The software I’m working on is also being developed in parallel for Android and iOS. So what does the process look like for the other platforms?

Android

The Android Cloud to Device Messaging framework also requires developers to signup to access the service. The difference being two seconds after you fill out the signup form you are ready to go!

Slightly off topic, but the C2DM API is extremely simple to use. It was a pleasure to work with since I could use basic Unix shell scripting to send requests to the service during testing.

Apple

After you register as an iOS developer (and you drop $100 US for the privilege), Apple gives you access to something called the iOS Provisioning Portal which is a web interface for generating all manor of certificates and mobile provisioning bits and bobs. In this portal access to the APNS (Apple Push Notification Service) can also be enabled with a few mouse clicks… instantly.

So what have I done so far?

Not wait on RIM, that’s for sure! In the last two weeks I have managed to completely finish the iOS and Android versions of my app including full push integration.

Let me say that again. In the time it’s taking RIM to grant me permission to write software that uses their services, I have created the same thing on competing platforms.

If you attempt to sign a .cod file larger than 64KB (ie: contains sibling files or child files or whatever files) then you will likely get a file created in your current working directory that looks something like kd8m34jkf/my_lame_app-1.cod. When you load the file to the device it will result in an error of some sort. The fix for this is relatively simple. Read on for the details.

Edit class file

Go back and read my other post for instructions on extracting the jar file and for packaging it up again. I’m going to leave out some details here since they are covered previously.

Note: this is based on the SignatureTool.jar file from JDE 4.7. I have not verified that this procedure is the same for other versions of the JDE.

We will use good old ClassEditor again. Open q.class and change string constant at index 253 from a back slash to a forward slash.

Edit string constant at index 253 in q.class

The simulator doesn’t sim and the signing tool doesn’t sign. These two issues cause a serious problem.

First the simulator. If you take an OS 4.3.1 simulator and a device also running OS 4.3.1 the two will behave very differently in a few subtle, but important ways. My gripe is localized to the mmapi api (handles audio/video playback). Long story short, the simulator can’t be trusted so we have to debug device/OS idiosyncrasies on the device itself.

But hold on a second. The signing servers aren’t working so I can’t debug on the device in all it’s developer friendly glory with lack of usable debugger and lack of even something as simple as a means to read text written to freaking standard output!

Thank you RIM! Thank you for wasting my valuable time. Thank you for reminding me that no amount of money can justify writing software for this platform any longer.

(10:30, signing servers still not working)

1. Rapc compiler
2. Signature tool
3. Simulator
4. Javaloader

At the time of writing the first two worked fine. The third, Simulator, is if’y at best. And the fourth was just not possible.

Until now. In late December, a saviour by the name of Nicolas Vivien dropped by the barry mailing list to let everyone know that he was hard at work on reverse engineering the Javaloader USB protocol. Within a few weeks he had developed a proof of concept that enabled users to load cod files to the BlackBerry.

Since then several developers (including myself) have chipped in to implement all of the crucial aspects of the Javaloader needed for day-to-day use.

Which Commands are Supported?

• list modules on handheld, including siblings
• get handheld info
• load one or more cod files onto the handheld
• save one or more cod files from the handheld
• erase cod files from the handheld
• dump the eventlog
• clear the eventlog
• get screenshot of the handheld and save as bmp
• set time on the handheld
• wipe the applications and/or file system on the handheld

As you can see, most of the important stuff is already finished. And more is being added all the time.

I’ve Heard Enough… Gimme, Gimme, Gimme!

The Javaloader is slated to be included in the next release of Barry; version 0.15. If you would like to test it right now, you’ll need to compile from source or find a binary package for your OS/distro.

Here are some instructions for compiling from source. If you get stuck, someone on the mailing list might be able to lend a hand.

While this is effective, it has a few caveats:

• makeMenu gets messy
• separators must be added explicitly
• doesn’t play nice with context menu items added by framework

Well what does the API say?

The obvious first place to look is the API documentation right? OK, lets have a look at the class description and constructor parameters.

Each item has an ordinal specifying the sort order within the menu, and a priority specifying which one should be the default. A lower value indicates a higher priority.

• ordinal – Ordering parameter, lower values are placed closer to the top of the menu screen.
• priority – The priority of the menu item. A lower value indicates a higher priority, conversely a higher value indicates a lower priority.

Ordinal seems pretty straight forward. But what relation (if any) does this have to separator placement? And what if I want my menu items be placed below the context menu items of the fields on the screen? What should my ordinal value be then?

The priority from what I can tell just determines the likelihood the menu item will be picked as the default. Lower values indication higher priority and higher values a lower priority.

Verdict

Here’s a summary first, because the details are rather dry (skip them if you like):

• ordinal values < 10,000 are placed above context items
• ordinal values >= to 300,000 are placed below context items
• ordinal values with a difference larger than 100,000 have separators between them
• use priority values >= 1000 play nice with context item priorities

Using a basic sample containing a couple fields and a prefab save menu item, I printed the ordinal ordinal and priority of each menu item in the menu generated by the MainScreen class (see source code at the end of the article).

Field context menu items have an ordinal somewhere in the neighbourhood of 10,000 to 200,000. The prefab menu item for save has an ordinal around 300,000. And based on the ordinal from the save menu item, it’s probably safe to assume that to place your menu items below context items, 300,000 is the magic number.

To have separators automagically placed between menu items, simply use an ordinal value at least 100,000 greater than the previous menu item.

Context menu items seem to have a priority value in the range of 100 to 500. The prefab menu item has a priority of 1000 so I’m guessing this is an appropriate value for most menu items.

MainScreen – Cause I’m Lazy

The MainScreen class has a method addMenuItem. The documentation for this method reads:

A main screen brings up a menu by default when Screen.onMenu(int) is invoked. If the makeMenu(net.rim.device.api.ui.component.Menu, int) method is not overridden, it will add these menu items to the menu. The close menu item is added by default.

So as long as the ordinal and priority values are set appropriately, you don’t need to bother with the makeMenu method override at all! Right?

Well there is one gotcha; MainScreen adds all all of the menu items to the menu. If you need to omit items from the menu resulting from the trackball press, then you will have to add these items exclusively in the makeMenu method.

public MyScreen() {
   addMenuItem(_saveMenuItem);
   // makeMenu will handle adding this now
   //addMenuItem(_deleteMenuItem);
}
 
protected void makeMenu(Menu menu, int instance) {
   super.makeMenu(menu, instance);
 
   // add delete menu item when NOT making context menu (trackball-click)
   if ( instance != Menu.INSTANCE_CONTEXT ) {
      menu.add(_deleteMenuItem);
   }
}

Sample Code

import net.rim.device.api.ui.MenuItem;
import net.rim.device.api.ui.UiApplication;
import net.rim.device.api.ui.component.EditField;
import net.rim.device.api.ui.component.Menu;
import net.rim.device.api.ui.component.ObjectChoiceField;
import net.rim.device.api.ui.container.MainScreen;
 
public class Hello extends UiApplication {
   public Hello() {
      MainScreen screen = new MainScreen(MainScreen.DEFAULT_MENU) {
         protected void makeMenu(Menu menu, int instance) {
            super.makeMenu(menu, instance);
 
            for (int i=0, len=menu.getSize(); i<len; i++) {
               MenuItem item = menu.getItem(i);
               System.out.println("DEBUG " + item.getOrdinal() +
                     " " + item.getPriority() + " " + item);
            }
         }
      };
 
      screen.addMenuItem(MenuItem.getPrefab(MenuItem.SAVE_CLOSE));
 
      screen.add(new EditField("Edit: ", ""));
      screen.add(new ObjectChoiceField("Choice: ", new Object[] {"1", "2"}));
 
      pushScreen(screen);
   }
 
   public static void main(String[] args) {
      new Hello().enterEventDispatcher();
   }
}

Disclaimer: I am not an expert with wine nor have I ever claimed to be. But if you setup an override for the msxml dll it seems as though the BlackBerry simulator is a bit more reliable in Linux. In fact, the JDWP seems to actually work for debugging your applications.

Fix JDWP

First you need to fix the JDWP. I outlined the steps required to do this in my last article about running the simulator in Linux.

DLL Overrides

Copy the DLL files from a Windows box. Again, I actually have no idea what I’m doing so I just went and copied all the DLL’s named msxml* from C:\WINDOWS\system32 to my local wine directory ~/.wine/drive_c/windows/system32. It’s probably not necessary to copy all of them, but I don’t think it will hurt anything either.

Start up winecfg and select the Libraries tab. Type msxml3 in the New override for library: field and click Add.

Winecfg with dll override

Start the JDWP and connect to it from your favorite IDE and watch the madness ensue. Now I have not tested this thoroughly but my first couple of tries seem to work well. So well in fact that I might event be able to ditch the VM running windows for debugging. As always, YMMV.

1. Rapc compiler
2. Signature tool
3. Simulator
4. Javaloader

The first two components works fabulously in Linux. The second (signature tool) needs a kick in the pants to work (shame shame shame!).

The third is hit or miss when using Wine. So a less painful option is to put windows in a virtual machine and run the simulator there (more on that later).

And the fourth I just can’t get past. There is the barry project which has made impressive progress in the last few months and lists javaloader support as on the roadmap for milestone 4. RIM, if you have any sympathy for us non-windows users show these guys some love and send them some specs for heaven sakes!

This article presents the basics for compiling, signing, and debugging BlackBerry applications with Linux.

Rapc Compiler

First you need a copy of the JDE or the JDE components package. In my last article I covered a method of extracting the install packages distributed by RIM, but if you have access to a Windows box copying the installed JDE works just fine too.

The rapc compiler needs two external commands:

1. Java Compiler (javac)
2. Preverification Tool (preverify)

Install the Sun JDK to get javac:

sudo apt-get install sun-java5-jdk

The preverify command can be obtained by downloading the Sun WTK.

chmod +x sun_java_wireless_toolkit-2_5_2-linux.bin
./sun_java_wireless_toolkit-2_5_2-linux.bin


For some reason this package complains it can’t find Java interpreter. If this happens, the Java 5 JDK is located /usr/lib/jvm/java-1.5.0-sun/bin on Ubuntu.

Once extracted, unless you plan on doing vanilla J2ME development, you can copy the preverify tool and then delete the WTK.

mkdir ~/bin
cp ~/lib/WTK2.5.2/bin/preverify* ~/bin
rm -r ~/lib/WTK2.5.2


Don’t forget to put the ~/bin directory in the execution path. To have this set automatically each time you login, put this line in your ~/.bash_profile file (create it if necessary).

export PATH=~/bin:$PATH

Compile “Hello, World!”

I’ll be using Apache Ant for building because it’s the common denominator when it comes to building Java projects (like it or not). Most IDE’s have some level of support for running ant scripts.

Install ant:

sudo apt-get install ant

Download bb ant tools and put it where ant can find it:

unzip bb-ant-tools-0.7-bin.zip
mkdir -p ~/.ant/lib
cp bb-ant-tools.jar ~/.ant/lib


Download the hello world project and run "ant" in the project directory:

wget http://www.slashdev.ca/download/hello.tar.gz
tar -xzf hello.tar.gz
cd hello
ant


Output from the ant should look something like this:

josh@ubuntu:~/hello$ ant
Buildfile: build.xml

build:
    [mkdir] Created dir: /home/josh/hello/build
     [rapc] Compiling 1 source files to ca_slashdev_hello.cod

BUILD SUCCESSFUL
Total time: 1 second

Signing

To sign your cod files, follow my guide on how to fix the signature tool. Then just run the sign target in the build script.

ant sign

MDS simulator log output

All you have to do is create a new script to replace the various batch files that ship with the MDS simulator. Here is the replacement for run.bat

#!/bin/bash
 
for i in classpath/* webserver/lib/*; do
  MDS_CP=$i:${MDS_CP}
done
 
java -classpath ${MDS_CP} -Xmx512M -DKeystore.Password=password \
   net.rim.application.ipproxyservice.IPProxyServiceApplication \
   -log.console.dump

Don’t forget to set the execute permission:
chmod +x run.sh

To shutdown MDS, just hit <ctrl>+c in the terminal window to kill the process.

Call me crazy but I just can’t seem to accept that if you want to (or have to) write BlackBerry software you’re stuck with Windows. It bugs me.

There is good news and bad news regarding the simulator. The good news is with a recent version of Wine you can actually get the BlackBerry simulator to run. The bad news is that it’s a bit buggy, and debugging hardly works at all.

What to expect

Your millage may vary! Generally I find that the simulator is usable for most testing. Attaching a debugger to it (via JDWP) just seems to cause the simulator to hang. Mind you, I have a rather slow system (AMD XP 2500+) and I would love to hear about your experiences if you have a faster computer.

Ingredients

Wine – You need the most recent version you can get your hands on. The package in the Ubuntu repository won’t due. So if you have installed this one, remove it and follow these instructions to get the latest and greatest.

For some reason until I performed the following steps, the simulator would always crash on startup:

1. rm -rf ~/.wine
2. Start up winecfg
3. Applications tab – select Windows 2000 from the dropdown list
4. Audio tab – select ALSA Driver from the drivers list

It should go without saying that you will need a copy of the JDE or one of the simulator packages. If you have access to a Windows box, just zip up the JDE or simulator and copy over. Otherwise, there are a few tricks to extracting the contents of the setup package.

Extract the setup package:
wine BlackBerry_JDE_4.2.1.exe /extract:c:\\

Which extracts the files to ~/.wine/drive_c/

Use msiexec to install the package:
wine msiexec /a c:\\jde_installer.msi /qn

Which extracts the file to ~/.wine/drive_c/Program Files/Research In Motion/BlackBerry JDE 4.2.1. This is a cumbersome path at best so move this somewhere more usable:
mv ~/.wine/drive_c/Program Files/Research In Motion/BlackBerry JDE 4.2.1 ~/lib/JDE4.2.1

Cleanup:
rm ~/.wine/drive_c/jde_*

Run the simulator

I created a shell script from one of the existing batch files for launching:
cat 8800.bat | dos2unix > 8800.sh
gvim 8800.sh


Replace the @echo off with #!/bin/bash and insert the wine command before the fledge.exe. Mine looks like this:

#!/bin/bash
wine fledge.exe /app=Jvm.dll /handheld=8800 /session=8800 \\
/app-param=DisableRegistration /app-param=JvmAlxConfigFile:8800.xml \\
/data-port=0x4d44 /data-port=0x4d4e /pin=0x2100000A

Set the execute permission:
chmod +x 8800.sh

Now just run it!
./8800.sh

Don’t give up if the first time you run it the thing crashes. Give it a few tries.

Debugger

So you fancy yourself an adventurer do ya!?! Well, good luck.

Thankfully, RIM hasn’t made the same mistake with JDWP as they did with the signature tool but you will still have to modify a number of text files to convert the Windows path names to Unix ones.

cd ~/lib/JDE4.2.1/bin
cp SimPackage-JDE.rc SimPackage-JDE.orig
cat SimPackage-JDE.orig | sed 's/\(Command.*\)=%/\1=wine %/' | sed 's/\\/\//g' > SimPackage-JDE.rc

cp FixedBuild.rc FixedBuild.orig
cat FixedBuild.orig | sed 's/\(Command.*\)=%/\1=wine %/' | sed 's/\\/\//g' > FixedBuild.rc

cp DefaultBuild.rc DefaultBuild.orig
cat DefaultBuild.orig | sed 's/\\/\//g' > DefaultBuild.rc


If you copied the JDE from a Windows box you might need to update other files. You must also remove Settings.rc if it already exists.

Make sure you have the Sun Java Runtime installed by running java -version. Now all that’s left is to start up the JDWP:
java -jar JDWP.jar

On startup, there will be an error message about RIMUsbJni.dll but you can safely ignore this. And that’s just about it; your JDWP client connects as usual (jdb, eclipse, netbeans). Feel free to leave your successes/failures in the comments.

I don’t know what the legality is regarding redistribution of RIM’s development tools, specifically ones that have been modified so what follows is a description of the steps required to modify SignatureTool.jar to work in a Unix style file system (forward slash for directory separators).

Get SignatureTool.jar and extract it

It goes without saying that you will need the Java Runtime installed before starting. I would recommend the Sun JRE. In Ubuntu, you can install this by running sudo apt-get sun-java5-jre

Grab a copy of the SignatureTool.jar and put it somewhere along with your .csk and .db files. I am using SignatureTool.jar from version 4.3 of the JDE. The remainder of this guide might be specific to this version as I will be editing constants and referring to them by index numbers.
~/lib/RIM43/bin/SignatureTool.jar
~/lib/RIM43/bin/sigtool.csk
~/lib/RIM43/bin/sigtool.db


Extract the SignatureTool.jar file into a temporary directory:
cd ~/lib/RIM43/bin
mkdir tmp
jar -xf SignatureTool.jar -C tmp


Editing the class files

Now I tried using a hex editor and a decompiler but by far the easiest method was to use a program called ClassEditor to modify the two files in question. Download the binary release and extract it, and run it with the command:
java -jar ce.jar

Open two class files extracted earlier: q.class and ad.class. ClassEditor will look something like this:

ClassEditor with q.class and ad.class

Start with q.class and select the Constant Pool tab. In this file we need to change string constants containing just a single backslash character. There are actually two constants in the pool but the first is just a reference to the second. The string we want is at index 223.

String constant at index 223

By default ClassEditor starts in read only mode so click the bright green Modify Mode(Off) button in the top right. The fields in the details area are now editable. Simply change the forward slash to a backslash and click Modify. Click save and that’s it for q.class!

String constant 223 modified

Select ad.class on left and locate constant 117, again in the Constant Pool tab.

String constant at index 117

Change the value from \sigtool.set to /sigtool.set and click Modify. Click save and that’s it for ad.class.

String constant 117 modified

Repack SignatureTool.jar and test

To repack SignatureTool.jar just cd to the temporary directory where you extracted the jar and rejar the class files
cd ~/lib/RIM43/bin/tmp
jar -cmf META-INF/MANIFEST.MF ../SignatureTool.jar *


Go up one directory and remove the temp directory
cd ..
rm -rf tmp


To test that everything worked, just run the command
java -jar SignatureTool.jar

If should prompt you to locate a cod file. Just cancel this and click the Properties button in the main signature tool window. If you see a list of Registered Signers it worked!

Properties windows with registered signers

RIM are you listening?

At the off chance that someone from RIM is reading this, please forward this to whoever maintains this tool:

• Locate class containing the string “\sigtool.set”
• Change to java.io.File.separator + “sigtool.set”
• Locate class containing the strings “\\” + “sigtool.db”
• Change to java.io.File.separator + “sigtool.db”
• Locate class containing the strings “\\” + “sigtool.csk”
• Change to java.io.File.separator + “sigtool.csk”

Thanks!