protobuf-net: Beware of setting an optional int primitive to zero

Here’s another gotcha with protobuf-net, setting an optional int to 0 causes it the property to be skipped during serialization.

private int? _myProperty;
[ProtoBuf.ProtoMember(21022, IsRequired = false, Name = @"myProperty", DataFormat = ProtoBuf.DataFormat.TwosComplement)]
public int myProperty {
	get { return _myProperty ?? default(int); }
	set { _myProperty= value; }
}
[System.Xml.Serialization.XmlIgnore]
[System.ComponentModel.Browsable(false)]
public bool myPropertySpecified {
	get { return _myProperty != null; }
	set { if (value == (_myProperty == null)) _myProperty = value ? fillreportpolicy : (int?)null; }
}
private bool ShouldSerializemyProperty() { return myPropertySpecified; }
private void ResetmyProperty() { myPropertySpecified = false; }

After you set myProperty to “0”, you’ll notice that myPropertySpecified will return true. You can also use reflection to verify that the property is indeed set. But, it won’t get serialized in the end.

ProtoBuf-net: A helper method

The current implementation of protobuf-net does not contain a simple way of writing the proto as a string, or key-value pairs (unlike the C++ library).

Here’s a simple extension method to aid in logging protos, in case you ever have a similar need.

/// 
/// Returns the contents of the proto object log-friendly string format.
/// This method will also cycle through the repeating groups.
/// Eg: MyProto { fielda [100] = Foo, fieldb [101] = bar }
/// 
public static string ToDebug(this IExtensible proto) {
	StringBuilder sb = new StringBuilder();
	sb.AppendFormat("{0} {{", proto.GetType().Name);
	ReadProto(proto, sb);
	sb.Append("}");
	return sb.ToString();
}

private static void ReadProto(IExtensible proto, StringBuilder sb) {
	object val;
	SortedDictionary props = GetProtoAttributeProperties(proto);
	int[] keys = new int[props.Count];
	PropertyInfo info = null;
	MethodInfo mi = null;

	props.Keys.CopyTo(keys, 0);
	for (int i = 0; i < props.Keys.Count; i++) {
		info = props[keys[i]];
		val = info.GetValue(proto, null);
		if (val == null)
			continue;

		mi = proto.GetType().GetMethod(
					string.Format("ShouldSerialize{0}", props[keys[i]].Name), 
					BindingFlags.NonPublic | BindingFlags.Instance);
		if (mi != null && !Convert.ToBoolean(mi.Invoke(proto, null)))
			continue;
		sb.AppendFormat("{0} [{1}] = {2}",
			info.Name, keys[i], 
			val == null ? string.Empty :
				val is IExtensible ? val.GetType().Name : val.ToString());

		if (val is IExtensible) {
			sb.Append(" {");
			ReadProto((IExtensible)val, sb);
			sb.Append("}");
		} else {
			IList list = val as IList;
			if (list != null) {
				ReadRepeatingGroups(list, sb);
			} else if (i != keys.Length - 1)
				sb.Append(", ");
		} 
	}
}

private static void ReadRepeatingGroups(IList list, StringBuilder sb) {
	if (list == null)
		return;
	if (list.Count == 0) {
		sb.Append("{}");
		return;
	}
	sb.Append(" {");
	for (int i = 0; i < list.Count; i++) {
		sb.Append(" {");
		ReadProto((IExtensible)list[i], sb);
		sb.Append("}");
		if (i != list.Count - 1)
			sb.Append(", ");
	}
	sb.Append("}");
}

private static SortedDictionary GetProtoAttributeProperties(IExtensible proto) {
	SortedDictionary props = new SortedDictionary();
	PropertyInfo[] properties = proto.GetType().GetProperties();
	foreach (PropertyInfo pi in properties) {
		if (!pi.IsDefined(typeof(ProtoBuf.ProtoMemberAttribute), false))
			continue;
		ProtoMemberAttribute attr = null;
		object[] attributes = pi.GetCustomAttributes(false);
		foreach (object a in attributes) {
			attr = a as ProtoMemberAttribute;
			if (attr != null)
				break;
		}
		if (attr != null)
			props.Add(attr.Tag, pi);
	}
	return props;
}

QPID.net oddities

I noticed a couple of odd behavior in the QPID.net implementation so figured I’d document them here.

First, the code…

_client = new Client();
_client.connect( host, port, virtualHost, userName, password );
_session = _client.createSession( -1 );

// Create a response queue so the server can send us responses
_responseQueue = string.Format( "GUI_{0}_{1}", Environment.UserName, DateTime.Now.Ticks );
// Use the name of the response queue as the routing key
_session.queueDeclare( _responseQueue, new Option[] {
	// Setting EXCLUSIVE option makes the queue hang around
	// even when the session is closed
	Option.AUTO_DELETE //, Option.EXCLUSIVE
} );
_session.exchangeBind( _responseQueue, "amq.direct", _responseQueue );

1. Apparently, specifying Option.Exclusive on the queue, as the comment indicates, causes the queue to hang around even after you cleanly tear down the session and connection. (You can use qpid-stat to find out about the open sessions;
   qpid-stat -q localhost:6001 -S queue
   with the port where your daemon is listening to, of course.)

2. Even though I’m not specifying a timeout when creating the session, and given that QPID sessions typically are set to infinite timeouts, the session is closed by the broker after some period of inactivity. You’ll notice that the session has closed the next time you attempt to send a message back to the server. (If you have QPID configured to log everything, you can easily search the log file for your IP address to see if it’s disconnected; you should see entries similar to the following)
   debug DISCONNECTED [10.1.223.77:3356]

   For now, until I find a better solution, to fix this requires a hack. Essentially, you need to check whether the session is closed or not before attempting to send a message (code below). You’ll notice a Sleep call which I found was necessary as sending a message immediately after the session connects causes it to go into the ether

///
/// HACK! Attempt to reconnect if the session was closed.
/// TODO Find a way to keep the session open
///
protected void CheckSessionState() {
	if (_session != null && _session.Closed == false)
		return;
	bool isConnected = false;
	int tryCnt = 5, i = 0;
	while (!isConnected && i < tryCnt) {
		OnMessageWorkerLogMessage(this.GetServiceName() + " was closed, attempting to reconnect");
		Disconnect();
		Connect(this.Host, this.Port, this.VirtualHost, _user, _pwd);
		Thread.Sleep(100);
		isConnected = _session != null && (_session.Closed == false);
		i++;
		if (isConnected) {
			break;
		}
		OnMessageWorkerLogMessage(this.GetServiceName() +
			(isConnected ? " reconnected" :
			" still closed. Attempting " + (tryCnt - 1 - i) + " more times"));
	}
}

Visualizing economics

A nice set of graphics from Visualizing Economics, via Zero Hedge.

Articles on FIXatdl

With the formal introduction of FIXatdl at the last FPL’s EMEA meeting, v1.1 is now an official FIX standard. Here, then, are the latest articles published in FIXGlobal on the new standard.

LHC: some details about the software

Yesterday represented a large milestone for the LHC. Here’re some details on the software.

You can also view LHC’s real time status here and here, the latter being the more interesting from a technology and visualization perspective.

Protobuf-Net: Couple of gotchas

Here’re a couple of issues with Protobuf-net that I ran into recently.

1. C# classes generated with protogen /p:detectMissing flag creates string getters as
   get { return _myStringField ?? ""; }
   This means that string fields that were supposed to be nulls still get sent to the server as an empty string; not an ideal situation.
   The fix is to manually edit the generated class to remove the coalesce from all string getters.
    get { return _myStringField; }
2. If you try to serialize a class where you do not populate an optional enum with no enum value defined for zero (0), you’ll get an exception “The default enum value is not defined for the optional property”. This happens even though Google spec says that in such cases, the 1st value should be used; although, in my opinion, Protobuf-Net should check whether to serialize the enum at all before attempting to get to the valid enum values.
   For example, this enum will throw the aforementioned exception:
   public enum MyEnum {
   MyEnumValueA = 10,
   MyEnumValueB = 100
}
   Unfortunately, there’re no workarounds for this problem at the moment except adding a “MyEnumValueUnknown=0” to the enum.

The author, Marc Gravell, is aware of these issues. Here’s hoping he’ll get a chance to fix these problems quickly.

Sybase acquiring Aleri?

Looks like Sybase is in the process of acquiring Aleri.

It’ll be interesting to see how Sybase manages Aleri and Coral8 products, in both development and deployment (licensing).

FIXatdl v1.1 Release Candidate

The Algoithmic Trading Working Group uploaded the v1.1 Release Candidate (internal build v2.7.0) for FIXatdl recently. You’ll need to register in order to download the files.

So start coding up those algo UIs already!

Here’s an earlier post that highlights some of the breaking changes with previous builds.

ProtoBuf.net – Handling optional primitives

If you are dealing with protos containing optional primitives, it’s not immediately obvious how you’d deal with them in C#.

For example, if the proto contains an optional int and you generated the C# class, the results would as follows:

optional int32 bar_ukey = 13;

>protogen -i:BarTest.proto -o:BarTestProto.cs

private int _bar_ukey =default(int);
[ProtoBuf.ProtoMember(13, IsRequired = false, Name=@"bar_ukey", DataFormat = ProtoBuf.DataFormat.TwosComplement)][System.ComponentModel.DefaultValue(default(int))]
public int bar_ukey
{
get { return _bar_ukey; }
set { _bar_ukey = value; }
}


As you can see, the highlighted default value doesn’t take the optional nature of the field into account.

To fix this, protogen needs to be run with detectMissing argument as follows:
protogen -i:BarTest.proto -o:BarTestProto.cs -p:detectMissing
which results in the property getting generated as


private int? _bar_ukey;
[ProtoBuf.ProtoMember(13, IsRequired = false, Name=@"bar_ukey", DataFormat = ProtoBuf.DataFormat.TwosComplement)]
public int bar_ukey
{
get { return _bar_ukey?? default(int); }
set { _bar_ukey = value; }
}
[System.Xml.Serialization.XmlIgnore]
[System.ComponentModel.Browsable(false)]
public bool bar_ukeySpecified
{
get { return _bar_ukey != null; }
set { if (value == (_bar_ukey== null)) _bar_ukey = value ? bar_ukey : (int?)null; }
}
private bool ShouldSerializebar_ukey() { return bar_ukeySpecified; }
private void Resetbar_ukey() { bar_ukeySpecified = false; }


Side note: Also tucked away in the -p template property observable and preObservable options for the observable pattern.