Using Protobuf-Net, I Suddenly Got an Exception About an Unknown Wire-Type

Using Protobuf-net, I suddenly got an exception about an unknown wire-type

First thing to check:

IS THE INPUT DATA PROTOBUF DATA? If you try and parse another format (json, xml, csv, binary-formatter), or simply broken data (an "internal server error" html placeholder text page, for example), then it won't work.

---

What is a wire-type?

It is a 3-bit flag that tells it (in broad terms; it is only 3 bits after all) what the next data looks like.

Each field in protocol buffers is prefixed by a header that tells it which field (number) it represents,
and what type of data is coming next; this "what type of data" is essential to support the case where
unanticipated data is in the stream (for example, you've added fields to the data-type at one end), as
it lets the serializer know how to read past that data (or store it for round-trip if required).

What are the different wire-type values and their description?

• 0: variant-length integer (up to 64 bits) - base-128 encoded with the MSB indicating continuation (used as the default for integer types, including enums)
• 1: 64-bit - 8 bytes of data (used for double, or electively for long/ulong)
• 2: length-prefixed - first read an integer using variant-length encoding; this tells you how many bytes of data follow (used for strings, byte[], "packed" arrays, and as the default for child objects properties / lists)
• 3: "start group" - an alternative mechanism for encoding child objects that uses start/end tags - largely deprecated by Google, it is more expensive to skip an entire child-object field since you can't just "seek" past an unexpected object
• 4: "end group" - twinned with 3
• 5: 32-bit - 4 bytes of data (used for float, or electively for int/uint and other small integer types)

I suspect a field is causing the problem, how to debug this?

Are you serializing to a file? The most likely cause (in my experience) is that you have overwritten an existing file, but have not truncated it; i.e. it was 200 bytes; you've re-written it, but with only 182 bytes. There are now 18 bytes of garbage on the end of your stream that is tripping it up. Files must be truncated when re-writing protocol buffers. You can do this with FileMode:

using(var file = new FileStream(path, FileMode.Truncate)) {
    // write
}

or alternatively by SetLength after writing your data:

file.SetLength(file.Position);

Other possible cause

You are (accidentally) deserializing a stream into a different type than what was serialized. It's worth double-checking both sides of the conversation to ensure this is not happening.

How to fix Invalid wire-type issue in c# protobuf-net

After I did some digging, I found out that I had put the parameters in the wrong order. Deserialization should look like this

using var memStream = new MemoryStream(bytes);
var result = Serializer.Deserialize(type, memStream);

Invalid Wire Type Exception on Protobuf-net Deserialize with Nested Data (C++ to C#)

My incoming data had about 50 extra bytes of "stuff" I wasn't expecting (i.e. not the message header), which didn't adhere to the defined message format, so the data was essentially corrupt. It was hard to tell this from looking at a stream of bytes; what gave it away was the difference in length of a message that I serialized on the C# side compared to the bytes I read from wireshark of a message coming in. I then looked at those bytes and found equivalent bytes to my serialized message a little ways in.

Why there is extra "stuff" in the incoming message is another matter, but this is an implementation detail, so I'll consider this question closed.

If it helps anyone in a similar situation, I made a little test loop to keep trying to deserialize byte-by-byte until it works (could be improved but works for now):

var rawBytes = msg.GetBytes(); // The raw incoming message
bool success = false;
OuterMsgType outer;
while (!success)
{
    try
    {
        rawBytes = rawBytes.Skip(1).ToArray();
        outer = ProtoBuf.Serializer.Deserialize<OuterMsgType>(new MemoryStream(rawBytes));
        if (outer?.InnerMsg != null)
             success = true;
    }
    catch (Exception e)
    {
        // Wire type exception: Ignore it, don't care
    }
}

protobuf-net: incorrect wire-type exception deserializing Guid properties

Agreed, you shouldn't need an explicit dependency - DataMember is fine. And protobuf-net uses the same logic re ignore etc. How / where are you storing the data? In my experience the most common cause of this is that people are over-writing a buffer (or file) with different data, and not truncating it (leaving garbage at the end of the stream), as discussed here. Is this related to your scenario?

Error Unexpected wire type in a protobuf response in PHP

In this case the error is because the string is invalid.

And it is invalid because the value (string) returned by CURL includes all the data from the HTTP request, for example:

HTTP/2 200 
content-type: application/x-protobuf
date: Wed, 02 Jun 2021 23:12:37 GMT
content-length: 172

╔╔eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjI1MTQwMTAsInVzZXJuYW1lIjoiNTM1MTAyODA2MCIsInZlcnNpb24iOiIyMTgwNiJ9.Li-bp3bIPdIrsRhuTWEWToS0ds62VCG-a2PCGaKSrigڲօ═"

I didn't notice this because I thought those details were due to the CURLOPT_VERBOSE => true option.

Now, how to extract the body from the value returned by CURL is a bit complex, in this question you will see why.

My solution is as follows (I repeat, read the question I mentioned, maybe a simpler solution will work for you):

// I can't trust the Content-Length header (it may or may not be present).
// However, I can reliably calculate the length of the headers and use it instead.

$responseHeadersLength = 0;

$handle = curl_init($url);

$curl_options = [
    // ...
    CURLOPT_HEADERFUNCTION => function ($curl, $header) use (&$responseHeadersLength) {
            $len = strlen($header);
            $responseHeadersLength += $len;
            return $len;
    }
];

curl_setopt_array($handle, $curl_options);

$response = curl_exec($handle);

$body = trim(substr($response, $responseHeadersLength));

protobuf-net: ProtoException in serializing and deserializing nullable enum with length prefix

When you pass a Nullable<T> into an API that takes object, it is boxed, and the boxing rules for Nullable<T> are special: it boxes to either a regular null, or a box of T, not a box of T?; in other words: an object value never contains a nullable value-type. Because of this, from the library's perspective you sent a ConsoleColor, not a ConsoleColor?, so:

Type Resolver(int fieldNumber)
{
    return typeof(ConsoleColor);
}

As a side note, though: the resolver API is very specific and niche, and there are usually better ways to do things. If you can tell me what you're trying to achieve, I can probably offer more guidance.

(I'm trying to think whether there is ever a scenario where the resolver could meaningfully return a typeof(Nullable<T>) for some T - we could perhaps have the library just unwrap the nullable itself, or raise a more meaningful message)

---

Related Topics