I will show you how to serialize a C# object to Protobuf and deserialize it in Python.
Protobuf Serialization from C# to Python 🚀
In this example, I will show how to use Protobuf-net to serialize complex data structures in C# and then deserialize them in Python using protobuf.
Data Object Structure
Below is a diagram representing a complex C# data object (Person) that contains all primitive data types, a nested object (Address), a list of objects (PhoneNumber), and a dictionary. This structure is suitable for Protobuf serialization.
@startuml
class Person {
int Id
string Name
bool IsActive
double Salary
float Height
long Timestamp
byte Age
char Initial
Address Address
List<PhoneNumber> Phones
Dictionary<string, string> Tags
DateTime BirthDate
decimal Balance
short Score
uint Points
ushort Level
sbyte Rating
}
class Address {
string Street
string City
string Zip
}
class PhoneNumber {
string Type
string Number
}
Person "1" -- "1" Address : has
Person "1" -- "*" PhoneNumber : phones
@enduml
Legend:
- Person contains all C# primitive types, a nested Address, a list of PhoneNumber, and a dictionary.
- Address and PhoneNumber are referenced from Person.
- Arrows indicate object references and collections.
- This structure is ideal for demonstrating Protobuf serialization across languages.
Below comes the sample code for the Person class and its nested classes as well as its serialization and deserialization methods.
using System;
using System.IO;
using ProtoBuf;
using System.Collections.Generic;
[ProtoContract]
public class Person
{
[ProtoMember(1)]
public int Id { get; set; }
[ProtoMember(2)]
public string Name { get; set; }
[ProtoMember(3)]
public bool IsActive { get; set; }
[ProtoMember(4)]
public double Salary { get; set; }
[ProtoMember(5)]
public float Height { get; set; }
[ProtoMember(6)]
public long Timestamp { get; set; }
[ProtoMember(7)]
public byte Age { get; set; }
[ProtoMember(8)]
public char Initial { get; set; }
[ProtoMember(9)]
public Address Address { get; set; }
[ProtoMember(10)]
public List<PhoneNumber> Phones { get; set; }
[ProtoMember(11)]
public Dictionary<string, string> Tags { get; set; }
[ProtoMember(12)]
public DateTime BirthDate { get; set; }
[ProtoMember(13)]
public decimal Balance { get; set; }
[ProtoMember(14)]
public short Score { get; set; }
[ProtoMember(15)]
public uint Points { get; set; }
[ProtoMember(16)]
public ushort Level { get; set; }
[ProtoMember(17)]
public sbyte Rating { get; set; }
}
[ProtoContract]
public class Address
{
[ProtoMember(1)]
public string Street { get; set; }
[ProtoMember(2)]
public string City { get; set; }
[ProtoMember(3)]
public string Zip { get; set; }
}
[ProtoContract]
public class PhoneNumber
{
[ProtoMember(1)]
public string Type { get; set; }
[ProtoMember(2)]
public string Number { get; set; }
}
public class Program
{
public static void Main(string[] args)
{
// Create a sample Person object
var person = new Person
{
Id = 1,
Name = "John Doe",
IsActive = true,
Salary = 50000.75,
Height = 5.9f,
Timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds(),
Age = 30,
Initial = 'J',
Address = new Address
{
Street = "123 Main St",
City = "Anytown",
Zip = "12345"
},
Phones = new List<PhoneNumber>
{
new PhoneNumber { Type = "Mobile", Number = "123-456-7890" },
new PhoneNumber { Type = "Home", Number = "098-765-4321" }
},
Tags = new Dictionary<string, string>
{
{ "Occupation", "Engineer" },
{ "Hobby", "Photography" }
},
BirthDate = DateTime.Now.AddYears(-30),
Balance = 1000.50m,
Score = 95,
Points = 1000,
Level = 1,
Rating = -1
};
// Serialize the Person object to a file
using (var fileStream = File.Create("person.bin"))
{
Serializer.Serialize(fileStream, person);
}
// Serialize using protobuf-net to a byte array and output to stdout
byte[] serializedData;
using (var memoryStream = new MemoryStream())
{
Serializer.Serialize(memoryStream, person);
serializedData = memoryStream.ToArray();
}
using (var stdoutStream = Console.OpenStandardOutput())
{
stdoutStream.Write(serializedData, 0, serializedData.Length);
}
Console.WriteLine("Serialized data written to stdout.");
}
}
Python Deserialization
To deserialize the Protobuf data in Python, you need to have a proto message as well as the auto-generated Python classes. Below is the proto message definition and the Python code to deserialize the data.
syntax = "proto3";
message Address {
string Street = 1;
string City = 2;
string Zip = 3;
}
message PhoneNumber {
string Type = 1;
string Number = 2;
}
message Person {
int32 Id = 1;
string Name = 2;
bool IsActive = 3;
double Salary = 4;
float Height = 5;
int64 Timestamp = 6;
uint32 Age = 7;
string Initial = 8;
Address Address = 9;
repeated PhoneNumber Phones = 10;
map<string, string> Tags = 11;
int64 BirthDate = 12;
double Balance = 13;
int32 Score = 14;
uint32 Points = 15;
uint32 Level = 16;
int32 Rating = 17;
}
To generate the Python classes from the proto file, use the following command:
protoc --python_out=. person.proto
I have encountered issues with the protoc command on Windows, so I recommend using the following command to specify the include path for the Protobuf library. Make sure to replace %username% with your actual username in the path.
protoc -I="C:/Users/%username%/AppData/Local/Microsoft/WinGet/Packages/Google.Protobuf_Microsoft.Winget.Source_8wekyb3d8bbwe/include" -I="." --python_out=. person.proto
One sample generated Python class is as follows in person_pb2.py:
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: test.proto
"""Generated protocol buffer code."""
from google.protobuf.internal import builder as _builder
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\ntest.proto\"4\n\x07\x41\x64\x64ress\x12\x0e\n\x06Street\x18\x01 \x01(\t\x12\x0c\n\x04\x43ity\x18\x02 \x01(\t\x12\x0b\n\x03Zip\x18\x03 \x01(\t\"+\n\x0bPhoneNumber\x12\x0c\n\x04Type\x18\x01 \x01(\t\x12\x0e\n\x06Number\x18\x02 \x01(\t\"\xee\x02\n\x06Person\x12\n\n\x02Id\x18\x01 \x01(\x05\x12\x0c\n\x04Name\x18\x02 \x01(\t\x12\x10\n\x08IsActive\x18\x03 \x01(\x08\x12\x0e\n\x06Salary\x18\x04 \x01(\x01\x12\x0e\n\x06Height\x18\x05 \x01(\x02\x12\x11\n\tTimestamp\x18\x06 \x01(\x03\x12\x0b\n\x03\x41ge\x18\x07 \x01(\r\x12\x0f\n\x07Initial\x18\x08 \x01(\t\x12\x19\n\x07\x41\x64\x64ress\x18\t \x01(\x0b\x32\x08.Address\x12\x1c\n\x06Phones\x18\n \x03(\x0b\x32\x0c.PhoneNumber\x12\x1f\n\x04Tags\x18\x0b \x03(\x0b\x32\x11.Person.TagsEntry\x12\x11\n\tBirthDate\x18\x0c \x01(\x03\x12\x0f\n\x07\x42\x61lance\x18\r \x01(\x01\x12\r\n\x05Score\x18\x0e \x01(\x05\x12\x0e\n\x06Points\x18\x0f \x01(\r\x12\r\n\x05Level\x18\x10 \x01(\r\x12\x0e\n\x06Rating\x18\x11 \x01(\x05\x1a+\n\tTagsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x62\x06proto3')
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'test_pb2', globals())
if _descriptor._USE_C_DESCRIPTORS == False:
DESCRIPTOR._options = None
_PERSON_TAGSENTRY._options = None
_PERSON_TAGSENTRY._serialized_options = b'8\001'
_ADDRESS._serialized_start=14
_ADDRESS._serialized_end=66
_PHONENUMBER._serialized_start=68
_PHONENUMBER._serialized_end=111
_PERSON._serialized_start=114
_PERSON._serialized_end=480
_PERSON_TAGSENTRY._serialized_start=437
_PERSON_TAGSENTRY._serialized_end=480
# @@protoc_insertion_point(module_scope)
Then you can use the following Python code to deserialize the data:
import person_pb2
import sys
import io
import os
import struct
import datetime
import time
import json
def deserialize_person(serialized_data):
person = person_pb2.Person()
person.ParseFromString(serialized_data)
return person
def convert_to_dict(person):
person_dict = {
"Id": person.Id,
"Name": person.Name,
"IsActive": person.IsActive,
"Salary": person.Salary,
"Height": person.Height,
"Timestamp": datetime.datetime.fromtimestamp(person.Timestamp).strftime('%Y-%m-%d %H:%M:%S'),
"Age": person.Age,
"Initial": person.Initial,
"Address": {
"Street": person.Address.Street,
"City": person.Address.City,
"Zip": person.Address.Zip
},
"Phones": [{"Type": phone.Type, "Number": phone.Number} for phone in person.Phones],
"Tags": {key: value for key, value in person.Tags.items()},
"BirthDate": datetime.datetime.fromtimestamp(person.BirthDate).strftime('%Y-%m-%d %H:%M:%S'),
"Balance": person.Balance,
"Score": person.Score,
"Points": person.Points,
"Level": person.Level,
"Rating": person.Rating
}
return json.dumps(person_dict, indent=4)
def main():
# Read serialized data from stdin
serialized_data = sys.stdin.buffer.read()
# Deserialize the data
person = deserialize_person(serialized_data)
# Convert to dictionary and print
person_dict = convert_to_dict(person)
print(person_dict)
Conclusion
By following the steps above, you can seamlessly serialize complex C# objects using Protobuf-net and deserialize them in Python with the official protobuf library. This approach ensures efficient, cross-platform data exchange between .NET and Python applications. If you encounter issues with field types or serialization, double-check your .proto definitions and make sure they match your C# class structure. Protobuf is a powerful tool for building robust, language-agnostic APIs and data pipelines.
Happy coding! 🚀