JSON vs Protobuf and gRPC: Performance Benchmarking

Separate the codec from the RPC framework

JSON and Protobuf are serialization choices; gRPC is an RPC system commonly using Protobuf messages. Comparing JSON over an existing HTTP endpoint with gRPC changes several variables at once: framing, connection behavior, generated clients, error handling, and possibly streaming. A faster end-to-end result does not reveal which variable caused the improvement. Run a codec experiment and a service experiment separately.

The codec experiment should encode and decode equivalent logical records in memory. The service experiment should include the actual client, transport, middleware, deadlines, and handler. Preserve application work and concurrency across both paths. If one handler skips validation or returns fewer fields, the comparison is not measuring transport alone. Document intentional differences rather than hiding them behind a single requests-per-second number.

Start with the business constraint. A public integration may prioritize inspectability and broad client support. An internal high-volume stream may prioritize efficient typed messages and generated contracts. Both can be sensible. The text-format benchmark guide introduces equivalent-model testing; the same discipline applies when one candidate is binary and the surrounding framework has more built-in capabilities.

Understand where binary size savings come from

Protobuf identifies fields numerically and encodes values using wire types rather than repeating textual property names. Small integer values and packed repeated numeric fields can be compact. Large strings still contain their content, so a payload dominated by long text may show less dramatic savings. Field naming length affects JSON size but not the binary field identifier, which is one reason synthetic benchmarks can exaggerate results.

Measure representative distributions, including optional fields, empty collections, long strings, and large numeric values. Compression changes the result because repeated JSON names are highly redundant. Compare uncompressed bodies and the actual compression settings used in production. Do not compress a large batch for one format and individual records for the other unless batching is itself the feature under evaluation.

Presence semantics also affect equivalence. An omitted scalar and a scalar set to its default can have different meanings to an application, depending on the schema's presence discipline. Make optionality explicit when absence matters. A smaller message obtained by discarding that distinction is not a free optimization. Before adopting a schema, specify how null-like values, defaults, and unknown fields should behave across old and new consumers.

Run a small reproducible codec experiment

Create the schema file below, generate the Python module, and run the benchmark from the same directory. Install protobuf and grpcio-tools in a virtual environment and retain the resolved versions in a lock file. The experiment uses a string identifier, a name, and repeated integers. It reports raw size and encode/decode timing for application-equivalent data without claiming that these numbers represent a network service.

Generated modules are part of the experiment's inputs. Keep the schema and generator version with the results because generated code and runtime libraries can affect performance and compatibility. The JSON path constructs a dictionary while the Protobuf path constructs a generated message object; inspect the downstream code to determine whether either needs an additional conversion before use.

Use several independent runs and add realistic records before drawing conclusions. The example's small schema is a reproducible starting point, not a comprehensive workload. Include validation and any mapping to domain objects in a second measurement. If a service immediately converts a Protobuf message into a dictionary and then back into another model, that allocation work belongs in the architectural comparison even though it is absent from the raw codec timing.

syntax = "proto3";
package bench;
message Record {
  string id = 1;
  string name = 2;
  repeated int32 samples = 3;
}

Measure encode and decode independently

Encoding and decoding can have different costs and deployment locations. A producer sending to many consumers may justify optimizing decode even if encoding becomes slightly more expensive. A write-heavy ingestion path may have the opposite balance. Report both directions, serialized bytes, and the shape of the returned value. Avoid a single combined timing that conceals where CPU is actually spent.

The benchmark below performs repeated operations with no file or network I/O inside the timed loop. This isolates the codec but also removes costs present in production. Treat its results as a component measurement. Add process-level resident memory measurements and allocation profiles separately; Python-level allocation tracing does not necessarily observe every allocation in native implementations.

CPU cycles are hardware-specific and should be collected with suitable platform tooling when needed. Wall-clock timing is not a cycle count. Record processor model, frequency policy, runtime build, and competing load. Avoid interpreting a tiny timing difference as meaningful without variability estimates. A production decision should remain stable across representative payloads, not depend on one favorable run of a short sample.

import json
import statistics
import timeit
import record_pb2

data = {"id": "a17", "name": "sample", "samples": list(range(100))}
message = record_pb2.Record(**data)
json_bytes = json.dumps(data, separators=(",", ":")).encode()
proto_bytes = message.SerializeToString()

operations = {
    "json encode": lambda: json.dumps(data, separators=(",", ":")).encode("utf-8"),
    "json decode": lambda: json.loads(json_bytes),
    "proto encode": message.SerializeToString,
    "proto decode": lambda: record_pb2.Record.FromString(proto_bytes),
}
assert list(record_pb2.Record.FromString(proto_bytes).samples) == data["samples"]
print("bytes", {"json": len(json_bytes), "protobuf": len(proto_bytes)})
for name, operation in operations.items():
    samples = timeit.repeat(operation, repeat=7, number=5000)
    print(name, statistics.median(samples) / 5000, "seconds/op")

Design schema evolution before optimizing

Binary field identifiers become durable protocol meaning. Do not reuse a removed field number for an unrelated value, because old serialized data can then be interpreted as the new field. Reserve retired identifiers and names where appropriate. Review type changes for wire compatibility and application meaning instead of assuming that similar language types are interchangeable.

Unknown-field behavior matters during rolling deployments and message forwarding. A service that parses a message, maps only known fields into a custom object, and serializes a new message can discard information that a direct binary relay would preserve. Test the actual transformation path. JSON intermediaries can lose unknown fields too when response schemas or explicit projections remove them.

Presence, defaults, and repeated fields deserve migration fixtures. Distinguish a client intentionally setting a value from one that never knew the field existed. Keep old-reader/new-writer and new-reader/old-writer tests, plus replay tests for stored messages. Generated code gives useful structure, but it cannot decide whether a semantic change is acceptable to downstream business logic. A successful compilation is weaker evidence than an end-to-end compatibility corpus.

Benchmark gRPC as a running system

For the service experiment, use the same data, handler logic, TLS policy, and server resources. Reuse connections according to each client's normal production behavior. Include deadlines and cancellation, because requests that continue after the caller has given up consume capacity without delivering useful work. Measure successful requests separately from timeouts and rejected requests.

Observe latency distributions under increasing concurrency, not just maximum throughput at saturation. Queueing can cause tail latency to rise sharply before average CPU appears alarming. Record request and response sizes, active connections, worker utilization, and downstream dependency time. If streaming is the proposed improvement, compare equivalent application flows rather than a stream against an artificially inefficient sequence of new connections.

Also measure operational costs: gateway compatibility, browser access strategy, debugging tools, generated-client distribution, and schema release coordination. A service team may gain efficiency while imposing complexity on external consumers. The GraphQL and REST guide explores a related tradeoff where response shape, caching, and client ergonomics matter as much as serialization speed. Architecture decisions should include those costs instead of treating them as implementation details to solve later.

Memory and allocation can dominate the outcome

A compact wire representation does not guarantee a compact in-memory graph. Generated objects, strings, repeated-field containers, and domain-model copies all allocate memory. The peak often occurs while both serialized bytes and decoded structures remain live. Under concurrency, multiply that retained working set by the number of active requests rather than considering only one isolated parse.

Measure decode-only memory and complete-handler memory. Inspect whether application code retains messages in caches, queues, or tracing contexts. Reusing buffers can reduce allocation, but ownership must be clear so one request cannot observe another request's data. Pooling a large buffer may also retain more memory than allocating a modest temporary buffer, especially after an unusually large request.

For Go services, benchmark allocations per operation and review the typed representation before replacing the codec. The Go JSON performance guide shows why generic maps, repeated conversions, and lifetime choices can dominate parser selection. Optimize the representation and processing path first, then compare codecs under the improved design. Otherwise the migration may preserve the expensive architecture while changing only the wire syntax.

Choose with an explicit decision record

Prefer JSON when broad compatibility, simple inspection, and loosely coordinated clients outweigh measured codec costs. Prefer a typed binary protocol when controlled clients, strong schema discipline, and high-volume communication make generated contracts and transport capabilities worthwhile. Neither choice eliminates schema evolution, validation, or observability requirements.

Write the decision using measured constraints: the payload corpus, software versions, concurrency, CPU budget, latency objective, and compatibility obligations. State which results are measured locally and which remain hypotheses pending deployment. Keep the raw benchmark outputs so future engineers can reproduce the comparison when the service changes. Do not turn a vendor benchmark into a promise about your workload.

Roll out through a bounded compatibility period with metrics for protocol adoption, errors, and fallback use. Keep authorization equivalent across endpoints and avoid a weaker alternate path. Retire unused adapters after clients migrate. The best outcome is not a winning chart; it is a protocol whose performance, failure behavior, and evolution rules are understood well enough that the team can operate it confidently.

Build a benchmark manifest and interpret break-even points

Store a manifest beside benchmark output containing schema revision, generator version, runtime version, machine architecture, payload generator seed, record count, and compression settings. Include the exact command used to run each phase. This is enough context to explain many apparent regressions months later, when a compiler or dependency upgrade has changed the execution environment.

Measure at several message sizes. For very small messages, framing and scheduling overhead may dominate; for larger messages, encoding and allocation become more visible. Plot or tabulate the results rather than reporting only one ratio. A useful decision can identify a workload range where one approach helps and another range where it makes little difference.

Keep throughput and latency experiments distinct. A tight in-memory loop measures how quickly one process repeats a component operation. A concurrent client test measures queueing, network behavior, and service capacity. Both are valuable, but neither should be relabeled as the other. A component that is twice as fast cannot make a request twice as fast when it originally accounted for only a small fraction of total work.

Check error-path costs as well. Malformed messages, incompatible schema versions, and cancelled requests are part of a production workload. The server should reject them predictably without repeated expensive retries. If a generated client automatically retries, include that behavior in the service experiment and confirm that it matches operation idempotence.

Finally, calculate migration effort against the measured benefit. Generated-client distribution, gateway support, operational training, and dual-protocol maintenance consume engineering time. A modest CPU reduction can still justify that work at large scale, but it should be quantified against actual traffic and infrastructure costs. Conversely, keeping JSON can be the correct decision when the measured bottleneck lies elsewhere and the existing client ecosystem is valuable.

Document the break-even assumptions explicitly. If traffic volume, payload size, or deployment topology changes, the team can rerun the experiment instead of defending an old conclusion as a universal rule. A benchmark is most useful when it explains when a decision should be reconsidered.

For a schema migration, include a fixture that passes through an older intermediary before returning to a newer reader. This catches unknown-field loss caused by domain-model conversion. A direct old-reader test can miss that forwarding behavior, especially when the intermediary rebuilds a message rather than preserving its original representation.

Engineering Comparison

No universal latency or size ratio is asserted. Run the provided codec benchmark and a separate service-level experiment before selecting a protocol.