Protobuf Ch.6: Optional Is Just a Synthetic Oneof

0:00 Welcome to Learning Podcasts. Protobuf, chapter 6: Optional Is Just a Synthetic Oneof. By the end of this chapter, the 2nd feature is going to disappear into the 1st one. You are designing a notification service. A notification can be delivered 3 ways: email, SMS, or push. Each delivery method needs different fields. Email needs a recipient address and a subject line. SMS needs a phone number. Push needs a device token. The obvious schema is 1 big message with all 3 sets of fields, and a producer that fills in only the ones that matter.

0:35 So picture the producer that sets the email block, then sets the SMS block, then sets the push block. All 3. The message is happily serialized. The downstream consumers each look at their own fields and find data. You just sent the same notification 3 times. Hope is not a schema constraint. Protobuf gives you a feature called oneof. Inside a message, you write the keyword oneof, you give the group a name, and inside curly braces you list member fields. Each member gets its own type, name, and field number, exactly like a regular field.

1:09 So a notification message gets a oneof named delivery_method, and inside that block you put Email email equals 1, Sms sms equals 2, and Push push equals 3. 3 field numbers, 3 types, all bound together by 1 logical group. 1 slot for 1 of 3 things. Now the runtime starts having opinions. At most 1 member of the oneof can hold a value at any time. Set the email member, and the email is there. Then set the sms member, and the email is gone. Automatically. The runtime cleared it for you. There is no path through the API where you can have 2 members populated at the same time.

1:48 The mutual exclusion is enforced by the generated code. Worth 1 detail. The name of the oneof, delivery_method, is not a field on the wire. It is a logical grouping that the generated code uses so you can ask which member is currently set. The members themselves are encoded the same way they would be outside a oneof. Mutual exclusion, free of charge. Once you set a member, you usually need to ask which one. Each language exposes this differently. In Python, the generated message class gives you a method called WhichOneof.

2:21 You call it with the name of the oneof group as a string, and it returns the field name of whichever member is currently populated, or None if nothing is set. So notification dot WhichOneof, with the string "delivery_method", returns "sms" if you set the sms member. In Java, the compiler generates an enum called DeliveryMethodCase, with 1 value per member field plus a not-set value. The message class gets a method called getDeliveryMethodCase, which returns the enum, and you feed that into a switch statement.

2:53 Same split we saw in chapter 3. Python checks at runtime, Java checks at compile time. There is 1 important restriction. You cannot put repeated fields or map fields inside a oneof. The reason is presence ambiguity. A oneof needs to know definitively whether a member is set or not set. Repeated and map fields have no encoded distinction between set-but-empty and not-set-at-all. An empty list and a missing list look identical on the wire. That is incompatible with mutual exclusion. The workaround, when you need a collection inside a oneof, is to wrap the list in its own message type and use that message as the oneof member.

3:38 Wrap the list, then it fits. The notification scenario is 1 pattern. The cleaner one shows up in API design. Picture a lookup endpoint that either finds a user or returns an error. The response message gets a oneof named outcome, with a User user member and an Error error member. The consumer calls WhichOneof, gets back either "user" or "error", and branches accordingly. Compare that to the older approach: both fields present, only 1 populated, a comment in the proto file saying "only 1 of these will be set."

4:10 Documentation does not stop bugs. The oneof does. The schema replaces the comment. So oneof solves mutual exclusion. There is a 2nd, related problem that oneof does not solve, and that is field presence. Imagine a configuration service. A field called rate_limit is an int32. The sender writes 0, on purpose, to mean "no rate limit." The receiver deserializes the message and reads 0. Now imagine the 2nd case. The sender never set rate_limit at all. The receiver deserializes the message and reads 0.

4:43 After the wire, those 2 cases look identical. There is no presence bit on a proto3 scalar. 0 is the default, and the default looks exactly like absent. Chapter 5 made us put UNSPECIFIED at 0 for enums for exactly this reason. Now apply that thinking to every scalar. This part has a history. Proto2 had 2 presence keywords. Required, which said the field had to be set or the message was invalid. And optional, which said the field carried a presence bit so you could distinguish set-from-0 from never-set.

5:14 Required caused real outages, because removing a required field broke every existing reader. Optional carried per-scalar memory overhead and complexity. When proto3 shipped in 2016, the team dropped both keywords. Every scalar became implicitly optional with no presence tracking. That was the right call for required. It was an over-correction for optional. Configuration services and update operations and partial-merge APIs all needed presence, and proto3 had taken it away. The community pushed back for years.

5:45 In version 3.12, the team shipped optional as an experimental feature. In 3.15, February of 2021, it graduated to stable and was enabled by default. Marking a scalar field optional does 1 specific thing. It enables explicit presence tracking for that 1 field. So if you write optional int32 score equals 5 inside a message, the generated code now exposes a has-method. In Java, you get hasScore returning true only if the sender explicitly wrote the field, even if the value happens to be 0. In Python, you call HasField with the string "score" and get the same answer.

6:25 Without the keyword, those checks do not exist for scalars. 0 is indistinguishable from absent. 1 keyword, 1 bit of clarity per field. The wire has to cooperate too. With optional, when the sender writes 0, the encoder still emits the bytes. Tag, varint, 0. The bytes look like any other varint from chapter 4, just present this time instead of stripped. Without optional, 0 matches the default and the encoder skips the field entirely. So an explicit 0 with optional and an absent field without optional are now genuinely different bytes.

7:01 The presence bit is real, all the way down. Different bits, different meanings. And here is the title of this chapter. Optional is not a separate mechanism. The compiler implements optional by wrapping the field in an unnamed oneof with 1 member. Internally it is sometimes called a synthetic oneof. You never see it in your proto file, but it gives the field a presence bit using exactly the same machinery a regular oneof uses to know which member is set. The 2 features in this chapter are mechanically the same machine.

7:34 Mutual exclusion across many members on 1 side, and explicit presence for 1 member on the other. 1 mechanism, 2 APIs. That is why optional could ship without inventing new wire format. The protobuf documentation now recommends adding optional to every proto3 scalar field. 2 reasons. You get presence tracking, which prevents a whole class of 0-versus-unset bugs. And it gives you a smoother migration path to protobuf editions, the successor to proto2 and proto3, which uses explicit presence by default.

8:09 Editions are a chapter 21 topic. The short version: if you are writing new proto3 today, mark your scalars optional. New code, new default. 2 features, 1 machine. Reach for oneof when you have multiple fields that are mutually exclusive by nature, and you want the runtime to enforce that exclusion across all of them. Reach for optional when you have a single scalar field and you need to tell explicit 0 apart from never set. Mechanically, optional is a oneof with a single member. The mental models are different, the use cases are different, but the wire format and the presence bit come from the same place.

8:46 1 machine, 2 APIs. You now have protobuf's tools for mutual exclusion and explicit presence. 2 APIs, 1 mechanism, both compile-time safe. Next chapter we look at default values and the 0-value trap, the bugs that come from 0 being the implicit default for every scalar. Thanks for listening to Learning Podcasts.