This change introduces a new experiment `call_tracer_send_initial_metadata_is_an_annotation`. When enabled, the `CallTracer::RecordSendInitialMetadata` method will now record a `SendInitialMetadataAnnotation` and call a new `MutateSendInitialMetadata` method on the underlying `CallTracerInterface`.
The `CallTracerInterface` and its implementations (including XorMetrics, OpenCensus, OpenTelemetry, and test fakes) have been updated to include the new `MutateSendInitialMetadata` virtual method. The existing `RecordSendInitialMetadata` implementations are modified to check the experiment flag and delegate to `MutateSendInitialMetadata` if the experiment is active.
A new `SendInitialMetadataAnnotation` class is added, which inherits from `CallTracerAnnotationInterface::Annotation`. This annotation type is used to capture the state of the initial metadata for immutable tracing purposes.
Additionally, `ForEachKeyValue` methods are added to `MetadataInfo` and `HttpAnnotation` to facilitate iterating over metadata key-value pairs for annotation recording. The experiment configuration files are updated to include the new experiment.
PiperOrigin-RevId: 854227812
When a `std::multimap` has multiple entries with the same key, calling `m.find(key)` returns an unspecified element.
Historically, this returns the first matching element. However, this is not guaranteed, and recent libc++ changes make this return an arbitrary element.
Using `m.equal_range(key)` is a replacement that will preserve the current behavior. The behavior of this is guaranteed to return a range of all matching elements in insertion order, and the beginning of the range is the same element as what's normally returned by `m.find(key)`.
Closes#41279
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/41279 from rupprecht:multimap-find 70b116441d03eff80523e010b25336f5a75c70c2
PiperOrigin-RevId: 852844558
gRPC is currently getting formatted with two different clang-format implementations, and due to some weirdness they have different include file orderings. This change introduces clang-format configuration to ensure that the two systems align - it's *highly* expected that this will need some maintenance going forward as the two systems evolve.
Closes#40905
PiperOrigin-RevId: 819606209
See `grpc_check.h`. This code redefines the abseil `CHECK*` macros using custom gRPC macros when building tests. In `bazel test ...` builds, on check failure, `PostMortemEmit()` will dump state to the log before crashing.
Caveat: to prevent circular dependencies, code that `postmortem` relies on cannot use the custom gRPC CHECK macros. This is not much code, ~50 source files. grep for the `absl/log:check` bazel dependency.
Closes#39945
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/39945 from drfloob:grpc_check ca8e46718f2021e0df79aa67a3a0b0c751b3ce44
PiperOrigin-RevId: 807452496
Change up latent-see so that it's an always-on thing.
Most of the testing code we've got so far should be updated to use `grpc_core::latent_see::Collect()` - there's a JSON exporter there, and I plan to implement a Fuchsia Trace Format one eventually too (much more compact trace representaton!).
Also added is a service (visibility protected!) that we'll be able to share with partners to allow collecting this data from real systems. That part is strictly opt-in for the moment.
Closes#39781
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/39781 from ctiller:nicety b2e855d9279ee4724961f479ba5c714c823e48a2
PiperOrigin-RevId: 786425381
At the same time, makes the change to elevate channelz to a first class thing a transport should provide.
We still (for the moment) allow a transport to return a null SocketNode, but the transport must explicitly opt to do so.
In turn, update subchannel, server code to use the transport API to retrieve this object, instead of using an alternative squirrly method that's different between client, server.
Closes#39233
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/39233 from ctiller:cg-tracer 7141b9bd8ccb802b21e1ba428f80c0fde05083f4
PiperOrigin-RevId: 749971664
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#39189
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/39189 from ctiller:http2-trace 547f78604124cc3005fbee6098487a2110afb926
PiperOrigin-RevId: 746174350
We never want to call the connection recording methods on calls, nor call level metrics on connections - so there's really two interfaces here.
This CL separates those two interfaces to ease tracing data flow through the stack.
PiperOrigin-RevId: 743264459
An experiment was conducted over the past few months to bypass the ApplicationCallbackExecCtx, instead running all application callbacks on an EventEngine. The rollout is completing ~today (2025-01-13). After another week or so without issues, this PR can be merged to delete the ApplicationCallbackExecCtx.
Closes#38444
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/38444 from drfloob:rm-acec 79a6047af0406e614973c13fdfbfc1d0de9026ed
PiperOrigin-RevId: 725691141
Fix https://github.com/grpc/grpc/issues/38092
This change fixes the alarm object reuse on cancellation cases for the CQ-based alarm and callback alarm. The solution of letting `Set` wait for `cq.Next` is proposed by @drfloob in https://github.com/grpc/grpc/pull/38160.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#38114
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/38114 from yijiem:38092 e1aceffc2a86b22cf3bf081ed862f1f2026fea29
PiperOrigin-RevId: 702067567
This change upgrades the sanity test to use Clang 19, including clang-format and clang-tidy. (It's a partial implementation of the changes proposed in #38038)
Key updates:
- Docker images now utilize Clang 19.
- Code has been reformatted using the updated clang-format.
- Resolved `readability-math-missing-parentheses` warnings raised by clang-tidy.
Note that the other part of the clang-19 upgrade, "using clang-19 for C++ test" will be done once opentelemetry-cpp fixes the clang-19 build error.
Closes#38070
PiperOrigin-RevId: 693833548
In https://github.com/grpc/grpc/pull/37077, the "different message size" benchmarks were made identical to the benchmarks defined on the lines above them.
Closes#37935
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37935 from drfloob:rm-duplicate-callback-benchmarks ad3f5694148e9a68ef70576a6932526e6665146c
PiperOrigin-RevId: 686647838
These look too large for the configured timeouts internally... will revisit later once the system is starting to be more ready.
Closes#37905
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37905 from ctiller:lol-nope 6d44e515b31405cd4b70954cb9ebbd2391c3d6c0
PiperOrigin-RevId: 685223450
We'll probably disable some next week :)
But I want to watch a good selection and refine criteria for acceptance.
Closes#37903
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37903 from ctiller:ALL-the-things 5f829db87019ae81c64fceaf679d5adc50e115dd
PiperOrigin-RevId: 685010911
Continuation of #37541 but focused on C++.
Closes#37755
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37755 from NathanBaulch:typos-cpp 9a7fd9d65fa97a689d529772534df1a501031e9f
PiperOrigin-RevId: 684600898
The following files have been moved:
- src/core/lib/avl/*
- src/core/lib/backoff/*
- src/core/lib/debug/event_log*
- src/core/lib/iomgr/gethostname*
- src/core/lib/iomgr/grpc_if_nametoindex*
- src/core/lib/matchers/*
- src/core/lib/uri/* (renamed from uri_parser.* to uri.*)
- src/core/lib/gprpp/* (existing src/core/util/time.cc was renamed to gpr_time.cc to avoid conflict)
Closes#36792
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36792 from markdroth:reorg_util d4e8996f481c611ffbb06a8b04924ff81bc1bc2b
PiperOrigin-RevId: 676947640
[Gpr_To_Absl_Logging] Remove gpr logging header include from other headers
Some of the cc files are using gpr_log_verbosity_init() functions.
So I removed the #include <grpc/support/log.h> from all headers and put it selectively only in the cc files that used gpr_log_verbosity_init()
Closes#37513
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37513 from tanvi-jagtap:remove_gpr_headers_from_headers_01 612ca6d0f7bc76a382c2f4b84371ee60977eb1e9
PiperOrigin-RevId: 663811895
[Gpr_To_Absl_Logging] Remove logging header from example and test/cpp folder
Closes#37488
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37488 from tanvi-jagtap:remove_header_examples 1423ee77fff3136f3157993c07a5288552768e0d
PiperOrigin-RevId: 663597077
The microbenchmarks are tools for us, and it doesn't seem like we're buying much by maintaining them for different build systems.
Closes#37391
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37391 from ctiller:pickthis 24184f2b443fedbe16c213939107f5764d6d7ad6
PiperOrigin-RevId: 658861012
Sample output:
```
➜ grpc git:(otel-metrics-benchmark) ✗ bazel-bin/test/cpp/microbenchmarks/bm_stats_plugin
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1721695619.615582 2126186 config.cc:257] gRPC experiments enabled: call_status_override_on_cancellation, call_tracer_in_transport, event_engine_dns, event_engine_listener, monitoring_experiment, pick_first_new, trace_record_callops, work_serializer_clears_time_cache
2024-07-22T17:46:59-07:00
Running bazel-bin/test/cpp/microbenchmarks/bm_stats_plugin
Run on (48 X 2450 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x24)
L1 Instruction 32 KiB (x24)
L2 Unified 512 KiB (x24)
L3 Unified 32768 KiB (x3)
Load Average: 1.16, 0.85, 0.85
***WARNING*** Library was built as DEBUG. Timings may be affected.
---------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------
BM_AddCounterWithFakeStatsPlugin 1738 ns 1738 ns 404265
BM_AddCounterWithOTelPlugin 757 ns 757 ns 928142
I0000 00:00:1721695621.304593 2126186 test_config.cc:186] TestEnvironment ends
```
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#37282
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37282 from yijiem:otel-metrics-benchmark eeba3dfb5e2991e7093bfaa613ac89e9215043f5
PiperOrigin-RevId: 655286398
Instead of passing the transport byte counts back up through the filter
stack to be reported to the `CallTracer`, we now have the transport
pass the transport byte counts directly to the `CallTracer` itself.
This will eventually allow us to avoid unnecessarily storing these byte
counts in cases where no `CallTracer` actually cares about the data, which
will reduce per-call memory. (In the short term, it actually increases
memory usage, but we can separately do some work to avoid the memory
usage in the transport by removing the `grpc_transport_stream_stats`
struct from the legacy filter API.)
This is a prereq for supporting `CallTracer` in the new call v3 stack,
which does not include the transport byte counts as part of the
receieve-trailing-metadata hook, unlike the legacy filter stack.
This change is controlled by the `call_tracer_in_transport` experiment,
which is enabled by default.
As part of this experiment, we also fix a couple of related bugs:
- On the client side, the chttp2 transport was incorrectly adding
annotations to the parent `ClientCallTracer` instead of the
`CallAttemptTracer`.
- The OpenCensus `ServerCallTracer` was incorrectly swapping the values
of sent and received bytes.
PiperOrigin-RevId: 650728181
Since these were disabled they stopped working, and we really need to be tracking overheads here.
Closes#37077
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37077 from ctiller:it-must-work aa19a4aa89e8e706b60928abfd2501f5f4a99a07
PiperOrigin-RevId: 648469428
As we've learned what configuration is needed for our benchmarks the settings have been growing more and more bespoke for each binary. Try to consolidate that into some useful defaults.
Also ensure we always `linkstatic=1`. `cc_binary` defaults to this, so it's reasonable to assume that's the performance our customers see. It also deeply impacts performance for small microbenchmarks, and so enabling it gives us more apples:apples, and saves chasing things that don't matter.
Closes#37012
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/37012 from ctiller:benchmark-bin b84cab58c61c75fc286c2ed7c0b393f2048915b6
PiperOrigin-RevId: 645483183
Specifically:
- use `OrphanablePtr<>` for `grpc_endpoint`
- use `absl::AnyInvocable<>` instead of `grpc_closure`
- use `EventEngine::Run()` instead of `ExecCtx::Run()`
- use `SliceBuffer` instead of `grpc_slice_buffer`
- use `absl::Status` instead of `grpc_error_handle`
- use `absl::string_view` instead of `const char*` for handshaker names
Also pass acceptor via `HandshakerArgs` instead of as a separate parameter.
Also changed chttp2 and httpcli to use `OrphanablePtr<>` for the endpoint.
PiperOrigin-RevId: 644551906
