This change adds `MetricsDomainNode` and `MetricsDomainStorageNode` to channelz. `QueryableDomain` and `DomainStorage` now inherit from `channelz::DataSource`, exposing domain metadata, labels, and metric values through channelz.
PiperOrigin-RevId: 806081893
This reverts PR #40481#40481 was work in progress and doesn't fix the mentioned observability test yet. It was wrongly submitted due to an earlier approval.
Closes#40662
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40662 from sreenithi:revert_wrong_commit bc8f55c2ed6c5fb8220f37827a43bb83a7d6511d
PiperOrigin-RevId: 805405106
#40417 has caused the Python Basic Tests job to fail continuously since this morning with the error
```
from grpc_observability import _open_telemetry_observability
File "/var/local/git/grpc/py39/lib/python3.9/site-packages/grpc_observability/_open_telemetry_observability.py", line 23, in <module>
from grpc_observability import _cyobservability
ImportError: /var/local/git/grpc/py39/lib/python3.9/site-packages/grpc_observability/_[cyobservability.cpython-39-x86_64-linux-gnu.so](https://www.google.com/url?q=http://cyobservability.cpython-39-x86_64-linux-gnu.so&sa=D): undefined symbol: _ZN9grpc_core17instrument_detail15QueryableDomain19AllocateDoubleGaugeESt17basic_string_viewIcSt11char_traitsIcEES5_S5_
```
This PR fixes it by adding the required dependency in grpcio-python-observability.
Passing Basic Tests Run using this fix:
https://btx.cloud.google.com/invocations/239da2ca-1394-42f7-aa5c-ac63b938b200/targetsCloses#40481
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40481 from sreenithi:fix_basic_test_breakage 124d6da56ccca7eae92a64d911ab6eb31f05175f
PiperOrigin-RevId: 805276991
This PR changes the logic of `shutdown_grpc_aio` to skip `_actual_aio_shutdown` python interpreter is already [being finalized](https://docs.python.org/3.14/glossary.html#term-interpreter-shutdown) (cleaning up resources, destroying objects, preparing for program exit, etc). `_actual_aio_shutdown` involves `PollerCompletionQueue` shutdown, followed by core [`grpc_shutdown`](https://grpc.github.io/grpc/core/grpc_8h.html#a35f55253e80714c17f4f3a0657e06f1b) API call.
Reasoning:
1. During finalizations, in come cases resources we're accessing may already be freed, and the order is not deterministic. Some of the resources being unloaded prior the `_actual_aio_shutdown` call: `_global_aio_state`, `AsyncIOEngine` enum, or even python libraries like `sys`. This leads to errors like `AttributeError: 'NoneType' object has no attribute 'POLLER'`.
2. `PollerCompletionQueue.shutdown()` will try to wait on its poller thread to finish gracefully. In py3.14, `PythonFinalizationError` is raised when `Thread.join()` is called during finalization. I think the logic here is similar to (1): these threads may have already been deallocated.
Note that in some cases users were able to prevent `_actual_aio_shutdown` from being called by manually calling `init_grpc_aio` prior to initializing any grpc objects. This resulted in an incorrect positive refcount, which prevents `_actual_aio_shutdown` from being run. Before the above finalization check was added this side-effect was sometimes misused to avoid deadlock on finialization (#22365).
This PR:
- Fixes#39520
- Fixes#22365
- Fixes#38679
- Fixes#33342
- Fixes#36655Closes#40447
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40447 from sergiitk:fix/aio/shutdown 11114f6feffd7380e9fde56e7581eb19cf001597
PiperOrigin-RevId: 804971756
This commit introduces the `event_engine_poller_for_python` experiment, allowing for controlled rollout and testing of the EventEngine's Posix poller specifically within gRPC Python bindings.
**Crucially, this change does *not* alter the default behavior for builds where `GRPC_DO_NOT_INSTANTIATE_POSIX_POLLER` is *not* defined.** In such configurations, the Posix EventEngine poller will continue to be enabled unconditionally, preserving existing functionality.
The primary impact of this change is when `GRPC_DO_NOT_INSTANTIATE_POSIX_POLLER` *is* defined (e.g., in gRPC Python's build system). In this scenario, the enablement of the Posix poller transitions from being implicitly disabled to being configurable via the `event_engine_poller_for_python` experiment flag. This enables a controlled, experimental rollout of the EventEngine poller in environments that previously opted out of its direct instantiation.
Closes#40243
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40243 from eugeneo:python-no-backup-poller-experiment 17906e6501b8e6fe7d2ccc63439ec121ef47d43b
PiperOrigin-RevId: 804472655
This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions.
## Distribtests
Required pre-requisite changes to enable 3.14 artifacts are covered in #40289 .
## Bazel tests
Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in #40602
## Basic tests
The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR:
### 1) No running event loop for AsyncIO when run outside an async function
```
Traceback (most recent call last):
File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop
RuntimeError: no running event loop
```
This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in #39507 with the root cause.
Following some investigation, the fix is being worked on by @sergiitk in PR #40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above.
### 2) Pickling error from the `multiprocessing` library
```
_pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types
when serializing dict item '_target'
when serializing multiprocessing.context.Process state
when serializing multiprocessing.context.Process object
```
This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library.
Error root cause:
- The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14.
- `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods))
- All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them.
Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method.
Closes#40403
PiperOrigin-RevId: 804290760
Roll forward of a prior change, this change includes fixes and also memory reclamation support for orphaned domain storage.
PiperOrigin-RevId: 802204833
As title
Closes#40484
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40484 from apolcyn:bump_dev_version_202508191952 e788be57e9dc7f5e8316bee4baadec26fba3f6e6
PiperOrigin-RevId: 798331971
This change introduces a new `ResourceQuotaDomain` and registers Resource Quota related counters (`rq_calls_dropped`, `rq_calls_rejected`, `rq_connections_dropped`) within this domain. Each `MemoryQuota` now holds a reference to a `ResourceQuotaDomain` Storage instance, allowing these metrics to be tracked per resource quota. The usage sites in `chttp2_transport.cc` and `parsing.cc` are updated to use the new per-quota telemetry storage. The old global stats definitions for these counters are removed.
Introduce gauges also, and use them to report current memory pressure.
PiperOrigin-RevId: 796613444
Fixes macOS compilation warnings:
```
clang: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
```
c- and cpp-specific arg filters were out of sync.
For Darwin platform, `-stdlib=libc++` is in EXTRA_ENV_COMPILE_ARGS, but not excluded during in BuildExt.build_extensions#new_compile().
This also adds a reminder note to keep these in sync when updating one or another.
Repro:
```sh
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_WITH_CYTHON=1
python setup.py build_ext
```
Closes#40446
PiperOrigin-RevId: 796609793
Now every time we build grpcio wheel locally, it removes the copyright
header from
[`src/python/grpcio/grpc/_grpcio_metadata.py`](https://github.com/grpc/grpc/blob/368c313ffe54a723cb70ac294f5f040f7096a7c4/src/python/grpcio/grpc/_grpcio_metadata.py).
This happens because setup.py command fully overrides the content of
this file with `__version__ = ...` (even if the version is the same).
This is not a problem on its own, but it gets pretty tedious to git
reset/stash this file every time.
This PR allows to `export
GRPC_PYTHON_BUILD_SKIP_METADATA_ON_VERSION_MATCH=1`, and
`_grpcio_metadata.py` will only be changed when there's an actual
version change.
I think it'll make sense to just keep this behavior by default (only
update the file when there version is different), but for now, for the
sake of not accidentally breaking CIs for this local-specific change,
let's guard this behavior with the
`GRPC_PYTHON_BUILD_SKIP_METADATA_ON_VERSION_MATCH` var.
Future improvement to consider: instead of comparing versions, we should
consider only updating the `__version__`, while preserving the rest of
the file.
Enable integer exception checking for performance improvement. This also resolves Asyncio segfaults with free-threaded Python.
Takes care of Cython performance hint:
```
performance hint: src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi:126:22:
Exception check after calling '_poll' will always require the GIL to be acquired.
Possible solutions:
1. Declare '_poll' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2. Use an 'int' return type on '_poll' to allow an error code to be returned.
```
Details in https://cython.readthedocs.io/en/3.1.x/src/userguide/language_basics.html#error-return-values:
> There’s a known performance pitfall when combining `nogil` and `except *` `@cython.exceptval(check=True)`. In this case Cython must always briefly re-acquire the GIL after a function call to check if an exception has been raised. This can commonly happen with a function returning nothing (C `void`). Simple workarounds are to mark the function as `noexcept` if you’re certain that exceptions cannot be thrown, or to change the return type to `int` and just let Cython use the return value as an error flag (by default, `-1` triggers the exception check).
Closes#39993
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/39993 from aditya-1112:aio-segfault-fix bdc3310d4650dafaca5e97b1d4999e933674ee61
PiperOrigin-RevId: 796464140
This change renames the existing interface types to have `Interface` in the name, and prepares the ground for concrete types for the call tracers.
PiperOrigin-RevId: 790173028
1. update cfstream endpoint to open accepted socket
2. add cfsocket listener
3. enabled event engine server test in mac and ios
4. fix an issue cfstream endpoint not releasing streams
5. endpoint config is not supported
5. unix and vsock are not supported
6. e2e tests and samples will be added later
Closes#40097
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40097 from HannahShiSFB:cfstream-server bb504c91cb5ea23d55a1093848391fc31d9a1924
PiperOrigin-RevId: 789089100
### Description
## Part 5 of Introducing Ruff
* In this PR - the suppression for `D417`, `DTZ001`, `E402`, `E713`, `E722` and `EM101` has been removed on the root `ruff.toml`
## Related:
* Next: #40181
* Prev: #40180
* b/423755915
Closes#40180
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40180 from asheshvidyut:feature/setup-ruff-part-5 1b2e57d05f7d41053afec20b5ea337d2565fa7d3
PiperOrigin-RevId: 787933947
### Description
## Part 4 of Introducing Ruff
* In this PR - the suppression for `C408`, `D100`, `D101`, `D202`,`D207`, `D209`, `D403`, `D404`, `D410` and `D411` has been removed on the root `ruff.toml`
## Related:
* Next: #40180
* Prev: #40172
* b/423755915
Closes#40173
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40173 from asheshvidyut:feature/setup-ruff-part-4 27808176fea90679b7a7057a3c3746814192b571
PiperOrigin-RevId: 787931275
### Description
## Part 3 of Introducing Ruff
* In this PR - the suppression for `ARG004`, `ARG005`, `B007`, `B011`, `B018`, `B027`, `C402` and `C405` has been removed on the root `ruff.toml`
## Related:
* Next: #40173
* Prev: #40171 (not merged)
* b/423755915
Closes#40172
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40172 from asheshvidyut:feature/setup-ruff-part-3 f00ab4565e525ac6ff7bd8e888a0016c0d19bf1a
PiperOrigin-RevId: 787009943
### Description
Part 3 of Add Typeguard to AIO stack - runs only in tests, fixes type hints for `grpc.aio._utils` and `grpc.aio._call`
### Testing
CI
### Related work
* Part1: #40201
* Part2: #40215Closes#40217
This CL also has changes from https://github.com/grpc/grpc/pull/40286/files, because it was merged on Github without copybara.
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40217 from asheshvidyut:feature/type-hints/typeguard-aio-stack-part-3 cd82052353f2486538ccb0505d6b9db02e68cb9b
PiperOrigin-RevId: 786961247
### Description
## Part 6 of Introducing Ruff
* In this PR - the suppression for `EM102`, `EM103`, `F523`, `F541`, `F821`, `FIX004`, `FLY002`, `FURB188`, `G003`, `G004`, `INP001`, `ICN001`, `LOG015` and `N803` has been removed on the root `ruff.toml`
Closes#40181
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40181 from asheshvidyut:feature/setup-ruff-part-6 a82aae8073663bc525a7119c21f6ba3b821ab425
PiperOrigin-RevId: 786950190
Roll forward after rollback of #39708 with the change to move `JsonLoader` impls from the `spiffe_utils.h` to `spiffe_utils.cc`. There was potential to cause a linker issue with the impls in the header.
Closes#40279
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40279 from gtcooke94:spiffe_xds 66fce6de7a546d4cb638d79cacdd0f18718e52b4
PiperOrigin-RevId: 786762336
### Description
Part 2 of Add Typeguard to AIO stack - runs only in tests, fixes type hints for `grpc.aio._interceptor` and `grpc.aio._utils`
### Type hint for Public API change for Experimental API
**Fixing the error in public API** - `ClientCallDetails` where `method` was `str` but actually it should be `bytes`.
It was detected by `typeguard` as well.
### Testing
CI
### Related Work
- Part 1: #40201
- Part 3: #40217
- b/423755915
Closes#40215
This CL has changes from https://github.com/grpc/grpc/pull/40286/files because it was merged to master on Github without copybara.
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40215 from asheshvidyut:feature/type-hints/typeguard-aio-stack-part-2 3fc7a9d6aff8d84f2ee004c7ba8303919546378c
PiperOrigin-RevId: 786606656
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Includes:
- An experiment for the v1 service, start using only the v2 api from channelz, and downgrade using the conversion library.
- An experiment to move the C APIs that wrapped languages can use to implement channelz v1.
PiperOrigin-RevId: 786039931
This is a large PR, so I wanted to provide an outline:
In the end, this is basically just changing the piping of roots around gRPC. We used to just have a single `std::string root_certificate`. Now, we can have that **OR** a `SpiffeBundleMap`. Thus, the piping was changed to use `std::variant`. The `SpiffeBundleMap` is passed around as a `shared_ptr` to reduce copies while working within the existing complex lifetime management structure.
This PR pipes from the configuration of a provider to the OpenSSL integration layer in `ssl_transport_security.cc`. It _does not_ actually configure this value in OpenSSL or use this value to do verification yet, that will be a follow-up PR.
It also handles the live updates of roots, being either spiffe bundles or a raw certificate. There's a lot of existing complex conditionals here, so I tried to add named booleans to make the different states clear and make the code more readable.
In addition, I took many of the existing tests, copied them, and swapped root certificates for SpiffeBundleMaps. As a warning, several of the testing setups are pretty dense. Further confidence with this will be built in the follow-up PR when we are actually using these values for verification in end2end flows.
Closes#39708
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/39708 from gtcooke94:spiffe_xds 1e1503af150e697dc66f4a96ee6ba9990a703acb
PiperOrigin-RevId: 784333570
The target is still a no-op for mobile platforms like android and ios
Closes#40231
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40231 from Vignesh2208:fusion-integrate dcf2ce1b6723e2e2827ea0f728d7cf2afde478af
PiperOrigin-RevId: 783907814
We have a set of C APIs that wrapped languages can use to implement channelz v1.
Migrate these to use the conversion library, and isolate them into a new file.
An experiment is added so we can do a gradual rollout.
PiperOrigin-RevId: 783581214
### Description
Add typeguard to AIO stack - runs only in tests
### Testing
To test if this change works on or not, I added return type `-> str` in this line -
https://github.com/grpc/grpc/blob/master/src/python/grpcio/grpc/aio/_server.py#L207
Then command
```
bazel test //src/python/grpcio_tests/tests_aio/unit:all
```
started failing with error of typeguard.
Closes#40201
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40201 from asheshvidyut:feature/type-hints/typeguard-aio-stack cea4d6b7dceb55238a53680be71b5c1f3f3a63e4
PiperOrigin-RevId: 783243843
[PH2] Fixing circular dependency
`Http2SettingsManager` needs `frame.h`.
Our new validation code in frame.h needs `Http2Settings`.
So we split `Http2Settings` and `Http2SettingsManager` into 2 separate header files to resolve the circular dependency.
Closes#40204
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40204 from tanvi-jagtap:ph2_fixing_circular_dependency 17718955eb455aa8d20840d9d0a9a8712c63dd01
PiperOrigin-RevId: 783114803
In preparation for migrating it from python3.7 to python3.11.
Pylint version had to be upgraded because pylint==2.2.2 doesn't support python 3.11.
The image will be updated separately to remove compiling python3.7 from source in #40139.
Pylint v2.17.7 docs: https://pylint.readthedocs.io/en/v2.17.7Closes#40169
PiperOrigin-RevId: 783026287
### Description
Minor fix of path of `ruff.toml` for tests.
Closes#40175
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40175 from asheshvidyut:fix-correct-path-for-ruff.toml 2eedddcbd760b63ba65e76a9858ba84bd1e0193d
PiperOrigin-RevId: 782030862
## Description
### Part 1 of Introducing Ruff
* This is Part 1 of introducing ruff series.
* The following PRs will remove some of the supressed rules introduced in this PR.
* This PR has all the folders which PyLint has currently.
* In the final part of these PRs, PyLint will be removed compeletly.
## Testing
CI
Closes#40160
PiperOrigin-RevId: 781990339
Craig Tiller