This commit introduces the `event_engine_poller_for_python` experiment, allowing for controlled rollout and testing of the EventEngine's Posix poller specifically within gRPC Python bindings.
**Crucially, this change does *not* alter the default behavior for builds where `GRPC_DO_NOT_INSTANTIATE_POSIX_POLLER` is *not* defined.** In such configurations, the Posix EventEngine poller will continue to be enabled unconditionally, preserving existing functionality.
The primary impact of this change is when `GRPC_DO_NOT_INSTANTIATE_POSIX_POLLER` *is* defined (e.g., in gRPC Python's build system). In this scenario, the enablement of the Posix poller transitions from being implicitly disabled to being configurable via the `event_engine_poller_for_python` experiment flag. This enables a controlled, experimental rollout of the EventEngine poller in environments that previously opted out of its direct instantiation.
Closes#40243
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40243 from eugeneo:python-no-backup-poller-experiment 17906e6501b8e6fe7d2ccc63439ec121ef47d43b
PiperOrigin-RevId: 804472655
Fixes macOS compilation warnings:
```
clang: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
```
c- and cpp-specific arg filters were out of sync.
For Darwin platform, `-stdlib=libc++` is in EXTRA_ENV_COMPILE_ARGS, but not excluded during in BuildExt.build_extensions#new_compile().
This also adds a reminder note to keep these in sync when updating one or another.
Repro:
```sh
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_WITH_CYTHON=1
python setup.py build_ext
```
Closes#40446
PiperOrigin-RevId: 796609793
### Description
Change typing extensions version to 4.12.2 which is same as in G3
### Testing
CI
Closes#40331
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/40331 from asheshvidyut:typing-extension-version 19d13ed2ecc4fad0c84278bfece746428a615c71
PiperOrigin-RevId: 787931830
Fixes https://github.com/grpc/grpc/issues/39026
A few notheworthy changes here besides the version bump itself:
- Reorder the way we pull in `ares.h` in various places, so that we make sure to pull in necessary windows header ahead of time. In older c-ares, the `ares.h` public header was more self-contained w.r.t. windows headers including `winsock2.h`. E.g. see [old way of inclusion](6360e96b5c/include/ares.h (L64)) vs. [new way of inclusion](42ddbc14ec/include/ares.h (L52)). I'm not crazy about this fix here but couldn't see a much cleaner way (we could maybe set `CARES_HAVE_WINSOCK2_H` in `port_platform.h` alternatively)
- Upated hand-crafted `config_windows/ares_config.h` to set newly required build defs for the c-ares build itself to work on windows
- Fix ruby macos cross-compilation build to correctly set `SYSTEM` Makefile var to `Darwin` (so that we pull in `config_darwin/ares_config.h`, so far we've been getting away with the linux build mode)
- Change ruby macos MINGW32 cross-compilation build to use our hand crafted `config_windows/ares_config.h` header
- Explicitly link `iphlpapi` on Windows. I think we used to pick this up during c-ares library build from [this pragma](6360e96b5c/include/ares.h (L71)), but now that's [hidden under build define](42ddbc14ec/include/ares.h (L59))
Closes#39508
PiperOrigin-RevId: 783875246
This solves several issues:
1. #39588: Selecting unstable Cython releases when building gRPC Python artifacts.
2. Using inconsistent Cython version in released platform-specific wheels within the same grpcio release.
3. cython/cython#6878 Several published grpcio wheels were built with Cython 3.1.0 and are affected by Cython memory leak issue when using AsyncIO APIs (`grpc.aio.*`).
This change will be backported to 1.73.x, 1.72.x and 1.71.x.
Note: bazel Cython version wasn't upgraded because the latest cython BCR module version is still `3.0.11-1`.
Closes#39609
PiperOrigin-RevId: 761343405
This PR makes necessary changes to Python build scripts for C++17 support, which were missed in #37919
- Fixed handling of C and C++ standard options for MSVC.
- Previously, a workaround relied on a specific method, `_compiler` found in the compiler class. [unixcompiler](ff11eed0c3/distutils/unixccompiler.py (L191)) has it but [msvccompiler](ff11eed0c3/distutils/_msvccompiler.py (L367)) doesn't. Therefore this won't work for MSVC even though it's supposed to work across all compilers.
- To resolve this, a new workaround has been implemented. This involves adding logic directly into the `_commandfile_spawn` function, which is considered the most suitable location for this fix. Although not ideal, this ensures that C and C++ standard options are managed correctly for MSVC.
- Removed `compiler_ok_with_extra_std function` as isolating C and C++ standard options is always recommended.
- Updated comments to correctly refer to C++17.
Closes#38410
PiperOrigin-RevId: 713442384
This bumps the minimum version of C++ to 17 from 14 per https://github.com/grpc/proposal/blob/master/L120-requiring-cpp17.md
Changes:
- Used C++17 instead of C++14 when building gRPC.
- Disabled `modernize-unary-static-assert` clang-tidy rule for the smooth transition. This rule will be re-enabled in a future update, along with a fix to address any identified issues.
- Added a post-install script for XCode/Cocoapod examples to enforce the use of C++17 across all projects, preventing build errors. (I'd like to have a better solution here but couldn't find it)
Closes#37919
PiperOrigin-RevId: 709073401
This PR adds templating for Python versions and updates the maximum supported Python version to 3.13. The following major changes related to templating are added:
- Minimum supported Python version and list of supported versions in `setup.py` are fetched using new template generated files called `python_version.py`
- Dockerfiles for the different Python Linux builds are now template generated.
- The "Supported Python Versions" section from READMEs of ancillary and main packages have been removed
Note: All the `python_version.py` files and Linux build `Dockerfiles` except `tools/dockerfile/grpc_artifact_python_linux_armv7/Dockerfile` in the PR are generated from the respective templates.
Further non-templated additions to add support for Python 3.13:
- install scripts and artifacts for windows, macos and linux are added manually. Later, these can be templated as well.
- updated cython bounds to 3.x
- updated twine version to solve [cgi module import error](https://github.com/pypa/twine/issues/1046)
- the twine update introduces a dependency on cryptography>=2.0. But the cryptography package doesn't support 32-bit Linux images and hence `twine check` has been disabled for x86 manylinux and x86 musllinux artifacts.
Closes#37643
PiperOrigin-RevId: 678954495
Add GRPC_PYTHON_BUILD_WITH_SYSTEMD environment variable to setup.py
The variable sets the required compile time define and adds the required
link time library to the build process
### NOTE
* We shouldn't merge this PR until GCP cloud functions drops support for Python 3.7 ([Currently scheduled for GCF](https://cloud.google.com/functions/docs/runtime-support#python))
As part of supporting Python 3.12, we're now officially drop support for Python 3.7.
This PR:
* Changed supported Python version from 3.7 to 3.8 in README.
* Replaced distribution test image from `debian:buster` to `debian:bullseye` since the default Python version in buster is 3.7.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#34450
PiperOrigin-RevId: 622899511
In Cython 3, cdef functions that really will not raise exceptions must be declared as `noexcept`. Fixed by this commit.
Update requirements to `cython >= 3.0` in requirements*.txt and setup.py.
Fixes issue #33918.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35995
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35995 from badshah400:master b3277bac1585ddee88a170b0a95c260d909cce9c
PiperOrigin-RevId: 621214091
Change was created by the release automation script. See go/grpc-release.
Additional Changes:
* Boring SSL started to [Require SSE2 when targetting 32-bit x86](56d3ad9d23), thus added `-msse2` to fix some build failures.
Closes#36089
PiperOrigin-RevId: 614822548
Sort input file list
so that the `.so` file builds in a reproducible way
in spite of non-deterministic filesystem readdir order and
http://bugs.python.org/issue30461
See https://reproducible-builds.org/ for why this is good.
This patch was done while working on reproducible builds for openSUSE.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
---------
Co-authored-by: Richard Belleville <rbellevi@google.com>
Discuss thread about this change: [link](https://mail.google.com/mail/u/0/#sent/QgrcJHsBpWNGRlrMktwbppGGfFTVCFLcQgL?compose=new)
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35849
PiperOrigin-RevId: 607144827
Change was created by the release automation script. See go/grpc-release.
On top of changes made by the script, `setup.py` had to be updated to have `/std:c11` option for MSVC to use c11 features for boringssl.
Closes#35768
PiperOrigin-RevId: 603403329
`GPR_BACKWARDS_COMPATIBILITY_MODE` was devised to support old Linux kernel with old glibc but gRPC is now expected to support glibc 2.17 and later (2.17 is derived from the oldest but supported Linux distro, CentOS 7). Effectively, gRPC doesn't need `GPR_BACKWARDS_COMPATIBILITY_MODE` anymore. Hence, let's remove it.
gRPC Python and Ruby that distribute binary artifacts already began to use `manylinux` environment dealing with this issue in a better way so they don't need this flag anymore.
Closes#35602
PiperOrigin-RevId: 599895270
This commit upgrades gRPC to protobuf v25.0 and makes some fixes to
account for upb changes. One major change is that upb has been merged
into the protobuf repo, so we can now drop the separate `@upb`
dependency. Another is that `.upb.c` files no longer exist and there are
new `.upb_minitable.h` and `.upb_minitable.c` files. The longer
filenames exceeded a Windows restriction, so to work around that I
renamed the `upb-generated` directory to just `upb-gen`, and likewise
for `upbdefs-generated`.
Some user reported an issue that symbols from CoreFoundation are missing
when install grpc/grpc-tools inside Conda:
* https://github.com/grpc/grpc/issues/33714
* https://github.com/grpc/grpc/issues/34135
This PR adds the `-framework CoreFoundation` flag so no workarounds are
required from user.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Support Python 3.12.
### Testing
* Passed all Distribution Tests.
* Also tested locally by installing 3.12 artifact.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
### Background
* `distutils` is deprecated with removal planned for Python 3.12
([pep-0632](https://peps.python.org/pep-0632/)), thus we're trying to
replace all distutils usage with setuptools.
* Please note that user still have access to `distutils` if setuptools
is installed and `SETUPTOOLS_USE_DISTUTILS` is set to `local` (The
default in setuptools, more details can be found [in this
discussion](https://github.com/pypa/setuptools/issues/2806#issuecomment-1193336591)).
### How we decide the replacement
* We're following setuptools [Porting from Distutils
guide](https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html#porting-from-distutils)
when deciding the replacement.
#### Replacement not mentioned in the guide
* Replaced `distutils.utils.get_platform()` with
`sysconfig.get_platform()`.
* Based on the [answer
here](https://stackoverflow.com/questions/71664875/what-is-the-replacement-for-distutils-util-get-platform),
and also checked the document that `sysconfig.get_platform()` is good
enough for our use cases.
* Replaced `DistutilsOptionError` with `OptionError`.
* `setuptools.error` is exporting it as `OptionError` [in the
code](https://github.com/pypa/setuptools/blob/v59.6.0/setuptools/errors.py).
* Upgrade `setuptools` in `test_packages.sh` and changed the version
ping to `59.6.0` in `build_artifact_python.bat`.
* `distutils.errors.*` is not fully re-exported until `59.0.0` (See
[this issue](https://github.com/pypa/setuptools/issues/2698) for more
details).
### Changes not included in this PR
* We're patching some compiler related functions provided by distutils
in our code
([example](ee4efc31c1/src/python/grpcio/_spawn_patch.py (L30))),
but since `setuptools` doesn't have similar interface (See [this issue
for more details](https://github.com/pypa/setuptools/issues/2806)), we
don't have a clear path to replace them yet.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
~~NB: I haven't tested this at all and am hoping the CI will tell me
where I've (undoubtedly) messed something up.~~ Edit: looks like CI is
now clear!
BoringSSL's gas-compatible assembly files, like its C files, are now
wrapped with preprocessor ifdefs to capture which platforms each file
should be enabled on. This means that, provided the platform can process
.S files it all (i.e. not Windows), we no longer need to detect the
exact CPU architecture in the build.
Switch gRPC's build to take advantage of this. I've retained
BUILD_OVERRIDE_BORING_SSL_ASM_PLATFORM, on the off chance anyone is
using it to cross-compile between Windows and non-Windows, though I
doubt that works particularly well.
As part of this, restore assembly optimizations in a few places where
they were seemingly disabled for issues relating to this:
- https://github.com/grpc/grpc/pull/31747 had to disable the assembly,
because at the time assembly required the library be built differently
for each architecture and then stitched back together. This should now
work.
- tools/run_tests/run_tests.py disabled x86 assembly due to some issues
with CMAKE_SYSTEM_PROCESSOR in a Docker image. This too should now be
moot.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
- Switched from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies
Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.
To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
Fix for Cython build issue in aarch64.
We're seeing this error in aarch64 distribution test:
```
In file included from ./src/core/lib/slice/slice.h:36,
from ./src/core/lib/slice/slice_buffer.h:29,
from ./src/core/lib/transport/transport.h:60,
from ./src/core/lib/channel/channel_stack.h:75,
from ./src/core/lib/channel/call_tracer.h:32,
from src/python/grpcio/grpc/_cython/cygrpc.cpp:2230:
./src/core/lib/slice/slice_refcount.h: In member function 'void grpc_slice_refcount::Ref(grpc_core::DebugLocation)':
./src/core/lib/slice/slice_refcount.h:55:25: error: expected ')' before 'PRIdPTR'
"REF %p %" PRIdPTR "->%" PRIdPTR, this, prev_refs, prev_refs + 1);
^~~~~~~~
)
```
Based on [this
post](https://stackoverflow.com/questions/26182336/priuptr-preprocessor-bug-in-gcc),
it's caused by including `<inttypes.h>` before define
`__STDC_FORMAT_MACROS` marco.
`<inttypes.h>` was included in `core/lib/channel/call_tracer.h` and this
macro should already be defined in `grpc/grpc.h` through
`port_platform.h`, but we're still having issue in aarch64, so we
manually define the macro in this PR.
This is a hack to get around an issue on Apple devices caused by the
PosixEventEngine's `poll` poller not supporting fork. This PR disables
the EventEngine poller entirely in Python builds. It will therefore
prevent the release of the EventEngine generally, and prevent any
testing of EventEngine integration with gRPC Python, until the `poll`
poller is fixed.
The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.
This is best reviewed commit-by-commit, I tried to group changes in
logical areas.
Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.
TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import
---------
Co-authored-by: Craig Tiller <ctiller@google.com>
* fix: grpcio fails to build from source on Apple Silicon mac #(25082)
Boringssl currently does not support macOS arm64, so installation on these platform with default config will undoubtedly fail, that means the env var 'BUILD_WITH_SYSTEM_OPENSSL' must be set to true.
* Use system installation of openssl by default on macOS arm64
* Format code by yapf
* Support Python 3.11
* Update build images for 3.11
* Whoopsie
* The architecture of this thing is garbage
* Silence ownership warning
* Account for change in git behavior
* Fix directory
* I am in great pain
* Update Windows and arm linux
* Agh
* Clean up
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
