Commit Graph

199 Commits

Author SHA1 Message Date
Mark D. Roth
0944410d6c [reorg] move test/core/util -> test/core/test_util (#36446)
Closes #36446

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36446 from markdroth:reorg_test 5dcc85e006581a8fc52a3a914baa5f33e4a21589
PiperOrigin-RevId: 629229220
2024-04-29 17:06:40 -07:00
Jan Tattermusch
5d405b2314 [bazel RBE] add direct link to Fusion UI in bazel rbe invocations (#34101)
Currently the bazel invocations provide a link back to the original
kokoro jobs for resultstore and sponge UIs.
This adds another back link to Fusion UI, which has the advantage of 
- being able to navigate to the kokoro job overview
- there is a button to trigger a new build (in case the job needs to be
re-run).


![image](https://github.com/grpc/grpc/assets/9939684/42f0bbc2-14f5-45db-89f7-73e48e32e7c9)
2023-08-21 09:33:20 +02:00
Sergii Tkachenko
de6ed9ba9f [Python] Migrate from yapf to black (#33138)
- Switched  from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies 

Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.

To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
2023-06-09 15:08:55 -07:00
Jan Tattermusch
07a75427bd Simplify running bazel with structured test results on CI (#29353)
* add bazel_report_helper.py

* simplify bazel invocations in selected builds

* introduce __main__

* update gitignore

* introduce sleep constant

* add type annotations

* use f-strings

* Revert "use f-strings"

This reverts commit f970d6a40bd134cada2f01d8bac1c224138cb35f.
2022-04-12 11:31:41 +02:00
Jan Tattermusch
a3be072605 Switch .py shebang to python3 (#28963)
* switch shebang to python3

* prepare_build_macos: install latest available patch for each python
2022-03-14 10:41:30 +01:00
Craig Tiller
5d81683faa Add per-call memory bloat report to CI (#29037)
* Add per-call memory bloat report to CI

* fix

* test

* Automated change: Fix sanity tests

* Update grpc_bloat_diff_in_docker.sh

* Update memory_diff.py

* review feedback

* prep for new job

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-03-12 11:56:48 +01:00
Craig Tiller
27662173e7 Clip long summaries (#28401) 2021-12-28 21:03:15 +00:00
Jan Tattermusch
82abb46ed3 improve tty output of run_tests.py and other scripts (#28308) 2021-12-09 07:23:02 +01:00
Jan Tattermusch
3a024ea40c Avoid extremely slow kokoro job finalization by moving workspace to a non-synced directory on kokoro workers. (#28259)
* support "move and respawn" for kokoro jobs

* delete_nonartifacts.sh no longer needed

* bazel distribtest dont publish any artifacts

* try the altsrc respawn for selected jobs

* use altsrc for all windows CI jobs

* use altsrc for all macos CI jobs

* only move files under src/github

* fix uploading artifacts on windows
2021-12-03 19:41:14 +01:00
Richard Belleville
2eb686e8ad Fix rbe_upload SSL issue (#27982)
* Attempt to fix rbe_upload SSL issue

* Fix comparison
2021-11-09 14:58:59 -08:00
Richard Belleville
73dcfd375d Fix relative imports for Python 3 (#27971)
* Fix relative imports for Python 3

* Remove space

* isort
2021-11-08 15:37:32 -08:00
Richard Belleville
03b46b228e Fix RBE upload (#27969) 2021-11-08 12:55:26 -08:00
Richard Belleville
59693b7e72 Migrate Infrastructure Scripts to Python 3 (#27135)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Migrate tools/distrib directory to python 3

* Remove unnecessary shebang

* Restore line_count directory

* Immediately convert subprocess.check_output output to string

* Take care of Python 2 shebangs

* Invoke scripts using a Python 3 interpreter

* Yapf. Isort

* Try installing Python 3 first

* See if we have any Python 3 versions installed

* Add Python 3.7 to Windows path

* Try adding a symlink

* Try to symlink differently

* Install six for Python 3

* Run run_interop_tests with python 3

* Try installing six in python3.7 explicitly

* Revert "Try installing six in python3.7 explicitly"

This reverts commit 2cf60d72f388a95d642b2c99a775d88a6248f788.

* And debug some more

* Fix issue with jobset.py

* Add debug for CI failure

* Revert microbenchmark changes
2021-11-08 08:55:25 -08:00
Craig Tiller
8cbf308ce2 Use PUT not POST to avoid duplicate bloat labels (#27952) 2021-11-05 13:56:31 -07:00
Craig Tiller
59ed9d09a2 Reland bloat labelling code. (#27926)
* Revert "Revert "Assign a label for level of bloat diff (#27880)" (#27925)"

This reverts commit f2b96000a7.

* fix
2021-11-03 09:57:13 -07:00
Craig Tiller
f2b96000a7 Revert "Assign a label for level of bloat diff (#27880)" (#27925)
This reverts commit 882cc1259e.
2021-11-03 08:14:19 -07:00
Craig Tiller
882cc1259e Assign a label for level of bloat diff (#27880)
* upgrade bloaty

* Update check_submodules.sh

* fix

* label bloat

* set labels based on severity

* measure severity of changes

* Automated change: Fix sanity tests

* test

* test

* Update bloat_diff.py

* Update check_on_pr.py

* Automated change: Fix sanity tests

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2021-11-02 22:39:15 -07:00
Richard Belleville
2c889a9921 Fix stdout handling in jobset.py (#27493)
* Fix stdout handling in jobset.py

* Apparently ascii was a bad assumption

* Or how about this

* Just ignore non-utf8 characters
2021-10-04 16:00:40 -07:00
Jan Tattermusch
c1bf4e1a2a Revert "Enable Alpine portability test to run under Python 3. (#27464)" (#27485)
This reverts commit 54d4e8f68b.
2021-09-27 10:53:34 -07:00
Richard Belleville
54d4e8f68b Enable Alpine portability test to run under Python 3. (#27464)
* Enable Alpine portability test to run under Python 3.

* Fix other scripts
2021-09-24 10:12:19 -07:00
Richard Belleville
eada6c6e56 Update Alpine Test Image to Use Python 3.8 (#27415)
* Upgrade Alpine to Python 3.8

* Upgrade docker image to use 3.8
2021-09-20 19:38:36 -07:00
Richard Belleville
b6cc72f129 Revert "Run 2to3 on tools directory (#26002)" (#27183)
This reverts commit a388361c3d.
2021-08-30 12:29:20 -07:00
Richard Belleville
a388361c3d Run 2to3 on tools directory (#26002)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Restore line_count directory
2021-08-30 09:25:21 -07:00
yifeizhuang
48b194f737 release v1.40.0: build gcr images for release, fix python2.7 (#27045) 2021-08-18 08:22:11 -07:00
Lidi Zheng
2231c2ba77 Introduce Python import sorting to our sanity test suite (#26768)
* Add isort_code.sh to sanity tests

* Run tools/distrib/isort_code.sh

* Fine tune the import order for relative imports

* Make pylint and project generation happy

* Fix a few corner cases

* Use --check instead of --diff

* The import order impacts test result somehow

* Make isort print diff and check output at the same time

* Let tools/run_tests/python_utils be firstparty library

* Run isort against latest HEAD
2021-07-26 12:31:21 -07:00
Sergii Tkachenko
368b9077e2 xds-k8s kokoro buildscripts: exclude from tests suites (#26098) 2021-04-27 19:26:53 -04:00
AJ Heller
6ff042277e Fix invalid use of python's traceback.print_exc (#25743)
The exception is being interpreted as a `limit` (int) to cap the number
of stack trace entries to print.

https://docs.python.org/3/library/traceback.html#traceback.print_exc
2021-03-17 12:04:15 -07:00
AJ Heller
5139a012e7 Migrate the bm_diff benchmarks to python3 (#25619)
* Migrate the bm_diff benchmarks to python3

Includes a requirements.txt pinned at ~2017 versions, when this script
was first written.

* Replace p2 with p3 scipy/numpy dependencies.

* py2->3 for benchmark setup scripts

* upgrade pip to resolve python3 cryptography/setuptools-rust problem

* re-add jobset import (accidentally removed)

* re-add six's urllib import. This file is still used in py2 tests

* force py3 in run_if_c_cpp_modified.sh

* Fix another instance of subprocess.check_output binary mode

* Use the requirements.txt for CI perf environment setup

* Try to upgrade PyJWT. (v2.0.0 was problematic, #25053)

v2.x makes encode return strs from jwt.encode in both py2 and py3.
Previously, py3 would return bytes, and py2 a str.

* upgate cryptography lib version requirements for jwt.

* Wrap pip requirements specifier in quotes '>=x,<y'

* Decode subprocess output once instead of for every line

* Revert "Decode subprocess output once instead of for every line"

This reverts commit 28d14026431622ac7afd3535a8d7118e5be96628.

py2 doesn't support the `text` argument to subprocess.check_output.

* Address reviewer requests

* Pin a valid scipy version

* Remove scipy and tabulate dependencies from macos tests
2021-03-09 15:44:00 +01:00
Lidi Zheng
39cc797025 [xDS Proto] Modernize buildgen scripts (#25512)
* [xDS Proto] Modernize buildgen scripts

* This PR only covers some buildgen scripts that will be used by xDS
proto
* This PR also improves the debuggability of some scripts
* Merge with master

* Adopt reviewer's advices
2021-03-02 11:25:01 -08:00
Yash Tibrewal
9526a55776 Remove disrespectful term 2021-02-08 15:23:18 -08:00
Sergii Tkachenko
2258cee7b2 xds-k8s exclude fix: dirs end with a slash 2021-01-19 22:42:57 -05:00
Sergii Tkachenko
83501b822d xds-k8s driver: exclude from tests suites 2021-01-19 22:36:58 -05:00
Jan Tattermusch
d446ec036f Merge pull request #24861 from dapeng-mi/master
Bug fix: decode bytes object to string for subprocess.check_output ou…
2021-01-15 17:09:10 +01:00
Sergii Tkachenko
8db79e2e71 Python style: reformat code using updated yapf 0.30.0 2021-01-13 14:00:12 -05:00
dapeng-mi
965aa55c9e Bug fix: add __lt__ method for JobSpec class.
Another exception is found when running "./tools/run_tests/run_tests.py
 -l c++ -c dbg" test in python3 environment. The exception complains
the "<" method is not supported when calling sorted() function to sort
the JobSpec class.

Here is the exception.

./tools/run_tests/run_tests.py -l c++ -c dbg
PASSED: make [time=1.6sec, retries=0:0]

Omitting EPOLLEXCLUSIVE tests

2020-12-01 10:49:42,122 detected port server running version 21
2020-12-01 10:49:42,182 my port server is version 21
Traceback (most recent call last):
  File "./tools/run_tests/run_tests.py", line 1919, in <module>
    build_only=args.build_only)
  File "./tools/run_tests/run_tests.py", line 1800, in _build_and_run
    one_run = set(spec for language in languages
  File "./tools/run_tests/run_tests.py", line 1801, in <genexpr>
    for spec in language.test_specs()
  File "./tools/run_tests/run_tests.py", line 425, in test_specs
    return sorted(out)
TypeError: '<' not supported between instances of 'JobSpec' and 'JobSpec'

Signed-off-by: dapeng-mi <clark_mdp@163.com>
2020-12-04 09:57:57 +08:00
Lidi Zheng
511af22c78 Hopefully fixed 27 build on winserver2016
* Download the working msys64 and use it to compile 27
* Remove the cygwin detection override logic
* Only install Python interpreters in jobs running on winserver2016
* Fix the batch script substring check
* Increase the timeout for related jobs
* TIL that there is an allow list for Kokoro env vars
2020-11-23 14:17:58 -08:00
Lidi Zheng
0e275f3d2b Change to explicitly encode 2020-11-13 12:07:45 -08:00
Lidi Zheng
e191e98f37 Unify the encoding to ascii 2020-11-13 12:07:44 -08:00
Lidi Zheng
892e7bacb1 Explicitly set encoding to ascii to prevent TypeError in 3.7+ 2020-11-13 12:07:44 -08:00
Esun Kim
eeb2b0f3a9 Fix pagination in upload_rbe_results 2020-10-19 11:26:30 -07:00
zhungxd
e4cdcb7b25 fix wrong path 2020-09-09 19:41:32 +08:00
Jan Tattermusch
3456636150 Merge pull request #23455 from jtattermusch/port_server_python3
Make port server python3 compatible
2020-07-14 20:21:53 +02:00
Jan Tattermusch
e329d1ddd0 bytes(string, encoding) is not supported in python2
results in error "TypeError: str() takes at most 1 argument (2 given)"
2020-07-14 17:46:27 +02:00
Jan Tattermusch
375b8551f7 update bigquery partition expiration 2020-07-13 15:04:53 +02:00
Jan Tattermusch
4847f85367 make port server python3 compatible 2020-07-13 11:35:07 +02:00
Vijay Pai
9d82fcc1a2 Fix test filter for C++ include files 2020-06-16 08:38:04 -07:00
Jan Tattermusch
6ad89b6369 also upload className in upload_rbe_results.py 2020-03-29 11:01:21 -04:00
Eric Gribkoff
9e1110cb11 Split logs by test case and run client separately for each test 2020-03-22 23:56:44 -07:00
Jan Tattermusch
fea074384f sometimes not all RBE results are uploaded 2020-03-16 07:32:20 -04:00
Jan Tattermusch
e20ed51e6f easier debugging for upload_rbe_results.py 2020-03-16 07:32:20 -04:00