Introduce the TAILCALL VM, a more efficient variant of the CALL VM:
* Each opcode handler tailcalls the next opcode handler directly instead of
returning to the interpreter loop. This eliminates call and interpreter loop
overhead.
* Opcode handlers use the preserve_none calling convention to eliminate
register saving overhead.
* preserve_none uses non-volatile registers for its first arguments, so
execute_data and opline are usually kept in these registers and no code is
required to forward them to the next handlers.
Generated machine code is similar to a direct-threaded VM with register pinning,
like the HYBRID VM.
JIT+TAILCALL VM also benefits from this compared to JIT+CALL VM:
* JIT uses the registers of the execute_data and opline args as fixed regs,
eliminating the need to move them in prologue.
* Traces exit by tailcalling the next handler. No code is needed to forward
execute_data and opline.
* No register saving/restoring in epilogue/prologue.
The TAILCALL VM is used when the HYBRID VM is not supported, and the compiler
supports the musttail and preserve_none attributes: The HYBRID VM is used when
compiling with GCC, the TAILCALL VM when compiling with Clang>=19 on x86_64 or
aarch64, and the CALL VM otherwise.
This makes binaries built with Clang>=19 as fast as binaries built with GCC.
Before, these were considerably slower (by 2.8% to 44% depending on benchmark,
and by 5% to 77% before 76d7c616bb).
Closes GH-17849
Closes GH-18720
JIT doesn't recognize that variables may be used after returning from a
trace due to YIELD, so some effects may never be stored to memory.
YIELD ops terminate trace recordings with ZEND_JIT_TRACE_STOP_RETURN, and are
handled mostly like RETURN. Here I change zend_jit_trace_execute() so that
YIELD terminates recordings with ZEND_JIT_TRACE_STOP_INTERPRETER instead,
to ensure that we recognize that variables may be used after returning from
the trace due to YIELD.
Fixes GH-19493
Closes GH-19515
JIT used to not have a compile-time dependency on VM kind, such that a single
build of opcache could work with different VM kinds at runtime. This has been broken
over time and would be difficult to restore. Additionally, as opcache is now
built-in, this would not be useful anymore.
Remove the zend_jit_vm_kind variable.
Reuse the helper zend_foreach_op_array() that we move to the
zend_optimizer.h header to be usable in opcache.
Note that applying this to other op_array loops is not easy because they either:
- start from EG(persistent_classes_count)
- or only apply to classes
This removes the --enable-opcache/--disable-opcache configure switch. OPcache
is now always builtin. The default value of opcache.enable and
opcache.enable_cli is unchanged.
RFC: https://wiki.php.net/rfc/make_opcache_required
Closes GH-18961.
Co-authored-by: Tim Düsterhus <tim@tideways-gmbh.com>
We use linker relocations to fetch the TLS index and offset of _tsrm_ls_cache.
When building Opcache statically, linkers may attempt to optimize that into a
more efficient code sequence (relaxing from "General Dynamic" to "Local Exec"
model [1]). Unfortunately, linkers will fail, rather than ignore our
relocations, when they don't recognize the exact code sequence they are
expecting.
This results in errors as reported by GH-15074:
TLS transition from R_X86_64_TLSGD to R_X86_64_GOTTPOFF against
`_tsrm_ls_cache' at 0x12fc3 in section `.text' failed"
Here I take a different approach:
* Emit the exact full code sequence expected by linkers
* Extract the TLS index/offset by inspecting the linked ASM code, rather than
executing it (execution would give us the thread-local address).
* We detect when the code was relaxed, in which case we can extract the TCB
offset instead.
* This is done in a conservative way so that if the linker did something we
didn't expect, we fallback to a safer (but slower) mechanism.
One additional benefit of that is we are now able to use the Local Exec model in
more cases, in JIT'ed code. This makes non-glibc builds faster in these cases.
Closes GH-18939.
Related RFC: https://wiki.php.net/rfc/make_opcache_required.
[1] https://www.akkadia.org/drepper/tls.pdf
This reduces the chances of confusion between opcode handlers used by the
VM, and opcode handler functions used for tracing or debugging. Depending
on the VM, zend_vm_opcode_handler_t may not be a function. For instance in
the HYBRID VM this is a label pointer.
Closes GH-19006
Property hooks were not handled for JIT+trait+preloading.
Split the existing functions that handle op arrays, and add iterations
for property hooks.
Closes GH-18923.
ZEND_FUNC_INFO() can not be used on internal CE's. If preloading makes a
CE that's an alias of an internal class, the invalid access happens when
setting the FUNC_INFO.
While we could check the class type to be of user code, we can just skip
aliases altogether anyway which may be faster.
Closes GH-18915.
JIT'ed ASSIGN_OBJ expressions will exit to VM when the prop is undef. However,
in a constructor it's very likely the case. Therefore most traces with `new`
expressions will exit to VM.
Here I ensure that we don't need to exit to VM when it's likely that the
prop will be undef.
In the function JIT we compile a slow path to handle such properties,
but not in the tracing JIT, assumingly to reduce code size. Here I enable
compilation of the slow path in the tracing JIT when it's likely the prop
will be undef. Quite conveniently we already record the prop type during
tracing, so I use that to make the decision.
This results in a 1.20% wall time improvement on the symfony demo benchmark
with 20 warmup requests.
Closes GH-18576
Polymorphic calls pass this and the function to side traces via snapshotting.
However, we assume that this/func are in registers, when in fact they may be
spilled.
Here I update snapshotting of poly_func/poly_this to support spilling:
- In zend_jit_snapshot_handler, keep track of the C stack offset
of the spilled register, in a way similar to how stack variables.
- In zend_jit_start, do not pre-load the registers if they were spilled.
- In zend_jit_trace_exit / zend_jit_trace_deoptimization, load from the
stack if the register was spilled.
- Store a reference to poly_func/poly_this in zend_jit_ctx so we can use that
directly in the side trace.
Closes GH-18408
Add a new exit flag (ZEND_JIT_EXIT_CHECK_EXCEPTION) that enables exception
checking during exit/deoptimization.
We already checked for exceptions during exit/deoptimization, but only when
ZEND_JIT_EXIT_FREE_OP1 or ZEND_JIT_EXIT_FREE_OP2 were set (presumably to
handle exceptions thrown during dtor). The new flag makes it possible to request
it explicitly.
This also fixes two issues in zend_jit_trace_exit():
- By returning 1, we were telling the caller (zend_jit_trace_exit_stub()) to
execute the original op handler of EG(current_execute_data)->opline, but in
reality we want to execute EX(opline), which should be EG(exception_op).
- EX(opline) is set to the value of %r15 in zend_jit_trace_exit_stub() before
calling zend_jit_trace_exit(), but this may be the address of a
zend_execute_data when the register is being reused to cache EX(call).
Fixes GH-18262
Closes GH-18297
This changes the signature of opcode handlers in the CALL VM so that the opline
is passed directly via arguments. This reduces the number of memory operations
on EX(opline), and makes the CALL VM considerably faster.
Additionally, this unifies the CALL and HYBRID VMs a bit, as EX(opline) is now
handled in the same way in both VMs.
This is a part of GH-17849.
Currently we have two VMs:
* HYBRID: Used when compiling with GCC. execute_data and opline are global
register variables
* CALL: Used when compiling with something else. execute_data is passed as
opcode handler arg, but opline is passed via execute_data->opline
(EX(opline)).
The Call VM looks like this:
while (1) {
ret = execute_data->opline->handler(execute_data);
if (UNEXPECTED(ret != 0)) {
if (ret > 0) { // returned by ZEND_VM_ENTER() / ZEND_VM_LEAVE()
execute_data = EG(current_execute_data);
} else { // returned by ZEND_VM_RETURN()
return;
}
}
}
// example op handler
int ZEND_INIT_FCALL_SPEC_CONST_HANDLER(zend_execute_data *execute_data) {
// load opline
const zend_op *opline = execute_data->opline;
// instruction execution
// dispatch
// ZEND_VM_NEXT_OPCODE():
execute_data->opline++;
return 0; // ZEND_VM_CONTINUE()
}
Opcode handlers return a positive value to signal that the loop must load a
new execute_data from EG(current_execute_data), typically when entering
or leaving a function.
Here I make the following changes:
* Pass opline as opcode handler argument
* Return next opline from opcode handlers
* ZEND_VM_ENTER / ZEND_VM_LEAVE return opline|(1<<0) to signal that
execute_data must be reloaded from EG(current_execute_data)
This gives us:
while (1) {
opline = opline->handler(execute_data, opline);
if (UNEXPECTED((uintptr_t) opline & ZEND_VM_ENTER_BIT) {
opline = opline & ~ZEND_VM_ENTER_BIT;
if (opline != 0) { // ZEND_VM_ENTER() / ZEND_VM_LEAVE()
execute_data = EG(current_execute_data);
} else { // ZEND_VM_RETURN()
return;
}
}
}
// example op handler
const zend_op * ZEND_INIT_FCALL_SPEC_CONST_HANDLER(zend_execute_data *execute_data, const zend_op *opline) {
// opline already loaded
// instruction execution
// dispatch
// ZEND_VM_NEXT_OPCODE():
return ++opline;
}
bench.php is 23% faster on Linux / x86_64, 18% faster on MacOS / M1.
Symfony Demo is 2.8% faster.
When using the HYBRID VM, JIT'ed code stores execute_data/opline in two fixed
callee-saved registers and rarely touches EX(opline), just like the VM.
Since the registers are callee-saved, the JIT'ed code doesn't have to
save them before calling other functions, and can assume they always
contain execute_data/opline. The code also avoids saving/restoring them in
prologue/epilogue, as execute_ex takes care of that (JIT'ed code is called
exclusively from there).
The CALL VM can now use a fixed register for execute_data/opline as well, but
we can't rely on execute_ex to save the registers for us as it may use these
registers itself. So we have to save/restore the two registers in JIT'ed code
prologue/epilogue.
Closes GH-17952
The JIT helper `zend_jit_assign_op_to_typed_ref` expects a `zval*` as an
argument, so we have to store to the stack if OP1_DATA(=op3) is in a
register.
Closes GH-18299.
Arnaud Le Blanc