Skip to content

Report activity schedule to start latency once per task without the activity type - #3101

Open
RaphaelFakhri wants to merge 2 commits into
temporalio:mainfrom
RaphaelFakhri:fix-activity-schedule-to-start-metric-once
Open

RaphaelFakhri wants to merge 2 commits into
temporalio:mainfrom
RaphaelFakhri:fix-activity-schedule-to-start-metric-once

Conversation

@RaphaelFakhri

@RaphaelFakhri RaphaelFakhri commented Sep 29, 2026 •

Copy link
Copy Markdown

What was changed

  • Removed the activity_schedule_to_start_latency timer from ActivityPollTask and AsyncActivityPollTask.
  • Moved the remaining recording from ActivityWorker.TaskHandlerImpl.handleActivity into TaskHandlerImpl.handle, just before the call to handleActivity. It now uses workerMetricsScope, so the metric carries no activity type or workflow type tags.
  • Added a TestStatsReporter.assertTimer overload that takes a Predicate<StatsAccumulator>, modelled on assertGauge.
  • TestStatsReporter.assertNoMetric now also checks timers. It only checked counters before.
  • MetricsTest.testWorkerMetrics asserts that the timer is reported exactly once on the activity worker tags and never on the tags that include the activity type.

Why?

Each activity task recorded the schedule to start latency twice: once in the poll task and once in handleActivity with the activity type tags. Schedule to start latency does not depend on the activity type, so it is now recorded once per task on the worker scope.

How was this tested?

./gradlew :temporal-sdk:test --tests io.temporal.workflow.MetricsTest

With the change, all 9 tests pass. With the three source files reverted to main, testWorkerMetrics fails because the timer is reported with activity_type=Execute and workflow_type=NoArgsWorkflow.

Closes #2733

@RaphaelFakhri
RaphaelFakhri requested a review from a team as a code owner September 29, 2026 14:32
@CLAassistant

CLAassistant commented Sep 29, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@maciejdudko maciejdudko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @RaphaelFakhri, thank you for your contribution! However, this isn't the full fix for the issue.

Emitting the metric was correctly removed from ActivityPollTask and AsyncActivityPollTask. However, emitting it inside ActivityWorker.TaskHandlerImpl.handleActivity is still wrong. It should be moved into TaskHandlerImpl.handle method, just before the call to handleActivity. It also should use workerMetricsScope instead of the local metricsScope, because attaching worker and activity type to this metric is wrong (schedule-to-start latency is independent of activity type).

I left a separate comment about tests, please address it too.

Feel free to reach out to me on Temporal Community Slack if you need assistance with implementation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new test suite is unnecessary. It's better to amend the existing io.temporal.workflow.MetricsTest. To ensure the metric is emitted only once, you can add a new overload of TestStatsReporter.assertTimer that takes Predicate<StatsAccumulator>, similar to how assertGauge works. To ensure the metric is not sent with wrong set of tags, you can use assertNoMetric.

@RaphaelFakhri
RaphaelFakhri force-pushed the fix-activity-schedule-to-start-metric-once branch from fbacec8 to d11802d Compare October 3, 2026 17:20
@RaphaelFakhri RaphaelFakhri changed the title Report activity schedule to start latency once per task with the activity type Report activity schedule to start latency once per task without the activity type Oct 3, 2026
@RaphaelFakhri

Copy link
Copy Markdown
Author

Thanks for the review. The timer is now recorded once in TaskHandlerImpl.handle on workerMetricsScope, and the assertions live in MetricsTest with the new assertTimer overload.

@RaphaelFakhri

Copy link
Copy Markdown
Author

The requested changes are in. The separate test suite is gone, and MetricsTest covers the timer through the new assertTimer overload and assertNoMetric. Can you take another look?

@maciejdudko maciejdudko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good now, except there's a logic bug (see comment). Please fix that and then we can get this merged.

Comment on lines +341 to +342
ProtobufTimeUtils.toM3Duration(
pollResponse.getStartedTime(), pollResponse.getCurrentAttemptScheduledTime()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize this was in the original code too, but the calculation is incorrect - it uses server started time instead of worker current time.

Suggested change
ProtobufTimeUtils.toM3Duration(
pollResponse.getStartedTime(), pollResponse.getCurrentAttemptScheduledTime()));
ProtobufTimeUtils.toM3DurationSinceNow(
pollResponse.getCurrentAttemptScheduledTime()));

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 3edc9fd. The latency now uses toM3DurationSinceNow(pollResponse.getCurrentAttemptScheduledTime()).

@maciejdudko

Copy link
Copy Markdown
Contributor

Please also add an entry to CHANGELOG.md. It's our new policy that every PR requires a changelog entry before merging.

Don't worry about CI failure here, it's a known flaky test. The test from this PR passes.

@RaphaelFakhri
RaphaelFakhri force-pushed the fix-activity-schedule-to-start-metric-once branch from d11802d to 3edc9fd Compare October 28, 2026 14:37
@RaphaelFakhri

Copy link
Copy Markdown
Author

Added a Fixed entry to CHANGELOG.md. The branch is rebased onto main so the entry sits with the current unreleased notes. MetricsTest passes locally (9 tests).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

activity_schedule_to_start_latency reports activity_type unexpectedly

3 participants