Repository navigation
Report activity schedule to start latency once per task without the activity type - #3101
RaphaelFakhri wants to merge 2 commits into
Conversation
maciejdudko
left a comment
There was a problem hiding this comment.
Hi @RaphaelFakhri, thank you for your contribution! However, this isn't the full fix for the issue.
Emitting the metric was correctly removed from ActivityPollTask and AsyncActivityPollTask. However, emitting it inside ActivityWorker.TaskHandlerImpl.handleActivity is still wrong. It should be moved into TaskHandlerImpl.handle method, just before the call to handleActivity. It also should use workerMetricsScope instead of the local metricsScope, because attaching worker and activity type to this metric is wrong (schedule-to-start latency is independent of activity type).
I left a separate comment about tests, please address it too.
Feel free to reach out to me on Temporal Community Slack if you need assistance with implementation.
There was a problem hiding this comment.
This new test suite is unnecessary. It's better to amend the existing io.temporal.workflow.MetricsTest. To ensure the metric is emitted only once, you can add a new overload of TestStatsReporter.assertTimer that takes Predicate<StatsAccumulator>, similar to how assertGauge works. To ensure the metric is not sent with wrong set of tags, you can use assertNoMetric.
fbacec8 to
d11802d
Compare
|
Thanks for the review. The timer is now recorded once in TaskHandlerImpl.handle on workerMetricsScope, and the assertions live in MetricsTest with the new assertTimer overload. |
|
The requested changes are in. The separate test suite is gone, and |
maciejdudko
left a comment
There was a problem hiding this comment.
The code looks good now, except there's a logic bug (see comment). Please fix that and then we can get this merged.
| ProtobufTimeUtils.toM3Duration( | ||
| pollResponse.getStartedTime(), pollResponse.getCurrentAttemptScheduledTime())); |
There was a problem hiding this comment.
I realize this was in the original code too, but the calculation is incorrect - it uses server started time instead of worker current time.
| ProtobufTimeUtils.toM3Duration( | |
| pollResponse.getStartedTime(), pollResponse.getCurrentAttemptScheduledTime())); | |
| ProtobufTimeUtils.toM3DurationSinceNow( | |
| pollResponse.getCurrentAttemptScheduledTime())); |
There was a problem hiding this comment.
Applied in 3edc9fd. The latency now uses toM3DurationSinceNow(pollResponse.getCurrentAttemptScheduledTime()).
|
Please also add an entry to CHANGELOG.md. It's our new policy that every PR requires a changelog entry before merging. Don't worry about CI failure here, it's a known flaky test. The test from this PR passes. |
d11802d to
3edc9fd
Compare
|
Added a |
What was changed
activity_schedule_to_start_latencytimer fromActivityPollTaskandAsyncActivityPollTask.ActivityWorker.TaskHandlerImpl.handleActivityintoTaskHandlerImpl.handle, just before the call tohandleActivity. It now usesworkerMetricsScope, so the metric carries no activity type or workflow type tags.TestStatsReporter.assertTimeroverload that takes aPredicate<StatsAccumulator>, modelled onassertGauge.TestStatsReporter.assertNoMetricnow also checks timers. It only checked counters before.MetricsTest.testWorkerMetricsasserts that the timer is reported exactly once on the activity worker tags and never on the tags that include the activity type.Why?
Each activity task recorded the schedule to start latency twice: once in the poll task and once in
handleActivitywith the activity type tags. Schedule to start latency does not depend on the activity type, so it is now recorded once per task on the worker scope.How was this tested?
With the change, all 9 tests pass. With the three source files reverted to
main,testWorkerMetricsfails because the timer is reported withactivity_type=Executeandworkflow_type=NoArgsWorkflow.Closes #2733