What is a good tail sampling configuration for long running celery tasks

[skippaydevs]

Hi,

I have a few celery tasks that run for a very long time (which is not the best design but it is what it is). The time they can run for is 1 hour. After that, they get killed and the SoftTimeLimitException is raised. I would like to sample traces for these tasks in 100 % of cases. For tasks that finish successfully, I want to trace only 10 % of these.

This is my configuration:

  tail_sampling/celery-very-long:
    decision_wait: 3606s
    num_traces: 40
    expected_new_traces_per_sec: 1
    decision_cache:
      sampled_cache_size: 8
      non_sampled_cache_size: 80
    sample_on_first_match: true
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: baseline_sampling
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

My understanding is this:

• decision_wait - I want to wait for the whole trace, which is 1 hour and a bit longer to make sure the tasks failed with soft time limit get to be sampled correctly
• num_traces - There are about 20 celery tasks per hour being processed, so just to be sure, this is twice the necessary size
• expected_new_traces_per_sec - that would be about 0.005, not sure I can set float, so 1
• decision_cache - 8 for sampled, 80 for non_sampled, probably a bit more than necessary

Does this configuration make sense? What is the recommended approach for sampling long running celery tasks?