CFE-3041: change regular acceptance test to .error.cf

[sarakthon]

As far as I understand, this test deliberately causes an error considering it tries to insert lines to a file that does not exist. Based on this I think the test is a potential error.cf candidate.

Before change:
After adding a check to testall that flags tests which report "Pass", but still print error: messages during execution, the following output was observed:

...
./00_basics/03_bodies/112.cf FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)
...

Passed tests:    1432
Failed tests:    340
Skipped tests:   238
Soft failures:   39
Flakey failures: 0
Total tests:     2049

The output in error.log before changing the filename to error.cf was as follows:

----------------------------------------------------------------------
./00_basics/03_bodies/112.cf 
----------------------------------------------------------------------
2026-08-11T21:13:11+0200    error: File '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_cf/tmp/TEST.cfengine' was marked for editing but could not be opened
2026-08-11T21:13:11+0200    error: Errors encountered when actuating files promise '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_cf/tmp/TEST.cfengine'
2026-08-11T21:13:11+0200    error: Method 'init' failed in some repairs
R: ok: class 'repair_failed' was set (should be)
R: ok: class 'cancel_kept' was set (should be)
R: ok: class 'cancel_repaired' was set (should be)
R: ok: class 'promise_kept' was not set (should not be)
R: ok: class 'promise_repaired' was not set (should not be)
R: ok: class 'repair_denied' was not set (should not be)
R: ok: class 'repair_timeout' was not set (should not be)
R: ok: class 'cancel_notkept' was not set (should not be)
R: /home/ubuntu/core/tests/acceptance/./00_basics/03_bodies/112.cf Pass
2026-08-11T21:13:11+0200    error: Method 'test_run' failed in some repairs

Return code is 0.

  ==> FAIL (Test passed, but printed error message(s)) (UNEXPECTED FAILURE)

After change:
Output when running testall after changing filename to error.cf:

...
./00_basics/03_bodies/112.error.cf Pass
...
Passed tests:    1433
Failed tests:    339
Skipped tests:   238
Soft failures:   39
Flakey failures: 0
Total tests:     2049

Output in error.log after changing filename to error.cf:

----------------------------------------------------------------------
./00_basics/03_bodies/112.error.cf (should log error(s), but exit with 0)
----------------------------------------------------------------------
2026-08-11T22:02:56+0200    error: File '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_error_cf/tmp/TEST.cfengine' was marked for editing but could not be opened
2026-08-11T22:02:56+0200    error: Errors encountered when actuating files promise '/home/ubuntu/core/tests/acceptance/workdir/__00_basics_03_bodies_112_error_cf/tmp/TEST.cfengine'
2026-08-11T22:02:56+0200    error: Method 'init' failed in some repairs
R: ok: class 'repair_failed' was set (should be)
R: ok: class 'cancel_kept' was set (should be)
R: ok: class 'cancel_repaired' was set (should be)
R: ok: class 'promise_kept' was not set (should not be)
R: ok: class 'promise_repaired' was not set (should not be)
R: ok: class 'repair_denied' was not set (should not be)
R: ok: class 'repair_timeout' was not set (should not be)
R: ok: class 'cancel_notkept' was not set (should not be)
R: /home/ubuntu/core/tests/acceptance/./00_basics/03_bodies/112.error.cf Pass
2026-08-11T22:02:56+0200    error: Method 'test_run' failed in some repairs

Return code is 0.

  ==> Pass

[cf-bottom]

Thank you for submitting a PR! Maybe @larsewi can review this?

[nickanderson]

@sarakthon

Do .error.cf suffixed tests just pass if they emit an error or do they require an error to be emitted and still check the test result?

If the former, I don't think this should move (or perhaps maybe we should reconsider what the desired behavior is). If the latter then I think good to move.

[github-actions]

Marking this PR as stale due to inactivity; it will be closed in 7 days.

[olehermanse]

@cf-bottom test this in Jenkins, please

[olehermanse]

Do .error.cf suffixed tests just pass if they emit an error or do they require an error to be emitted and still check the test result?

If the former, I don't think this should move (or perhaps maybe we should reconsider what the desired behavior is). If the latter then I think good to move.

@nickanderson see my 2 added commits.

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14764/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14764/

[craigcomstock]

looks great!

[craigcomstock]

it looks like acceptance test doesn't quite agree that this is OK

2026-10-07T12:57:59+0000    error: Method 'test_run' failed in some repairs

Return code is 0.

  ==> FAIL (Pass not printed!) (UNEXPECTED FAILURE)

[olehermanse]

@craigcomstock strange...

----------------------------------------------------------------------
./00_basics/03_bodies/112.error.pass.cf (should log error(s), print pass message and, exit with 0)
----------------------------------------------------------------------
2026-10-07T12:57:58+0000    error: File '/home/runner/work/core/core/tests/acceptance/workdir/__00_basics_03_bodies_112_error_pass_cf/tmp/TEST.cfengine' was marked for editing but could not be opened
2026-10-07T12:57:58+0000    error: Errors encountered when actuating files promise '/home/runner/work/core/core/tests/acceptance/workdir/__00_basics_03_bodies_112_error_pass_cf/tmp/TEST.cfengine'
2026-10-07T12:57:59+0000    error: Method 'init' failed in some repairs
R: ok: class 'repair_failed' was set (should be)
R: ok: class 'cancel_kept' was set (should be)
R: ok: class 'cancel_repaired' was set (should be)
R: ok: class 'promise_kept' was not set (should not be)
R: ok: class 'promise_repaired' was not set (should not be)
R: ok: class 'repair_denied' was not set (should not be)
R: ok: class 'repair_timeout' was not set (should not be)
R: ok: class 'cancel_notkept' was not set (should not be)
R: /home/runner/work/core/core/tests/acceptance/./00_basics/03_bodies/112.error.pass.cf Pass
2026-10-07T12:57:59+0000    error: Method 'test_run' failed in some repairs

Return code is 0.

  ==> FAIL (Pass not printed!) (UNEXPECTED FAILURE)

Pass is printed and it succeeded in Jenkins 😅

[olehermanse]

@cf-bottom jenkins again please :)

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14765/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14765/

[olehermanse]

@cf-bottom jenkins again please :)

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14766/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14766/