Skip to content

Let CodeTailor accept passing solutions on python3 questions - #1607

Merged
bnmnetp merged 1 commit into
RunestoneInteractive:mainfrom
aspadiyath:codetailor-strip-unittest-main
Oct 14, 2026
Merged

bnmnetp merged 1 commit into
RunestoneInteractive:mainfrom
aspadiyath:codetailor-strip-unittest-main

Conversation

@aspadiyath

Copy link
Copy Markdown
Contributor

CodeTailor fell back to the example solution on Python questions whose language is python3, even when the solution it generated was correct.

Before CodeTailor uses a generated solution, it runs the question's tests on JOBE. It appends its own runner and looks for a final PASS line. clean_python_testcase removes the question's own runner call so the two don't clash, but coach.py only called it when the language was exactly "python". Questions marked python3 kept their own unittest.main(), which runs the tests and exits before PASS is printed. JOBE's output showed Ran 1 test ... OK with empty stdout, so every correct solution was counted as a failure.

Changes

  • Clean the test code for any language starting with python.
  • Strip any unittest.main(...) or myTests().main(...) call, with or without arguments. Before, only myTests().main() was stripped.
  • Add test/bases/rsptx/book_server_api/test_clean_python_testcase.py.

Testing

  • The new tests pass.
  • Checked locally on a CSV question whose tests end in unittest.main(): Get Help now returns a personalized solution instead of example_solution.

🤖 Generated with Claude Code

CodeTailor only cleaned a question's test code when its language was
exactly "python", so "python3" questions kept their own unittest.main()
call. That call exits before JOBE prints PASS, so every generated
solution read as failing and help fell back to the example solution.
Clean for any Python language, and strip unittest.main(...) as well as
myTests().main().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aspadiyath
aspadiyath requested a review from bnmnetp as a code owner October 13, 2026 16:59
@bnmnetp

bnmnetp commented Oct 13, 2026

Copy link
Copy Markdown
Member

What is causing the inconsistency in python vs python3?

As I recall python3 was explicitly meant to run on jobe whereas python meant the in browser version of python (which approximates python3)

@aspadiyath

Copy link
Copy Markdown
Contributor Author

Yes you're correct. Xinying wrotecoach.py to only clean the test code when the language was exactly "python". So python3/JOBE tests (which we're currently using for I/O because the csv module is not available in Skulpt) went to JOBE with their unittest.main() still at the end, which ran the tests and exited before CodeTailor's own runner could print PASS, causing a failure in CodeTailor.

@bnmnetp
bnmnetp merged commit a34243e into RunestoneInteractive:main Oct 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants