JOSS Review: Failing Test And Automated Testing
As a reviewer for the JOSS paper, I have identified a failing test in the bayes_traj repository. In this review, I will discuss the failing test and provide suggestions for setting up automated testing using GitHub Actions.
Failing Test
The failing test is test_compute_prior_info_5 located in the bayes_traj/tests/test_generate_prior.py file. When running pytest within the repository directory, this test fails with an assertion error. The log provided at the end of this issue shows the error message.
============================= test session starts ==============================
platform darwin -- Python 3.10.9, pytest-8.2.2, pluggy-1.5.0
rootdir: /Users/gchure/Desktop/bayes_traj
configfile: pyproject.toml
plugins: anyio-3.5.0, html-4.1.1, cov-4.1.0, metadata-3.0.0, codecov-0.5.1
collected 24 items
bayes_traj/tests/test_fit_stats.py . [ 4%]
bayes_traj/tests/test_generate_prior.py ........F [ 41%]
bayes_traj/tests/test_mult_dp_regression.py ............ [ 91%]
bayes_traj/tests/test_utils.py .. [100%]
=================================== FAILURES ===================================
__________________________ test_compute_prior_info_5 ___________________________
def test_compute_prior_info_5():
"""Model and data, different preds, new target
"""
model_file_name = os.path.split(os.path.realpath(__file__))[0] + \
'/../resources/models/model_1.p'
data_file_name = os.path.split(os.path.realpath(__file__))[0] + \
'/../resources/data/trajectory_data_1.csv'
mm = pickle.load(open(model_file_name, 'rb'))['MultDPRegression']
mm.cast_to_torch()
mm.ranef_indices_ = None
df = pd.read_csv(data_file_name)
df['y2'] = df.y.values + 10
targets = ['y', 'y2']
preds = ['intercept', 'age', 'age^2']
pg = PriorGenerator(targets, preds)
pg.set_model(mm)
pg.set_data(df, 'id')
pg.compute_prior_info()
# The following values are from a prior that -- upon visual inspection of
# random draws -- appears reasonable. This test, then, should be considered
# a regression test as opposed to a test of "correct" values
prior_info_gt = \
{'w_mu0': {'y': {'intercept': 8.102185344934247,
'age': -1.7021817743184755,
'age^2': 0.018416334689522876},
'y2': {'intercept': 18.10218534493426,
'age': -1.7021817743184746,
'age^2': 0.01841633468952318}},
'w_var0': {'y': {'intercept': 1.9028074002722193,
'age': 0.1804899197433961,
'age^2': 0.0007043047299188618},
'y2': {'intercept': 1.9028074002722182,
'age': 0.18048991974339593,
'age^2': 0.0007043047299188608}},
'lambda_a0': {'y': 14.399999999999999, 'y2': 14.399999999999999},
'lambda_b0': {'y': 355.56217578065207, 'y2': 355.56217578065207},
'w_mu': {'intercept': {'y': np.array([5.73256565, 9.27107768]),
'y2': np.array([15.73256565, 19.27107768])},
'age': {'y': np.array([-1.91955117, -0.86877812]),
'y2': np.array([-1.91955117, -0.86877812,])},
'age^2': {'y': np.array([-0.0042961, -0.00525871,]),
'y2': np.array([-0.0042961, -0.00525871,])}},
'w_var': {'intercept': {'y': np.array([ 0.06041792, 0.30262284,]),
'y2': np.array([0.06041792, 0.30262284,])},
'age': {'y': np.array([0.00333069, 0.01713044]),
'y2': np.array([0.00333069, 0.01713044])},
'age^2': {'y': np.array([9.82818683e-06, 5.18071486e-05]),
'y2': np.array([9.82818683e-06, 5.18071486e-05])}},
'lambda_a': {'y': np.array([203.50423622, 11.95284849]),
'y2': np.array([203.50423622, 11.95284849])},
'lambda_b': {'y': np.array([201.74450982, 48.8934525]),
'y2': np.array([201.74450982, 48.8934525])},
'v_a': np.array([51., 51.]),
#'v_b': np.array([50.74102343, 0.74102343]),
'v_b': np.array([0.74102343, 50.74102343]),
'traj_probs': np.array([0.5, 0.5]),
'alpha': 0.5}
> assert pg.prior_info_['w_mu0']['y']['intercept'] == \
prior_info_gt['w_mu0']['y']['intercept'], "Error in prior"
E AssertionError: Error in prior
E assert 8.10218534493425 == 8.102185344934247
bayes_traj/tests/test_generate_prior.py:456: AssertionError
=============================== warnings summary ===============================
bayes_traj/tests/test_mult_dp_regression.py::test_update_u
bayes_traj/tests/test_mult_dp_regression.py::test_update_u
bayes_traj/tests/test_mult_dp_regression.py::test_update_u
bayes_traj/tests/test_mult_dp_regression.py::test_update_u
/Users/gchure/Desktop/bayes_traj/bayes_traj/mult_dp_regression.py:1024: FutureWarning: DataFrameGroupBy.grouper is deprecated and will be removed in a future version of pandas.
groupby_col = self.gb_.grouper.names[0]
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED bayes_traj/tests/test_generate_prior.py::test_compute_prior_info_5 - A...
=================== 1 failed, 23 passed, 4 warnings in 2.53s ===================
The error message indicates that the assertion pg.prior_info_['w_mu0']['y']['intercept'] == prior_info_gt['w_mu0']['y']['intercept'] fails, resulting in an AssertionError.
Possible Causes
- Numerical Precision: The assertion error might be due to numerical precision issues. The values in
pg.prior_info_andprior_info_gtmight be slightly different due to floating-point arithmetic. - Data Loading: The test might be loading the model and data files incorrectly, leading to inconsistent results.
- Prior Generation: The
PriorGeneratorclass might be generating the prior information incorrectly, resulting in the assertion error.
Setting Up Automated Testing
To ensure the stability and reliability of the bayes_traj repository, it is essential to set up automated testing using GitHub Actions. This will enable continuous integration and continuous deployment (CI/CD) pipelines, which will run the tests automatically upon push to the main branch.
GitHub Actions
GitHub Actions is a powerful tool for automating software development workflows. To set up automated testing using GitHub Actions, follow these steps:
- Create a new workflow file: In the repository's
.github/workflowsdirectory, create a new file namedtest.yml. - Define the workflow: In the
test.ymlfile, define the workflow using YAML syntax. The workflow should include the following steps:- Checkout the code: Checkout the code in the repository using the
actions/checkoutaction. - Install dependencies: Install the dependencies required for the tests using the
pipaction. - Run the tests: Run the tests using the
pytestaction.
- Checkout the code: Checkout the code in the repository using the
- Trigger the workflow: Trigger the workflow by pushing changes to the main branch.
Example Workflow File
Here is an example test.yml file that defines a simple workflow for running the tests:
name: Test
on:
push:
branches:
- main
jobs:
test:
<br/>
**JOSS Review: Failing Test and Automated Testing - Q&A**
=====================================================
As a reviewer for the [JOSS](https://github.com/openjournals/joss-reviews/issues/7323#issuecomment-2677094547) paper, I have identified a failing test in the `bayes_traj` repository. In this Q&A article, I will address some common questions related to the failing test and automated testing.
**Q: What is the failing test?**
---------------------------
A: The failing test is `test_compute_prior_info_5` located in the `bayes_traj/tests/test_generate_prior.py` file. When running `pytest` within the repository directory, this test fails with an assertion error.
**Q: Why is the test failing?**
---------------------------
A: The test is failing due to an assertion error in the `PriorGenerator` class. The assertion error is caused by a discrepancy between the expected and actual values of the prior information.
**Q: What are the possible causes of the failing test?**
------------------------------------------------
A: The possible causes of the failing test are:
1. **Numerical Precision**: The assertion error might be due to numerical precision issues. The values in `pg.prior_info_` and `prior_info_gt` might be slightly different due to floating-point arithmetic.
2. **Data Loading**: The test might be loading the model and data files incorrectly, leading to inconsistent results.
3. **Prior Generation**: The `PriorGenerator` class might be generating the prior information incorrectly, resulting in the assertion error.
**Q: How can I set up automated testing using GitHub Actions?**
---------------------------------------------------------
A: To set up automated testing using GitHub Actions, follow these steps:
1. **Create a new workflow file**: In the repository's `.github/workflows` directory, create a new file named `test.yml`.
2. **Define the workflow**: In the `test.yml` file, define the workflow using YAML syntax. The workflow should include the following steps:
* **Checkout the code**: Checkout the code in the repository using the `actions/checkout` action.
* **Install dependencies**: Install the dependencies required for the tests using the `pip` action.
* **Run the tests**: Run the tests using the `pytest` action.
3. **Trigger the workflow**: Trigger the workflow by pushing changes to the main branch.
**Q: What is the benefit of automated testing?**
-----------------------------------------
A: Automated testing provides several benefits, including:
1. **Improved stability**: Automated testing ensures that the code is stable and reliable.
2. **Reduced bugs**: Automated testing reduces the number of bugs in the code.
3. **Faster development**: Automated testing enables faster development by reducing the time spent on testing.
**Q: How can I troubleshoot the failing test?**
--------------------------------------------
A: To troubleshoot the failing test, follow these steps:
1. **Check the test code**: Check the test code for any errors or inconsistencies.
2. **Check the expected values**: Check the expected values in the `prior_info_gt` dictionary.
3. **Check the actual values**: Check the actual values in the `pg.prior_info_` dictionary.
4. **Use debugging tools**: Use debugging tools such as print statements or a debugger to identify the source of the error.
**Q: How can I improve the test coverage?**
-----------------------------------------
A: To improve the test coverage, follow these steps:
1. **Write more tests**: Write more tests to cover different scenarios and edge cases.
2. **Use mocking**: Use mocking to isolate dependencies and make the tests more reliable.
3. **Use parameterized testing**: Use parameterized testing to run the same test with different inputs.
By following these steps and using the Q&A article as a guide, you can improve the test coverage and stability of the `bayes_traj` repository.