[Data] Infinite Recursion In Ansitowin32.py (under Tqdm_ray)
Data Infinite Recursion in ansitowin32.py (under tqdm_ray)
In this article, we will be discussing a bug that was encountered in the ansitowin32.py file under the tqdm_ray module. The bug causes an infinite recursion, resulting in a RecursionError being raised. We will delve into the details of the bug, provide a reproduction script, and discuss the possible causes and solutions.
When executing a dataset in a service, logs were routed to a file. Occasionally, the code would hit a seemingly infinite recursion under the tqdm_ray stuff. This caused the program to crash with a RecursionError.
The expected behavior was for the program to run without any infinite recursion.
We tried setting RAY_COLOR_PREFIX=0 to turn off colored logs, which seemed like it might be the issue. However, this did not work.
The dataset plan was as follows:
InputDataBuffer[Input] -> TaskPoolMapOperator[FlatMap(OpenSearchReader._to_parent_doc)] -> AllToAllOperator[Sort] -> TaskPoolMapOperator[MapBatches(map_reduce_parent_id)->Map(OpenSearchReader.reconstruct)->MapBatches(materialize)]
The stack trace was as follows:
- FlatMap(OpenSearchReader._to_parent_doc) 1: 0.00 row [00:00, ? row/s]2025-03-13 16:25:11,367 ERROR exceptions.py:73 -- Exception occurred in Ray Data or Ray Core internal code. If you continue to see this error, please open an issue on the Ray project GitHub page with the full stack trace below: https://github.com/ray-project/ray/issues/new/choose
2025-03-13 16:25:11,368 ERROR exceptions.py:81 -- Full stack trace:
Traceback (most recent call last):
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/exceptions.py", line 49, in handle_trace
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/_internal/plan.py", line 424, in execute_to_iterator
bundle_iter = execute_to_legacy_bundle_iterator(executor, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/_internal/execution/legacy_compat.py", line 51, in execute_to_legacy_bundle_iterator
bundle_iter = executor.execute(dag, initial_stats=stats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor.py", line 124, in execute
self._topology, _ = build_streaming_topology(dag, self._options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 390, in build_streaming_topology
i += op_state.initialize_progress_bars(i, options.verbose_progress)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/_internal/execution/streaming_executor_state.py", line 220, in initialize_progress_bars
self.progress_bar = ProgressBar(
^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/data/_internal/progress_bar.py", line 86, in __init__
self._bar.set_description(self._desc)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/experimental/tqdm_ray.py", line 104, in set_description
self._dump_state()
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/experimental/tqdm_ray.py", line 140, in _dump_state
instance().process_state_update(copy.deepcopy(self._get_state()))
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/experimental/tqdm_ray.py", line 306, in process_state_update
self._process_state_update_locked(state)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/experimental/tqdm_ray.py", line 350, in _process_state_update_locked
process.allocate_bar(state)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/experimental/tqdm_ray.py", line 236, in allocate_bar
self.bars_by_uuid[state["uuid"]] = _Bar(state, self.pos_offset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/experimental/tqdm_ray.py", line 180, in __init__
self.bar = real_tqdm.tqdm(
^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/asyncio.py", line 24, in __init__
super().__init__(iterable, *args, **kwargs)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/std.py", line 1098, in __init__
self.refresh(lock_args=self.lock_args)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/std.py", line 1347, in refresh
self.display()
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/std.py", line 1495, in display
self.sp(self.__str__() if msg is None else msg)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/std.py", line 459, in print_status
fp_write('\r' + s + (' ' * max(last_len[0] - len_s, 0)))
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/std.py", line 452, in fp_write
fp.write(str(s))
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/tqdm/utils.py", line 196, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/thirdparty_files/colorama/ansitowin32.py", line 47, in write
self.__convertor.write(text)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/thirdparty_files/colorama/ansitowin32.py", line 177, in write
self.write_and_convert(text)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/thirdparty_files/colorama/ansitowin32.py", line 205, in write_and_convert
self.write_plain_text(text, cursor, len(text))
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/thirdparty_files/colorama/ansitowin32.py", line 210, in write_plain_text
self.wrapped.write(text[start:end])
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/thirdparty_files/colorama/ansitowin32.py", line 47, in write
self.__convertor.write(text)
File "/home/admin/aryn-docstore-0.20250305.134508/.venv/lib/python3.11/site-packages/ray/thirdparty_files/colorama/<br/>
**Q&A: Data Infinite Recursion in ansitowin32.py (under tqdm_ray)**
**Q: What is the issue with the ansitowin32.py file under the tqdm_ray module?**
A: The issue is that the file causes an infinite recursion, resulting in a RecursionError being raised.
**Q: What is the expected behavior of the program?**
A: The expected behavior is for the program to run without any infinite recursion.
**Q: What is the dataset plan that was being executed when the issue occurred?**
A: The dataset plan was as follows:
`InputDataBuffer[Input] -> TaskPoolMapOperator[FlatMap(OpenSearchReader._to_parent_doc)] -> AllToAllOperator[Sort] -> TaskPoolMapOperator[MapBatches(map_reduce_parent_id)->Map(OpenSearchReader.reconstruct)->MapBatches(materialize)]`
**Q: What is the stack trace that was generated when the issue occurred?**
A: The stack trace is a long and complex sequence of function calls that ultimately leads to the RecursionError. It is included in the original article.
**Q: What is the cause of the infinite recursion?**
A: The cause of the infinite recursion is not immediately clear from the stack trace. However, it is likely related to the way that the tqdm_ray module is handling the progress bars.
**Q: How can the issue be reproduced?**
A: Unfortunately, the issue has not been able to be reproduced locally. However, it is likely that the issue can be reproduced by running the same dataset plan with the same configuration.
**Q: What is the severity of the issue?**
A: The severity of the issue is unknown. However, it is likely that the issue is causing significant performance problems and may be preventing the program from running correctly.
**Q: What is the solution to the issue?**
A: The solution to the issue is not immediately clear. However, it is likely that the issue can be solved by modifying the tqdm_ray module to handle the progress bars in a different way.
**Q: What are some possible causes of the issue?**
A: Some possible causes of the issue include:
* A bug in the tqdm_ray module that is causing the infinite recursion
* A misconfiguration of the dataset plan that is causing the infinite recursion
* A problem with the way that the progress bars are being handled
**Q: How can the issue be debugged?**
A: The issue can be debugged by using a debugger to step through the code and identify the point at which the infinite recursion is occurring. Additionally, the issue can be debugged by using print statements to print out the values of variables at different points in the code.
**Q: What are some best practices for debugging infinite recursion issues?**
A: Some best practices for debugging infinite recursion issues include:
* Using a debugger to step through the code and identify the point at which the infinite recursion is occurring
* Using print statements to print out the values of variables at different points in the code
* Modifying the code to prevent the infinite recursion from occurring in the first place
* Using a tool such as a memory profiler to identify the point at which the memory usage is increasing and the infinite recursion is occurring.