Non Monotonic Reads, E.g. Prior Local Reads/writes Reverted, Due To Disconnecting/connecting, Particularly On A Flaky Network.

by ADMIN 127 views

Non-Monotonic Reads in PowerSync: A Breakdown of the Issue

Introduction

PowerSync is a distributed database system that allows for seamless replication across multiple nodes. However, like any complex system, it is not immune to issues that can arise from network instability and disconnections. In this article, we will delve into the issue of non-monotonic reads in PowerSync, a problem that can occur when a client reads a value that is older than the current value, causing the read to appear non-monotonic.

Background

The issue of non-monotonic reads in PowerSync is a subset of a larger problem, "db.connect()/disconnect() issues: db.prop and db.syncStatus.prop values, delayed replication, and read your own writes anomaly" #226. This issue was first identified in version 1.10.0 of PowerSync and became more difficult to reproduce in version 1.11.0. However, with the addition of network instability, such as occasional network partitions, it is now possible to reliably trigger the bug in version 12.1.

The Bug

The bug presents itself in two ways:

  1. A {key: value} is read that is older than the current value, which was previously read in prior reads. This causes the read to appear non-monotonic to the client, as if it is going backwards in time.
  2. The client reads an empty table, even though there have been successful prior reads.

Fuzzing Application

To reproduce this bug and other anomalies, a fuzzing application has been created. This application uses a combination of network instability and disconnections to trigger the bug. The fuzzing application is designed to be run in a Docker container and can be triggered using a script.

GitHub Actions

The fuzzing application is also integrated with GitHub Actions, which allows for automated testing of PowerSync. The GitHub Actions runs a matrix of 240 jobs, which includes a variety of tests, including non-monotonic reads. The results of these tests are used to identify and fix issues in PowerSync.

Local Script

In addition to the fuzzing application, a local script has been created to reproduce the bug. This script uses a combination of network instability and disconnections to trigger the bug and can be run on a local machine.

Example Output

The output of the fuzzing application and local script can be seen in the following example:

[2025-03-11 02:18:57.576457] [main] [INFO] nemesis: disconnect/connect: start listening to stream of disconnected/connected messages
[2025-03-11 02:18:57.576504] [main] [INFO] nemesis: partition: start listening to stream of partition messages
[2025-03-11 02:19:01.847681] [main] [INFO] nemesis: disconnect/connect: disconnected: clients: (1, 3)
[2025-03-11 02:19:03.575833] [main] [INFO] nemesis: disconnect/connect: connected: clients: (1, 5, 3, 2, 4)
[2025-03-11 02:19:05.290393] [main] [INFO] nemesis: partition: bidirectional
[2025-03-11 02:19:09.693580] [main] [INFO] nemesis: partition: none
[2025-03-11 02:19:13.468192] [main] [INFO] nemesis: disconnect/connect: disconnected: clients: (2)
[2025-03-11 02:19:18.185368] [main] [INFO] nemesis: partition: outbound
[2025-03-11 02:19:21.954030] [main] [INFO] nemesis: disconnect/connect: connected: clients: (3, 4, 2, 1, 5)
[2025-03-11 02:19:25.980436] [main] [INFO] nemesis: partition: none
...

This output shows the nemesis activity, which includes disconnecting and reconnecting clients, as well as partitioning the network.

Debugging

To debug the issue, the easiest cause to identify is "expected because myPreviousRead". This indicates that the client is reading a value that is older than the current value, causing the read to appear non-monotonic.

Conclusion

In conclusion, the issue of non-monotonic reads in PowerSync is a complex problem that can arise from network instability and disconnections. The fuzzing application and local script have been created to reproduce this bug and other anomalies, and the GitHub Actions runs a matrix of 240 jobs to test PowerSync. By understanding the causes of this issue and using the fuzzing application and local script, developers can identify and fix issues in PowerSync.

Recommendations

Based on the results of the fuzzing application and local script, the following recommendations are made:

  1. Improve Network Stability: PowerSync should be designed to handle network instability and disconnections more effectively.
  2. Enhance Causal Checker: The causal checker should be improved to detect and prevent non-monotonic reads.
  3. Fuzz Testing: Fuzz testing should be integrated into the PowerSync testing process to identify and fix issues like non-monotonic reads.
  4. Automated Testing: Automated testing should be used to test PowerSync and identify issues like non-monotonic reads.

By following these recommendations, developers can improve the stability and reliability of PowerSync and prevent issues like non-monotonic reads.
Q&A: Non-Monotonic Reads in PowerSync

Q: What is a non-monotonic read in PowerSync?

A: A non-monotonic read in PowerSync is a situation where a client reads a value that is older than the current value, causing the read to appear non-monotonic. This can occur when a client reads a value that was previously written and then overwritten by a newer value.

Q: What causes non-monotonic reads in PowerSync?

A: Non-monotonic reads in PowerSync can be caused by a variety of factors, including network instability and disconnections. When a client disconnects and reconnects, it may read a value that is older than the current value, causing the read to appear non-monotonic.

Q: How can I reproduce non-monotonic reads in PowerSync?

A: To reproduce non-monotonic reads in PowerSync, you can use the fuzzing application and local script provided by the PowerSync team. These tools can be used to simulate network instability and disconnections, causing non-monotonic reads to occur.

Q: What are the symptoms of non-monotonic reads in PowerSync?

A: The symptoms of non-monotonic reads in PowerSync can include:

  • A client reading a value that is older than the current value
  • A client reading an empty table, even though there have been successful prior reads
  • A client experiencing errors or inconsistencies in its reads

Q: How can I debug non-monotonic reads in PowerSync?

A: To debug non-monotonic reads in PowerSync, you can use the following steps:

  1. Check the client's logs for errors or inconsistencies in its reads.
  2. Use the fuzzing application and local script to reproduce the issue.
  3. Analyze the output of the fuzzing application and local script to identify the cause of the non-monotonic read.
  4. Use the PowerSync debugger to step through the code and identify the source of the issue.

Q: How can I prevent non-monotonic reads in PowerSync?

A: To prevent non-monotonic reads in PowerSync, you can use the following strategies:

  1. Improve network stability and reduce the likelihood of disconnections.
  2. Enhance the causal checker to detect and prevent non-monotonic reads.
  3. Use automated testing to identify and fix issues like non-monotonic reads.
  4. Implement a mechanism to detect and recover from non-monotonic reads.

Q: What are the implications of non-monotonic reads in PowerSync?

A: Non-monotonic reads in PowerSync can have significant implications for the reliability and consistency of the database. If left unchecked, non-monotonic reads can lead to data corruption, inconsistencies, and errors.

Q: How can I get help with non-monotonic reads in PowerSync?

A: If you are experiencing issues with non-monotonic reads in PowerSync, you can contact the PowerSync team for assistance. They can provide guidance on how to debug and fix the issue, as well as offer suggestions for preventing non-monotonic reads in the future.

Q: What is the current status of non-monotonic reads in PowerSync?

A: The PowerSync team is actively working to address the issue of non-monotonic reads. They have implemented various fixes and improvements to prevent non-monotonic reads, and are continuing to work on further enhancements to the system.

Q: How can I contribute to the resolution of non-monotonic reads in PowerSync?

A: If you are interested in contributing to the resolution of non-monotonic reads in PowerSync, you can:

  1. Report any issues you encounter with non-monotonic reads to the PowerSync team.
  2. Participate in the PowerSync community to discuss and share knowledge about non-monotonic reads.
  3. Contribute to the development of PowerSync by implementing fixes and improvements to prevent non-monotonic reads.
  4. Provide feedback and suggestions to the PowerSync team on how to improve the system and prevent non-monotonic reads.