[Proofpoint TAP] - SIEM Client Should Request A Minimum 30 Seconds Interval Duration
Introduction
The Proofpoint TAP (Threat Acceleration Platform) is a powerful security information and event management (SIEM) system that provides real-time threat detection and incident response capabilities. However, when using the SIEM client, it is essential to ensure that the interval duration is set correctly to avoid errors and ensure seamless data retrieval. In this article, we will discuss the importance of setting a minimum 30 seconds interval duration for the SIEM client and provide a detailed analysis of the issue.
Description
The SIEM API client of the connector can request intervals shorter than 30 seconds, leading to a server error response. This issue was observed while running the connector on a 48-CPU instance, and it is suspected that the optimization mechanism splitting requests into smaller intervals may be causing the problem.
Environment
The following environment details were used to reproduce the issue:
- OS (where OpenCTI server runs): Linux
- OpenCTI version: 6.5.5
- OpenCTI client: Python
- Other environment details: 48 CPUs
Reproducible Steps
To create the smallest reproducible scenario, follow these steps:
- Launch the connector on a 48 CPUs server, with DURATION_PERIOD set to 30 minutes and the TAP_EXPORT_EVENTS set to True.
Expected Output
The expected output is a successful data retrieval with no errors.
Actual Output
However, the actual output is a server error response with the following message:
Data retrieval error: Bad Request - The requested interval is too short. Requests must be for at least 30.00s of data - query: https://tap-api-v2.proofpoint.com/v2/siem/issues?format=JSON&interval=2025-02-27T15:22:51.224683%2B00:00/2025-02-27T15:22:58.450581%2B00:00
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/client_api/common.py", line 143, in _process_raw_response
response.raise_for_status()
File "/usr/local/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1161, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request',
url='https://tap-api-v2.proofpoint.com/v2/siem/issues?format=JSON&interval=2025-02-27T15:22:51.224683%2B00:00/2025-02-27T15:22:58.450581%2B00:00'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/app.py", line 264, in _process_events
events: list[EventPort] = self._events.fetch()
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/adapters/events.py", line 370, in fetch
for result in asyncio.run(_coro()):
File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
return future.result()
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/adapters/events.py", line 366, in _coro
return await asyncio.gather(*tasks)
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/adapters/events.py", line 332, in _fetch
return await method(start_time, stop_time)
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/client_api/v2/siem.py", line 578, in fetch_issues
return await self._get_siem_data()
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/client_api/v2/siem.py", line 494, in _get_siem_data
response = await self.get(query_url=query_url, response_model=SIEMResponse)
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/client_api/common.py", line 239, in get
data = await self._process_raw_response(response)
File "/usr/local/lib/python3.12/site-packages/proofpoint_tap/client_api/common.py", line 151, in _process_raw_response
raise ProofpointAPIError(message) from e
proofpoint_tap.errors.ProofpointAPIError: Bad Request - The requested interval is too short.
Requests must be for at least 30.00s of data - query:
https://tap-api-v2.proofpoint.com/v2/siem/issues?format=JSON&interval=2025-02-27T15:22:51.224683%2B00:00/2025-02-27T15:22:58.450581%2B00:00
Conclusion
In conclusion, it is essential to set a minimum 30 seconds interval duration for the SIEM client to avoid errors and ensure seamless data retrieval. The issue discussed in this article highlights the importance of setting the correct interval duration and provides a detailed analysis of the problem. By following the reproducible steps outlined in this article, you can create a smallest reproducible scenario and test the issue on your own environment.
Additional Information
For more information on this issue, please refer to the following Notion page:
Introduction
In our previous article, we discussed the importance of setting a minimum 30 seconds interval duration for the SIEM client to avoid errors and ensure seamless data retrieval. In this article, we will provide a Q&A section to address common questions and concerns related to this issue.
Q: What is the minimum interval duration required for the SIEM client?
A: The minimum interval duration required for the SIEM client is 30 seconds.
Q: Why is it essential to set a minimum 30 seconds interval duration?
A: Setting a minimum 30 seconds interval duration is essential to avoid errors and ensure seamless data retrieval. If the interval duration is set too short, the SIEM client may request data that is not available, leading to errors and data inconsistencies.
Q: What happens if I set an interval duration shorter than 30 seconds?
A: If you set an interval duration shorter than 30 seconds, the SIEM client will return a server error response with a message indicating that the requested interval is too short.
Q: Can I set an interval duration longer than 30 seconds?
A: Yes, you can set an interval duration longer than 30 seconds. However, it is recommended to set the interval duration to a value that is not too long, as this may lead to data inconsistencies and delays in data retrieval.
Q: How can I troubleshoot issues related to interval duration?
A: To troubleshoot issues related to interval duration, you can follow these steps:
- Check the interval duration setting in your SIEM client configuration.
- Verify that the interval duration is set to a value of at least 30 seconds.
- Check the SIEM client logs for any error messages related to interval duration.
- Contact your SIEM client support team for further assistance.
Q: Can I use a different interval duration for different data sources?
A: Yes, you can use a different interval duration for different data sources. However, it is recommended to use a consistent interval duration across all data sources to ensure data consistency and avoid errors.
Q: How can I ensure that my SIEM client is configured correctly?
A: To ensure that your SIEM client is configured correctly, you can follow these steps:
- Review your SIEM client configuration settings to ensure that they are correct.
- Verify that the interval duration is set to a value of at least 30 seconds.
- Check the SIEM client logs for any error messages related to configuration.
- Contact your SIEM client support team for further assistance.
Conclusion
In conclusion, setting a minimum 30 seconds interval duration for the SIEM client is essential to avoid errors and ensure seamless data retrieval. By following the Q&A section outlined in this article, you can address common questions and concerns related to this issue and ensure that your SIEM client is configured correctly.
Additional Information
For more information on this issue, please refer to the following Notion page: