github.com/lerianstudio/fetcher/pkg/engine

Asynchronous external data retriever designed to securely fetch, process, and store data from various external sources. It handles data retrieval in a fully non-blocking way, encrypts the fetched content, and stores it as files in SeaweedFS for efficient and scalable storage