An advanced data management app, it can consume data in any format and offers search engine, data manipulation 🛠, report generation, SQL and script actions on the data.

This app support background jobs, and excels at processing large datasets 🚀 with billions of rows.

This project continued for 1.5 years and we worked closely with the client and eventually by the end of it the app had dozens of modules, each module having 100s of features.

This complex app is carefully designed and separated into modules as below :

🔵 Each module and features support batch operation, background job, scheduled job, cloning, sharing, access level, deep logging, Excel like data editing and raw access to run low level code (SQL,Pyhton,R) on the resources.

• Data: The app has hierarchial data layer viz. Datacloud > Database > Datatable > Schema. The data can be manipulated indirectly with spreadsheet like feature, directly with SQL and with custom features. The data can be in any format,can be local or online as well it can fetch data from FTP server.
• Files: The app supports single and multiple file upload and file management through a file manager. This module also supports fetching through FTP server of the logged in user. Upon user account creation a FTP account is also created for the user and user can upload files using any FTP client with his credential.
• Code: This module supports running low level code in Sql, Python and R on the resources. The code is sanitized and checked before it is allowed to run directly. Also giving user access to running low level code is not safe so it is ensure user can only run the code on his resources with restricted permission. The module has provision to add support for any other language as well.
• Jobs / Logs / Action: Each job is made up of actions that can run separately. Each action and job can be cancelled manually and all the data and result is logged at each step of the job. The job can be scheduled as well as the actions order can be set. The job progress is streamed live using websocket.
• ElasticSearch / OpenSearch: All the data is indexed using Opensearch (Elasticsearch) which enables querying data fast and efficient.