The next logical step was to look for database optimisations and I started diving into the amazing world of SQLite. Secondly, I wanted the code to be simple and near to the daily usage version. The script I had written is very simple, so I assumed there isn’t much room for optimisation.
Sqlite browser running slow full#
Here is the full code SQLite Optimisations With this simple change, the running time was reduced to 10 minutes. I tried different sizes of batch inserts, found out 100,000 to be a sweet spot. Each transaction guarantees that it is written to disk thus could be slow. In SQLite, each insertion is atomic and is a transaction. This version took close to 15 minutes, sparked my curiosity and made me explore further to reduce the time. In this script, I tried to insert 10M rows, one by one, in a for loop. The standard library provides a nice SQLite module, using which I wrote my first version. Python is my go to language for any kind of scripting. No need of using true random methods, pseudo-random methods from stdlib are just fine.
0 kommentar(er)
