GoKawiilHello! I’ve been working on a Django site recently, and I decided to use SQLite as the database. When I was getting started with using SQLite as database for a website I read a bunch of blog posts about how it is totally fine to use SQLite in production for a small site and I think it is totally fine, but what I did not fully appreciate is that SQLite is still a database, databases are complicated, and I do not know a lot about operating databases.
So here are a couple of small things I’ve been learning about running SQLite. This is the 4th website I’ve used SQLite for, and I think this one is harder because with the power of the Django ORM I’ve been making the database do more work than I was previously without Django.
I started by turning on WAL mode like all the blog posts said to do and hoping for the best.
Today I was running a query (using SQLite’s FTS5 for full-text search) on a table with 4000 rows and it took 5 seconds. That seemed wrong to me: computers are fast!
It turned out that what I needed to do was to run ANALYZE ! Immediately the problem query went from taking 5 seconds to like 0.05 seconds (or some other number small enough that I didn’t care to investigate further). I still don’t know exactly what went wrong in the query plan, but my best guess is that it was some sort of accidentally quadratic thing.
ANALYZE generates “statistics” (I guess about the number of rows in each table? and presumably other things?) so that the query planner can make better choices.
Maybe one day I’ll learn to read a query plan.
Occasionally I’ve run into situations where I accidentally put a bunch of rows in my database that I don’t want to be there (for example completed tasks from django-tasks-db), and I want to clean them up.
What’s happened to me a few times in this case is:
I run some kind of command to clean up the rows The command takes more than 5 seconds, since there are a lot of rows (though I still have some questions about why these DELETE statements are so slow honestly, maybe there’s a bunch of Python code running inside a transaction, I’m not sure) One of the other workers tries to write the database while this is happening, and times out after 5 seconds (I have a timeout of 5 seconds set) The worker crashes because it couldn’t write to the database and the VM shuts down
... continue reading